Ori Livson

HTMX with Python's HTBuilder & FastAPI

Interactive websites all in one server generating HTML snippets.

• Software Engineering
• Web Development
• Python

Introduction

HTMX is a JavaScript library that lets you create interactive websites entirely out of a single backend server that generates HTML snippets. Critically, this can mean avoiding a lot of the complexity surrounding passing data back and forth between a frontend client and a backend server, let alone setting up a frontend client.

I’ve personally been using it often at my job to setup internal, data entry and management websites with Python, and I have really been enjoying it. However, it took a few tries to get the solution to scale, i.e., I had to figure out some design patterns to prevent other sorts of complexities popping up and bogging the development of those projects.

So, this post is tutorial on that, and will hopefully save you some of the troubles I encountered. To explain this, we’ll go over a simple example project I’ve developed, available in this public repo.

Moreover, this post is also about encouraging another approach I really enjoy - functional HTML generation. In other words, Python + HTMX users are often steered towards using HTML templating languages such as jinja and mako, and I’ve always found HTML templating very lacking in the following two senses:

1. These templates often become cluttered, unreadable, and don’t look like the HTML they actually generate.
2. They often don’t have great support for composing templates like one can with pure functions or components in modern JS frameworks.

What I’m interested in, is simply generating HTML with functions, and when it’s done right (with a good code formatter), the functions can be automatically indented to mirror the HTML you expect on the screen. I’be written a bit about that in an earlier post about how this website was made using Haskell, and I plan to write a version of this post for Haskell in the future. Note, my first introduction to this was through elm, which I highly recommend looking into too.¹

What is HTMX?

The basic feature of HTMX is giving HTML elements special attributes that:

• Lets them send HTTP requests on any standard web event, e.g., clicks, key-presses, mouse hovers, etc.

• Specifies where to put the server response — in particular, if the server responds with an HTML snippet, you can switch out a corresponding part of the page with that snippet.

A prototypical example of HTMX is as follows.

<input
  type="text"
  name="name_contains"
  placeholder="Type to filter rows by name..."
  hx-get="/table/filter"
  hx-trigger="input changed delay:300ms"
  hx-target="#table"
  hx-swap="outerHTML"
/>

What does it do? It makes an ordinary text input like so: and filters a table to only contain records whose “name” contains the contents of the input. Specifically, HTMX drives this process with the follwoing attributes:

1. (hx-trigger="input changed") Every time you edit its contents of the input to X, it -
2. (hx-get="/table/filter") fires an HTTP request GET /table/filter?name_contains=X then -
3. (hx-swap="outerHTML") takes the HTTP response, and uses it to replace the entire HTML element that matches id=table (i.e., the table’s outerHTML is swapped out).

Moreover:

4. (hx-trigger="...delay:300ms") Means requests are fired only after 300ms without an input change, this stops the website from overloading the server or creating jittery changes on those responses.

Typically, this functionality would have to be split-up over the following 4 pieces of code, disconnected from the HTML <input> tag.

1. Some JavaScript event handler would have to listen for changes to the input.
2. Some JavaScript code would have to send the HTTP request, wait for, and capture the response.
3. Some webpage state would have to be changed to trigger the replacement of the HTML surrounding #table.
4. Some other state would also have to keep track of how long it has been since the <input> tag has been changed.

Yes, a modern JavaScript framework like React could keep part 1 close to the <input> tag by bundling them together in a component. The logic of parts 2-3 may also be part of the component (although there would likely be a module providing the functions to do the HTTP request). Likewise, the state needed to drive parts 3 and 4 could be part of the component too. However, driving the refresh of the component according to the state change of 3 requires one to understand the often complicated lifecycle system of the framework. I’ve certainly worked on projects where setting up an <input> like the above could require editing 4 or 5 files, just on the frontend.

So, HTMX can sidestep a lot of the work and failure points, but there are two natural questions: - What if a single client-side change requires many HTML elements to be changed in response? - Are we really going to send HTML between the backend and frontend? Surely, passing JSON back and forth is more efficient.

HTMX has a number of fine-grained solutions for the first question that we will discuss, although just replacing the closest parent of everything that needs to be changed often works fine. Secondly, by sending HTML snippets (AKA partials), there really isn’t that much text to pass through, it just looks wrong. That said, you can add a text compression middleware (e.g., gzip or brotli) to your server.

Generating HTML with HTBuilder?

Htbuilder is a python package for generating HTML by effectively exposing a function for every HTML tag. HTML attributes can be passed as arguments to those functions, and critically, so can child-HTML elements.

For example, the following htbuilder expression:

from htbuilder import div, li, ul

div(id="container")(
    ul(_class="greetings")(
        li("hello"),
        li("hi"),
        li("whattup"),
    )
)

Generates the following HTML:

<div id="container">
  <ul class="greetings">
    <li>hello</li>
    <li>hi</li>
    <li>whattup</li>
  </ul>
</div>

(I.e., if we print or cast the expression to str).

This first example, satisfies the following key tenant of good HTML generation:

Tenant 1: The function arguments indent to match the nesting of the HTML; that way, the source Python and resulting HTML have matching visual logic. In other words, you can mentally write out the HTML as your eyes follow the nesting of the code.

The way I reliably get this indenting is by setting up the Python black formatter to format on-save (the reference HTML formatting was produced using the prettier formatter).

However, the functionally generating HTML really shines when rendering HTML based on programmatic logic (i.e., variables, if-else logic and loops). For example, we can rewrite the above example as follows.

div(id="container")(
    ul(_class="greetings")(
        li(greeting)
        for greeting in [
            "hello",
            "hi",
            "whattup",
        ]
    )
)

This second example, satisfies another key tenant of good HTML generation:

Tenant 2: Any mixing of programmatic logic with HTML should not make any surrounding HTML less readable, and the final product should still have a matching visual logic to the HTML it generates.

This property is only somewhat satisfied in templated approaches such as JSX (React), Jinja (Python), Hugo (Go) due to programmatic parts cluttering the solutions with brackets and other special symbols (see the examples in Footnote 2).

Moreover, unlike many templated solutions, the above expressions can be assigned to variables, or the return value of functions. This allows us to express the high level design of our websites by composing high level functions. For example, in the full website example we’re about to overview, the homepage is rendered from a top-level function like so.

from htbuilder import HtmlTag, form, h3, head, html, p, script, ...

def homepage(session: SessionState) -> HtmlTag:
    return html(
        head(script(src="/static/htmx.min.js")),
        body(
            h3("HTMX Demo"),
            form(
                name_contains_input(session),
                bulk_select_button(session),
            ),
            main_table(session),
            submit_button(session),
            p(id="result"),
        ),
    )

Note, When HTML elements or attributes are python keywords (e.g., input and class), they are prefixed with an underscore (e.g., _input and _class). Likewise, underscores replace hyphens when they would be invalid in Python syntax, e.g., _input(hx_get="/table/filter") generates `.

Finally, adding in HTMX is just a matter of adding more arguments (i.e., HTML attributes), e.g., hx_get="/table/filter".

FastAPI

The final piece of this puzzle is an API server for listening to the HTMX requests and serving the response HTML that we render with htbuilder.

Here, nothing special is required, I just prefer FastAPI because it’s relatively lightweight and doesn’t force you into mountains of mandatory directories and config files.

One just needs to figure out what kind of HTTP requests HTMX generates (e.g., whether the data you’re expecting will be in query parameters, form data, etc.).

A First Project

For the remainder of this post, we will go over some key portions of this example project (available here). Here’s a non-functioning preview of what this project produces.

The key requirements are:

1. Typing in the text input should instantly filter the table to rows whose “name” contains that text anywhere (case insensitively).
2. Clicking select all, should check all rows in the table, and when all rows are checked, the button turns into a “Deselect all” button.
3. Clicking the arrow symbol of any column sorts the table by that column (up arrow ascending) and reverses the ordering if clicked again; the arrow also turns red to indicate how the table is sorted.
4. Clicking submit prints the names of all the selected rows (excluding all filtered-out rows).

Implementing these requirements alone is straightforward, but one has to be careful when combining them. For example, requirement 1 requires us to first render elements like so:

from htbuilder import input_, table, thead, tbody, trow, th, td, tr

TABLE_ID = "table"

def name_contains_input() -> HtmlTag:
    return input_(
        type="text",
        name="name_contains",
        placeholder=placeholder,
        hx_get="/table/filter",
        hx_trigger="input changed delay:300ms",
        hx_target=f"#{TABLE_ID}",
        hx_swap="outerHTML"
    )

def main_table(rows: list) -> HtmlTag:
    ...
    heading_row: HtmlTag = thead(tr(...))
    rows: HtmlTag = tbody(make_row(row) for row in rows)
    return table(
        heading_row,
        rows,
        id=TABLE_ID,
    )

and an API endpoint like so:

from fastapi import FastAPI
from fastapi.responses import HTMLResponse

app = FastAPI()

@app.get("/table/filter")
def filter_names(name_contains: str, session_id: str) -> HTMLResponse:
    rows = [row for row in get_rows() if name_contains.lower() in row.name.lower()]
    return HTMLResponse(content=str(main_table(rows)))

However, if we then implement requirement 2 or 3, we presumably make another hx-get on main_table such that we don’t forget about our name filter. A naive solution is loading up our endpoints with query parameters, but this becomes cumbersome quickly. Instead, we will use session state retrievable from a single session_id attached to every request.

Session State

Our general procedure for keeping track of client state is to define classes:

@dataclass
class Row:
    name: str
    selected: bool  # checkbox column


@dataclass
class SessionState:
    id: str  # referenced by ?session_id=x in allrequests
    name_contains: str  # Requirement 1
    rows: list[Row]
    sort_by: tuple[str, bool]  # Requirement 3 (attr of Row, ascending?)

    def sort(self):
        sort_col, sort_ascending = self.sort_by
        self.rows = sorted(
            self.rows,
            key=lambda r: asdict(r)[sort_col],
            reverse=not sort_ascending,
        )

And having the API assign each user a session_id used to access their session state, e.g., via a flow like:

from uuid import uuid4

from fastapi import FastAPI, HTTPException
from fastapi.responses import HTMLResponse, RedirectResponse

SESSIONS: dict[str, SessionState] = {}

@app.get("/")
def render_homepage(session_id: str | None = None) -> HTMLResponse:
    if not session_id:
        # If first time visiting, assign a session_id and attach it to their URL
        session_id = generate_session_id()
        return RedirectResponse(
            url=f"/?session_id={session_id}",
        )

    if session_id not in SESSIONS:
        raise HTTPException(
            status_code=404,
            detail=f"Session ID {session_id} not found.",
        )

    return response(
        dom=homepage(session=SESSIONS[session_id]),
    )

def generate_session_id() -> str:
    result = str(uuid4())
    SESSIONS[result] = initialise_session(session_id=result)
    return result

Then, our HTMX powered endpoints update the session before generating response HTML, e.g.,

@app.get("/table/filter")
def filter_names(name_contains: str, session_id: str) -> HTMLResponse:
    # Render a table with the updated name filter
    session = SESSIONS[session_id]
    session.name_contains = name_contains
    return response(
        dom=main_table(session=session),
    )

Note: if the session state contains sensitive data, one should not use query parameters to manage session ids, but rather something like authorization headers, or signed, encrypted tokens in secure cookies.

Updating Multiple HTMX Targets

Recall the following part of requirement 3: “when all rows are checked, the button turns into a”Deselect all” button”. There are 2 events that should cause this outcome: the checking of a final unchecked checkbox, and the clicking of the “Select all” button. In the latter case, two portions of the HTML need to be updated: the whole table, and the button itself.

Because there is so little outside the table, one could just hx-target the <body>, but it’s worth looking at another option: hx-swap-oob, which stands for
hx swap out of bounds.

The way this works, is that we mark certain elements as ‘out of bounds swappable’, and if a matching element (e.g., by element id) is returned by the server it automatically swaps out that first element.

For example, say we render the buttons as follows:

def bulk_select_button(session: SessionState) -> HtmlTag:
    all_selected = all(row.selected for row in session.rows)
    button_text = "Deselect All" if all_selected else "Select All"
    href = f"/table/{'deselect' if all_selected else 'select'}/all"
    return button(
        button_text,
        id=SELECT_ALL_ID,
        hx_post=add_session(href, session),
        hx_target=f"#{TABLE_ID}",
        hx_swap_oob="true",
    )

If all rows are selected, it reads “Deselect All” and hx-posts to /table/deselect/all, and importantly becomes hx-swap-oob enabled. If we then deselect a row manually — triggering the /table/{row_id}/toggle-select endpoint — attaching a copy of the “Select All” button to the response automatically causes the desired swap.

Attaching simply means string concatenation like so:

def oob_response(dom: HtmlTag, oob: HtmlTag) -> HTMLResponse:
    return HTMLResponse(content=str(dom) + str(oob))

So, the toggle-select endpoint can work like so:

@app.post("/table/row/{row_id}/toggle-select")
def select(row_id: str, session_id: str) -> HTMLResponse:
    session = SESSIONS[session_id]
    if row := get_row(session=session, row_id=row_id):
        row.selected = not row.selected
        return oob_response(
            dom=some_target(session),
            oob=bulk_select_button(session),
        )

    raise HTTPException(
        status_code=404,
        detail=f"Row ID {row_id} not found in session",
    )

There are yet more options for targeting multiple HTML elements, discussed by the creator of HTMX in this short post.

Conclusion

At this point you hopefully want to learn more HTMX, in particular, there are lots of other cool HTMX features for producing triggers, targets, swapping behaviour, etc, which we do not go into in this post.

The HTMX homepage is a great reference with lots of examples, but I also highly recommend and the companion Hypermedia Systems ebook. Besides being practical and exercise driven, I found it provided some really valuable perspective about the history of web development, that I never learned.

The rest of this post is just an assortment of tricks and extra snippets that may be valuable to you.

Happy coding!

HT Builder Tricks

HTBuilder has a number of additional interesting ways to initialise HTML elements by way of overwriting the __getitem__ method (i.e., obj[idx] syntax) and __call__ method (i.e., obj(arg) syntax) of python objects.

Firstly, if one has a variable element: HtmlTag that already has children and HTML attributes, one can call element again to add more elements and attributes, for example, if we have:

def generate_checkbox(row: Row, session: SessionState) -> HtmlTag:
    attrs = dict(
        type="checkbox",
        name="selected",
        value=row.id,
        hx_post=add_session(f"/table/row/{row.id}/select", session),
    )
    # need an absence of the checked attribute to render an unchecked checkbox
    if row.selected:
        attrs["checked"] = "checked"  # any string works

    return input_(**attrs)

We can give it a id attribute afterwards by calling:

base_checkbox = generate_checkbox(row, session)
special_checkbox = base_checkbox(id="special_id")

Note: adding attributes to a second call will overwrite existing, matching attributes in the HtmlTag.

Secondly, if we have a parent element parent: HtmlTag and list of child elements [element1, element2, element3]: list[HtmlTag], we can initialise the parent via the expression parent[element1, element2, element3] rather than the more cumbersome parent(*[element1, element2, element3]).

CSS

Htbuilder reduces a lot of the friction for writing inline CSS styles, which can be useful in situations where things don’t have to be pretty (e.g., internal websites). However, once you want the design of parent HTML elements to determine the styles of their children, one probably should resort to working with a proper external .css file. In other words, htbuilder code tends to be become clunky / poorly designed when you try to inject styles into functions responsible for rendering child elements.

For example, say we have a function render_paragraph(text) that generates a <p> tag used in several parts of the webpage, but when that paragraph is a child of a special <div>, we want the paragraph to have margin-left: 20px.

I wish we could do something like:

from htbuilder import styles

base_paragraph = render_paragraph(text)
special_paragraph = base_paragraph(style=styles(margin_left="20px"))

But this will just remove all other styles that base_paragraph may have. Likewise, we don’t want to create a duplicate render_special_paragraph(text) function that adds this style. So that really just leaves patterns like:

def parent():
    return div(
        "Parent",
        base_paragraph(
            extra_styles=dict(margin_left="20px"),
        ),
    )


def base_paragraph(extra_styles: dict = None):
    p_style = dict(color="red")
    p_style |= extra_styles or {}
    return p("World", style=styles(color="red"))

And this can get a bit cumbersome, especially since one has to be careful as to when they actually chose to render the dict as an inline style.

HTMX Extras

Here are some HTMX + Htbuilder additions that you might find yourself needing for similar, real-world projects.

Loading Spinners

Simply add the following style globally:

.htmx-indicator {
    opacity: 0;
    transition: opacity 500ms ease-in;
    z-index: -9999;
}
.htmx-request .htmx-indicator {
    opacity: 1;
    z-index: 9999;
}
.htmx-request.htmx-indicator {
    opacity: 1;
    z-index: 9999;
}

And include element like so in your webpage:

from htbuilder import HtmlTag, div, img, styles

SPINNER_ID = "spinner"

def spinner() -> HtmlTag:
    overlay_style = styles(
        position="fixed",
        top=0,
        left=0,
        width=percent(100),
        height=percent(100),
        display="flex",
        justify_content="center",
        align_items="center",
        background=rgba(0, 0, 0, 0.1),
    )

    spinner_style = styles(
        width=px(200),
        height=px(200),
    )
    return div(id=SPINNER_ID, style=overlay_style, _class="htmx-indicator")(
        img(id=SPINNER_ID, src="/static/spinner.gif", style=spinner_style),
    )

It can then be used in HTMX enabled elements by adding the attribute hx_indicator=f"#{SPINNER_ID}"

Use with HTTPS

If your server runs with HTTPS enabled, you need the following <meta> tag in your <head>:

from htbuilder import meta

meta(http_equiv="Content-Security-Policy", content="upgrade-insecure-requests")

Nice Dropdowns with select2

While I love vanilla, HTML, CSS and JS when possible, select inputs (AKA dropdowns) are very lacking (i.e., one typically type to filter and multiselect functionality). For this, I’ve found the JS library select2 works well to close this gap, however, it needs a bit of help to work with HTMX, namely, serve the select2 css and js files and add the following script.

function refreshSelect2() {
    // Initialise Select2 drop downs
    $('.select2-element').select2();

    // Ensure Select2 triggers HTMX events
    $('select').on('select2:select', function (e) {
        $(this).closest('select').get(0).dispatchEvent(new Event('change'));
    });
}

$(document).ready(function() {
    refreshSelect2();
});

// Reinitialize Select2 after the content has been updated by HTMX
document.addEventListener('htmx:afterSettle', function (event) {
    refreshSelect2();
});

Then you can create compatible dropdowns with something like:

from dataclasses import dataclass, field
from htbuilder import select

@dataclass
class Select:
    name: str
    selected: str | list[str] | None = None  # list for multi_select
    options: list[str] = field(default_factory=list)
    mutli_select: bool = False
    endpoint: str
    target_id: str

def select_element(s: Select):
    options = []
    for val in s.options:
        option_attrs = {"value": val}

        is_selected = (
            s.selected is not None
            and (val in s.selected and s.mutli_select)
            or (val == s.selected and not s.mutli_select)
        )

        if is_selected:
            option_attrs["selected"] = "selected"
        options.append(option(**option_attrs)(val))

    select_attrs = {
        "_class": "select2-element",
        "name": s.name,
    }
    if s.mutli_select:
        select_attrs["multiple"] = "multiple"  # the pressence of just attr key is all we need
    elif s.on_change:
        select_attrs |= dict(
            data_hx_get=s.endpoint,
            data_hx_trigger="change",
            data_hx_target=f"#{s.target},
        )

    return select(**select_attrs)(*options)

const jsx = (
  <div id="container">
    <ul className="greetings">
      {["hello", "hi", "whattup"].map((greeting) => (
        <li key={greeting}>{greeting}</li>
      ))}
    </ul>
  </div>
);

<div id="container">
  <ul class="greetings">
    {% for greeting in ["hello", "hi", "whattup"] %}
      <li>{{ greeting }}</li>
    {% endfor %}
  </ul>
</div>

<div id="container">
  <ul class="greetings">
    {{ range slice "hello" "hi" "whattup" }}
      <li>{{ . }}</li>
    {{ end }}
  </ul>
</div>