Local First HTMX Part 2

Overview

Welcome back to Meme Driven Development (MDD) for the second and more technical entry in our series on Local First HTMX. In this post I'll go over the technical details for my Local First HTMX Todo App. In the first post we discussed the general ideas and the general architecture, I'll recap just a little bit of that here and then start diving into the technical details of the project. If you, like me, just want to see the github here ya go.
Disclaimer: I am not an experienced web dev — I have literally written far more assembly than I have javascript. I am just having fun messing around with technology and learning a little bit more about the web.

In this example we are going to side step a lot of the complexity of local first, syncing data, etc.

Architecture Overview

For our architecture we will have the main thread, a service worker, and a remote server. Nothing crazy so far.

Going in one level of detail in the main thread we have HTML and some HTMX — not that much as this is a very simple example app that I modified to make Local First. We have a service worker that is running a Go program compiled to WASM. This service worker is responsible for proxying the fetch requests and returning rendered HTML to the main thread. The server runs the same code. In a real world example the server would have some additional code and be authoritative, but I'm bypassing that for the purpose of this POC. If for some reason the service worker is not installed when a fetch request is made, that request will go to the server, be handled by the server and rendered HTML will be returned just as if it was a SSR app.
The difference is that once the service worker is installed we do not have to make a full round trip request to the server, instead we make a very fast request to the service worker. Finally in the background we have a simple sync that will run to keep our data somewhat in sync.

Web Main Thread

If you run the app on your local host and then go to the URL you'll get a page load. This page load should be noticeably slow. The reason is I've added a 1 second delay on handling all requests to simulate a case where you have a great deal of network latency. This demonstrates a bad case for the initial page load feeling slow. Any additional requests made to the server will incur this same 1 second artificial delay.

This page is immediately usable because it has the benefits of being fully SSR. We can add todos, etc. while the service worker is loading it because the calls will go directly to the server and rendered HTML will be returned.
In this initial page load we load a script called start_worker.js. It contains a lot of very important logic. I'll go over this in its entirety.

navigator.serviceWorker.register("sw.js");

Ok. That actually wasn't bad at all. Here's some MDN docs.

• loads from server rendered
• usable
• starts service worker

Service Worker

A service worker is a separate thread in the browser that has special privileges. MDN documentation describes it: "Service workers essentially act as proxy servers that sit between web applications, the browser, and the network (when available)."
Now the service worker does have many of the same restrictions as a web worker, such as no access to the DOM. It can also only be used over https because of security concerns.
The main point for us is that a service worker acts as a proxy and can choose how to handle network requests. This is what allows us to use HTMX and do a post to a URL and capture that request and render the HTML in client or as I like to call it SSR in the browser.
As we get into some Javascript code snippets I just want to remind you, dear reader, that JS is not my mainstay. I've written more assembly than I've written JS. So if something stands out as particularly bad please tell me.

At the beginning of our sw.js file we need to import wasm_exec.js which provides bindings for Go to call.

importScripts("wasm_exec.js");

I could not find much in the way of official documentation on this, but I believe it provides the syscall/js package bindings. It is important your wasm_exec.js exactly matches your Go version. The best way to do this is to copy it from your Go installation.

cp "$(go env GOROOT)/misc/wasm/wasm_exec.js" . ```
{% endraw %}


Next we need to fetch our WASM binary and install it and start it.
{% raw %}

```js
  WebAssembly.instantiateStreaming(fetch(wasm), go.importObject).then((
    { instance },
  ) => go.run(instance));

We also need to add an event listener for "fetch". This registers that we are proxying network requests. For more in depth info see MDN.

  addEventListener("fetch", (e) => {
    const url = new URL(e.request.url);
    const { pathname } = new URL(e.request.url);
    if (
      pathname === "/wasm_exec.js" || pathname == "/sw.js" ||
      pathname === "/start_worker.js"
    ) {
      e.respondWith(fetch(e.request));
      return;
    } else if (url.hostname === "localhost") {
      e.respondWith(handlerPromise.then((handler) => handler(e.request)));
    } else {
      // For requests to other domains, just pass them along to the network
      e.respondWith(fetch(e.request));
    }
  });

I do a little bit of hacking to ignore certain requests and pass them along to the underlying network. I don't think this part is particularly clean, but I left it like this to show how I handled an error where once the service worker was registered I couldn't load those files because I was proxying the network requests.
The real magic is here.

      e.respondWith(handlerPromise.then((handler) => handler(e.request)));

Where the definition of that is.

  const handlerPromise = new Promise((setHandler) => {
    self.wasmhttp = {
      path,
      setHandler,
    };

So what is happening is I'm using a package called go-wasm-http-server. It takes the JS network request and does some magic (I am not going to get into it this blog post because I don't fully understand all the internals yet) and then we get a network http request in Go.
So let's start talking about our Golang some more in the next section.

• service worker starts
• loads WASM
• listen for fetch events
• does a little bit of routing logic (this is probably a skill issue)

Golang WASM

I am using Go for Local First HTMX because of a few reasons. The first and most important is Go can be compiled to WASM and run in the browser. The second and also important is I like writing Go and I have a fair bit of experience with the language from using it over the years. It is a fast language, it is simple to write, understand and has excellent tooling built around it. I think if I was truly doing all of the memes this should be written using Rust, but that is not a journey I need to dive into at the moment. TLDR - I like writing Go.

The big downside of using Go in the browser is that the WASM binary size is large. I think mine is like 10mb at the moment. That is pretty large. There are some optimizations you can do for the binary size, but it remains pretty large no matter what you do. I did play around with TinyGo but it does not support a lot of the standard library packages we need. So the argument I'll make for why having a 10mb WASM download is not that bad is that this site will load and be useful before the WASM download completes. You can even navigate around while it is downloading. It is a fully SSR site, but after the service worker is up and running you get a full local first experience. I would love to see the Go WASM binary get trimmed down by a lot — tho Go binaries are just large because of the language's design.

Go WASM HTTP Server

The real magic is the go-wasm-http-server package. This is what enables us to have a Go server that can then route the network requests from the sw.js. This package is a little bit old, only has 286 stars on Github and hasn't been updated in 2 years. But it works so we are going to use it. I may attempt to dive into the internals in another blog post in the future, we will just have to see what the future holds and where my interests lead.

Compiling for WASM/JS

I was initially using Go Fiber, because that is what the example app I was piggybacking off of used. It however does not compile when building for the browser. I tested out Gin and Echo. They both compiled and since my current project at work uses Echo I chose to use that here.

Go in the browser

Let's start digging into the code a little. First thing you can see is we have a build constraint — this file is only built when we compile for JS target. We import normal echo imports and wasmhttp server.

//go:build js

package api

import (
    "github.com/labstack/echo/v4"
    "github.com/labstack/echo/v4/middleware"
    wasmhttp "github.com/nlepage/go-wasm-http-server"
)

Next we have our EchoStart which registers routes and handlers and then takes the echo.Server.Handler which is of type http.Handler and passes that into wasmhttp.Serve(). Then we do a bit of weirdness select {}. Why do we do this? Well the wasmhttp documentation says to and it didn't work when I took it out.

func EchoStart() {
    // Echo instance
    e := echo.New()

    // Middleware
    e.Use(middleware.Logger())
    e.Use(middleware.Recover())
    e.Use(SyncToServer)

    // Routes
    e.GET("/", renderTodosRoute)
    e.POST("/toggle/:id", toggleTodoRoute)
    e.POST("/add", addTodoRoute)

    // Start server
    wasmhttp.Serve(e.Server.Handler)
    select {}
}

Other than passing the handler to wasmhttp.Serve and the empty select this looks like very normal router code that you would have on the server. This is part of the idea that I am going for in this Local First HTMX app: we can write code once that runs in the browser and on server.
Now we have our main.go for the service worker "server". This is very simple, it merely syncs data from the server and then starts the server code.

func main() {
    err := api.GetData()
    if err != nil {
        fmt.Println("error syncing data with server")
    }

    api.EchoStart()
}

For handling data syncing while the app is running I have a middleware function SyncToServer that syncs the current data state with the server in a Go routine. This is a very rudimentary data sync method and it definitely has issues. It is another idea I'd like to follow up with to see how to do this better and more robustly. That is not the main point of this post, so I just got something very basic working.

func SyncToServer(next echo.HandlerFunc) echo.HandlerFunc {
    return func(c echo.Context) error {
        SyncData()
        return next(c)
    }
}

func SyncData() {
    go syncDataRoutine()
}

A single language stack (kinda)

There are a lot of ways to render HTML on the client. The reason I have chosen this architecture is because there is a movement to write JS or really TS on both the browser and the server and that way you can have a single language stack. You can see Atwood's law for the first instance of this thinking.
This argument has its merit, but I think it also has serious downsides and so as this is all MDD I wanted to do a single language stack. Obviously this is not truly a single language, but most of the logic is in Go. It also has the distinct upside that the code you write and run on the server will be far faster than JS or TS.
I also wanted to make sure that you only have to write your logic once and then run it in the browser and on the server. This gets back to that idea of making our technology stack simple and productive. This is at the heart of HTMX's philosophy.
I work in a big enterprise on big projects with huge teams of engineers. We do a lot of unnecessary work and add process to help compensate for all of this complexity. I believe there needs to be concerted thought put into designing software that enables individuals and small teams to create great software that scales.
A single language stack has benefits especially if you can write something once that can both work for your client side rendering and server side rendering. Or as I like to say SSR in your browser. I think this is doubly beneficial if the language you write is highly performant on the server.

What I mean by single language stack

main.go running on the server.

package main

import (
    "github.com/elijahmorg/lhmtx/api"
)

func main() {
    api.EchoStart()
}

EchoStart for the server is slightly different than for the browser.

func EchoStart() {
    fmt.Println("server side code")
    e := echo.New()
    e.Use(middleware.Logger())
    e.Use(middleware.Recover())
    e.Use(ServerDelay)

    // Routes
    e.GET("/", renderTodosRoute)
    e.POST("/toggle/:id", toggleTodoRoute)
    e.POST("/add", addTodoRoute)
    e.GET("/sync", getTodos)
    e.POST("/sync", syncTodos)

    e.Static("/", "../../public/")
    e.Logger.Fatal(e.Start(":3000"))
}

The different EchoStart functions are compiled based on build constraints.
As you can see I have very little custom code to handle the difference between my server code and client code. I think this is pretty nifty. I am sure that if I did more complex stuff, with real databases, auth, etc. I would have to handle a lot more corner cases and this would become more complex. All that said, I think this is pretty interesting and I hope it makes you think about the possibilities of what you could do with something like this.

Local First

Local First HTMX has the benefits of both being a SSR web framework. When the site first loads it behaves like an SSR. Here is an example of it in action.

Then after the service worker is installed and running it behaves like a local first app.

So now we have a Local First HTMX app. It can behave like both an SSR web app giving it benefits for SEO, first page load, etc. and a local first app. Thanks for reading this, I hope you enjoyed it and you will check back in the future for my next post.