DEV Community: David Mohl

Building your own distributed pocket ChatGPT in Elixir with GenServer

David Mohl — Mon, 13 Mar 2023 10:00:00 +0000

From my blog: https://david.coffee/mini-chatgpt-in-elixir-and-genserver/

I recently built an Elixir SDK for OpenAI APIs for a project I was working on to automatically fix and correct Japanese grammar (https://fixmyjp.d.sh). The SDK is actually fully auto-generated with metaprogramming which is something I wanted to do in Elixir for a while, but that’s not the topic of today’s post.

Today, let’s use that SDK for something fun and build a smol ChatGPT service for your shell from scratch, in true Elixir fashion!

The end result will be an ai shell command that is usable from wherever we want, to send messages to a distributed gptserver node for interacting with ChatGPT, that can either be on the same computer, or anywhere on the same network

Quick refresher: GenServers, and how state in Elixir is implemented

Elixir is an immutable functional programming language, meaning you can’t mutate things. If you do

a = 1
a = a + 1

you are not mutating a to be a+1, instead you create a new a with the value of a+1. That means you can’t do things like this in Elixir:

a = %{}
a["foo"] = "bar"

instead, you would do:

a = %{}
a = Map.put(a, "foo", "bar")

again, creating a new a with a copy of the previous a, but one item added. The old a is not mutated, it still exists until the garbage collector gets rid of it.

So how tf do you actually keep any state across different modules then? From the sounds of it, you would need to constantly do something with a value, you can’t just create an object that different parts of the code can mutate and share.

In Elixir/Erlang-land, instead of having a shared object/container somewhere, we use processes managed within the BEAM VM, and the most common way to define those processes is GenServer.

GenServer is a behaviour that looks like this:

defmodule Foo do
  use GenServer

    # stuff that's running inside a separate process

    @impl true
  def init(val), do: {:ok, val}

    @impl true
  def handle_call({:add, i}, _from, sum) do
      {:reply, sum+i, sum+i}
  end

    # stuff that's running inside the main process

  def add(pid, num) do
      GenServer.call(pid, {:add, num})
  end
end

foo = GenServer.start_link(Foo, 0)
Foo.add(foo, 2) # 0 + 2
Foo.add(foo, 3) # 2 + 3

To hold state, we effectively created a new process whose whole purpose is to hold a value in memory, then listen for a message :add, to then add whatever got passed to create a new value (remember? we can’t mutate things) to hold in memory and start listening for new messages again. Imagine it as a (very simplified representation):

function add(startValue) {
    add(startValue + waitForUserInput()); // yay, no variables
}

Level 1: ChatGPT in a GenServer

Okay now that we remembered how to handle state in Elixir, we know what to do to get a ChatGPT-like system going: Wait for user input -> append to messages so far -> send to OpenAI -> output -> wait for user input again

Let’s summon a new mix project:

mix new mixgpt

add my ex_openai library as deps to mix.exs:

defp deps do
  [
    {:ex_openai, ">= 1.0.2"}
  ]
end

(follow the instructions for configuration with an API key)

and conjure up a basic GenServer with a new {:msg, m} message listener:

defmodule Mixgpt do
  use GenServer

  @impl true
  def init(_opts) do
      {:ok, []} # start value is empty list
  end

  @impl true
  def handle_call({:msg, m}, _from, msgs) do

  end
end

To make our life easier, we’ll also add a start_link function to start the thing:

def start_link(_opts) do
  GenServer.start_link( __MODULE__ , [], name: :gptserver)
end

Notice the name: :gptserver argument here - we’re giving the GenServer a unique name so that we don’t need to worry about remembering its PID somewhere.

To send a message to this new genserver, we’ll also create a new send():

def send(msg) do
    GenServer.call(:gptserver, {:msg, msg}))
end

thanks to name: :gptserver we already know how to reach this instance, so we no longer need the PID to be passed. Now, all we have to do is Mixgpt.send "hi!"

In the actual handle_call, what we have to do is: construct a new OpenAI ChatGPT message with the passed message, send it to the OpenAI API, and add whatever it returned to the internal state of this server (remember from above? how to handle state). Something like this:

defp new_msg(m) do
  %ExOpenAI.Components.ChatCompletionRequestMessage{
    content: m,
    role: :user,
    name: "user"
  }
end

@impl true
def handle_call({:msg, m}, _from, msgs) do
  # create temporary "msgs" with existing list + the new message from the user 
  # effectively appending the new message to the existing messages
  with msgs <- msgs ++ [new_msg(m)] do

      # call OpenAI ChatGPT API with the new msgs list
    case ExOpenAI.Chat.create_chat_completion(msgs, "gpt-3.5-turbo") do
      {:ok, res} ->
          # return the content of the result, add it to the msgs list and continue the GenServer loop with {:reply}
        first = List.first(res.choices)

        # second value is what's returned, third value is the new state of this server
        {:reply, first.message.content, msgs ++ [first.message]}

      {:error, reason} ->
        {:error, reason}
    end
  end
end

The cool thing about Elixir and other functional languages is that they usually come with a REPL, or a shell that you can use to interact with different bits and pieces of your program. In Elixir, that’s iex. Run iex -S mix, wait until it starts, then try to interact with our new GenServer:

iex> Mixgpt.start_link(nil)
{:ok, #PID<0.280.0>}
iex> Mixgpt.send "Hi!"
"Hello! How can I assist you today?"
iex> Mixgpt.send "tell me a joke in 5 words"
"\"Why did the tomato blush?\""

Cool, now we already got our own pocket ChatGPT that we can access through mix, that was pretty quick, wasn’t it? Let’s make it a bit more automatic and close the loop. To wait for user input, we can use IO.gets, let’s use that in a new function to directly ask for the next message:

def wait_for_input_and_send() do
  IO.gets("> ")
  |> send
  |> IO.puts() # print the message out to have it nicely formatted

  wait_for_input_and_send() # recurse 
end

Type recompile into iex to recompile the project and try again:

iex> Mixgpt.wait_for_input_and_send
> Hi
Hello, how can I assist you today?
> Do you like Elixir?
As an AI language model, I don't have personal preferences or feelings. However, I can tell you that Elixir is a popular programming language among developers and has some unique features that make it stand out, such as its scalability, fault tolerance, and ease of use for distributed systems.
> what was the first message I sent you?
The first message you sent me was "hi".
>

Level 2: Distributed ChatGPT service

The example above is easy to do in any programming language, but let’s go one step further and do something cool that you can do beautifully in Elixir and Erlang, but not as easily in other programming languages: Let’s turn this thing into a proper distributed system!

As a reminder, our mini chatgpt is running inside a server! We did this because that’s how state is handled in Elixir. Even though we created the interface so that we can easily just call send("why is the sky green?"), on the back it still sends a new message to this gpt-server, then waits until it responds. It just does this so transparently that we don’t even notice.

But what’s even cooler: GenServers can already be accessed from other nodes within same BEAM cluster, so we already have something that is usable over a network! Let’s take a look at how to do that, with only a little bit of black magic.

Close the iex shell we had open, and create a new one, but this time with the --name parameter:

iex --name gptserver@127.0.0.1 -S mix 
iex> Mixgpt.start_link
{:ok, #PID<0.276.0>}

--name tells elixir to start a distributed node with the given name, on the given network. So we created a distributed node gptserver on the network 127.0.0.1

In a new terminal window, start another iex session, this time without -S mix, but still with --name:

iex --name client@127.0.0.1

To now execute things on another cluster node, Elixir provides the Node.spawn_link function among other things. With this you could do stuff like:

iex> Node.spawn_link(:"gptserver@127.0.0.1", fn -> IO.puts("hi there, I'm running on another node") end)

But we already have a GenServer running, so no need for that and just hit the GenServer directly. This is also very easy to do, just by changing the GenServer.call(:gptserver, {:msg, msg}) call to also include the node information where :gptserver is running: {:gptserver, :"gptserver@127.0.0.1"}:

iex> GenServer.call({:gptserver, :"gptserver@127.0.0.1"}, {:msg, "Helloo, anyone there?"})
"Hello! Yes, I'm here. How can I assist you?"

Do you realize what just happened? Our completely separate iex shells just connected to each other to send a message to the GenServer running under the :gptserver name, which then hits the OpenAI API, returns the data, and relays it back to the other iex shell!

We just created a distributed ChatGPT service on our local network (well, localhost), all without any extra setup required! This even works across machines, as long as the nodes are running reachable on the same network :)

Final step: A global shell command

You can already guess what comes next. We can run Elixir code directly with Elixir -e "IO.puts \"Hi\"". The last lego piece missing is hooking this up to a global shell command, so that we can access our mini-ChatGPT whenever we need through a terminal, while retaining previous context.

To package it all up into a one-liner:

$ elixir --name client@127.0.0.1 -e "GenServer.call({:gptserver, :'gptserver@127.0.0.1'}, {:msg, \"hi\"}) |> IO.puts"
Hello! How can I assist you today?

That’s still the same call as before, except we’re running it through elixir -e "", and pipe the result into IO.puts to write it to the shell. This is still using the same GenServer that’s still running (unless you shut it down), and it still retains the same message history from earlier.

Lastly, package into a bash function:

ai() { elixir --name client@127.0.0.1 -e "GenServer.call({:gptserver, :'gptserver@127.0.0.1'}, {:msg, Enum.join(System.argv, \" \")}) |> IO.puts"; $@; };

Tadaa! We now have a system-wide (or network-wide) mini-ChatGPT that is fully stateful

$ ai is the sky blue?
Yes, on clear days, the daytime sky typically appears blue due to the scattering of sunlight by Earth's atmosphere.
$ ai what is the last message I sent you?
Your last message was "is the sky blue?"

We could easily put this on a home server and have it available across all of our machines without any extra setup if we really wanted to

Of course, this is still super barebones and in reality, we’d do a few more things:

A Supervisor to handle crashes
Changing :gptserver to be global, so we don’t need to know the hostname
Some way to reset, or clear the message history to not have messages endlessly pile up
Better error handling

among others

Check out the code for this on GitHub

Refactoring code with ChatGPT

David Mohl — Sun, 04 Dec 2022 00:00:00 +0000

From my blog: https://david.coffee/refactoring-code-with-chat-gpt/

ChatGPT is really damn impressive and the more I read about it, the more it feels like withcraft. I see people using it for game development, coming up with titles, improving prompts for MidJourney and and and. It's shaping up to be one of those universally useful tools that are great to keep around.

Here's one thing I had on my mind: Can it help me when I code?

I was working on a little pet project and had some pretty inefficient code I slapped together for a middleware while watching TV:

package middleware

import (
    "context"
    "encoding/json"
    "net/http"
)

var parsedJsonBodyContextKey struct{}

func ParseJsonBody(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        var data map[string]interface{}
        err := json.NewDecoder(r.Body).Decode(&data)
        if err != nil {
            ctx := context.WithValue(r.Context(), parsedJsonBodyContextKey, nil)
            next.ServeHTTP(w, r.WithContext(ctx))
            return
        }

        ctx := context.WithValue(r.Context(), parsedJsonBodyContextKey, nil)
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

The code takes whatever is passed into the request body, tries to JSON decode it, then puts the result into context. If no body exists or isn't JSON, it will pass nil instead.

I wanted to shorten and make it a bit more efficient but though it'd be a cool experiment to let ChatGPT do it:

Me: Can you refactor the following code to make it more efficient?

<code from above here>

ChatGPT: Here are a few ways the provided Go code can be imporved:

The ParseJsonBody() function defines a data variable to store the parsed JSON body, but it doesn't appear to be used anywhere in the function. This variable can be removed.

The ParseJsonBody() function creates a new context.Context with a key-value pair for the parsed JSON body, but it sets the value to nil in both branches of the if statement. This means that the value of the parsed JSON body will always be nil, regardless of whether the JSON body was parsed successfully or not. The value of the parsed JSON body should be set to the parsed JSON data if parsing was successful, and nil if parsing failed.

The ParseJsonBody() function passes the original http.Request object to the next handler, even if the JSON body was parsed successfully. In this case, the http.Request object should be updated with the new context.Context that contains the parsed JSON body before being passed to the next handler.

Here is how the code can be improved:
package middleware

import (
  "context"
  "encoding/json"
  "net/http"
)

var parsedJsonBodyContextKey struct{}

func ParseJsonBody(next http.Handler) http.Handler {
  return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
      var data map[string]interface{}
      err := json.NewDecoder(r.Body).Decode(&data)
      if err != nil {
          ctx := context.WithValue(r.Context(), parsedJsonBodyContextKey, nil)
          next.ServeHTTP(w, r.WithContext(ctx))
          return
      }

      ctx := context.WithValue(r.Context(), parsedJsonBodyContextKey, data)
      next.ServeHTTP(w, r.WithContext(ctx))
  })
}

Huh, turns out I copy pasted and forgot to change the normal branch to actually add data into the context - whoops. ChatGPT spotted this, understood what this was supposed to do and fixed it.

Can we make it more compact though?

Me: Can you make the code more compact and more DRY?

ChatGPT: Yes, the provided code can be made more compact and DRY (Don't Repeat Yourself) by removing the duplicate code and using a single context.Context object.

Here is how the code can be improved:

package middleware

import (
  "context"
  "encoding/json"
  "net/http"
)

var parsedJsonBodyContextKey struct{}

func ParseJsonBody(next http.Handler) http.Handler {
  return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
      var data map[string]interface{}
      err := json.NewDecoder(r.Body).Decode(&data)

      ctx := context.WithValue(r.Context(), parsedJsonBodyContextKey, data)
      next.ServeHTTP(w, r.WithContext(ctx))
  })
}

This is exactly what I had in mind for refactoring this code: No need for the if condition at all because var data map[string]interface{} is nil by default. Only nitpick would be that we assign err and never use it, so the compiler will complain about this code.

Let's add a function to get the data out of the context again. Can ChatGPT summon this function for us?

Me: Please add a function that takes the body out of the context again

ChatGPT: To extract the parsed JSON body from the context.Context object, a new function can be added to the middleware package. This function can take a context.Context object as an argument and return the parsed JSON body, if it exists in the context, or nil otherwise.

Here is an example implementation of this function:

package middleware

import (
  "context"
  "encoding/json"
)

var parsedJsonBodyContextKey struct{}

func ParseJsonBody(next http.Handler) http.Handler {
  return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
      var data map[string]interface{}
      err := json.NewDecoder(r.Body).Decode(&data)

      ctx := context.WithValue(r.Context(), parsedJsonBodyContextKey, data)
      next.ServeHTTP(w, r.WithContext(ctx))
  })
}

func GetParsedJsonBody(ctx context.Context) map[string]interface{} {
  val := ctx.Value(parsedJsonBodyContextKey)
  if val == nil {
      return nil
  }
  return val.(map[string]interface{})
}

To use this function, you can pass a context.Context object as an argument and check the return value for nil to determine whether the JSON body was parsed successfully or not.

Here is an example of how this function can be used:

package main

import (
  "context"
  "fmt"
  "net/http"

  "github.com/myproject/middleware"
)

func main() {
  http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
      parsedBody := middleware.GetParsedJsonBody(r.Context())
      if parsedBody == nil {
          w.Write([]byte("JSON body was not parsed"))
          return
      }
      w.Write([]byte(fmt.Sprintf("

Short and concise. It understood what I wanted it to do, it delivered a solution and even gave an explanation with examples how to use it!

After a bit of hacking I noticed that using map[string]interface{} is kind of a PITA because we have to do many type assertions and checks if keys exist. I got curious if we can tell it to swap out the type into something custom that has a nicer interface:

Me: Instead of using map[string]interface{}, let's use a custom type that has methods for getting the value for a given key, or a default

ChatGPT: To use a custom type that has methods for getting the value for a given key or a default value, you can define a new ParsedJsonBody type that satisfies the context.Context interface. This type can be a struct that contains a map[string]interface{} field to store the parsed JSON body.

Here is an example implementation of the ParsedJsonBody type:
type ParsedJsonBody struct {
  data map[string]interface{}
}

func (p *ParsedJsonBody) Value(key interface{}) interface{} {
  if key == parsedJsonBodyContextKey {
      return p
  }
  return nil
}

func (p *ParsedJsonBody) Get(key string) interface{} {
  return p.data[key]
}

func (p *ParsedJsonBody) GetString(key string) string {
  val := p.data[key]
  if val == nil {
      return ""
  }
  return val.(string)
}

func (p *ParsedJsonBody) GetInt(key string) int {
  val := p.data[key]
  if val == nil {
      return 0
  }
  return val.(int)
}
The ParseJsonBody() middleware function can be updated to use the ParsedJsonBody type instead of map[string]interface{}:

Hmm okay, this is where things get a bit weird, but that could have also been my way of explaining what I wanted.

The code looks ok on first skimming, but it ignored my instruction to have a default. I think it interpreted that as "we'll return a default", but not one I can specify.

There's lots of type assertions in there that don't have any checks, so this will panic if we were to try to GetInt something that isn't an int. Also, you can define a new ParsedJsonBody type that satisfies the context.Context interface is nonsense, and so is the Value() method.

What I actually wanted was a custom type like type MyType map[string]interface{}, not a struct with a type field, but whatever.

Let's ask directly if it thinks anything is wrong with that code:

Me: Thanks for the code, do you see any problems with this implementation?

ChatGPT: There is a potential problem with the provided implementation of the ParsedJsonBody type. The Value() method returns a pointer to the ParsedJsonBody struct, but the Get(), GetString(), and GetInt() methods return values of the underlying data field. This means that if the ParsedJsonBody struct is modified after it is added to the context.Context object, the modifications will be reflected in the context.Context, but not in the return values of the Get(), GetString(), and GetInt() methods.

To fix this problem, the Get(), GetString(), and GetInt() methods should return copies of the values in the data field, rather than the values themselves.

Here is an updated implementation of the ParsedJsonBody type that fixes this problem:
type ParsedJsonBody struct {
  data map[string]interface{}
}

func (p *ParsedJsonBody) Value(key interface{}) interface{} {
  if key == parsedJsonBodyContextKey {
      return p
  }
  return nil
}

func (p *ParsedJsonBody) Get(key string) interface{} {
  val, ok := p.data[key]
  if !ok {
      return nil
  }
  return val
}

func (p *ParsedJsonBody) GetString(key string) string {
  val, ok := p.data[key].(string)
  if !ok {
      return ""
  }
  return val
}

func (p *ParsedJsonBody) GetInt(key string) int {
  val, ok := p.data[key].(int)
  if !ok {
      return 0
  }
  return val
}

The good: It fixed the type assertions and now has a checks, neat!

The bad: The explanation it gave on why it added those doesn't make sense. It didn't actually change those to return copies of values. Let's do one more optimization pass.

Me: Please further optimize this code

ChatGPT: Here are a few ways the provided implementation of the ParsedJsonBody type can be further optimized:

The Value() method currently returns p if the key argument matches parsedJsonBodyContextKey, and nil otherwise. Since parsedJsonBodyContextKey is a global variable that is only used in the ParseJsonBody() middleware function, this check is unnecessary. The Value() method can simply return p without checking the key argument.

The Get(), GetString(), and GetInt() methods currently check if the specified key exists in the data field and return a default value if the key does not exist. This check can be removed, because the data field is a map[string]interface{} and trying to access a non-existent key in a map will return the zero value for the type of the map's value (interface{} in this case, which is nil).

The GetString() and GetInt() methods currently use the ok return value of the type assertion to determine whether the value for the specified key is of the expected type. This check can be removed, because

Me: Please show me an implementation with those points

ChatGPT:
type ParsedJsonBody struct {
  data map[string]interface{}
}

func (p *ParsedJsonBody) Value(key interface{}) interface{} {
  // The key argument is not used, so we can simply return p
  return p
}

func (p *ParsedJsonBody) Get(key string) interface{} {
  // We don't need to check if the key exists, because trying
  // to access a non-existent key in a map will return the zero
  // value for the type of the map's value (nil in this case)
  return p.data[key]
}

func (p *ParsedJsonBody) GetString(key string) string {
  // We don't need to check the type of the value, because trying
  // to assert a non-string value to a string type will return
  // the zero value for the string type ("" in this case)
  return p.data[key].(string)
}

func (p *ParsedJsonBody) GetInt(key string) int {
  // We don't need to check the type of the value, because trying
  // to assert a non-int value to an int type will return the
  // zero value for the int type (0 in this case)
  return p.data[key].(int)
}

It got more compact for sure, but the code is wrong. It got rid of the type checks with the confident explanation that type-asserting a non-int value to an int type will return the zero value for the int type, but this is not correct and will panic. Here's the code in Golang playground to try out: https://play.golang.com/p/8jjBJyrJ1uC

While impressive, at this point it's internal state of my program and what I'm trying to do is a bit too messed up, and the confidently wrong explanation+implementation can be dangerous. For a moment I self-doubted my own knowledge of Golang and had to try this out in the playground to make sure.

The network isn't specifically trained on code, so I guess it just doesn't fully understand the inner-workings of Golang, without providing enough context to it first.

Generating unit tests

Let's do something fun - unit tests (groans)! If it can spit out functional unit tests for stuff I'm too lazy to write, that would already be very helpful. After all, it seems to understand code and is able to give explanations for what's going on.

The code here is a snippet from a JWT token parser. It validates the token, extracts sid/uid scopes (UUID) and checks if those are actually valid UUIDs.

Me: Consider the following Golang code:

// other code here
func (c *Client) validateToken(t jwt.Token) (*Token, error) {
  if err := jwt.Validate(t); err != nil {
      return nil, err
  }

  // parse sid
  sid, ok := t.Get("sid")
  if !ok {
      return nil, domain.ErrTokenInvalidSID
  }
  sidUUID, err := uuid.Parse(sid.(string))
  if err != nil {
      return nil, domain.ErrTokenInvalidSID
  }

  // parse uid
  uid, ok := t.Get("uid")
  if !ok {
      return nil, domain.ErrTokenInvalidUID
  }
  uidUUID, err := uuid.Parse(uid.(string))
  if err != nil {
      return nil, domain.ErrTokenInvalidUID
  }

  return &Token{
      SessionID: sidUUID,
      UserID:    uidUUID,
  }, nil
}
// other code here

Please generate some testcases for when sid or uid are invalid

ChatGPT: Here are some test cases for when the sid or uid values in the token are invalid:

func TestValidateTokenInvalidSID(t *testing.T) {
 // Create a new client
 client := NewClient("some-secret-key")

 // Create a test token
 token := jwt.New()
 token.Set("uid", uuid.New().String())
 // Set the sid value to something invalid
 token.Set("sid", "invalid-sid")
 token.Set(jwt.ExpirationKey, time.Now().Add(time.Minute * 5).Unix())

 // Sign the token
 signedToken, err := jwt.Sign(token, jwt.WithKey(client.alg, client.hmacSecret))
 if err != nil {
     t.Errorf("unexpected error signing token: %s", err)
 }

 // Validate the token
 _, err := client.validateToken(signedToken)
 if err != domain.ErrTokenInvalidSID {
     t.Errorf("expected error to be %s, got %s", domain.ErrTokenInvalidSID, err)
 }
}

func TestValidateTokenInvalidUID(t *testing.T) {
 // Create a new client
 client := NewClient("some-secret-key")

 // Create a test token
 token := jwt.New()
 // Set the uid value to something invalid
 token.Set("uid", "invalid-uid")
 token.Set("sid", uuid.New().String())
 token.Set(jwt.ExpirationKey, time.Now().Add(time.Minute * 5).Unix())

 // Sign the token
 signedToken, err := jwt.Sign(token, jwt.WithKey(client.alg, client.hmacSecret))
 if err != nil {
     t.Errorf("unexpected error signing token: %s", err)
 }

 // Validate the token
 _, err := client.validateToken(signedToken)
 if err != domain.ErrTokenInvalidUID {
     t.Errorf("expected error to be %s, got %s", domain.ErrTokenInvalidUID, err)
 }
}

Sweet as. Sadly it's missing a bit of context here too, so I can't copy-paste the test as is. jwt.Sign() isn't returning a jwt.Token, but the jwt token in signed form, so it can't get passed to validateToken, but there's no way that it could have known that, without feeding the the jwt package into it first. Besides that it looks good, and the tests run after fixing that!

And to go full circle, now that we have more unit tests, let's ask it to optimize my validateToken function as well:

One possible optimization for the validateToken function would be to move the code that parses the sid and uid values into a separate function, and then call that function from validateToken instead of duplicating the code. This would make the code easier to read and maintain, and would also avoid repeating the same error-checking code multiple times.

Here is an example of how the optimized validateToken function could look like:

func (c *Client) validateToken(t jwt.Token) (*Token, error) {
 // Validate the token
 if err := jwt.Validate(t); err != nil {
     return nil, err
 }

 // Parse the sid and uid values from the token
 sid, err := c.parseSID(t)
 if err != nil {
     return nil, err
 }
 uid, err := c.parseUID(t)
 if err != nil {
     return nil, err
 }

 return &Token{
     SessionID: sid,
     UserID:    uid,
 }, nil
}

func (c *Client) parseSID(t jwt.Token) (uuid.UUID, error) {
 // parse sid
 sid, ok := t.Get("sid")
 if !ok {
     return uuid.UUID{}, domain.ErrTokenInvalidSID
 }
 sidUUID, err := uuid.Parse(sid.(string))
 if err != nil {
     return uuid.UUID{}, domain.ErrTokenInvalidSID
 }

 return sidUUID, nil
}

func (c *Client) parseUID(t jwt.Token) (uuid.UUID, error) {
 // parse uid
 uid, ok := t.Get("uid")
 if !ok {
     return uuid.UUID{}, domain.ErrTokenInvalidUID
 }
 uidUUID, err := uuid.Parse(uid.(string))
 if err != nil {
     return uuid.UUID{}, domain.ErrTokenInvalidUID
 }

 return uidUUID, nil
}

I was hoping it'd catch the naked .(string) type assertion, but code splitting is also nice :D

Although this iteration doesn't really avoid repeating the same error-checking code multiple times like it claims.

Collaborative hacking with an AI buddy

I use Copilot quite a lot so I'm already used to having an AI pair-programming buddy, but I strongly believe that as these tools get smarter, they will drastically change the way how we as engineers work.

While not there yet, AI will be able to generate more efficient code faster than us, spot errors we'd miss, or flat-out generate entire applications based on spec input.

When Stable Diffusion came out, some people integrated it into their photo editors to quickly generate content based on a prompt. I am sure we'll see something like this, but for code, in no time. Imagine reading through some code, hitting an "AI" button, then ask it to refactor a routine, or give an explanation what that code does. The plugin will then send all the necessary context to the neural net and ask it to do the task.

The next generation of AI powered plugins could also do things like quickly provide feedback when it strongly believes a mistake has been made, or when it has a high confidence score for a completion / refactoring.

I think that would be really dang cool!

Syncing app configurations across machines with unisync

David Mohl — Sat, 19 Nov 2022 00:00:00 +0000

tl;dr : unisync is a little tool I wrote that diffs and syncs your application settings across Macs, powered by unison

Config management across machines is kind of a PITA. Some apps luckily natively support syncing settings, but most don’t.

There are some solutions like the excellent Mackup that try to fix this issue by moving all config files into a shared folder (like Dropbox), then creating symlinks from the application settings folders (eg ~/Library/Preferences/xxxx.plist) into that Dropbox folder.

I used mackup for a while (even with my own config files), but eventually got frustrated when symlinks randomly broke, for example, because the app decided it wanted to write the full file fresh to disk, effectively overwriting the symlink. Then some other apps detect symlinks and refuse to read them.

I tried to mitigate that by creating a PR to mackup that adds support for hardlinks, but tl;dr: hardlinks in APFS are not really hardlinks, and this is even more fragile with cloud storage than using symlinks.

Unison is great

unison is one of these really old tools that’s been around and developed forever. It’s a

*“Unison is a file-synchronization tool for POSIX-compliant systems (e.g. BSD and GNU/Linux), macOS and Windows, with the caveat that the platform must be supported by OCaml”

basically, it syncs files, kind of like a bi-directional rsync: Where rsync syncs one way (and alternatively deletes files), unison goes both ways and finds files that changed, got added or deleted across multiple directory nodes.

Bi-directional sync makes it great for those pesky config dirs that could change on one host without us noticing. By diffing the directories against, say, a directory in Dropbox, unison can determine what actually changes, and copies the files accordingly.

If a file is updated on Host B and is then synced through Dropbox to Host A, unison on Host A would see that the file in Dropbox is now newer than on the local one, and overwrite it with the version from Dropbox.

unison can even sync across machines directly!

Mackup + Unison = Unisync

That brings us to unisync. Unisync is using the Mackup idea of having a library of app configurations that state where config files are, but instead of using symlinks, it calls into unison to do the actual copying of files. So in theory, if files changed on one machine and then get synced to other machines, those files would get marked as changed / new, and get copied over the existing local file.

Because unisync is using unison under the hood, we can do more fancy things like providing a tree with files we want to handle, but add regex or naming patterns to ignore rules:

name: Raycast
friendlyName: raycast
files:
  - basePath: ~/Library/Preferences/
    includedFiles:
      - com.raycast.macos.plist
    ignoredFiles:
      - Name somethingToIgnore.plist

Unisync also supports the entire Mackup app library for syncing to get you started, and comes with LaunchAgents that make it run automatically in the background.

Check it out on GitHub and let me know if this tool is useful for you: https://github.com/dvcrn/unisync

Why and how I use k8s for my personal stuff (and love it)

David Mohl — Sat, 05 Nov 2022 00:00:00 +0000

First published on my personal blog

I’m one of those odd nuts that love kubernetes. I like it so much that it’s currently powering a lot of my personal stuff and hobby projects. So instead of the usual “k8s is bloated”, “k8s is overkill” or “why you don’t need k8s” posts, today let’s talk about why k8s is actually great for personal stuff, and why you should maybe also consider using it? :)

1. Why I love kubernetes for personal stuff

Let’s start with the why. Why would I use something as bloated and heavy as kubernetes for personal stuff, isn’t it a huge overkill? I could go on and on about why it’s great, but let’s take a look at the big reasons.

I want to also preface that small managed clusters can be really cheap. My main managed “cluster” (putting it in quotes because it only has 1 node) on digitalocean costs $12/month. Compare that with other hosting providers.

Infra as code 👨‍💻

k8s, same as with terraform, allows me to specify my infrastructure as code. I write manifest files that specify how stuff is going to get run. If I want to make a change to my infra, all I need to do is update those manifests, apply them, and I’m done. Look at this snippet from a manifest that tells k8s that I want to run a webdav container:

      containers:
        - name: webdav
          image: bytemark/webdav
          ports:
            - containerPort: 80
          env:
            - name: "USERNAME"
                value: "myuser"
            - name: "PASSWORD"
              valueFrom:
                secretKeyRef:
                  name: webdav-credentials
                  key: password

Because I have everything in code, I can set up my entire cluster from scratch on any provider I want. If I no longer like digitalocean, all I have to do is create a cluster somewhere else and do a kubectl apply. The new setup will be identical (or almost identical) to my previous one, it’s almost provider agnostic (there are some provider-specific things like persistent volumes, but we’ll get to those later).

Manifests are also idempotent, I can re-apply the same files a dozen times without worry of breaking anything. If resources already exist, they stay existent and if there are no changes, the manifest will just get ignored.

Extremely easy to add storage or IP addresses 🔄

Managed kubernetes clusters (like the digitalocean one) are integrated with the entire cloud platform of that provider, so it’s very easy to do things like adding block storage or loadbalancers. How easy? Check this out:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: some-name
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: do-block-storage

This manifest when applied will tell DigitalOcean (through the kubernetes controller) to provision a 1Gb block storage volume. If that 1Gb fills up at some point, I change it to 10Gb, re-apply and DigitalOcean will automatically resize the volume to 10GB for me - neat!

Even cooler, I can now use this storage volume on any of my containers that are running inside my cluster. All I need to do is add a mount point into the manifest, and suddenly ephemeral containers have a way to write persistent data:

      volumes:
        - name: some-name-volume
          persistentVolumeClaim:
            claimName: some-name
            readOnly: false

If I want to get rid of the volume, I do kubectl delete pvc some-name and it gets wiped. (There is a retain setting if you don’t want volume content to get deleted)

This isn’t just storage though, the same thing happens with stuff like LoadBalancers and egress IPs. Provisioning resources on kubernetes causes DigitalOcean to get a static IP + LoadBalancer ready for us, wait until it’s good to go, then assign it to the cluster.

It makes running stuff and keeping stuff running a breeze 🍃

K8s is very battle-tested. A lot of companies run on kubernetes so it’s grown to be robust and fault-tolerant. And if something does end up not working or acting up, I can be sure that I will find the answer to my issue in no time.

Besides running containers, k8s does things like keeping those running (if you want that). Let’s take a look at the webdav example from above, but conjure up some more yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webdav
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webdav
  template:
    metadata:
      labels:
        app: webdav
    spec:
      containers:
        - name: webdav
          image: bytemark/webdav
          ports:
            - containerPort: 80
          env:
            - name: "USERNAME"
                value: "myuser"
            - name: "PASSWORD"
              valueFrom:
                secretKeyRef:
                  name: webdav-credentials
                  key: password

The webdav container is now in a Deployment with 1 replica. This means k8s will make sure there is always 1 webdav container running. You can even kubectl delete pod <webdav-xxxx> to destroy the webdav container, and kubernetes will just respawn a new one.

My server can crash, the container can crash - doesn’t matter, don’t care. In the end, kubernetes will jump in and just scale a new container up again.

And if I want more containers? All I have to do is change replicas: 1 to replicas: 2, and now I will always have 2 webdav containers. This makes scaling things up and down as easy as changing a yaml file and applying it.

But there are other cool things that just work like CronJobs! I can specify my CronJob yaml manifests, tell kubernetes when it should run those, and I can be sure they are getting run. No messing with crontab or some cloud scheduling thing, I can do it right from within k8s, together with the rest of my infra.

Easy scaling when it becomes necessary 📈

So we saw in the previous section that we can scale containers however we please. It gets even cooler if we deal with multiple nodes on the cluster. Say my hobby project is getting more popular - instead of a simple web server I had to add a database and maybe a queue server. Resources are getting tight, and stuff doesn’t run as nicely any more.

What I can do is, I can open up the DigitalOcean kubernetes admin, go to my node settings, and either:

Increase the nodes of the cluster to 2
Destroy the current 1-node setup and replace it with a stronger node

Because k8s is managing the cluster, even if we destroy the node and add a new, stronger node in its place, it will still do exactly what it did before: Make sure those containers are running in the configuration we specified as soon as the new machine is up and running.

If we increase the cluster size from 1 to 2, kubernetes will look at available resources on the cluster, see that a new node is available, and balance those containers for us, by maybe moving the queue server to the new machine, or having one of those 2 webdav replicas there instead. (Of course, if we want less magic we can also tell it specifically what it should do and how it should utilize those nodes, like always having one replica on each node)

It goes the other way as well: We’re getting tight on money so we decided to cut down on resources. Removing a node from the cluster will make k8s reshuffle our containers and consolidate them onto the remaining nodes, and THEN shut the node down, without downtime.

2. How I use kubernetes for my personal stuff

Now that you’ve heard the main reasons why I like kubernetes, let’s take a look at how I specifically am using it on a daily basis.

Let’s start with the provider - I mentioned that I use DigitalOcean kubernetes for my cloud of choice. I like them because the cost is predictable and the managed cluster is cheap. I am always sure how my bill will look like and there are no surprises. At $12/node for kubernetes I really have no complaints here. It’s great! This price doesn’t include a LoadBalancer/static IP but for most stuff I don’t really need those.

k8s can do a lot, but the parts that I actually use are:

Deployments: Fancy way of saying “keep this many replicas running at all time”
Services: Fancy way of saying “I need an cluster internal/external IP for these containers”
Ingress: A fancy router like “route domain david.coffee to this pod”
Cron

Exposing stuff for personal use only with tailscale

Most of my stuff doesn’t need to be public. It’s either a cronjob that does something and doesn’t need inbound connections, or it’s stuff that’s only public to me.

To make things public to myself only, I am using tailscale as my overlay network of choice (other great options are ZeroTier and Slack Nebula). Tailscale utilizes wireguard tunnels to build a Layer 3 p2p network that most of the times Just Works™️

The cool thing is, I can embed the sidecar container into any kubernetes pod I want to access, and it becomes available to my tailscale network. All I have to do is add some yaml to a pod:

      containers:
        - name: webdav
          image: bytemark/webdav
          ports:
            - containerPort: 80
          env:
            - name: "USERNAME"
                value: "myuser"
            - name: "PASSWORD"
              valueFrom:
                secretKeyRef:
                  name: webdav-credentials
                  key: password
        - name: ts-sidecar
          imagePullPolicy: Always
          image: "ghcr.io/tailscale/tailscale:latest"
          env:
            - name: TS_KUBE_SECRET
              value: "tailscale-webdav"
            - name: TS_USERSPACE
              value: "false"
            - name: TS_AUTH_KEY
              valueFrom:
                secretKeyRef:
                  name: tailscale-auth
                  key: TS_AUTH_KEY
          securityContext:
            capabilities:
              add:
                - NET_ADMIN

… and when my connection is established, I can just do http://webdav-tailscale:80 and I have a fully personal connection to my webdav container. No ingress or load-balancer needed ✌️

A non-tailscale option would be to use port-forwarding into the pod directly:

  # Listen on port 8888 locally, forwarding to 80 in the pod
  kubectl port-forward pod/mypod 8888:80

Exposing public stuff without static IP + LoadBalancer with Nginx (the frugal option)

Because my cluster has only 1 node, I almost never need a proper loadbalancer or static IP.

My ingress controller of choice is nginx, and we can tweak it to run off a ClusterIP service instead of a full blown load balancer:

helm install ingress-nginx ingress-nginx/ingress-nginx -f nginx-ingress.yml


---
controller:
  kind: DaemonSet
  daemonset:
    useHostPort: true
  dnsPolicy: ClusterFirstWithHostNet
  hostNetwork: true
  service:
    type: ClusterIP
  resources:
    requests:
      cpu: 10m
rbac:
  create: true

A DaemonSet is something we haven’t talked about yet, it’s similar to a Deployment with the exception that it makes sure the thing is running on all nodes of the cluster. So if the cluster has 1 node, there’ll be one of those. If there are 3 nodes, we’ll have 3 nginx controllers.

hostNetwork: true is running that service off the host directly, so we can hook up <machineip>:80 and let nginx handle the routing.

Then once we have that contorller running we can add actual Ingress resources to do something based on domain name like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
spec:
  ingressClassName: nginx
  defaultBackend:
    service:
      name: ingress-nginx-controller
      port:
        number: 80
  rules:
    - host: somethingsomething.david.coffee
      http:
        paths:
          - pathType: Prefix
            path: /
            backend:
              service:
                name: somethingsomething-service
                port:
                  number: 80

To make sure the IP stays up to date, we can hook up a cronjob and update the cloudflare DNS records automatically, for maximum money saving.

Of course if you have the cash and want to scale up, just get a proper static IP with LoadBalancer. It’s only $10 extra.

Happy whale-ing

As you can tell by this long post, I am a fan of kubernetes and use it for my personal stuff a lot. If you read until this point, congratulations! You made it. Let me know on twitter if you think this kind of post is useful, or if there are parts you’d like to know more about.

Happy whale-ing! 🐳

Venturing into WASM on React Native and Golang

David Mohl — Sat, 15 Oct 2022 00:00:00 +0000

Appeared first on my personal blog

I recently had the idea to create a little react native app for FastMails MaskedEmails, to quickly add new ones on my device, so I don’t have to open the FastMail app all the time.

I wanted to do this with as little code duplication as possible, so instead of rewriting a client from scratch, I decided to re-use my maskedemail-cli somehow. I could have probably compiled it down to a shared library and pulled it into Swift, but I am no iOS developer and know React already, so why not do it in React Native with WASM?

I’ve never actually used wasm and was looking for an excuse to play around with it. Here’s what I learned.

You can find the app I worked on during this writeup at https://maskedemailmanager.david.coffee

(Spoiler: You can’t use native WASM within JSC on React Native yet)

Golang and WASM, how does it actually work?

Golang supports compiling to wasm since version 1.11, with further improvements happening in 1.13. The basic gist of making it spit out wasm is

GOOS=js GOARCH=wasm go build -o main.wasm

Main interactions from Golang with JavaScript-land happen through the js/syscall package. For example, if we want to create a js function, we can do it like so:

cb = js.FuncOf(func(this js.Value, args []js.Value) any {
    fmt.Println("Hello World")
    return nil
})

The first argument is always this, and the second argument is always an array of actual arguments passed to the function.

We can also use js.Global() to get a ref to the global object and call into js:

js.Global().Call("alert", "Hello from wasm!")

… and of course, we can also make our functions available on the global object:

js.Global().Set("myCoolFunc", cb)

Using Golang WASM from JavaScript

Golang comes with some glue code that you need to run wasm files built by Golang properly, that’s this copy statement you see in most guides:

cp "$(go env GOROOT)/misc/wasm/wasm_exec.js" .

Besides setting up the environment, this code defines the globalThis.Go object that we’ll use to run our wasm code:

const go = new Go();
const res = await WebAssembly.instantiateStreaming(fetch("main.wasm"), go.importObject);
go.run(res.instance);

(instantiateStreaming supports fetch; that’s why everything is so compact. we can also use WebAssembly.instantiate(arrayBuffer, go.importObject) directly without fetch).

go.importObject provides values to be imported into the new instance, which defines all the supported syscalls

Preparing maskedemail-cli for WASM - async code and Promises

First, we need to create glue code that exports the necessary functions from maskedemail-cli to JavaScript, in Golang.

func main() {
    done := make(chan struct{}, 0)
    js.Global().Set("maskedemailList", js.FuncOf(list))
    <-done
}

func list(this js.Value, args []js.Value) interface{} {
    // (...) get maskedemails here (...)
    return maskedEmailArray
}

Now here’s a problem: We can’t block the main goroutine with things like HTTP requests. It would be kinda weird if we do an ajax request in Golang and the browser freezes while our functions get executed…

Instead, async code has to go into separate goroutines, which means in JavaScript… Promises!!

Specifically, to make our code compatible with js promises from Golang, we have to use the Promise constructor through js.Global(), effectively creating a Promise from scratch:

func list(this js.Value, args []js.Value) interface{} {
    handler := js.FuncOf(func(this js.Value, args []js.Value) interface{} {
        resolve := args[0]
        reject := args[1]
        go func() {
            // (...) get maskedemails here (...)
            resolve.Invoke("Hi there")
        }
    })
    promise := js.Global().Get("Promise")
    return promise.New(handler)
}

js.Global().Get("Promise") is getting the global Promise object, then calling New onto it with a custom handler.

Running main() from above would now populate globalThis.list, and it’s signature would look like this: list(): Promise<string>.

Creating a NPM package that’s using WASM

Next, I wanted to publish an NPM package to pack away the complexity, but most guides on Golang + wasm were mainly focused on the browser, not NodeJS.

Luckily the key steps are effectively the same:

Use the wasm_exec.js glue script to populate globalThis.Go
Load the wasm file through whatever method
Run WebAssembly.instantiate or WebAssembly.instantiateStreaming, pass go.importObject
Run go.run(instance)

In NodeJS, we can solve 1. by doing a dirty require("./wasm_exec.js"). The code runs fine in the nodejs runtime so no issues there. Too bad it’s polluting globalThis and doesn’t give us a nice-to-use package back, but we can live with that.

Point 2. is a tad trickier - we could of course, just load it with fs.readFileSync:

const go = new globalThis.Go()
const buffer = fs.readFileSync('./main.wasm');
const inst = await WebAssembly.instantiate(buffer, go.importObject);
go.run(inst.instance);
// run func
globalThis.list();

… but given that this is JavaScript, usage of fs means that this is already unusable in a browser environment, which is not great.

What’s more, wasm_exec.js depends on a few browser-specific modules, mainly crypto and TextEncoder. So now we have some stuff that doesn’t work in the browser, and also some stuff that doesn’t work within node 😀 Let’s fix that with some polyfills:

if (!globalThis.crypto) {
  const crypto = require("crypto");
  globalThis.crypto = crypto;
}

if (!globalThis.TextEncoder) {
  const te = require("text-encoding-polyfill");
  globalThis.TextEncoder = te.TextEncoder;
  globalThis.TextDecoder = te.TextDecoder;
}

const go = new globalThis.Go()
const response = await fetch('./main.wasm');
const buffer = await response.arrayBuffer();
const module = await WebAssembly.compile(buffer);
const instance = await WebAssembly.instantiate(module);
go.run(inst.instance);
// run func 
globalThis.list()

Instead of fs, we can use node-fetch (or the new NodeJS native one) and fetch to pull the wasm file, then continue instantiating like usual. Now it’s no longer node/browser or fetches implementation dependent.

My final solution

Because I wanted to package this for ESM and CJS simultaneously, I went with rollup as bundler. rollup-plugin-wasm can handle loading + instantiating of the actual wasm file similar to how we did above, so we can slim the code down like this:

// polyfills here 
require("./wasm_exec.js");
import wasmMain from "./static/main.wasm";

const go = new globalThis.Go();
const wasmObj = await wasmMain(go.importObject);
go.run(wasmObj.instance);
globalThis.list()

Packaging into a usable module

To make everything a tad nicer, let’s export some functions that other modules can use:

interface Mod {
  list: (token: string, accountId: string) => Promise<MaskedEmail[]>;
}

const instantiatedPromise = new Promise<Mod>(async (resolve, reject) => {
  const go = new (globalThis as any).Go();
  const wasmObj = await wasmMain(go.importObject);
  go.run(wasmObj.instance);

  resolve({
    list: (globalThis as any)["maskedemailList"],
  });
});

export const list = async (token: string, accountId: string) => {
  const mod = await instantiatedPromise;
  return mod.list(token, accountId);
};

We have some waiting to do before wasm is usable, so we package all of that into a new instantiatedPromise. The exported list() is then awaiting that promise before doing anything, then executing mod.list, our actual wasm function.

To make the API nicer to use I’ve added typescript type definitions for the module, that’s what the interface Mod is for. instantiatedPromise will resolve with Mod, so calling mod.list will have the same return signature as Mod.list

The package is available on npm, the code on my github

Investigating React Native + WASM

By now, we have packaged everything so neatly that it doesn’t matter that we used Golang to create the wasm binary. It’s all just an npm package that should be usable everywhere. (Ignore the big 7MB file size of the .wasm file for now 😛)

So all we have to do is yarn add maskedemail and that should be it… right? Right??

But life is not that simple, and adding that final lego piece was much harder than expected. Let’s go over what is making wasm + react native so hard?? (This was written by poking the simulator and debugging errors)

WebAssembly.instantiateStreaming / WebAssembly.compileStreaming only works with fetch if the server is returning application/wasm, which react-native doesn’t. In fact, using fetch to pull in resources the same way we do in NodeJS or the Browser is in RN flimsy at best
- Expo has Asset.fromModule to make it possible to load arbitrary assets from disk
WebAssembly.instantiateStreaming / WebAssembly.compileStreaming doesn’t even exist in the react native JavaScriptCore runtime (WebAssembly exists, but those functions don’t. Only WebAssembly.instantiate and WebAssembly.compile do, which work with ArrayBuffers
Current asset loading functionality through Asset.fromModule or fetch implementations for react native can pull assets, but rn’s FileReader does not implement readAsArrayBuffer, so no fetch().then((res) => res.arrayBuffer()). No ArrayBuffer means no easy way to use WebAssembly.instantiate, and also no easy trickery to convert Blob into an ArrayBuffer
RN’s JavaScriptCore does not have a crypto implementation
Expos packager is trying to resolve all require and build and doesn’t support dynamic imports, so having require("crypto") in an if statement (like a env-dependent polyfill) will still make the packager throw an error that the package doesn’t exist, effectively meaning that our generic solution ain’t gonna work

I probably forgot a few points but these were the most noticeable issues.

I banged my head against FileReader.readAsArrayBuffer to make it possible to read the .wasm file from disk, tried a couple polyfills and re-implementations, but eventually gave up. There are fs.readFile implementations that could work, but expo (the tool I am using to bootstrap my RN application) does not support linking libraries, so we can’t easily add things that require editing of the application bundle.

To make things worse, the package resolving the issue I mentioned above also meant that all the glue code that rollup-plugin-wasm generated to switch based on environment for browser + nodejs doesn’t work either, because if there is a call to require("fs"), expo breaks. Same thing for attempts to polyfill crypto in the correct environment with crypto-browserify.

Fixing RN issues one by one

Looking at wasm_exec.js, we can see that it doesn’t need the entire crypto package, just one single function:

if (!globalThis.crypto) {
    throw new Error("globalThis.crypto is not available, polyfill required (crypto.getRandomValues only)");
}

That’s much easier to polyfill with something like polyfill-crypto.getrandomvalues (though using Math.random() instead of proper entropy source…)

We can swap out our existing polyfill for this implementation:

import getRandomValues from "polyfill-crypto.getrandomvalues";
if (!globalThis.crypto) {
  globalThis.crypto = {
    getRandomValues,
  };
}

Now the import no longer breaks nodejs, browser, or react-native JavaScriptCore :)

Loading the .wasm file is a different beast, and the polyfills for FileReader.readAsArrayBuffer caused expo to crash consistently. So, what do we do if we can’t load something from disk in the format we need? We play dirty: Instead of async loading the .wasm file like a good citizen, we base64 that chunker and stuff it into our main JavaScript file!

rollup-plugin-wasm even has an option to inline wasm:

wasm({
  sync: ["main.wasm"],
  targetEnv: "auto-inline",
  maxFileSize: 0,
}),

The result will make any editor without proper optimizations for uber-large files crawl

… but it does work inside the Simulator! We’re no longer dependent on asset loading logic anymore. It’s all just js now.

To fix remaining FileLoader issues, there is this patch available that polyfills the functionality. This patch alone doesn’t work to load the wasm file from disk and crashes the simulator, but it does work for smaller under-the-hood functionality that relies on readAsArrayBuffer.

With that out of the way, my app finally runs in the Simulator 🎉

I thought this would be the end of the story and continued chipping away on my app. After days of work, it was finally ready to run it on my iPhone to do a final test before release. Compiling everything… running…

You gotta be kidding. WebAssembly does not exist at all in JavaScriptCore when running on an actual device, but it does exist inside the Simulator. Apparently, Apple shut it down in recent iOS versions, but from reading through posts, this used to be available in the past.

… the end? Did we fail? What next?

I’ve been considering what next steps to take to get this working, but without proper WebAssembly support in JSC on devices, performance will always be subpar. Here are some thoughts and directions I investigated:

Option 1: Fallback to WKWebView

This is the most promising but also a bit absurd. We create a WebView, inject our wasm binary into the webview and execute it there, then relay messages (with window.postMessage) back and forth with our main app.

In my tests, this actually worked pretty well, with the problem that making something generic that just works is a bit hard. Actually, polyfilling window.WebAssembly would mean we need to somehow bridge go.importObject into our webview. On top of that, we need to proxy the changes the wasm binary did (like modifying globalThis) back to our native app using a transparent message relay, so libraries that rely on it can just work.

Without a proper polyfill, window.WebAssembly won’t be available in global scope, so our maskedemail NPM package won’t be able to use it.

All in all, a lot of headache. Probably better to split native and bridged code altogether and write the glue code from scratch. (react-native-react-bridge tries to get around this by using a separate entrypoint that’s getting processed with babel)

Option 2: wasm2js

binaryen includes a tool called wasm2js that compiles wasm files back to JavaScript. I have tried this as well, but could not get it running with my library, probably due to complexity with some stuff like reflections and HTTP requests. Very promising but very hard to debug due to the way this tool spits out JavaScript. Either it works, or it doesn’t; for me, it didn’t. (Though the errors I got were actual Go errors related to HTTP, so it almost worked)

Option 3: gopherjs

Not a solution to wasm, but to make Go run within react-native. Instead of building a wasm binary, gopherjs can output functional JavaScript so we don’t have to deal with WebAssembly runtime at all.

Honestly, I think this might be the best option for code reusability until WebAssembly support is fully available in JSC.

What I ended up doing: WASM -> GopherJS

I’ve tried all options listed above, and the GopherJS variant was the nicest to work with. Even better: The Go API for GopherJS is almost identical to WASM! For example:

This is the WASM version:

func list(this js.Value, args []js.Value) interface{} {
    handler := js.FuncOf(func(this js.Value, args []js.Value) interface{} {
        resolve := args[0]
        reject := args[1]
        go func() {
            // (...) get maskedemails here (...)
            resolve.Invoke("Hi there")
        }
    })

    promiseConstructor := js.Global().Get("Promise")
    return promiseConstructor.New(handler)
}

This is the gopherjs version:

func list(this *js.Object, args []*js.Object) interface{} {
    handler := js.MakeFunc(func(this *js.Object, args []*js.Object) interface{} {
        resolve := args[0]
        reject := args[1]
        go func() {
            // (...) get maskedemails here (...)
            resolve.Invoke(out)
        }()

        return nil
    })

    promiseConstructor := js.Global.Get("Promise")
    return promiseConstructor.New(handler)

In fact, it’s so similar that I was able to flat-out generate the gopherjs main.go file with this script:

.PHONY: generate-gopherjs
generate-gopherjs:
    rm -rf cmd/gopherjs
    mkdir -p cmd/gopherjs
    cp main.go cmd/gopherjs/
    sed -i '' 's/syscall\/js/github\.com\/gopherjs\/gopherjs\/js/g' cmd/gopherjs/main.go
    sed -i '' 's/FuncOf/MakeFunc/g' cmd/gopherjs/main.go
    sed -i '' 's/js\.Value/\*js\.Object/g' cmd/gopherjs/main.go
    sed -i '' 's/js\.Global()/js\.Global/g' cmd/gopherjs/main.go

With a bit of JavaScript, I was able to add an automatic fallback to the gopherjs version to maskedemail-js, so it’s now usable even when window.WebAssembly isn’t available (although ships with 2x 8MB files).

No WASM in react-native as of now

A bit sad I wasn’t able to get my goal of using real WASM working within React Native, but learned a good bit with this adventure.

For now, I’ll use GopherJS when I want to re-use Go code, but I hope we’ll see actual WebAssembly support in JSC on devices soon-ish. With all the WASM hype, I doubt it’ll take long.

There is also stuff like Wasmer that can embed WASM binaries into other languages like Swift, so instead of going the JSC route, it should be possible to embed it into native modules directly or non-RN apps - for when we really want to re-use the same binary at different places. But that’s also doable with Golang, without WASM.

The app is available at https://maskedemailmanager.david.coffee

Relaying emails with Hide My Email back and forth

David Mohl — Sat, 24 Sep 2022 00:00:00 +0000

Mirrored from my blog - https://david.coffee/relaying-with-hide-my-email/

I’m a big fan of one-time-use emails (as you can probably tell by previous posts on this blog). However, one thing that iCloud’s Hide My Email is still unbeatable at compared to my own domain is full anonymity. When I sign up for a junk site to get free WiFi, or order at a restaurant, I don’t even want to expose my domain to them, because that domain is, well, registered to my real identity.

Hide My Email on the other hand is fully anonymous: Emails come from “@icloud.com”, no name in the headers, nothing. Just a fully anonymous blank email. Great! Services can’t even filter for it (compared to mozillas solution, or duckduckgo), because filtering would mean shutting access to everything coming from icloud.com.

Here’s one thing I wanted to get working for a while: Instead of having <site or person> -> one-time-use Email (iCloud) -> Me, I want it the other way around: A one-time-use email that allows me to anonymously send to a specific recipient whenever I send something to that email. Turns out you can actually do that in Apple’s Hide My Email! Let’s take a look how

How Hide My Email actually works

Hide My Email is acting as a relay in between the user account and the sender. You don’t actually send emails from that randomly generated @icloud.com address. In fact, you can’t. You can’t authenticate with that email on the iCloud SMTP server because that email is not yours.

Here’s what I mean: Let’s compose a new email and use Hide My Email

When I click “Send”, the email arrives successfully with the hidden iCloud email as sender, as expected:

… but when we inspect the actually sent email, it looks like this:

Do you see it? The email never got sent from “heave_balks_0g@icloud.com”, instead what Mail did is, it inserted a relay address as the actual receiver of the email. That’s the long test_at_davemail_io_5rvm…@icloud.com you see there. Just that Mail is doing this so transparently that you’ll likely never even notice what is happening.

Relaying back and forth

So now that we know this relay address, what happens when we send an email directly to it? Let’s try!

Sending, and hey look at this! Doing it manually gives us the same result! (duh)

This means that test_at_davemail_io_5rvrmv74x77007_62a85f4e@icloud.com is the relay address that iCloud mapped to test@davemail.io - Everything I send to it will always arrive at test@davemail.io, kind of like a mini portal that shovels content over to a specific sender.

The other end will always see heave_balks_0g@icloud.com, the other part of the relay, as sender. The final relay looks like this:

heave_balks_0g@icloud.com: Relay to my email
test_at_davemail_io_5rvrmv74x77007_62a85f4e@icloud.com: Relay to the recipient

Hide My Email is unique per recipient, but not fully random

Another question you might have while following this is: Wait a moment, aren’t Hide My Email addresses random? Well, kind of.

Each time you generate a hide-my-email-email (nice word), it will be random, but only per recipient: When I hide my email from plans@tripit.com, it will always be the same @icloud.com email, no matter how often I generate it. You can’t use iCloud to spam people, they’ll see everything coming from the same person.

Cool, but what can we actually do with this information??

Obviously this post was leading up to something, otherwise I would have not put so much time into writing it.

Now that we know that relay addresses don’t change, we can use this knowledge to automate sending completely anonymous emails to specific senders without exposing our main domain, name or any other information.

Even better: Using the relay even shields us from any mistakes, like accidentally leaking personal information in headers. It happened dozen times that I used “forward email” functionality in email clients to forward an email to a service, just to realize that the email service sent that email with my global account email, instead of the one-time-use email. (How would the service even know what email I want to use for forwarding?)

Let’s take a service like TripIt which gives me an email that I can send flight itineraries to: plans@tripit.com. I can link an email to it like “tripit.com@mydomain.com”, then whenever I send an email from this email to plans@tripit.com, it will add that itinerary to my account. This works great, but means I can’t really setup any quick-actions, shortcuts or automations to quickly forward to it, because of the issue I described above. Now we can fix that by directly using the relay address :)

Send an email to plans@tripit.com using Hide My Email
Copy the generated @icloud.com address and add it to TripIt
Copy the generated relay address (the long one aka test_at_davemail_io_5rvrmv74x77007_62a85f4e@icloud.com), create a new contact with it for quick access
Profit.

Now we can hit “forward” (or use a Siri Shortcut) to forward any PDF attachment we want to add to tripit to our relay address, iCloud will do the rest.

All of this effort just so we don’t expose our main email, huh 😉

Keep on shaving that yak

Teaching my UniFi UDM how to IPv6

David Mohl — Wed, 18 May 2022 06:26:28 +0000

Mirrored from my blog: https://david.coffee/teaching-my-unifi-udm-how-to-ipv6/

I recently ventured into IPv6 territory with one goal: Get rid of the router my ISP gave me.

I am a fan of UniFi gear and own an UniFi Dream Machine (upgraded from a UniFi Secure Gateway last year) which is more than capable of handling IPv6, so how hard can it be?

I went into this not knowing a thing about IPv6 and just assumed it was identical to IPv4, just bigger, but boy was I wrong. This is my writeup of what I learned and what my endresult is (spoiler: I got it working)

IPv6 misconceptions

As I explained above, I just assumed that IPv6 was IPv4, but longer. So you have some gateway/subnet-mask combo, your router gets an IP assigned from the ISP and all devices inside the network use that IP outwards facing when interacting with things (= “NAT”). Internally, the router assigns IPv4 addresses based on the gateway/subnet-mask, with each device receiving a unique IPv4 address from the router through DHCP. So far so good.

IPv6 on the other hand does not work at all like this. In IPv6-land, each device receives unique (actually multiple) IPv6 addresses. There is no NAT, so when my iPhone that’s connected to my WiFi communicates with a website, that website will see the unique IPv6 address my iPhone has assigned, not my router’s IP. There are also (unless specifically setup) no local network addresses - each IPv6 address is global. And how can the router assign IPv6 addresses? Enter prefix delegation or IPv6-PD.

The router, after authenticating with the ISP when PD is enabled, receives an entire address space (like /64 or /48) delegated to it. In simple words, the ISP tells the router: “Here’s a chunk of addresses that you can use for your devices, handle it.”

Not all IPv6 addresses are created equal

Let’s take a look at this ifconfig output of my Mac:

en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=50b<RXCSUM,TXCSUM,VLAN_HWTAGGING,AV,CHANNEL_IO>
        ether 4c:20:b8:e7:3b:be
        inet6 fe80::1449:6fa5:65a9:a313%en0 prefixlen 64 secured scopeid 0x6
        inet6 fd74:d39f:a55f::8c prefixlen 64 dynamic
        inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 240f:78:<snip> prefixlen 64 dynamic
        inet6 240f:78:<snip> prefixlen 64 autoconf secured
        inet6 240f:78:<snip> prefixlen 64 autoconf temporary

It doesn’t just have 1 IPv6 address, no that would be far too simple. Instead, it has 5!! Why? Because apparently, we have so many IPv6 addresses now that we don’t need to be sparse with them /s.

There are different kinds of IPv6 addresses assigned to my mac:

inet6 240f:78:<snip> prefixlen 64 dynamic
inet6 240f:78:<snip> prefixlen 64 autoconf secured
inet6 240f:78:<snip> prefixlen 64 autoconf temporary

The secured address is my permanent IPv6 address. This address rarely ever changes and will stick around even after reboots.

The temporary address is, you guessed it, a temporary address.

Lastly we have 2 more:

inet6 fe80::1449:6fa5:65a9:a313%en0 prefixlen 64 secured scopeid 0x6
inet6 fd74:d39f:a55f::8c prefixlen 64 dynamic

fe80::, while not obvious whatsoever, is a prefix for localhost or local link. This is an address that’s only usable locally.

fd74::, more specifically the fd:: prefix is designated for the local network. Didn’t I just earlier say that there are no local network addresses in IPv6? By default, yes, but I specifically setup my router to hand out local network addresses. Keyword here is “Unique local address” (ULA).

IPv6 implementations - Native, 6to4, IPv6oE

Are you still with me?? Okay, because there is more. Not all IPv6 implementations are equal either. Very common implementations are either IPv6 “native” or “IPv6-in-IPv4 tunnel”. There are more exotic implementations like SoftBanks “IPv6高速ハイブリッド IPv6 IPoE + IPv4” that’s utilizing IPoE for authentication and assigning addresses. (Note here: I used SoftBank in the past and was not able to get IPv6 without their router working due to the proprietary way their router authenticates)

I’m not going into detail here and don’t fully understand everything either, but a 6to4 tunnel is like the name implies an earlier implementation that tries to build the IPv6 stack on top of a IPv4 network stack. Read more about it here.

“Native” IPv6 is the one that’s nice to work with, but also the one not every ISP supports. I made sure to pick an ISP that has a native IPv6 stack and doesn’t use IPv6-over-IPv4

Getting started - figuring out what my ISP router is doing

The first step to getting IPv6 working was to actually figure out what my ISP router is doing when I plug it in. Specifically: How does it actually get the IPv6 prefix assigned, how does it authenticate, and what does the handhshake look like.

Now how do we actually do this? Wireshark and a couple of USB-C network dongles to the rescue!

Mac has a lot of nice networking features, one of them is the ability to bridge 2 network interfaces, effectively creating a tunnel between them. This can be done through the CLI, but also through the network preferences pane after plugging multiple network dongles into the Mac.

Specifically, we want to make it so the MacBook is in between the ISP Router and the network out and acting as a transparent bridge. So Internet <-> MacBook <-> ISP Router

Once the virtual bridge is created, we can use Wireshark to monitor the new device for any traffic that happens between the ISP router and the mothership. Now start Wireshark, plug the ISP router in, and….

… it works!

Lot’s of new information here for a non-network engineer like me, but also some familiar words: DHCPv6, ICMPv6, UDP, SSL. The *v6 protocols are IPv6 variants of the same things we have in IPv4, but beefed up and more important in IPv6 land.

Specifically, ICMPv6 is much more important than the IPv4 version that we use for pinging stuff and is used extensively for the Neighbor Discovery Protocol, that’s those “Router Solicitation” and “Router Advertisement” messages you see in the screenshot above.

So from this startup dump we can see:

My ISP router inquires with ISP uplink through Router Solicitation where the uplink router is
Uplink responds with Router Advertisement
ISP router advertises itself through DHCPv6, uses a DUID (“DHCP unique identifier”) to identify itself, this DUID is generated off the mac address
ISP responds to DHCPv6 request and assigns IPv6 prefix
Neighbor Discovery Protocol continues with Neighbor Solicitation and Neighbor Advertisement

(the XID stuff you see in there is a transaction id that’s being calculated on each request)

So to summarize : This ISP is using fully native IPv6, the IPv6 prefix space is assigned through DHCPv6, and authentication happens through a DUID identifier that’s being used in all DHCPv6 requests.

If we are able to replicate this handshake on the UDM, we should have IPv6 working. Now how do we actually do that…

Teaching the UDM how to DHCPv6

From the wireshark dump above, we know that the ISP router is using DHCPv6 to authenticate and get its prefix assigned. We also know which DUID the ISP router uses for identifying itself, and we know the prefix size that we get assigned from the ISP (/48).

Let’s enable IPv6 through DHCPv6 through the UniFi admin on WAN:

Then grep for dhcp to see what’s happening on the UDM:

# ps aux | grep dhcp
 3537 root /sbin/udhcpc --foreground --interface eth4 --script /usr/share/ubios-udapi-server/ubios-udhcpc-script -r 124.214.62.19
 3895 root /usr/sbin/odhcp6c -R -f -e -v -s /usr/share/ubios-udapi-server/ubios-odhcp6c-script -P 48 eth4
 7715 root grep dhcp

The UDM is spinning up udhcpc and odhcp6c with the closed-source binaries ubios-odhcp6c-script, both on the eth4 interface (my WAN port).

We can also print our own duid by running cat /config/dnsmasq.lease, and since the DUID being used is not the one that our ISP router uses, we have to do something about it.

odhcp6c that is being used here is is a DHCPv6 client by openwrt and it being open-source and widely used means we can dig deeper into what it’s doing with all those parameters:

Usage: odhcp6c [options] <interface>

Feature options:
        -S <time> Wait at least <time> sec for a DHCP-server (0)
        -N <mode> Mode for requesting addresses [try|force|none]
        -P <length> Request IPv6-Prefix (0 = auto)
        -F Force IPv6-Prefix
        -V <class> Set vendor-class option (base-16 encoded)
        -u <user-class> Set user-class option string
        -c <clientid> Override client-ID (base-16 encoded 16-bit type + value)
        -i <iface-id> Use a custom interface identifier for RA handling
        -r <options> Options to be requested (comma-separated)
        -R Do not request any options except those specified with -r
        -s <script> Status update script (/usr/sbin/odhcp6c-update)
        -a Don't send Accept Reconfigure option
        -f Don't send Client FQDN option
        -k Don't send a RELEASE when stopping
        -t <seconds> Maximum timeout for DHCPv6-SOLICIT (120)
        -m <seconds> Minimum time between accepting updates (30)

The flag we’re after is -c <clientid> Override client-ID (base-16 encoded 16-bit type + value). So theoretically, if we are able to pass -c into the startup of the odhcp6c command, we should be able to receive a valid IPv6 address from the ISP. Let’s give it a try!

Disable DHCPv6 again through the admin UI, wait until the process fully stopped, then manually run the command above, but with -c flag:

/usr/sbin/odhcp6c -R -f -c <DUID from our wireshark dump> -e -v -s /usr/share/ubios-udapi-server/ubios-odhcp6c-script -P 48 eth4

Wait a bit, then ip -6

# ip -6 addr show scope global
10: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 fd74:d39f:a55f::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 240f:78:<snip>/64 scope global dynamic
       valid_lft 108367sec preferred_lft 108367sec

Hello there, IPv6 :)

Back on the UDM admin, let’s enable IPv6 on the LAN network. We want to do prefix delegation and also advertise ourselves as a router (remember Neighbor Discovery Protocol above?)

Wait a bit, then let’s enable IPv6 on my mac and renew the DHCPv6 lease:

en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        inet6 240f:78:9464::<snip> prefixlen 64 dynamic
        inet6 240f:78:9464:0:<snip> prefixlen 64 autoconf secured
        inet6 240f:78:9464:0:<snip> prefixlen 64 autoconf temporary

Nice!!

And what about https://test-ipv6.com?

Persisting changes to the UDM

Now that we got an IPv6 address, let’s persist those changes so we don’t have to manually run odhcpv6 every time.

Pre-req: Extending the UDM with boot-scripts

UniFi gear is hackable which allows us to do more than just what’s possible through the admin UI. The USG had a very advanced customization flow baked in, but the UDM is a tad more annoying to work with.

Luckily there is this awesome project called udm-utilities by boostchicken, a “A collection of things to enhance the capabilities of your Unifi Dream Machine or Dream Machine Pro.”

Basically what this project does is, it installs a custom package that extends the UDM with the ability to run scripts on boot that persist in between reboots. We put our shell scripts into /mnt/data/on_boot.d, and udm-utilities makes sure those are run when the router boots.

There are a lot of really cool things you can do with it, from configuring the UDM to even running podman containers like homebridge or adguard, but we’ll be focusing on the network aspects only for this writeup.

Changing DUID on boot

Now the next problem is, how do we actually start odhcp6c with our own options? The UDM launches it on boot through ubios-udapi-server, which is closed-source. We could decompile the binary with ghidra and see if we can hack our own arguments in, how hard can it be?

But there is an easier (more hacky) way of achieving just that. What if the binary at /usr/sbin/odhcp6c isn’t actually a binary, but an executable script that runs odhcp6c for us? The UDM will continue to execute odhcp6c like usual and we can inject our custom flags. Here’s what I mean:

# cat /usr/sbin/odhcp6c
#!/bin/sh
# workaround script to inject odhcp6 options
/usr/sbin/odhcp6c-real -R -f -c 0003000<snip> "$@"

The custom script that we put in /usr/sbin is calling odhcp6c transparently and adds the extra -c flag.

Using udm-utilities, we can automate this with the following boot script:

#!/bin/sh
mv /usr/sbin/odhcp6c /usr/sbin/odhcp6c-real
cp /mnt/data/odhcp6c/odhcp6c /usr/sbin/odhcp6c
chmod +x /usr/sbin/odhcp6c
killall odhcp6c

We move odhcp6c to odhcp6c-real
Copy the custom odhcp6c to the old location
Make it executable
Kill odhcp6c

Once killed, ubios-udapi-server will automatically attempt to restart it by executing /usr/sbin/odhcp6c, which is no longer the real binary but our custom script.

And does it work?

# ps aux | grep dhcp
 3891 root {odhcp6c} /bin/sh /usr/sbin/odhcp6c -e -v -s /usr/share/ubios-udapi-server/ubios-odhcp6c-script -P 64 eth4
 3895 root /usr/sbin/odhcp6c-real -R -f -c 0003000<snip> -e -v -s /usr/share/ubios-udapi-server/ubios-odhcp6c-script -P 64 eth4

It absolutely does!

Was it worth it?

Worth it in terms of performance or network benefit? Probably not. The world doesn’t feel different when interacting with IPv6 and speed for me is pretty much on par as it was with IPv4.

But it has been a very nice learning experience. Without this venture into IPv6-land I would have never learned about the differences between IPv6 and IPv4.

My network is now fully IPv6 compatible which it wasn’t before, and that’s an achievement in itself.

Some tiny personal programs I've written

David Mohl — Tue, 22 Mar 2022 09:14:12 +0000

Mirrored from my blog - https://david.coffee/some-tiny-programs-i-ve-written

There was this post on HN the other day titled Some tiny personal programs I've written that I enjoyed a lot. It resonated with me so much because I've been pumping out smaller programs left and right for a while as well, but never thought of sharing those with the public, so let's change that.

I love hacking on mini projects. Fixing a problem with programming feels like a superpower - I can do something that other people that can't program can't do, and every time I find a situation where I can apply my skills to an issue, I feel happy.

Here are some programs I've written in the past few months, in no specific order:

a covid bot

Every day when there were new cases published, people posted Twitter links, links to news and other sources. Turns out Tokyo has an official dashboard that pulls data out of a GitHub repository and is updated daily.

About an hour after discovering those json files, we had a slackbot that posts updates once daily - nice!

apartment finder LINE bot

This one I'm especially proud of because I was able to get my current place thanks to it.

In Tokyo there are 2 'companies' that are owned by the government, and rent out pretty decent places for an ok price, without hidden costs, minimun contract runtime, cancellation fee, and so on. Only problem - these place are gone fast. Like in minutes after something is posted, it's gone.

People don't even bother visiting these places, they just put in an application right away, then visit them and cancel accordingly.

At first I was checking the page every couple hours once, but that was too inefficient, so I hacked together a bot that's using puppeteer to check for new place, and send me a message on LINE whenever something new is available. A couple weeks later and my dream place free'd up!

Fastmail masked aliases through the cli

I wrote a separate blogpost about this one so I'll skip the details, but when Fastmail + 1Password released their 'masked email' (domain specific email addresses) implementation I was stoked because I was doing something like this for a long long time. I wanted to migrate to this new feature, but it wasn't available to the public yet, and only usable through 1Password.

A put a couple hours in to reverse-engineer the fastmail JMAP API with proxyman and hacked together a Golang client that was able to create identical requests. Now I have masked emails everywhere I want by just hammering '@@@' into a dialogue.

work-hour tracking through the cli

At work we use this tool called King of Time to track working hours. It's usable through the web app but doing bulk edits was very tedious. I wanted to be able to hook checkins/checkouts to different events and actions, and the lack of an official API made this difficult.

puppeteer to the rescue (it involved way too much javascript to be usable without headless browser), and tadaa - a npm-installable company internal tool to interact with King of Time was born.

auto-importing Japanese words into my study app

When I was learning Japanese more actively I often encountered words I didn't know, as you usually do. I looked those up in a dictionary, and then usually added them to my flashcard app (Anki).

This became repetitive fast, so I did what engineers do and hacked together a chrome extension that does it for me. Bonus points for writing it in ClojureScript which I wanted to get better with at that time.

auto-compound crypto

This one is the definition of a micro-program. FTX is able to lend away unused crypto and gives interest back. I wanted this interest to be auto-compounded every hour without me having to open the site all the time and click some buttons.

Luckily FTX has an API to do just that, so hooking it up to a script that runs in cron, and we have automatic compounding :)

HomeKit-ifying my TV and aircon

homebridge is an amazing piece of software. It allows to integrate basically anything that has an API into Apples HomeKit, which is what I use at home.

Problem: Neither my Air-Conditioner, nor my TV are HomeKit-compatible. The TV is from a time way before HomeKit was even a thing (running 'netcast'), and my Air-Conditioner is only using through Sharps proprietory 'Cocoro Cloud'.

2 homebridge plugins got created during longer caffeinated sessions, and now I can automate both purely in HomeKit:

netatmo temperature getter

I use a netatmo weather station at home and wanted to get it's temperature to integrate into HomeKit, so I can always see what the current temperature is, but also automate things like my AirCon or fans. This is another micro-script that just pulls the stations from my account and stores it in a text file (/tmp/temperature), running in a CRON

for _, station := range dc.Stations() {
        for _, module := range station.Modules() {
            if module.ID == homeModuleID {
                fmt.Println("Found home module")
                homeModule = module
                break
            }
        }
    }

    temperature := fmt.Sprintf("%.1f", *homeModule.DashboardData.Temperature)
    fmt.Printf("Current temperature at home: %v \n", temperature)
    ioutil.WriteFile("/tmp/temperature", []byte(temperature), 0644)

... and lots more!

It's nice to just hack on things in between professional tasks and work. Low commitment, easy to explore new technology and usually makes something in my day-to-day life a bit easier.

VPN connections and local network traffic on Mac

David Mohl — Fri, 21 Jan 2022 05:55:30 +0000

Mirrored from my blog - https://david.coffee/vpn-connections-and-local-network-traffic-on-mac

These days I've been trying to be more conscious about my online footprint and privacy in general. I've been using one-off email addresses for years now (btw check out my recent post on masked emails if you haven't yet), and have a bunch of little scripts that spit out things like random usernames when I hit "!" 3 times.

But honestly speaking, IP addresses were one of those things I just kind of didn't really care about.

What changed my view drastically was when I hacked on a cloudflare workers project and for debug purpose decided to console.log() the Request object. Here's what got printed

{
      "message": [
        {
          "cf": {
            "longitude": "REDACTED",
            "latitude": "REDACTED",
            "continent": "AS",
            "country": "JP",
            "tlsVersion": "TLSv1.3",
            "colo": "NRT",
            "timezone": "Asia/Tokyo",
            "city": "REDACTED",
            "edgeRequestKeepAliveStatus": 1,
            "requestPriority": "",
            "httpProtocol": "HTTP/3",
            "region": "Tokyo",
            "regionCode": "13",
            "asOrganization": "REDACTED",
            "postalCode": "REDACTED"
          },
        }
      ],
    }

I cut out some of the details here, but the request had longitude, latitude, my internet provider name and a postal code that is very close to mine. Meaning every site I interact with basically knows where I live to a certain degree. That ain't good.

I now have a VPN running almost all of the time, which brings me to the main topic of this post:

When using a VPN, like NordVPN, and you want to let that run 24/7 so that you can move all traffic without leaks through the it, you'll sooner or later end up running into issues with things like local network no longer resolving because it's not part of the virtual network.

How do you fix that?

Temporary rules with route

Mac has a bunch of neat network tools that you can use to get around this, the easiest is route:

# route <target> <gateway>
sudo route -nv add -net 10.0 192.168.1.1

with the -net flag (there is also -host), 10.0 expands to 10.0.0.0, so the entire subnet of 10.0.* will get routed to the gateway 192.168.1.1 instead of the VPN network, sweet! (10.0.5.* is my network for services and smaller devices)

Routes added with route are temporary, so once you restart they're gone which makes them good for experimenting and quickly setting things up in a non-destructive way.

Making things persist

For persistent rules, mac comes with the networksetup tool:

❯ networksetup -listallnetworkservices

An asterisk (*) denotes that a network service is disabled.
Ethernet
USB 10/100/1000 LAN
Wi-Fi
Thunderbolt Bridge
NordVPN NordLynx

The command we want is -setadditionalroutes. This command takes a interface, and then triplets of <target> <netmask> <gateway>

networksetup -setadditionalroutes "Ethernet" 10.0.5.0 255.255.255.0 192.168.1.1

This command adds a route for 10.0.5.* (designated by the netmask 255.255.255.0) and routes all traffic to the gateway 192.168.1.1 instead.

Repeat that for all your interfaces that you use to connect, and done.

To add multiple routes you'll have to specify them in the same go, in triplets:

networksetup -setadditionalroutes "Ethernet" 192.168.0.0 255.255.0.0 192.168.1.1 10.0.5.0 255.255.255.0 192.168.1.1

Now everything from 192.168.* and 10.0.5.* will get routed to my router

About NordVPN

I never talked to NordVPN, but after trialing 4-5 different VPN services that were recommended for privacy, I ended up at Nord and am very happy with them (dark patterns like difficulty to unsubscribe aside). The speed is very consistent and it's one of the only options I was really able to keep turned on 24h a day.

If you're thinking of signing up for NordVPN, consider using my referal

Creating Fastmail Masked Emails without 1Password anytime and everywhere

David Mohl — Fri, 17 Dec 2021 06:35:15 +0000

Mirrored from my blog - https://david.coffee/creating-fastmail-masked-emails-without-1password-anytime-and-everywhere

I love Masked Emails, Fastmails take on one-time-use email addresses. I've been doing something similar myself for years now, but it is a bit of a shame that it's so tightly coupled to 1Password. Sure you can create them through the webinterface now, but it's still kind of a pain to use and not very convenient (unless you use 1Password).

You can't even manually trigger creation of one within 1Password, so when the extension doesn't detect an email signup field, you're sorta out of luck.

What I want is something like this:

Let's get it working.

The first thing I noticed is that the request to create a new masked email (through 1Password) isn't actually coming from the 1Password servers like I first thought, it goes directly from the client to api.fastmail.com, like seen on a reverse-proxy like Proxyman:

By the way, Fastmail also stated they put this functionality right into their open-source JMAP project in a blogpost.

After a bit of reverse-engineering the request format to the API server and replicating it in Golang:

I present you: https://github.com/dvcrn/maskedemail-cli

maskedemail-cli -accountid xxxx -token abcdef12345 create facebook.com

... and you get a new masked email for facebook.com, right in your shell!

Glueing the CLI to shortcuts with KeyboardMaestro

I wanted this to be available everywhere whenever I need to. A full extension for a product that isn't mine would have been too much work, and I wanted it to work outside browsers as well. With something like KeyboardMaestro this is trivial to do

Here's the macro from above:

Insert "..."
Retrieve the URL from the current browser window. I had to use some glue applescript here because Firefox has no scripting interface
Pipe it into the maskedemail-cli
Remove the "..." and paste the newly created email

Other thoughts

I am sure this will become more available in the near future. Fastmail likely has some kind of exclusivity thing with 1Password for now, but it's great to see that we can hack ontop of it so easily. It's also very cool that this is built right into JMAP. I hope we'll see more email providers implementing it!

PS - I just wanted an excuse to write something on dev.to