DEV Community: Boots

Retrieval Augmented Geese - Semantic Search with the HONC Stack

Boots — Wed, 13 Nov 2024 18:13:21 +0000

🪿 Register for November's HONCathon here - ship apps, win prizes - sponsored by Cloudflare, Drizzle, Fiberplane, and Neon

If you've heard the term RAG or "Retrieval Augmented Generation" lately, you may have asked yourself something like "What is that?" or "How did such a ridiculous acronym get so popular?"

I can't answer to the way it was named, but I can tell you this: One of the core pieces of RAG systems is semantic search, which helps map the underlying meaning (semantics) of a search query to the closest-related meaning of a set of documents in a database.

In this post, we'll go over how to implement basic semantic search with the HONC stack, using Hono as an api framework, Neon Postgres as a database, Drizzle as a typesafe ORM, and Cloudflare Workers as the serverless deployment platform.

We'll build a small semantic search engine for Cloudflare's documentation, giving us a system that understands the meaning behind search queries, not just keyword matches.

🔗 Follow along with the code on GitHub

A Conceptual Primer

Before we get started, let's go over some basic concepts. This will be especially helpful if you're not already familiar with vectors and embeddings.

💡 Already familiar with embeddings?
Feel free to skip past this section if you already know about vectors and embeddings and all that jazz.

Vectors

For our purposes, vectors are lists of numbers, like [1, 2, 3, 4, 5].

Vectors are described in terms of their length. An example of a vector of length two would be [.68, .79] and a vector of length three would look like [.883, .24, .6001]. Simple!

When vectors are all the same length, you can compare and manipulate them in interesting ways, by adding them together, subtracting them, or finding the distance between them.

All of this will be relevant in a bit, I promise 🙂

Embeddings

Put shortly, embeddings are vector representations of the meanings of words and phrases… But. Well. That's a little abstract.

My favorite analogy for why these are useful and how they work comes from from AI researcher Linus Lee. He draws a comparison to colors. There's a difference between describing a color with a name, like "blue", versus with an RGB value rgb(0, 0, 255). In this case, the RGB value is a vector of length three, (0, 0, 255).

If we wanted to mix some red into the named color "blue" and make a new color that's just a little more purple, how would we do that?

Well, if all we have is the name of the color, there's not much we can do. We'd just invent a new color and give it a new name, like "purpleish blue". With an RGB value, though, we can simply "add" some red:

rgb(20, 0 , 0) + rgb(0, 0, 255) = rgb(20, 0, 255)

Because we chose to represent our color with vectors of numbers, we can do math on it. We can change it around, mix it with other colors, and have an all around good time with it.

Embeddings are a way for us to do this kind of math on human language. Embeddings are vectors, like RGB values, except they're much larger.

How would you "do math on human language" though? Borrowing an example from the trusty Internet, let's say we have a vector for the word "king", and a vector for the words "man" and "woman".

If we subtract the vector for "man" from the vector for "king" , then add the vector for the word "woman". What would you expect we get?
Wild enough, we would get a vector very very close to the one for the word "queen".

Pretty neat, huh?

Vector Search

Searching across vectors usually refers to looking for vectors that are similar to one another.

In this case, we think of similarity in terms of distance. Two vectors that are close to one another are similar. Two vectors that are far apart are different.

So, in a database that stores vectors, we can calculate the distance between an input vector, and all vectors in the database, and return only the ones that are most similar.
In this post, you will see a reference to "cosine similarity", which is a way to calculate the distance between two vectors. So, don't get freaked if we start talking about cosines.

Basically, instead of looking for exact matches or keyword matches for a user's query, we look for "semantically similar matches" based off of cosine distance.

How Do We Start?

To perform semantic search, we need:

a database that supports vector embeddings
a way to vectorize text
a way to search for similar vectors in the database

To be entirely frank, the hardest part of building semantic search is knowing how to parse and split up your target documents text into meaningful chunks.

I spent most of my time on this project just pulling down, compiling, and chunking the Cloudflare documentation. After that, the search part was easy-peasy.
I will gloss over the tedious parts of this below, but I've provided a link to the script that processes the documentation, for anyone who is interested.

That said, let's go over the stack and database models we'll be using for the actual searching part.

The Stack

We want to expose a simple API for users to search the Cloudflare documentation. Then we want some tools for storing the documents and their embeddings, as well as querying them.

Here's the stack we'll be using for all of this:

Hono: A lightweight web framework for building typesafe APIs
Neon Postgres: Serverless postgres database for storing documents and vector embeddings
OpenAI: To vectorize documentation content and user queries
Drizzle ORM: For constructing type-safe database operations
Cloudflare Workers: To host the API on a serverless compute platform

Setting up the Database

First, we define a schema using Drizzle ORM.

When we craft our database models, we have to think of what kind of search results we want to return.
This leads us to the idea of "chunking", which is the process of splitting up the text into smaller chunks.

The logic is: We don't want to match a user's query to entire documents, because that would return a lot of irrelevant results.
Instead, we should split up each documentation page into smaller chunks, and match the user's query to the most semantically similar chunks.

Since we're working witha relational database, we can define a schema for the documents and chunks, where each document can have many chunks.

So, our Drizzle schema defines two main tables:

documents: Stores the original documentation pages
chunks: Stores content chunks with their vector embeddings

export const documents = pgTable("documents", {
  id: uuid("id").defaultRandom().primaryKey(),
  title: text("title").notNull(),
  url: text("url"),
  content: text("content"),
  hash: text("hash").notNull()
});

export const chunks = pgTable("chunks", {
  id: uuid("id").defaultRandom().primaryKey(),
  documentId: uuid("document_id")
    .references(() => documents.id)
    .notNull(),
  chunkNumber: integer("chunk_number").notNull(),
  text: text("text").notNull(),
  embedding: vector("embedding", { dimensions: 1536 }),
  metadata: jsonb("metadata").$type<Array<string>>(),
  hash: text("hash").notNull()
});

Once we define a schema, we can create the tables in our database.
If you're following along on GitHub, you can run the commands below to create the tables.
Under the hood, we use Drizzle to generate migration files and apply them to the database.

pnpm db:generate
pnpm db:migrate

Now, with our database set up, we can move on to processing the documentation itself into vector embeddings.

Processing Documentation

The heart of our system is the document processing pipeline. It's a bit of a beast.
I'm going to move through this quickly, but you can see the full implementation in
./src/scripts/create-vectors.ts.

This script:

Takes HTML documentation files as input
Uses GPT-4o to clean and chunk the content
Generates embeddings for each chunk
Stores everything in our Neon Postgres database

Once this runs, we have a database full of document chunks and their embeddings, and we can move on to building the search API.
Probably the most interesting part of this script is how we use GPT-4o to clean and chunk the content. In some cases, this might be
cost prohibitive, but for our use case, it was a no-brainer. We only need to run this once, and it was a lot smarter than any heuristics I would've defined myself.

If you're following along on GitHub, I commend you. You can run these commands to process the documentation. (Please file an issue if you run into any problems.)

# Process the Cloudflare documentation
cd data
bash copy-cf-docs.sh
cd ../
# Create the vector embeddings
pnpm run vectors:create

The Search API

When a user makes a search request, we do the following:

Convert their query into a vector embedding
Use cosine similarity to find the most relevant chunks for their query
Return the top matches back to the user

Why convert their query into a vector embedding? We want to find the most semantically similar chunks,
so we need to represent their query in the same format as our database chunks.

The search endpoint is surprisingly simple thanks to Hono.
We define a GET route that takes query and similarity parameters.

app.get("/search", async (c) => {
  // Parse query parameters
  const query = c.req.query("query");
  const similarityCutoff =
    Number.parseFloat(c.req.query("similarity") || "0.5") ?? 0.5;

  // ...
});

Then, we make a request to OpenAI to create an embedding for the user's search query.

app.get("/search", async (c) => {
  // Initialize the OpenAI client
  const openai = new OpenAI({ apiKey: c.env.OPENAI_API_KEY });

  // Create embedding for the search query
  const embeddingResult = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: query
  });
  const userQueryEmbedding = embeddingResult.data[0].embedding;

  // ...
});

Finally, we craft a similarity search based on the cosine distance between the query embedding and each chunk's embedding.
Here, the drizzleSql helper is the sql helper exported by Drizzle, which has been renamed for clarity.
It allows us to construct type-safe sql expressions.

app.get("/search", async (c) => {
  // Craft a similarity search query based on the cosine distance between
  // the embedding of the user's query, and the embedding of each chunk from the docs.
  const similarityQuery = drizzleSql<number>`1 - (${cosineDistance(chunks.embedding, queryEmbedding)})`;

  // Search for chunks with similarity above the cutoff score
  const results = await db
    .select({
      id: chunks.id,
      text: chunks.text,
      similarity: similarityQuery
    })
    .from(chunks)
    .where(drizzleSql`${similarityQuery} > ${similarityCutoff}`)
    .orderBy(drizzleSql`${similarityQuery} desc`)
    .limit(10);

  // ...
});

That leaves us with a list of chunks that are semantically similar to the user's query!
We can choose to render these results however we see fit.

When we use Fiberplane to test the search API, we can see the timeline of the request, including the embedding generation, similarity search, and result rendering.
We can also see the raw SQL that was executed, which is a little unwieldy since we're dealing with vectorized queries:

And that's it! We've built a semantic search engine with the HONC stack.

The Magic of Vector Search

What makes this more powerful than regular text search? Again, vector embeddings capture semantic meaning. For example, a search for "how to handle errors in Workers" will find relevant results even if they don't contain those exact words.

Neon makes this simple, easy, and scalable by allowing efficient similarity searches over high-dimensional vectors out of the box.

However, that doesn't mean that vector search is the only tool for retrieval. Any robust system should consider the trade-offs of vector search vs. keyword search, and likely combine the two.

Deployment

🔗 See it in action - Live Demo

Deploying to Cloudflare Workers is straightforward. We just push the code and set the secrets.

# Set your secrets
wrangler secret put DATABASE_URL
wrangler secret put OPENAI_API_KEY

# Deploy
pnpm run deploy

Conclusion

If you're looking for a quick way to get up and running with semantic search, this should give you a solid starting point.

Worth noting: there are a lot of libraries out there that can take care of the heavy lifting of building RAG systems.
At Fiberplane, we've built smaller systems with Llamaindex and Langchain,
but for simpler use cases, I found myself wading through documentation and GitHub issues more than I'd like.
A higher-level library can be very helpful for bigger applications, but for something this simple, I think the core concepts are valuable to implement and understand.

To that end, the full code for everything in this post is available on GitHub, and you can adapt it for your own needs.

🔗 Get the code on GitHub

Otherwise, don't forget to check out the HONC stack for more examples of building with Hono, Drizzle, Neon, and Cloudflare.

A Quick JS Reduction: My possibly-too-clever way to reduce an array into one object in javascript

Boots — Mon, 20 Apr 2020 11:48:32 +0000

Let's say you have an array of user objects, all with unique IDs, but you want to group them in a single object by their ids.

// These are your only users
const users = [
  { id: 1, name: "Paul" },
  { id: 2, name: "Ringo" },
  { id: 3, name: "George" },
];

An easy way to do this using Array.prototype.reduce without creating too many intermediary objects would be:

const usersGroupedById = users.reduce((res, u) => (res[u.id] = u, res), {})

I have a feeling every linter in the world would cry linty tears upon seeing this. But honestly, I kind of like it!

Does this code offend you? How would you do this?

My alternatives were:

/// Create objects willy-nilly because they are cheap
users.reduce((res, u) => ({ ...res, [u.id]: u }), {})

// Use Object.assign to avoid spread syntax, still create a new object for every iteration
users.reduce((res, u) => Object.assign(res, { [u.id]: u }), {})

// Arguably more readable version of the featured solution
users.reduce((res, u) => {
  res[u.id] = u;
  return res;
}, {})

Twenty-second article! An important difference between parseInt and + in javascript

Boots — Tue, 14 Apr 2020 07:48:43 +0000

I'll be brief!

parseInt('.3', 10)
- returns NaN
+'.3'
- returns 0.3 (a Number)

The + operator might be slower than parseInt, but it's a bit more intuitive imo... and my goodness the title was almost as long as the article itself.

If you could only get one tech newsletter, what would it be?

Boots — Fri, 10 Apr 2020 08:35:09 +0000

What's one newsletter or digest that you could not live without as a developer?

I realized I get close to ten weekly newsletters about tech-related things.

I certainly do not read them all every week. I try to glance through a few (especially when the subject line is interesting) when I'm waiting for builds and what have you, but yeah. I'm thinking of unsubscribing from a handful to reduce inbox clutter, which got me to thinking: Which one gives me the most consistent value?

As of this moment, I'd personally say Web Dev Monthly by Andrei Neagoie. He does an end-of-the-month summary of the most important web development happenings for that month. I know "most important" depends greatly from person to person, but in my opinion his round-ups are pretty spot on.

Does dev.to ever just hang and not load for you?

Boots — Sat, 04 Apr 2020 08:54:30 +0000

Hey! So. It never occurred to me to check if others were experiencing this bug... on dev itself!

Simple question though: Do you ever enter the URL, and nothing happens? Like the browser progress thingy in the address bar makes some moves, but then it just freezes?

If so, you might be entitled to compensation!

No. Not really. But! You might be hitting a ServiceWorker bug that I just submitted a PR for!

If you're experiencing this bug as well, could you comment on this issue on GitHub?

It'd be nice for me to know the impact of what I'm trying to fix!

Thanks,

Three data structures you do not need to know in 2020

Boots — Fri, 03 Apr 2020 22:34:50 +0000

It seems that you, dear reader, have been compelled to brush up on your data structures, or at least the ones that you will never, ever need to know. Ever.

I commend your studiousness. As a reward, here are three useless data structures that you'll never need to know about, neither now nor for the remainder of the current garbage year that we're calling ✨2020✨.

🌳 1. The giving trie

The giving trie is a variant of the classic German-speaking trie, except this data structure is only half German. (Don't worry, the other half is actually quite friendly¹.)

The giving trie has no interface to insert new items. It can only remove its elements until there’s nothing left but a nebulous melancholic log message that confuses young children and adults alike.

class GivingTrie {
  OLD = Symbol("I am now an old trie")
  constructor(...initialTrieStuff) {
    this._trieStuff = initialTrieStuff
  }
  takeUntil(signal = OLD) {
    if (signal === OLD) {
      const everything = this._trieStuff
      this._trieStuff = []
      return everything
    }
    throw new Error('I only get old. That is all I do. I am a Trie.')
  }
  takeMore() {
    const remainingTrieThings = this._trieStuff?.length
    if (remainingTrieThings) {
      return this._trieStuff.pop()
    }
    console.log("All I have left is this log message.")
    console.log("But I am happy.")
    console.log("But there is nothing left of me.")
    console.log("Also I am a trie.")
  }
}

/*********************
 *** Example Usage *** 
 *********************/

const gt = new GivingTrie(1, 2, 3)
gt.takeUntil(GivingTrie.OLD)
gt.takeMore()
// Prints:
// => All I have left is this log message.
// => But I am happy.
// => But there is nothing left of me.
// => Also I am a trie.

📝 Also worth noting is that the giving trie is, in fact, nothing like a trie at all.

🐒 2. The linked-by-a-barrel-of-monkeys list

Typically, linked lists have banal, straight pointers between their nodes.

The linked-by-a-barrel-of-monkeys list, on the other hand, is a linked list where each node is connected by tiny monkeys that link arms with each other. Hence, its name.

The thing about these tiny monkeys is that they are constantly moving around, changing places, etc, which makes the next and prev methods much more whimsical than a traditional linked list.

Usage of next and prev is subject to random monkey-business. Their values change on each access, and a node could possibly link back to itself or start flying through Oz.

Proper use of the next and prev accessors should guard against potential FlyingToOzErrors, since these are thrown roughly 10% of the time you access said properties. (The monkeys really like Oz.)

One last note: This data structure always has a tail property, which, when accessed, returns a gif of a surprised or angered monkey.

class FlyingToOzError extends Error {
  constructor(message) {
    super(message);
    this.name = 'FlyingToOzError';
  }
};

class LinkedBarrelOfMonkeysList {
  // Utility getter that might throw a FlyingToOzError
  pickRandomOrFlyToOz = list => {
    const shouldFlyToOz = !Math.floor(Math.random() * 10)
    if (shouldFlyToOz)
      throw new FlyingToOzError('OOaAOo I FLY TO OZ!')
    return list[Math.floor(Math.random() * list.length)]
  }

  constructor(...initialValues) {
    const bomInstance = this;
    this._items = initialValues
      .map(value => ({
         value,
         get prev() {
           return pickRandomOrFlyToOz(bomInstance._items)
         },
         get next() {
           return pickRandomOrFlyToOz(bomInstance._items)
         }
       }))
    Object.defineProperty(this, 'tail', {
      writable: false,
      get: () => console.warn(
        'OOoOOAAaAA NO POL TAIL!',
        'https://media.giphy.com/media/l378hyTQvH3bCdD2g/giphy.gif'
      )
    })
  }
}


/*********************
 *** Example Usage *** 
 *********************/

const bomList = new LinkedBarrelOfMonkeysList(1, 2, 3)
bomList.tail 
// This will print a warning, along with a link to the gif below

OOoOOAAaAA NO POL TAIL

Say what you will about these nondeterministic rascals—they’re a heck of a lot more fun than your traditional linked lists.

🤤 3. The lack of priorities queue

The lack of priorities queue uses a first-Doritos-last-whatever algorithm to resolve which element it should return next.

The function for accessing the next item in the queue is asynchronous, and does not have a timeout mechanism. You just have to chill and be patient.

A common workaround to this queue’s inconsistent priority ranking is to wrap known high priority items inside a struct-like Tortilla, as if it were a virtual burrito. Tortillas provide a priority boost to their wrapped values².

Note that a nap subroutine may be implemented to reduce lag, but it’s not guaranteed to work.

For this data structure, I provide no reference implementation. I might write one like next week or something or whatever.

🤦🏻‍♂️ Conclusion

There you have it. Three useless data structures that you will never use. I hope you learned something useless.

Frankly, I'm kind of surprised you made it this far. This is complicated stuff. Really complicated! Why not reward yourself with a little nap?

this guy knos what i'm talkin bout

💤
💤
💤
💤
💤
💤
💤
💤
💤
💤
💤
💤

...Hey! If you're still awake why not hit the follow button? You'll get a notification when I implement that LackOfPrioritiesQueue!

Also Entschuldigung. Ich muss es hier sagen dass die Deutsche Sprache, Kultur, und Menschen gefallen mir sehr. (Ich träge manchmal Lederhosen. Ernsthaft!) Ich wollte am Anfang nur einen kleinen Witz machen. Hoffentlich ist der nicht ganz in die Hose gegangen 😅 🇩🇪 🥨 🍻 ❤️ ↩
In some cases, Durum wraps may also be used. ↩

How did you estimate your coding experience level for dev.to?

Boots — Tue, 31 Mar 2020 20:43:09 +0000

I'm curious: How did others feel about evaluating their experience level inside their dev.to profile?¹

I'll be honest about something that I feel kind of weird about... I just set mine to 5, the max ooOOOoOoO.

Do I really feel like "la crème de la crème" of engineers? Like a 5 out of 5?? Like a 5x dev?!? No. Absolutely not.

However, I wanted to start seeing more challenging content in my feed, and I figured that would be the most straightforward way.

Hence, I'm curious if anyone else felt unsure about how they estimated their skill-level.

Opening the floor to discussion.

This is what Content Customization looks like as of 31 March, 2020

If you have no idea what I'm talking about, go to your UX Settings and scroll to Content Customization. ↩

My COVID-proof /etc/hosts file

Boots — Fri, 27 Mar 2020 18:20:13 +0000

There's a point at which the news ceases to serve you, and only feeds the part of your brain that wants to reduce uncertainty.

I have reached that point.

Unfortunately, the world is extremely uncertain right now. And I just gotta learn to live with that.

First, though, I wanted to break the habit of checking the news as often as I was. I figured I could, in the short term, block the news-related sites that were distracting me most often, e.g.: Twitter, The New York Times, and The Washington Post.

To block these sites, I reached for an age-old trick: Modifying /etc/hosts to map certain urls to localhost (example below).

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1   localhost
# TO STAY FOCUSED
127.0.0.1 twitter.com
127.0.0.1 www.twitter.com
127.0.0.1 mobile.twitter.com
127.0.0.1 www.mobile.twitter.com
127.0.0.1 nytimes.com
127.0.0.1 www.nytimes.com
127.0.0.1 washingtonpost.com
127.0.0.1 www.washingtonpost.com

255.255.255.255 broadcasthost
::1             localhost
# Added by Docker Desktop
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
# End of section

So, instead of the stream of "holy sh*t" headlines I was checking every hour, I now see (pardon the Dutch 🇳🇱):

This is what you'll see, except probably not in Dutch 🇳🇱

Works like a charm!

If, like me, you're on a Mac, you'll want to run sudo killall -HUP mDNSResponder afterward editing /etc/hosts in order to flush your DNS cache.

Also, I should note that messing up this file can have bad consequences... so make a copy before you toy with it, okay?

Otherwise, stay inside and enjoy your weekend.

✌️

Also, as a post script, I should mention that my new deal with myself is only to check this Axios one-pager on pandemic updates. We'll see how that goes.

"Wikipedia Races, But For GitHub!" ... Is this a genius idea?

Boots — Fri, 20 Mar 2020 02:33:05 +0000

I was thinking some thoughts today, and a random memory came to mind.

Many moons ago, two of my friends started a podcast where they would record themselves doing so-called "Wikipedia Races."

If this is a new term for you, then you should know that a Wikipedia Race refers to a game in which you take two wildly divergent wikipedia pages, and you then try to navigate from one to the other.

The twist is that you're only allowed to navigate using links to other wiki pages. No search. No crawlers. Just good, ole-fashioned hyperlink-clickin' fun!¹

Aside: It hit me while I was typing out the rules that this game is just thinly veiled breadth-first search for humans. Hah!

As an example, try starting from Spider-Man 3 and maneuvering yourself to the page on Poland-US Relations. There's a very intuitive solution if you think about it!²

The reason I bring this up is that, after thinking of Wikiraces, a truly profound thought³ entered my mind, which was: "Couldn't you do a Wikipedia Race on GitHub? Or on GitHub's new bff, NPM?? What if you tried to race from one repo to another???"

I paused for a second, and before any much-needed critical reflection could begin, I thought to myself, "Yeah! Sounds awesome! Why not?!" and I set out to tell the world about this awesome new game.

I originally had two questions for the DEV community, which you're still free to answer if you want:

Would you ever play this game?
What would be some good combinations of projects that I could try to race between on GitHub?

Now the thing is, after having written all of this down and thought about it for maybe six or seven more seconds, I'm not entirely sure if this somehow nerdier adaptation of Wikipedia Races actually sounds all-too fun.⁴

Just think about the ways you'd have to navigate between repos:

Cross-project mentions (in issues, PRs, comments, etc)
Users, via their interactions (contributions, stars, etc)
A dependency graph, if it exists

Indiscriminately clicking through GitHub users just doesn't seem as fun as a random walk through Wikipedia articles. In fact, "fun" might be an inappropriate description for either of those activities.

That said, even though a GitHub Race is sounding less "fun" by the moment, it just so happens that I am more or less stuck in my apartment for the foreseeable future.

In addition to that, I'm noting that I just wrote a post about how excited I was to have thought up a game that I'm now almost sure would be exceedingly boring.

So, I guess if current trends continue, my threshold for "fun" may drop dramatically over the next few days.

In which case, do not be surprised if when I announce my innovative, new tech podcast next week 😁

My research tells me this is how people did Internet in the 90s. ↩
So intuitive! Spider-Man 3 => James Franco => The Interview => CIA => Poland-United States Relations ↩
Okay yeah I didn't really check if someone already thought of this. It seems very likely that someone has already thought of this. ↩
You don't seem surprised? Why don't you seem surprised??? ↩

What would you name a read-only user role?

Boots — Thu, 12 Mar 2020 15:09:28 +0000

Assuming you implemented a role-based access control system, what would you name a "readonly" role?

Here are my proposals so far, preference-ordered:

* Observer
* Spectator
* Monitor
* Watcher
* Reporter
* Guest
* Rubbernecker
* Greg
* Wallflower
* Mall-Cop
* Tattle-Tale
* Sloth

A countdown clock? In an email?? In this economy?!

Boots — Tue, 10 Mar 2020 09:08:52 +0000

Yesterday, I got an email from Memrise, a popular language learning tool.

The email had an animated countdown clock in it, which unfortunately I didn't think to capture with a screen recorder. Nevertheless, here is what it looks like now:

Having worked on HTML emails in the past, I was impressed.

Email is the Wild West of web development. Best practices for email would make most of us cry. You can never use javascript, you have to do layouts with tables, and most of your fancy CSS just won't work.

I wanted to know how they did this countdown magic.

First, I notice that the countdown clock was an image. Duh. It had to be!

So then I followed the source of the image and saw it was a url without a file extension. I.e., there was no .png or .jpeg at the end of the url. It was just https://gen.sendtric.com/countdown/zkihae5gaj.

I opened this url and inspected the headers to see what was going on:

HTTP/1.1 200 OK
Pragma: no-cache
Content-Type: image/gif
Transfer-Encoding: Identity
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, max-age=0
ETag: 14126387014612041038
Date: Tue, 10 Mar 2020 08:51:54 GMT
Content-Encoding: gzip
Expires: -1
Vary: Accept-Encoding

Right! It's a gif.

But... how'd they get the gif to work like a countdown?

This part is just guesswork, but I'd imagine that this endpoint programmatically sends back a non-looping gif depending on the server time.

That is, the server calculates how many seconds are left until Memrise's sale finishes, then it sends back a gif that animates a new frame every second, starting from the time until sale expires and finishing at 00:00:00:00.

Now, when I download and inspect the gif, it is only one frame, 00:00:00:00.

I wish I had time to try to recreate this... but unfortunately, I just got to work!

If you have an alternative theory about how this magical countdown clock works, I'd love to hear it in the comments 😀

How should you label boolean columns in a user-facing table?

Boots — Thu, 05 Mar 2020 09:16:35 +0000

I'm having one of those mornings where I'm questioning all the things.

I prototyped a table where a boolean column was labeled with "Has Alert." This means the row is associated with an action item.

I'm the only native English speaker at my job, and this label looked funny to my colleagues. Now it looks funny to me!

I started to think about boolean columns, and I reasoned that there are broadly two types, those of Being and those of Existence¹:

Being: The row "is" or "is not" a certain way
Existence: The row "has" or "does not have" a property

For Being columns I usually omit the "is" in the label. E.g., if something can "be finished," I'd just write "Finished" in the column header.

For Existence columns, I'm not so sure. I thus invite you to overthink this with me 😁

How would you label an existence column? What feels natural to you?

I don't desire his approval, but I feel like Heidegger would be proud of this classification ↩