DEV Community: Ryan Cooke

Query caching using Nest.js and Typeorm

Ryan Cooke — Mon, 16 Jun 2025 21:56:52 +0000

tl;dr We added a request-scoped query-cache layer to our API backend (NestJS + TypeORM + CLS) that cut duplicate database reads by ~30%, with zero code changes to individual queries, no stale-data risk, and no cross-request leakage.

Recently one of our engineers was curious about something - how often does our API code runs the same SQL query multiple times while handling a single http request? A bit of local log analysis showed one query fired 29 times during a single page render. A page load didn’t equate to one HTTP request, but they knew it also didn’t call the API dozens of times.

Context: We run a monorepo with shared service layers, so a single HTTP path can traverse dozens of helpers.

Now our services code is pretty well decomposed and DRY, with shared helpers extracted into common packages, and clear boundaries that prevent teams from re-implementing the same logic in different places. There aren’t multiple modules interacting with the same database tables, each table has a single repository layer that owns all reads and writes, and the database acts as a single source of truth. But cross-cutting concerns still weave through many layers of the stack, sometimes triggering repeated database look-ups. For example, what are a user’s team memberships, what entitlements do they inherit from teams, and what are the settings for their environment?

So we set out to introduce a request-scoped in-memory query cache to our database adapter, so if the same query was run by multiple helpers at different points during the request, the results would be returned from the cache instead of sending the query to the database. Our hope was that this would reduce overall query load on the database.

The setup

The tech stack relevant to this project is:

NestJS - our HTTP backend, handling dependency injection of our database adapter across services code.
TypeORM - the library that issues queries to the database, with handy builder functions and typed object hydration.
PostgreSQL - not super relevant to this project, except that it’s the source of truth for state in our application.

To limit the scope of how we’d actually implement this cache, there were some ground rules for the project:

No mass code rewrites – it must work without touching thousands of query builders.
Request-scoped only – the cache dies when the HTTP request ends.
Never return stale data – any write to the database blows away the cache immediately.

Initial optimism

With those constraints in mind, we started our scoping. At first glance, this looked like an easy project! NestJS can inject request-scoped instances of a provider, which could hold the contents of our cache. And TypeORM has built-in support for caching queries, with the ability to create custom cache providers. A little bit of glue code, and we could ship the feature by lunch!

Immovable objects meet TypeORM

Implementation started and immediately the plan ran into problems. Reviewing the TypeORM docs on caching in more detail indicated that the only way to enable the cache was to include cache(true) as part of every query builder. This violated our first rule of the project - don’t rewrite every query. Some sleuthing of the cache code in TypeORM’s GitHub repository uncovered an undocumented option that enabled the cache for every query. Phew!

The next problem though was that TypeORM only supported cache invalidation on a time interval. There were no explicit calls to remove entries from the cache in TypeORM’s code, even though the cache provider interface had methods corresponding to expiration. It expected the provider to check for stale entries when reading the cache. This violated our third rule of never returning stale data from the cache - no period of time could guarantee the data wasn’t stale. It’s entirely reasonable that code would update the database then immediate read that data back out.

Here we had some flexibility in the use case we were using cacheing. It’s nice for our app to avoid redundant database reads, but not strictly necessary. So we could extend our custom TypeORM repository code (this was in place to correct TypeORM behavior that drifted across versions) to simply drop the entire cache before save/update/delete queries. Any HTTP requests that only read data wasn’t at risk of staleness, and any requests that mutated data during its lifecycle would always fetch freshly from the database.

CLS to the rescue

Solving the TypeORM issues we moved on to NestJS, but here too there were hurdles. Because of the way our app instantiated our database configuration, it wasn’t possible to inject a request-scoped custom cache provider to pass to TypeORM, without writing custom middleware. This felt dangerously close to violating rule 1 - minimize change surface area. While middleware for NestJS isn’t that complex (it’s running on top of express), if there was a bug introduced in middleware, it would affect the entire app. But also we found a much better approach, a library called nestjs-cls which is purpose built for our use case -

Continuation-local storage allows to store state and propagate it throughout callbacks and promise chains. It allows storing data throughout the lifetime of a web request or any other asynchronous duration. It is similar to thread-local storage in other languages.

NestJS CLS comes with a middleware which instantiates the provider and initializes the local storage. We can add the module to the root app and give it a setup function to create an empty cache object. Then we wrote a custom TypeORM query cache provider that wrapped the context-local storage, and injected the cache provider into our database configuration.

Going deeper

Time to look at the code, first the TypeORM query result cache. Note that we cache queries in a simple map of the query string to the query result object -

//file: src/.../cls-query-result-cache.ts
import { Inject, Injectable } from '@nestjs/common';
import { ClsService, ClsStore } from 'nestjs-cls';
import { QueryRunner } from 'typeorm';
import { QueryResultCache } from 'typeorm/cache/QueryResultCache';
import { QueryResultCacheOptions } from 'typeorm/cache/QueryResultCacheOptions';

// Add a type interface, so that we don't need to
// add type assertions when reading/writing the cache
interface QueryCache extends ClsStore {
  queryCache: Map<string | undefined, QueryResultCacheOptions | undefined>;
}

@Injectable()
export class ClsQueryResultCache implements QueryResultCache {
  constructor(
    @Inject(ClsService) private readonly cls: ClsService<QueryCache>,
  ) { }

  // nothing to do here
  async connect() { }
  async disconnect() { }
  async synchronize(_queryRunner?: QueryRunner) { }

  // TypeORM provides all of the query options when reading
  // from the cache, but we only care about the query string
  async getFromCache(
    options: QueryResultCacheOptions,
    _queryRunner?: QueryRunner,
  ): Promise<QueryResultCacheOptions | undefined> {
    if (!this.cls.isActive()) {
      return;
    }

    const cache = this.cls.get('queryCache');
    return cache.get(options.query);
  }

  // Just like reading, when writing to the cache
  // we key off of the query string
  async storeInCache(
    options: QueryResultCacheOptions,
    _savedCache: QueryResultCacheOptions | undefined,
    _queryRunner?: QueryRunner,
  ) {
    if (!this.cls.isActive()) {
      return;
    }

    if (options.query) {
      const cache = this.cls.get('queryCache');
      cache.set(options.query, options);
    }
  }

  // This function is intended to tell TypeORM if a ttl
  // has expired for the query, but we aren't using
  // time-based cache expiration
  isExpired(_savedCache: QueryResultCacheOptions): boolean {
    return false;
  }

  // We invoke this function to expire the entire cache on writes
  async clear(_queryRunner?: QueryRunner) {
    if (!this.cls.isActive()) {
      return;
    }

    const cache = this.cls.get('queryCache');
    cache.clear();
  }

  // This method is used by TypeORM's CLI for manually
  // interacting with the cache. We don't need it but
  // implement it for completeness
  async remove(identifiers: string[], _queryRunner?: QueryRunner) {
    if (!this.cls.isActive()) {
      return;
    }

    const cache = this.cls.get('queryCache');
    identifiers.forEach((identifier) => {
      cache.delete(identifier);
    });
  }
}

The query cache map gets instantiated when we define our root app’s module imports -

//file: src/app.module.ts
import { ClsModule } from 'nestjs-cls';
import { QueryResultCacheOptions } from 'typeorm/cache/QueryResultCacheOptions';
//... a whole bunch of other imports

// Relevant CLS configuration
@Module({
  imports: [
    ...,
    ClsModule.forRoot({
      global: true,
      middleware: {
        mount: true,
        setup: (cls) => {
          cls.set('queryCache', new Map<string, QueryResultCacheOptions>());
        },
      },
    }),
    ...,
  ],
  providers: [...]
})
export class AppModule implements NestModule {
  constructor() {
    initializeTransactionalContext();
  }
  //... other configuration and setup
}

Now that we have the cache set up, we need to add it to our TypeORM configuration. We also use the alwaysEnabled parameter to instruct TypeORM to always check the cache when executing queries -

//file: src/.../database-config.ts
import { ObjectLiteral } from 'typeorm';
import { ClsQueryResultCache } from '../database';
//... a whole bunch of other imports

interface MakeDatabaseTypeOrmOptionsOptions {
  ...,
  // need to set the cache option to false
  // for connections that don't need it
  cache: boolean | ObjectLiteral;
}

@Injectable()
export class DatabaseConfig {
  constructor(
    private readonly queryCache: ClsQueryResultCache,
  ) {}

  get mainDatabaseTypeOrmOptions(): TypeOrmModuleOptions {
    const cacheProvider = () => this.queryCache;

    return this.makeDatabaseTypeOrmOptions({
      ...,
      cache: {
        provider: cacheProvider,
        // undocumented TypeORM configuration that
        // eliminates the need to add cache option
        // to every query builder
        alwaysEnabled: true,
      },
    });
  }

  get otherDatabaseTypeOrmOptions(): TypeOrmModuleOptions {
    return this.makeDatabaseTypeOrmOptions({
      ...,
      cache: false,
    });
  }

  //... other stuff
}

Finally, we extended our custom TypeORM repository to include overrides that cleared the cache on writes. Here’s a sample -

//file: src/.../custom-typeorm-repository.ts
//... bunch of imports

export class CustomOrmRepository<
  Entity extends ObjectLiteral,
> extends Repository<Entity> {
  private readonly dataSource: DataSource;

  constructor(entity: EntityTarget<Entity>, dataSource: DataSource) {
    super(entity, dataSource.createEntityManager());
    this.dataSource = dataSource;
  }

  override save<T extends DeepPartial<Entity>>(
    entities: T[],
    options: SaveOptions & { reload: false },
  ): Promise<T[]>;
  override save<T extends DeepPartial<Entity>>(
    entities: T[],
    options?: SaveOptions,
  ): Promise<(T & Entity)[]>;
  override save<T extends DeepPartial<Entity>>(
    entity: T,
    options: SaveOptions & { reload: false },
  ): Promise<T>;
  override save<T extends DeepPartial<Entity>>(
    entity: T,
    options?: SaveOptions,
  ): Promise<T & Entity>;
  override save<T extends DeepPartial<Entity>>(
    entityOrEntities: T | T[],
    options?: SaveOptions & { reload?: boolean },
  ): Promise<T | T[] | (T & Entity) | (T & Entity)[]> {
    // Invalidate cache
    this.clearQueryResultCache();

    if (Array.isArray(entityOrEntities)) {
      return Promise.all(
        entityOrEntities.map((entity) => super.save(entity, options)),
      ) as Promise<(T & Entity)[]>;
    } else {
      return super.save(entityOrEntities, options) as Promise<T & Entity>;
    }
  }

  //... other overrides

  private clearQueryResultCache() {
    // If TypeORM is configured to use the query cache
    // clear the entire cache
    if (this.dataSource.queryResultCache) {
      this.dataSource.queryResultCache.clear();
    }
  }
}

Results

To test the effectiveness of our cache, we added some instrumentation to count how many cache hits we had, but since the cache was local to each request, so it was tough to measure across a browsing session. So instead we enabled the pg_stat_statements extension in our local development PostgreSQL database, and this would give us counts on the number of queries being run. We could reset the stats, click around in our web application, and get an accurate number of queries from the database directly.

Our testing showed that the cache was in fact very effective in reducing repeat queries. By navigating through the same pages with the query cache disabled and then with the cache enabled, we saw a number of queries executed reduced by 30 percent!

Conclusion

This post shows that with a couple hundred LOC, you can make sizable impact to your application performance, without wide refactoring or making risky changes. Setting up guard rails when scoping and adhering to them during implementation will force you to experiment until you find an optimal solution.

We are in a growth phase at WorkOS and the scale of load on our services is doubling every few months. Making precise and impactful optimizations without re-architecting
our tech stack allows the bulk of our engineering team to focus on advancing our products. If you are interested in this kind of work, check out our open roles here.

State management in Svelte apps

Ryan Cooke — Mon, 05 Aug 2024 18:35:00 +0000

HTTP was invented as a stateless protocol, which means that each request fully encapsulates all of the information necessary to return a correct response. So historically, web pages never had to worry about managing state - each request to a URL with parameters or with a form submission would receive a response with all of the HTML that the browser needed to render content.

But now on today's Internet, we have web applications - which are pretty complicated bundles of JavaScript code, and they load data in a variety of ways, sometimes fetching HTML, but just as often getting JSON data through APIs. And yet for a lot of web app frameworks, client state is treated as an add-on, offered via libraries, or worse, by comingling state with view components. This feels a lot less mature than server-side apps, where there are databases that can serve as a single source of truth for data, support flexible queries i.e. SQL, and offer guarantees for correctness i.e. ACID properties. (OK, Notion did build a SQL database for their web app, and boy did they make it complicated)

So it's up to the humble developer to decide how they will manage state, and for any non-trivial web app, this quickly turns into a tangled mess of cross-component dependencies, bi-directional props, and state duplication. Fortunately, Svelte ships with a fantastic module called "stores" which is simple and powerful and can be shared across a codebase.

In this post, I'll describe a tiered approach I've been using for client-side web application state. It separates different concerns that are common when it comes to state, making an opinionated division between component state, view state, client-side persistent state, and server-side state.

What am I optimizing for? The best User eXperience possible - I want interaction that is responsive without delay, changes to state should be optimistic (as in immediately re-rendered), and feedback is obvious when something goes wrong. But this comes with a tradeoff, namely tightly coupled layers of code. This will be more clear with code samples below.

The app I'm building is an email client. It has a bunch of state - each email message has headers and content, messages are grouped into threads, and threads have labels. There are filtered lists of threads, chronological lists of messages in a thread, and actions like adding a label to a thread or deleting it. This client talks to a mail server through an API, and that is the source of truth, but it also keeps a local database that is a semi-processed copy of all emails to support offline mode.

First a visual, if I were to stack the "layers" of state in my app, it would look something like this -

Let me explain each layer:

Components keep track of whatever state is necessary to render their portion of the view correctly. But this does not include the main content of the component! For example, think of a list of threads - the list has an order/pagination/filters/etc. that is kept in the component. However the list of threads itself is not managed by the component.
View state is the main space for whatever state the app is keeping in memory. Continuing with the thread list from above, it's the page of threads with their message data. Other examples of view state include - a selected (or "active") thread, any filters that apply to the list of threads, and all of the labels assigned to a thread. I use Svelte stores as the interface for this layer, so that state changes can be communicated to any component with a subscription.
Client state is what I call anything persisted between sessions, when the browser or tab is closed and re-opened. In my app, I leverage IndexedDB to store a copy of all emails synced from the server, which allows my app to work offline and load instantly. To load a page of threads, the view layer can query the database directly and populate the Svelte store, rather than making an HTTP request to the server.
Server state is everything not on the client, which could include SSR components or third-party APIs. In my case, it's the source-of-truth email server accessed through a REST API, and client-side state changes must be synced with the server for durability.

Here's an annotated version of the visual from above -

A lot of apps I see conflate component and view state. The problem with this is that changes become hard to manage, as state becomes deeply nested through props or local variables. Components might have different behavior across a change lifecycle, with some re-rendering optimistically while others need to wait for changes to be fully persisted through to the API.

Let's look at some code. Here's a message list component -

// ./src/components/ThreadList.svelte
<script>
  import { filters, activeThread, threadList } from './stores'
  import { getPage } from './view-state'

  let order = 'prev',
    stillLoading = true
  // reactively reload page if filters change
  $: loadPage($filters)

  function loadPage(filters) {
    getPage(filters, order)
      .then(() => {
        stillLoading = false
        if (
          $threadList.findIndex((t) => t.id == $activeThread?.id) == -1
        ) {
          activeThread.set($threadList.at(0))
        }
        return tick()
      })
      .then(() => scrollToActiveThread())
  }

  function scrollToActiveThread() {
    if ($activeThread == undefined) return
    //... a bunch of bounding box calculations
  }
</script>

<div>
  <ul class="mr-2 font-serif">
    {#each $threadList as thread (thread.id)}
      <li
        id="thread-{thread.id}"
        class="border-b border-l-4 .thread-item"
        class:bg-white={thread.id == $activeThread?.id}
      >
      <!-- a bunch of markup -->
      </li>
    {:else}      
      <li class="border-b border-gray-100 border-l-4">
        <div class="py-6 px-8 animate-ellipsis">
          {#if stillLoading}
            Loading
          {:else}
            No messages
          {/if}
        </div>
      </li>
    {/each}
  </ul>
</div>

The markup here is simplified, but it shows how the state for a list of threads is managed. The component never populates the list, in fact it doesn't have a local variable corresponding to a list at all. Instead, the list is contained in a Svelte store called threadList. The component does make a call to getPage(filters, order, offset), a method exposed by the view-state.js, which will fetch a page and set the threadList store. The component renders markup directly from the threadList store to leverage Svelte's excellent reactive capabilities.

Here's stores.js, it's pretty simple, just a place for declaring and exporting state that is shared across components. I've added comments that explain why each store is used to manage state -

//./src/lib/stores.js
import { writable } from 'svelte/store'

// A single page of threads fetched from local persistence
// NB: this list is only displayed on the ListThreads.svelte component,
// but is kept as a Svelte store so that optimistic updates to a single
// thread can be rendered without needing to re-fetch the entire page.
export const threadList = writable([])
// The currently selected thread, either from the threadList
// or via direct navigation.
// NB: This is the thread that all mutations apply to, such as
// adding a label or deleting a thread.
export const activeThread = writable(undefined)
// All of the labels in the mailbox, kept in memory for quick retrieval
export const labels = writable([])
// Any filters that should be used when fetching a page of threads
// NB: Filters are set via one component (TopNav), but used in another
// component (ListThreads) when fetching a page.
export const filters = writable([])

Now let's look at view-state.js, where the bulk of the logic is happening -

//./src/lib/view-state.js
import { get } from 'svelte/store'
import { labels, activeThread, threadList } from './stores'
import { Thread } from './structs'

export async function getPage(filters, order = 'prev') {
  const db = await database()
  const allLabels = get(labels)

  const txn = db.transaction('threads')
  // Query client-side database for a page of threads
  const query = txn.objectStore('threads').index('ts').openCursor(null, order)

  // Wrap query callback in a Promise, so that the
  // caller of getPage can do post-query processing of list
  return new Promise((resolve) => {
    const page = []
    query.onsuccess = (request) => {
      const cursor = request.target.result
      if (cursor) {
        const thread = cursor.value
        // IDB provides fast cursors, but no filtering capabilities
        // so we match filters in code. The thread must have all labels
        // that appear as filters, so do a conjunction test.
        const matchFilters =
          filterIds.filter((fi) => thread.labelIds.includes(fi)).length == filterIds.length

        if (matchFilters) page.push(new Thread(thread, allLabels))
        if (page.length < 50) {
          cursor.continue()
        } else {
          // Page if full with 50 threads
          // set Svelte store and resolve promise
          threadList.set(page)
          resolve()
        }
      } else {
        // Cursor is not set, meaning we've reached the end.
        // set Svelte store and resolve promise
        threadList.set(page)
        resolve()
      }
    }
    query.onerror = () => resolve({ error: `Error loading page: ${query.error}` })
  })
}

If you trace the code, the component ThreadList.svelte calls our view-state function getPage(...), which then queries our client-side state in IndexedDB. The result from the query gets set in the view-state Svelte store threadList, which ThreadList.svelte is reactively using to render the page of threads.

Here's what it looks like in our state diagram:

Ok, this looks like a lot of complexity for simply rendering a list of email threads. I don't disagree! Let's look at a better example, adding a label to an email thread. Optimistically mutating state is where this structure starts to make sense, as each layers of state can handle a single concern.

I'll skip the component (there's a bunch of modal stuff), and just show the function in view-state.js, which calls a function to update the client-side IndexedDB in client-state.js -

//./src/lib/view-state.js
import { get } from 'svelte/store'
import { labels, activeThread, threadList } from './stores'
import { Thread } from './structs'
import { addLabelToThreadDB } from './client-state'

export async function getPage(filters, order = 'prev') {
  //... code is above
}

export async function addLabelToActiveThread(label) {
  const thread = get(activeThread)

  if (thread === undefined) {
    return Promise.resolve({ error: 'Unable to add label to thread, no thread selected' })
  }

  // do nothing if thread already contains label
  if (thread.labelIds.includes(label.id)) {
    return Promise.resolve(thread)
  }

  // optimistically update Svelte stores with new label
  // this will trigger any components that have this state
  // to re-render
  thread.labelIds.push(label.id)
  updateThreadInStores(thread)

  const dbUpdate = addLabelToThreadDB(thread.id, label)

  return dbUpdate
    .then((result) => {
      // something went wrong, undo optimistic update
      if (result.error) {
        thread.labelIds = thread.labelIds.filter((l) => l !== label.id)
        updateThreadInStores(thread)
      }
      return result
    })
}

function updateThreadInStores(thread) {
  activeThread.update((t) => {
    // only update activeThread if it is
    // set to the same thread!
    if (t?.id !== thread.id) return t
    return thread
  })

  threadList.update((tl) => {
    return tl.map((t) => (t.id == thread.id ? thread : t))
  })
}

//./src/lib/client-state.js

async function database() {
  //...open connection to IndexedDB
}

export async function addLabelToThreadDB(threadId, label) {
  const db = await database()
  const txn = db.transaction(['threads'], 'readwrite')
  const store = txn.objectStore('threads')
  return new Promise((resolve, reject) => {
    const getThread = store.get(threadId)
    getThread.onsuccess = ({ target }) => {
      const dbthread = target.result
      if (!dbthread.labelIds.includes(label.id)) {
        dbthread.labelIds.push(label.id)
      }

      const putThread = store.put(dbthread)
      putThread.onsuccess = () => resolve(dbthread)
      putThread.onerror = () => resolve({ error: putThread.error })
    }
    getThread.onerror = () => resolve({ error: getThread.error })
  })
}

export async function removeLabelFromThreadDB(threadId, label) {
  const db = await database()
  const txn = db.transaction(['threads'], 'readwrite')
  const store = txn.objectStore('threads')
  return new Promise((resolve, reject) => {
    const getThread = store.get(threadId)
    getThread.onsuccess = ({ target }) => {
      const dbthread = target.result
      if (dbthread.labelIds.includes(label.id)) {
        dbthread.labelIds = dbthread.labelIds.filter((l) => l !== label.id)
      }

      const putThread = store.put(dbthread)
      putThread.onsuccess = () => resolve(dbthread)
      putThread.onerror = () => resolve({ error: putThread.error })
    }
    getThread.onerror = () => resolve({ error: getThread.error })
  })
}

This code is updating Svelte stores before writing the change to the client-side IndexedDB. The result is that the components will re-render the thread with the new label immediately. The method updateThreadInStores simultaneously updates the activeThread stores, as well as the threadList store that was populated with the getPage call above.

However if there is an error when persisting the change, view-state.js will simply roll back the stores to their previous state. Again, the user experience is updated almost instantly, and feedback given to the user about the failure.

Finally, this pattern can be stacked with an API call, to persist changes to the server side. This is implemented by chaining an additional call to the promise that is updating IndexedDB -

//./src/lib/view-state.js

export async function addLabelToActiveThread(label) {
  //...all of the code from above

  return dbUpdate
    .then((dbResult) => {
      // something went wrong in IndexedDB, undo optimistic update
      if (dbResult.error) {
        thread.labelIds = thread.labelIds.filter((l) => l != label.id)
        updateThreadInStores(thread)
        return dbResult
      }
      // client-side state updated, now update API
      return addLabelToThreadAPI(thread, label)
    })
  .then((apiResult) => {
      // something went wrong in the API, undo optimistic update
      if (apiResult.error) {
        thread.labelIds = thread.labelIds.filter((l) => l != label.id)
        updateThreadInStores(thread)
        // also undo database update
        return removeLabelFromThreadDB(thread.id, label)
      }
      return result
    })
}

So at the beginning of this post, I mentioned a tradeoff with this approach - the view-state.js ends up being very tightly-coupled to the client-state and API layers. The view-state code is doing a lot of coordination around updating the client-side database, API, and handling error scenarios between them. You may have also noticed that there are some race conditions in this code which means consistency cannot be guaranteed.

But I've still found this structure to help organize the code, and reduce decisions around the question of "where should this logic go?" that I have to answer pretty often while building apps. Hopefully this helps you with this as well!

Keep on coding

Breaking IndexedDB consistency to explore its transactions

Ryan Cooke — Wed, 21 Feb 2024 18:55:47 +0000

Did you know every browser has a built-in key-value database?

I’ve been building a Chrome extension, and learning about IndexedDB while using it for an offline mode. Everything that the app reads from the API gets processed and stored locally. Then the app renders and is reactive to changes from the database.

Having done loads of backend development, I'm curious to know how I can use IndexedDB (IDB for short) with strong consistency - basically without losing data. After reading in some threads that data is easy to corrupt in IDB, it's time to explore the transaction model, and suss out what code patterns might lead to unexepected inconsistency.

So I coded up a little app that would do lots of writes into the same "store" (IDBs name for table). The idea is to use web workers to execute simultaneous writes across threads, and see if we can stomp on parallel updates to lose data. "Stomping" means using concurrency to have code execute and overwrite each others' state.

Since I use Svelte for everything, I set up a simple app with tailwind and daisyUI. You can see an example of this in my post "Svelte Without Kit".

First, the docs

After perusing IDB docs on transactions, I didn't see any details about thread safety or anything about under the hood. Is there some kind of multiversion concurrency control happening? Hard to say. But there was one example, and it emphasized that transactions are applied in order by when the transaction is created. Still not sure what that means across threads, but an ordering guarantee is a good sign that transactions provide some predictability, so let's write some code to find out!

You can find the complete code of the following files:

App.svelte // Svelte component and UI
worker.js // web worker that interacts with IDB
pool.js // utility function for distributing work among workers
database.js // utility function for connecting to IDB

Test One: update inside transaction

My first test sends messages to a web worker to update a "thread" object in IDB, and then verifies that the update succeeded. The workers each add a random number "labelId" to an array on the thread object. The idea is to test - if workers update the same thread object's array simultaneously, will one overwrite array the other in IDB and lose a labelId?

Here's the database code for updating the thread object, notice I am adding the labelId after starting the transaction and fetching the thread object from the store -

// threadId and labelId variables are in scope
database() // returns a promise with an instance of IDBDatabase
.then((db) => {
  // start transaction
  const txn = db.transaction(['threads'], 'readwrite')
  txn.oncomplete = () => db.close()
  const store = txn.objectStore('threads')

  return new Promise((resolve, reject) => {
    // read thread object from IDB store
    const getThread = store.get(threadId)
    getThread.onsuccess = ({ target }) => {
      // create new thread object if it doesn't exist
      const thread = target.result || { id: threadId, labelIds: [], attr: true }
      // add labelId to array in thread object
      if (!thread.labelIds.includes(labelId)) {
        thread.labelIds.push(labelId)
      }

      // write thread object to IDB store
      const putThread = store.put(thread)
      putThread.onsuccess = () => resolve(thread)
      putThread.onerror = () => reject(new Error(`Error writing message: ${putThread.error}`))
    }
    getThread.onerror = () => reject(new Error(`Error getting message: ${getThread.error}`))
  })
})

For our App UI, we have a button to toggle the test running or not, and a running test will messages to a pool of workers. Refer to below for the full code, but it basically looks like this:

  function startStopTest() {
    if (testRunning) {
      const promises = []
      for (let step = 0; step < 15; step++) {
        const lId = Math.floor(Math.random() * 1000)
        const p = messageWorker({ threadId: step % 5, labelId: lId, action: 'addLabel' }, url)
          .then(//... another message)
          .then(//... another message)
        promises.push(p)
      }

      Promise.all(promises)
        .then(function (results) {
          //... process results
        })
        .then(tick)
        .then(startStopTest)
    }
  }

Let's fire it up!

Result

There are no missing labelIds in any thread object's list, so no data is being corrupted across simultaneous updates. Well that was underwhelming!

Test Two: Add another update

Next I'll try adding a second update to the worker, this time flipping a boolean on the object. The database code is the same as before - start transaction, get a thread object, modify it, put object back in store. My idea is to see if adding more updates to the same object can cause corruption.

./worker.js
function toggleAttr(threadId, labelId) {
  return database().then((db) => {
    const txn = db.transaction(['threads'], 'readwrite')
    txn.oncomplete = () => db.close()
    const store = txn.objectStore('threads')

    return new Promise((resolve, reject) => {
      const getThread = store.get(threadId)
      getThread.onsuccess = ({ target }) => {
        const thread = target.result
        thread.attr = !thread.attr
        const putThread = store.put(thread)
        putThread.onsuccess = () => resolve()
        putThread.onerror = () => reject(new Error(`Error writing thread: ${putThread.error}`))
      }
      getThread.onerror = () => reject(new Error(`Error getting thread: ${getThread.error}`))
    })
  })
}

Get it going!

Result

Same result, no corruption

Test Three: separate read and write transactions

I'm curious what happens if I read the object in one transaction, update the list, then write the object in a separate transaction. This will test if the consistency I've achieved so far is a result of transaction serialization or something.

Let's modify our worker code, it will fetch a thread object in one transaction:

function getThread(db, threadId) {
  const txn = db.transaction(['threads'], 'readonly')
  const store = txn.objectStore('threads')

  return new Promise((resolve, reject) => {
    const getThread = store.get(threadId)
    getThread.onsuccess = ({ target }) => {
      const thread = target.result || { id: threadId, labelIds: [], attr: true }
      resolve({ db, thread })
    }
    getThread.onerror = () => reject(new Error(`Error getting thread: ${getThread.error}`))
  })
}

Then it updates the object list, before starting a new transaction and writing the object to IDB:

database()
  .then((db) => getThread(db, threadId))
  .then(({ db, thread }) => {
    if (!thread.labelIds.includes(labelId)) {
      thread.labelIds.push(labelId)
    }

    return new Promise((resolve, reject) => {
      const txn = db.transaction(['threads'], 'readwrite')
      txn.oncomplete = () => db.close()
      const store = txn.objectStore('threads')
      const putThread = store.put(thread)
      putThread.onsuccess = () => resolve()
      putThread.onerror = () => reject(new Error(`Error writing thread: ${putThread.error}`))
    })
  })

Let 'er rip!

Result

Immediately we see errors, missing labelIds:

Corruption accomplished! Almost half of the objects are missing labelIds. That's bad!

This suggests that updating a thread object outside of a transaction, then writing it to the store, is susceptible to data loss. But before you jump up and say "obviously!", think about the complex applications you've built. It's pretty common for data modifications to happen outside of database transactions, for example in response to user input, and then to push that data to storage, for example through an API. But IDB is only consistent with modifications to an object that is also fetched/read within the same transaction. At least according to this test.

Test Four: mixing updates inside and outside transactions

Finally, I wanted to know if updates made outside transactions could conflict with updates happening inside transactions. So I added another "conflicting" action to the worker, one that deletes the labelId outside of a transaction. And reverted the original add labelId action to use an in-transaction update, which should be safe from data loss?

Result

Tests were still passing:

But labelIds were not being removed from the thread objects:

Once I update the delete labelId worker to modify inside a transaction, labelId lists look more reasonable:

Analysis

Ok so what's going on here? IndexedDB seems capable of handling concurrent updates as long as objects are read, updated and written in the same transaction.

If I go back to the IDB docs, I recall how it highlighted that transactions followed some kind of ordering, based on when they are created. This leads me to think that transactions are "serialized", meaning they act as if they are executed one at a time (very watered-down definition of serialization).

Here's a sketch of the pattern that leads to data loss -

Notice that there is no way to order the "put" transactions that doesn't lose data. If Worker1 writes first, its array of labelIds will get overwritten by Worker2, and vice versa.

Now what if we update within a transaction -

Order doesn't matter for these transactions, as long as they are executed serially, the object in IDB doesn't lose any data!

Conclusion

So it seems that IDB is safe as long as updates are made to objects fetched within the same transaction. I even tested transactions with long running code between the get and put, and data consistency held. (There are some situations where transactions get closed after they are passed between closures, but that raises an error).

While it's great to see that IDB can be used with consistency, this mechanism dramatically affects how code using IDB is written! You can't reliably accumulate changes to an object, and then write that object to the database, without potential for stomping on another update happening simultaneously. So if your app has multiple code paths for updating IDB, this is a risk.

I should also note that there are some updates where order does really matter. For example, adding then removing a value relies on the "add" happening before the "remove", otherwise the final value is wrong. Again, pretty common for this to happen in a web app, say a user clicks really fast.

These kind of consistency problems are exacerbated by the fact that IDB only supports a primitive put method - there's no way to make partial or conditional updates. So as a developer, I'm left with solving for consistency, and it requires me to be very deliberate about how state is updated.

Ah well, keep on coding!

App.svelte

Simple page with a button to start and stop tests, as well as a summary of the results of the running test.

<script>
  import { messageWorker } from './pool'
  import { tick } from 'svelte'

  let successCount = 0,
    failedCount = 0,
    totalCount = 0,
    lastError = undefined,
    testRunning = false

  const url = new URL('./worker.js', import.meta.url)

  function toggleTest() {
    testRunning = !testRunning
    if (testRunning) {
      successCount = failedCount = totalCount = 0
      lastError = undefined
    }
    startStopTest()
  }

  function startStopTest() {
    if (testRunning) {
      const promises = []
      for (let step = 0; step < 15; step++) {
        const lId = Math.floor(Math.random() * 1000)
        const p = messageWorker({ threadId: step % 5, labelId: lId, action: 'addLabel' }, url)
          .then(function ({ threadId, labelId }) {
            return messageWorker({ threadId, labelId, action: 'toggleAttr' }, url)
          })
          .then(function ({ threadId, labelId }) {
            return messageWorker({ threadId, labelId, action: 'verifyLabel' }, url)
          })
          .then(function ({ threadId, labelId }) {
            return messageWorker({ threadId, labelId, action: 'deleteLabel' }, url)
          })
        promises.push(p)
      }

      Promise.all(promises)
        .then(function (results) {
          results.forEach(({ error, threadId, labelId }) => {
            if (error) {
              failedCount = failedCount + 1
              lastError = { error, threadId, labelId }
            } else {
              successCount = successCount + 1
            }

            totalCount = totalCount + 1
          })
          return Promise.resolve()
        })
        .then(tick)
        .then(startStopTest)
    }
  }
</script>

<main>
  {#if testRunning}
    <button class="btn btn-primary" on:click={toggleTest}>Stop test</button>
  {:else}
    <button class="btn btn-warning" on:click={toggleTest}>Start test</button>
  {/if}
  <p class="text-sm font-light py-4">
    {totalCount} total tests, {successCount} passed, {failedCount} failed
  </p>
  {#if lastError != undefined}
    <p class="text-lg text-error">
      {lastError.error?.message || 'Unknown Error'}
    </p>
  {:else}
    <p class="font-semibold italic text-lg text-primary-content">No errors</p>
  {/if}
</main>

database.js

A utility function to open a connection to the database, and return the connection in a promise.

// ./database.js
export function database() {
  return new Promise((resolve, reject) => {
    const request = indexedDB.open('test_db', 1)
    request.onsuccess = () => resolve(request.result)
    request.onerror = () => reject(request.error)
    request.onblocked = () => console.warn('pending till unblocked')
    request.onupgradeneeded = (e) => {
      const up = request.result
      if (e.oldVersion < 1) {
        const threads = up.createObjectStore('threads', {
          keyPath: 'id',
          autoIncrement: false
        })
      }
    }
  })
}

pool.js

A utility function to manage a pool of web workers and message a worker. A promise is returned, that will be invoked when the worker responds. Promises are tracked by a nonce value is that is passed with the message.

// ./pool.js

const workers = []
const promises = {}

function resolveMessage(data) {
  promises[data.nonce].resolve(data)
  delete promises[data.nonce]
}

export function messageWorker(message, url) {
  const ix = Math.floor(Math.random() * 5)
  if (workers.at(ix) == undefined) {
    workers[ix] = new Worker(url, { type: 'module' })
    workers[ix].onmessage = (event) => resolveMessage(event.data)
  }

  const rand = Math.floor(Math.random() * 50000)
  const nonce = `${rand}`
  const promise = new Promise((resolve) => {
    promises[nonce] = { resolve }
  })

  workers[ix].postMessage({ nonce, ...message })
  return promise
}

worker.js

A background worker for manipulating data with various transaction patterns.

//./worker.js
import { database } from './database'

function handleMessage(message, respond) {
  let pr
  if (message.action == 'addLabel')
    pr = addLabel(message.threadId, message.labelId)
  else if (message.action == 'toggleAttr')
    pr = toggleAttr(message.threadId, message.labelId)
  else if (message.action == 'verifyLabel')
    pr = verifyLabel(message.threadId, message.labelId)
  else if (message.action == 'deleteLabel')
    pr = deleteLabel(message.threadId, message.labelId)
  else
    pr = Promise.reject(new Error('No action in message'))

  pr
   .then(() => respond({ success: true, ...message }))
   .catch((error) =>
    respond({ error, success: false, ...message })
  )
}

function addLabel(threadId, labelId) {
  return database().then((db) => {
    const txn = db.transaction(['threads'], 'readwrite')
    txn.oncomplete = () => db.close()
    const store = txn.objectStore('threads')

    return new Promise((resolve, reject) => {
      const getThread = store.get(threadId)
      getThread.onsuccess = ({ target }) => {
        const thread = target.result || { id: threadId, labelIds: [], attr: true }
        if (!thread.labelIds.includes(labelId)) {
          thread.labelIds.push(labelId)
        }

        const putThread = store.put(thread)
        putThread.onsuccess = () => resolve()
        putThread.onerror = () => reject(new Error(`Error writing thread: ${putThread.error}`))
      }
      getThread.onerror = () => reject(new Error(`Error getting thread: ${getThread.error}`))
    })
  })
}

function toggleAttr(threadId, labelId) {
  return database()
    .then((db) => getThread(db, threadId))
    .then(({ db, thread }) => {
      thread.attr = !thread.attr
      return new Promise(function (resolve, reject) {
        const txn = db.transaction(['threads'], 'readwrite')
        txn.oncomplete = () => db.close()
        const store = txn.objectStore('threads')
        const putThread = store.put(thread)
        putThread.onsuccess = () => resolve()
        putThread.onerror = () => reject(new Error(`Error writing thread: ${putThread.error}`))
      })
    })
}

function verifyLabel(threadId, labelId) {
  return database()
    .then((db) => getThread(db, threadId))
    .then(({ db, thread }) => {
      db.close()
      if (thread == undefined) {
        return Promise.reject(new Error(`Missing thread ${threadId}`))
      } else if (!thread.labelIds.includes(labelId)) {
        return Promise.reject(new Error(`Thread ${threadId} is missing label ${labelId}`))
      } else {
        return Promise.resolve()
      }
    })
}

function deleteLabel(threadId, labelId) {
  return database()
    .then((db) => getThread(db, threadId))
    .then(({ db, thread }) => {
      thread.labelIds = thread.labelIds.filter((lid) => lid != labelId)

      return new Promise((resolve, reject) => {
        const txn = db.transaction(['threads'], 'readwrite')
        txn.oncomplete = () => db.close()
        const store = txn.objectStore('threads')

        const putThread = store.put(thread)
        putThread.onsuccess = () => resolve()
        putThread.onerror = () => reject(new Error(`Error writing thread: ${putThread.error}`))
      })
    })
}

function getThread(db, threadId) {
  const txn = db.transaction(['threads'], 'readonly')
  const store = txn.objectStore('threads')

  return new Promise((resolve, reject) => {
    const getThread = store.get(threadId)
    getThread.onsuccess = ({ target }) => {
      const thread = target.result || { id: threadId, labelIds: [], attr: true }
      resolve({ db, thread })
    }
    getThread.onerror = () => reject(new Error(`Error getting thread: ${getThread.error}`))
  })
}

self.onmessage = (message) => {
  handleMessage(message.data, postMessage)
}

Security Workshop Part 1 - Put up a gate

Ryan Cooke — Thu, 02 Nov 2023 19:11:55 +0000

This post is part of a series that was developed for fintech_devcon 2023. You can see a full video of the workshop on YouTube

Your Architecture as a Security Defense

When it comes to securing a web application, many developers tend to focus on implementing specific security controls right away. You might hear statements like, "I need authentication," "Maybe Cloudflare can stop DDoS attacks," or "Let's require users to create strong passwords." While these measures are important, they often skip a crucial step in the process: considering the application's architecture design.

Designing your architecture with security in mind is a fundamental step in building a robust defense against potential threats. By leveraging architectural modularity, you can create a system that separates security concerns into its own dedicated layer. Just as a web server contains business logic and a database stores data, specific components can be configured to manage cross-cutting security concerns. This approach not only enhances security but also offers benefits such as scalability and improved refactoring capabilities as the application's complexity grows.

The importance of "Insecure design" has been underscored by the OWASP Top 10 list of common web vulnerabilities. In a new category introduced in 2021, the focus is on risks related to design and architectural flaws. The call is for more use of threat modeling, secure design patterns, and reference architectures. The community needs to move beyond "shift-left" in the coding space to pre-code activities that are critical for the principles of Secure by Design.

One powerful architectural module for web applications is an API Gateway. An API gateway is an application that sits in front of the API and acts as a single entry point for requests from outside the cluster. It routes these requests to the appropriate backend microservices. What makes it particularly valuable is that it allows you to add security controls without needing to modify the backend microservices themselves.

Using an API Gateway to stop Credential Stuffing

Let's look at a practical example of how an API Gateway can be used to enhance security. One common concern is limiting the rate of requests accepted by the API for a particular IP address. Rate limiting serves as a potent defense against attacks like "credential stuffing" where attackers attempt to authenticate with known username/password combinations at high speeds.

We'll use Traefik, an open source cloud native gateway that can plug into a Kubernetes cluster. It has the concept of "middleware" that can process API requests before passing them through to a backend. We can configuring a rate limit for all of our API endpoints by matching on the request path:

+---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: http-ratelimit
  namespace: default
spec:
  rateLimit:
    average: 50
    burst: 75
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: strip-api-prefix
  namespace: default

spec:
  stripPrefix:
    prefixes:
      - /api
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: api-ingress
  namespace: default
spec:
  entryPoints:
    - web
  routes:
  - kind: Rule
    match: PathPrefix(`/api/v1`)
    services:
    - name: api
      port: 80
    middlewares:
    - name: http-ratelimit
    - name: strip-api-prefix

We are configuring the gateway to allow on average 50 requests/second across all of our API endpoints for a single client. Up to 75 req/s are also allowed for short periods of time in bursts. Clients are identified by IP addresses, but other configuration options allow for clients to be identified by a header value. The full list of configuration options are documented here.

One gotcha with IP address rate-limited is that it's not uncommon for many different end-users to share an IP address. This is prevalent for customer apps but can also affect B2B, when several companies are in the same co-working space.

Traefik has a catalog of middlewares that can be downloaded and installed, or you can develop your own for custom functionality.

Consolidate functionality, and future proof

In addition to rate limiting, we can use the gateway to encrypt traffic with TLS and validate authentication for requests to our API. These are "cross-cutting concerns", or security principles that we expect every API backend to enforce. Rather than relying on each backend server implementing these controls (or risking that a backend gets deployed without one), our gateway consistently enforces them.

Not only does a gateway provide a layer of security to our architecture, but it also gives us flexibility for upgrading our backend services in the future. We can control our requests are routed through the gateway configuration, and forward version 2 of our API to new services while keeping backwards-compatible services running separately.

Another architectural design consideration is network segmentation. This strategy involves separating a network into distinct sub-networks (subnets), which restrict how machines on the network can communicate with each other or the Internet.

Implementing Network Segmentation

By limiting the sources from which a server can receive network traffic, you can effectively guard against unpatched vulnerabilities being exploited. "Zero day" flaws refer to vulnerabilities discovered after attackers have already started exploiting them to hack into systems. It's difficult to predict where the next "zero day" will emerge, but a server that is isolated from direct Internet traffic is less likely to be compromised by such vulnerabilities.

The strongest form of network segmentation is achieved by restricting what backend servers can receive traffic directly from the Internet. Given the Internet's vast and varied population, it's still one of the primary sources of threats.

A useful design strategy starts with the assumption that "none of your backend servers" need a direct connection to the Internet. For instance, your database likely doesn't need to accept connections from the Internet; it only needs to communicate with other backend servers. Therefore, start with the assumption that no Internet connectivity is required and then selectively add it to components that absolutely need it.

It's entirely feasible that, when using an API Gateway as described earlier, none of your servers need a direct Internet connection. The Gateway can handle all incoming traffic from the Internet, ensuring that the traffic is safe and authorized before passing it to a server located on a restricted network.

Incorporating these architectural considerations early in your application's design can significantly enhance your security posture. By thinking ahead and building security into the very structure of your application, you're better equipped to defend against evolving threats and vulnerabilities. In today's digital landscape, this proactive approach is a critical component of your overall security strategy.

Stay safe out there coders!

Ryan

Svelte without Kit

Ryan Cooke — Fri, 27 Jan 2023 01:54:50 +0000

updated August 2024 with node 22, express 4.19 and html-express-js 3.1

I love Svelte so much that I want to use it in all of my projects! Often these projects are micro apps that don't need a full-fledged app framework - they are a few HTML pages with a couple of server-side API endpoints. SvelteKit ends up being overkill for this use case, as it doesn't need routing, preloading, customized rendering, or pretty much all of the great features provided by SvelteKit.

The Svelte docs don't give a lot of guidance on how to deploy an app without SvelteKit, in fact there's only one paragraph in the docs that talks about how this might work -

If you don't need a full-fledged app framework and instead want to build a simple frontend-only site/app, you can also use Svelte (without Kit) with Vite by running npm init vite and selecting the svelte option. With this, npm run build will generate HTML, JS and CSS files inside the dist directory.

This post will outline how to use Svelte with Express, a minimalist web framework. It is split into two parts - the dev environment and building a container for production deployment, as we use slightly different build steps for each of them.

The setup

Express is a great web framework, it's super simple for setting up HTTP endpoints and processing requests/responses. There are a variety of middleware plugins to handle things like authnz and rendering. JSON request bodies are natively supported in the latest version of Express. While middleware can significantly extend the capabilities of Express, in this setup we are looking for absolute minimalism. We do need a rendering engine so that we can inject data as properties into our Svelte components from routes on the server. But that's about it.

Also critical to get working is the build step for using Svelte. Since Svelte is pre-compiled, it's not as simple as serving Svelte files directly through Express. Instead those assets must be built by Vite before they can be served to the browser client. Vite will package all of our Svelte components into minified Javascript and CSS files, along with any JS dependencies used by components as well as CSS libraries like Tailwindcss with Postcss.

To manage the compilation step, we are going to use some scripting in our local dev environment. It will trigger a build command whenever a .svelte file is created/modified/deleted, and copy the generated files for Express to serve. (Unfortunately, there's no hot-reload in the browser). For deployment, we'll compile Svelte as a step in building a Docker container to run on a hosting provider.

Express with html-express-js

Our app is a simple chatbot that receives a message from the Svelte front-end and echos back a random response. It stores chat sessions with a random identifier in memory. It's not a super complicated app, but we need to route a chat session based on an ID in the HTTP request path. This ID is an example of server-side state that is loaded as properties by Svelte.

We configure Express with a bit of middleware - cors and html-express-js, as well as built-in middleware express.static and express.json.

express.static will serve the compiled Svelte js/css from a local ./public directory. json and cors are necessary to allow for fetching data from our Svelte components. And finally, html-express-js provides minimal templating capabilities, so that we can also render data as HTML for GET requests.

Here's the basic Express backend, as server.js -

// ./server.js
import express, { response } from 'express';
import htmlExpress, { renderView } from 'html-express-js';
import cors from 'cors';
import { resolve } from 'path';
import fs from 'fs';

const port = process.env.PORT || 3000;
const hostname = process.env.HOSTNAME || 'http://localhost'

const __dirname = resolve();

const app = express();
// Configure express to parse json and allow CORS requests
app.use(express.json());
app.use(cors());

// Configure express to serve static assets from
// the local directory ./public. This is where
// compiled Svelte files and images will get copied into.
app.use(express.static('public'));

// Store chats in local memory
const chats = {};

// Configure html-express-js as the rendering engine.
// Templates will be in the local directory ./views
const { engine, staticIndexHandler } = htmlExpress({
  viewsDir: `${__dirname}/views`
})
app.engine('js', engine)
app.set('view engine', 'js')

// POST endpoint for the front-end to send chats
// The front-end sends a list of all chats on each request
app.post('/:cid', function(req, res) {
  chats[req.params.cid] = req.body;
  res.send("Ok");
});

// GET endpoint for the front-end to load chats by identifier
// The list of chats is sent as a JSON array
app.get('/chats/:cid', function (req, res) {
  let cachedchats = []
  if (req.params.cid && chats[req.params.cid]) {
    cachedchats = chats[req.params.cid]
  }

  res.send(cachedchats);
})

// GET endpoint for loading HTML from the template in views/homepage.js
// If a chat identifier is in the path, parse as :cid and render in template
app.get(['/', '/:cid'], function (req, res, next) {
  renderView('homepage', req, res, {
    title: 'Very Cool Chatbot',
    cid: req.params.cid
  });
});

app.listen(port, () => console.log(`Node app listening on port ${port}!`));

Take note of the GET endpoint for the root path - it also parses a cid parameter from the path and passes that into the html-express-js template. You'll see that parameter used in both the template as well as the root Svelte component below.

Svelte standalone

We set up the Svelte project in a subdirectory ./svelte using Vite, as recommended in the docs -

you can also use Svelte (without Kit) with Vite by running npm create vite@latest and selecting the Svelte option.

After initializing and running a build using npm run build, we can take a peek at the generated assets under ./svelte/dist. Of note is the HTML file that gets created, it's actually pretty simple -

<!-- ./svelte/dist/index.html -->
<!DOCTYPE html>
<html lang="en">
  <head>
    <!-- meta tags -->
    <script type="module" crossorigin src="/assets/index-acdd3185.js"></script>
    <link rel="stylesheet" href="/assets/index-972eb3c3.css">
  </head>
  <body>
    <div id="app"></div>
  </body>
</html>

The HTML references a .js and .css file that are created during the build step. And there is a single <div id="app"></div> that is used by Svelte to inject the compiled HTML/JS/CSS components that serve as the main web app. This means we can use the same compiled JS/CSS files in any HTML file, as long as there is a div with id="app" somewhere in the template.

html-express-js

We'll create a similar HTML file as our template that is rendered by Express. It will load the .js and .css file created by building Svelte, and include a single div with id="app". It also will have data rendered from the Express server backend, remember that path parameter called cid from above?

// views/homepage.js
import { html } from 'html-express-js';

export const view = (data, state) => html`
  <!DOCTYPE html>
  <html lang="en">
    <head>
      <title>${data.title}</title>
      <meta charset="UTF-8" />
      <meta name="viewport" content="width=device-width, initial-scale=1.0" />
      <!-- Precompiled JS/CSS files by Svelte during build step -->
      <script type="module" crossorigin src="/assets/index-acdd3185.js"></script>
      <link rel="stylesheet" href="/assets/index-972eb3c3.css">
    </head>

    <body>
      <!-- data.cid is the parsed path variable from Express -->
      <div id="app" data-chat-id="${data.cid}"></div>
    </body>
  </html>
`

Wiring it all together

How do we access the data from the template in our Svelte component? Here's the App.svelte that loads the root of our front-end app. It uses plain JS/HTML DOM to read the value of the data attribute out of the main app div, selecting by id="app". Super easy!

<!-- ./svelte/src/App.svelte -->
<script>
  import Chatbox from "./lib/Chatbox.svelte";

  // Lookup main app div
  const app = document.getElementById("app");
  // Access `data-chat-id` attribute from div
  let chatId = app.dataset.chatId;

</script>

<main class="font-mono text-primary h-full">
    <!-- other markup -->

    <Chatbox {chatId} />
</main>

In my testing, this worked great for simple parameters such as strings. For loading more complex JSON data, I found it didn't always render correctly in the html-express-js template, especially for embedded string properties that contained quotes. In that case, I would fetch JSON data when mounting the Svelte component instead.

Finally we need to copy the generated JS/CSS files from ./svelte/dist/assets into ./public/assets so that they can be served by Express. Given the file names include a random postfix for browser cache busting, we'll add a bit of code to read JS/CSS file names from the filesystem and render them in the template:

//./server.js
// ...other includes

import fs from 'fs';

const __dirname = resolve();

function readAssets() {
  const js = fs.readdirSync(`${__dirname}/public`).filter((f) => f.endsWith('.js'));
  const css = fs.readdirSync(`${__dirname}/public`).filter((f) => f.endsWith('.css'));

  return { js, css };
}

// ...other setup and endpoints

app.get(['/', '/:cid'], function (req, res, next) {
  const { js, css } = readAssets();
  res.render('homepage', {
    title: 'Very Cool Chatbot',
    cid: req.params.cid,
    js,
    css
  });
});

// views/homepage.js
import { html } from 'html-express-js'

const renderJs = (js) => {
  return js.map((j) => `<script type="module" crossorigin src="/${j}"></script>`).join('\n');
}

const renderCss = (css) => {
  return css.map((c) => `<link rel="stylesheet" href="/${c}" />`).join('\n');
}

export const view = (data, state) => html`
  <!DOCTYPE html>
  <html lang="en">
    <head>
      <title>${data.title}</title>
      <meta charset="UTF-8" />
      <meta name="viewport" content="width=device-width, initial-scale=1.0" />
      ${renderJs(data.js)}
      ${renderCss(data.css)}
    </head>

    <body data-theme="forest">
      <div id="app" data-chat-id="${data.cid}"></div>
    </body>
  </html>
`

DevEx

In review so far - we've set up an Express app to serve static assets, created a stand-alone Svelte app, and added a template that initializes the Svelte app from Vite compiled assets as well as renders data from the server. However this still requires building the Svelte app and copying the compiled asset files into a directory for Express to serve, which is a lot of commands to run while actively modifying the Svelte app. Let's set up some scripts to make this easier.

First, a simple Makefile to run the Vite build and copy the compiled files into Express asset directory -

# ./Makefile
SHELL := /bin/bash

clean:
    rm -rf ./public/* \
    && rm -rf ./svelte/dist/*

build:
    mkdir -p ./public \
    && pushd svelte \
    && npm install \
    && npm run build \
    && popd \
    && rm -rf ./public/* \
    && cp ./svelte/dist/assets/* ./public/ \
    && find ./svelte/dist -name \*.svg -exec cp {} public/ \;

run:
    npm install \
    && node server.js

We also create a small shell script called ./watch-svelte to watch for changes to *.svelte files and run the build automatically. This saves us from having to run a command every time we modify a Svelte component -

# ./watch-svelte
#!/bin/bash

inotifywait -q -m -r -e modify,create,delete ./svelte | while read DIRECTORY EVENT FILE; do
    if [[ $FILE == *.svelte ]]; then
        make build
    fi
done

Production build

To deploy the app to production, we'll create a Dockerfile that both builds the Svelte app and packages the Express app. The resulting container can be run in many hosting providers, for this example we deployed to fly.io

The Dockerfile uses a multi-stage build, which helps to keep the resulting container size smaller as we won't include any of the Vite or Svelte dev dependencies.

FROM node:22-alpine AS svelte-build

WORKDIR /app

ADD svelte ./
RUN npm install
RUN npm run build

FROM node:22-alpine

WORKDIR /app

ADD public ./public
ADD views ./views
COPY --from=svelte-build /app/dist/assets/* ./public/
COPY --from=svelte-build /app/dist/*.svg ./public/

COPY server.js ./
COPY package.json ./
RUN npm install

EXPOSE 3000

CMD node server.js

And that's it! Hopefully this is a useful reference for how you might use Svelte in other micro apps where SvelteKit's capabilities are extraneous.

Happy coding,

Ryan

Header image generated with the assistance of DALL·E 2

E2E Reactivity using Svelte with Phoenix LiveView

Ryan Cooke — Thu, 17 Nov 2022 21:51:50 +0000

This post is based on a talk given at ElixirConf 2022

Svelte is a reactive JS framework that is delightful to use - it finds a perfect balance between simplicity and power. But since it's running in a browser, there's still the issue of fetching data from the server, and keeping that data state consistent between clients and backend servers.

In contrast, LiveView (part of Elixir's Phoenix framework) takes a contrarian view and insists that state should be managed only on the server. This makes sense since data is stored in a database close to the server, and not accessible by the browser! However there are drawbacks to the user experience, as events need to make a round trip from the browser to the server and back whenever a user changes state.

If only there is a way to combine the snappy UX of client side rendering in Svelte, with the strong-consistency state management of LiveView. Well dear reader, what if I told you...

But first, why Svelte?

Few JavaScript frameworks I've used have captured "simplicity" in the way that Svelte is designed. In a gross generalization, code runs in the browser primarily to store state, capture events and manipulate DOM. Sometimes this is executing simple logic based on user interaction with the page, such as form entry validation. In other cases, there is complex interplay between elements displayed on the webpage. Often some part of the markup needs to be re-generated and rendered, as a result of changes.

A gross generalization: code runs in the browser primarily to store state, capture events and manipulate DOM

A lot of modern frameworks create significant abstractions on top of this basic utility, such as "lifecycles" for rendering, or finite state machines to manage data. They also like to reinvent markup, as though HTML/CSS/DOM are not sufficient for modern browsers on their own (cough JSX cough).

Svelte uses vanilla HTML/CSS/JS and adds just enough extra syntax to render JS variables in HTML, conditionally render blocks based on variables' state, loop and iterate over data, etc. Components logically organize and scope HTML/CSS/JS into a single file. This means you can grab some CSS from a css-tricks post and use it, without concerns about conflicting with styles outside of the component.

What is slick about Svelte is that it doesn't need a "virtual DOM" to determine which markup to re-render, but instead precompiles the page into JavaScript functions that may be called by the Svelte engine to generate updated markup, which is injected to existing DOM. This approach is not all that different than jQuery code you've probably written which manipulates DOM directly through selectors, and keeps rendering very snappy and efficient for the browser.

Svelte gives developers precise control over which JS variables should be treated as reactive, and this can include logical blocks in addition to plain variables, which means components can react to derived state.

On top of the reactive features, Svelte adds a lot of gravy - context/stores for state management across components, tweening and transitions for buttery smooth UX animations to give that native-app feeling, and actions for interfacing third-party libraries into the reactive system.

Everything everywhere all the time

All of this reactivity to local state changes is great, except for a glaring omission - a lot of state changes happen on the server, not just in the browser! Keeping state on the client up-to-date with a server creates a lot of challenges around consistency, especially if the client can update state as well. Heck, entire areas of academia are devoted to algorithms that try to keep state consistent across distributed nodes. And since the protocol of HTTP requires separate requests/responses for exchanging information, most solutions for keeping the browser up-to-date involve polling the server to refetch state.

This feels pretty clumsy compared to the smart reactivity of our client-side framework, which just figures out that state has changed and updates itself.

Enter Phoenix LiveView - a framework for server-rendered HTML that takes a similar reactive approach to re-rendering where LiveView does the hard work of tracking changes and sending the relevant diffs to the browser. This makes the server the single source of truth, regardless if state changes on the client or the server.

At the end of the day, a LiveView is nothing more than a process that receives events as messages and updates its state. The state itself is nothing more than functional and immutable Elixir data structures. The events are either internal application messages or sent by the client/browser.

While this solves hairy problems around data consistency, rendering everything on the server has a detrimental impact on UX, as browser events must be handled by the server as well. Here's an example of how to show modal with animation in LiveView, which IMO is a ridiculous amount of code. While JavaScript code can still execute in the browser, they are triggered through "hooks" supplied by LiveVew. In practice this makes it hard to design UX elements that leverage browser capabilities heavily, and using third-party libraries requires retrofitting into a LiveView hook.

But underneath the hood, something magical is happening - LiveView is using authenticated WebSockets to send diffs of HTML from the server to the browser for rendering.

When powers combine

What if we could take the best of both of these frameworks - using Svelte to drive a snappy browser UX while leveraging LiveView to keep Svelte property state up-to-date with changes from the server?

In this post, I describe exactly how to do that! Not only will we wire LiveView's lifecycle interface into Svelte's component lifecycle to update state, but we will also inject LiveView's WebSocket into Svelte components to make it easy to send updates from the client back to the server without needing an XMLHttpRequest.

This will be fun, let's get started.

NOTE: the full code for this post can be found here

I bootstrapped a new Phoenix app named "swiphly" using mix (note that LiveView is enabled by default in newer versions of Phoenix):

mix phx.new . --app swiphly --database sqlite3

Because Svelte is precompiled, we need a way to include it into the javascript package build pipeline of Phoenix. The latest version of Phoenix leverages esbuild, but a few modifications are necessary to use esbuild plugins. You can find the full instructions in the Phoenix documentation, but the commands I used are:

npm install esbuild svelte esbuild-svelte --save-dev
npm install ../deps/phoenix ../deps/phoenix_html ../deps/phoenix_live_view --save

I also need to add a custom ./assets/build.js file since we are overriding the built-in config. This will include an esbuild plugin to compile Svelte components into our package:

const esbuild = require('esbuild')
const sveltePlugin = require('esbuild-svelte')

const args = process.argv.slice(2)
const watch = args.includes('--watch')
const deploy = args.includes('--deploy')

const loader = {
  // Add loaders for images/fonts/etc, e.g. { '.svg': 'file' }
}

const plugins = [
  // Add and configure plugins here
  sveltePlugin(),
  // ... other plugins such as postCss, etc.
]

let opts = {
  entryPoints: ['js/app.js'],
  mainFields: ["svelte", "browser", "module", "main"],
  bundle: true,
  minify: false,
  target: 'es2017',
  outdir: '../priv/static/assets',
  logLevel: 'info',
  loader,
  plugins
}

if (watch) {
  opts = {
    ...opts,
    watch,
    sourcemap: 'inline'
  }
}

if (deploy) {
  opts = {
    ...opts,
    minify: true
  }
}

const promise = esbuild.build(opts)

if (watch) {
  promise.then(_result => {
    process.stdin.on('close', () => {
      process.exit(0)
    })

    process.stdin.resume()
  })
}

And that's about it, now Phoenix will precompile all ./assets/**/*.svelte files, and include them in the static JavaScript bundles under ./priv/static/assets. So far so easy!

Two lifecycles become one

Now for the fun part. We need a way to instantiate Svelte components as part of LiveView's rendering engine, and we'll do this with a Svelte-specific LiveView component with a custom hook for creating and updating a Svelte JavaScript object.

First, the LiveView component

We define a LiveView component in Phoenix, which creates a div with two data attributes:

data-name which will contain the name of the Svelte component to use for rendering, and
data-props to hold variable state that should be passed as properties to the Svelte component.

It also wires a phx-hook (Phoenix hook) to the element, which will map LiveView lifecycle methods into Svelte component lifecycle methods. We'll look at that shortly.

Data properties will be encoded as JSON when rendered by Phoenix, and parsed as JSON in the browser.

defmodule SwiphlyWeb.SvelteComponent do
  use SwiphlyWeb, :live_component

  def render(assigns) do
    ~H"""
    <div id={@id || @name} data-name={@name} data-props={json(@props)} phx-update="ignore" phx-hook="SvelteComponent"></div>
    """
  end

  defp json(nil), do: ""
  defp json(props) do
    # encode props as JSON, using Jason or other
  end
end

We can use this LiveView component when rendering from a controller action:

defmodule SwiphlyWeb.EndToEnd do
  use SwiphlyWeb, :live_view

  @impl true
  def mount(params, session, socket) do
    # do some server side logic
    # add state to the socket assignments
    socket.assign(:foo, "bar")
    {:ok, socket}
  end

  @impl true
  def render(assigns) do
    ~H"""
      <.live_component module={SwiphlyWeb.SvelteComponent} id="foo" name="MySvelteComponent" props={%{foo: @foo}} />
    """
  end
end

This renders the component defined above by referencing the module SwiphlyWeb.SvelteComponent, and passes server state from the socket assignments into a props variable of the module. There is a bit of indirection happening between the controller and the component, but it scopes only the state to the component that it needs to render.

So far, we have an HTML div that gets rendered by Phoenix as a LiveView component, with state encoded as a data attribute. The magic of the LiveView component is that it is aware of server-side state changes - whenever the props variable changes, LiveView detects this and will call a method in our custom hook in the browser.

Joining two frameworks

Let's take a look at the custom hook now, we define this in ./assets/js/hooks.js. Recall that we specified this hook in the LiveView component div with a phx-hook attribute above.

There's a lot of code here, so we'll start at the top. Svelte components compile into JavaScript objects, we can import them directly.

We specify which Svelte component to render from the LiveView component above by setting data-name to the variable name of the import. Note that the variable name of the import must match the data-name attribute of the LiveView component div. Yes this is souper hacky, but due to limitations in esbuild, a dynamic require isn't possible isn't possible 🤦.

The interface for a custom LiveView hook is the link between LiveView and Svelte lifecycles - we are instantiating the Svelte component during mounted(), updating its props during updated(), and cleaning up during destroyed().

Svelte objects are pretty simple as well, providing a constructor that takes a target DOM element and a list of properties. The DOM element I am using is simply the div being rendered by the LiveView component. I can reference an instance of this object through this._instance when updating properties or removing the component. And recall that LiveView will call these hook methods whenever state changes on the server socket assignments!

import MySvelteComponent from "./components/MySvelteComponent.svelte"

const components = {
  MySvelteComponent,
}

function parsedProps(el) {
    const props = el.getAttribute('data-props')
    return props ? JSON.parse(props) : {}
}

const SvelteComponent = {
    mounted() {
        // 
        const componentName = this.el.getAttribute('data-name')
        if (!componentName) {
            throw new Error('Component name must be provided')
        }

        const requiredApp = components[componentName]
        if (!requiredApp) {
            throw new Error(`Unable to find ${componentName} component. Did you forget to import it into hooks.js?`)
        }

        const request = (event, data, callback) => {
            this.pushEvent(event, data, callback)
        }

        const goto = (href) => {
            liveSocket.pushHistoryPatch(href, "push", this.el)
        }

        this._instance = new requiredApp({
            target: this.el,
            props: {...parsedProps(this.el), request, goto },
        })
    },

    updated() {
        const request = (event, data, callback) => {
            this.pushEvent(event, data, callback)
        }

        const goto = (href) => {
            liveSocket.pushHistoryPatch(href, "push", this.el)
        }

        this._instance.$$set({...parsedProps(this.el), request, goto })
    },

    destroyed() {
        this._instance?.$destroy()
    }
}

export default {
    SvelteComponent,
}

In updated() we are calling into Svelte internals, using $$set to update properties without re-instantiating the entire object. Similarly with $destroy, we call this internal Svelte function to release component resources and remove from DOM.

Bonus round

Now that we have end-to-end reactivity from the backend to the client, this is like a dream come true! But wait, there's more...

I'm also wrapping two methods from LiveView and adding them to the Svelte component props - pushEvent and pushHistoryPatch. pushEvent lets us send data to the server on the WebSocket, without needing XMLHttpRequest/fetch/axios, and is authenticated and encrypted. pushHistoryPatch helps with navigating to other pages, and skips full page reloads if not necessary.

Let's see how this looks in MySvelteComponent

Here's a simple component containing a form and button, with requests to the server passing a name, data and callback() parameter. That's all you need, no paths or parameters, methods or headers.

<script>
    export let foo, request
    let form
    let formResult = ''

    function createFoo() {
        const data = new FormData(form)
        request('create', Object.fromEntries(data.entries()), (result) => {
            if (result?.success) {
                setTimeout(() => { form.reset() }, 555)
                formResult = 'Foo created'
            } else if (result?.reason) {
                formResult = `Error creating foo: ${result.reason}`
            } else {
                formResult = 'An unknown error has occurred'
            }
        })
    }

    function deleteFoo() {
        request('delete', { foo_id: 123 }, () => {})
    }
</script>

<form
    class="grid grid-cols-2 gap-4 form-control"
    bind:this={form}
    on:submit|preventDefault={createFoo}
>
    <!-- some form fields -->
    <div class="font-semibold text-accent">{formResult}</div>
</form>
<div class="button" on:click={deleteFoo}>

Pattern matching FTW

And on the server, Elixir can pattern match on the name and properties to decompose logic:

defmodule SwiphlyWeb.PushEvent do
  # same code as above...

  @impl true
  def handle_event("create", %{"foo" => "bar"}, %{assigns: %{}} = socket) do
    with {_, _} <- Foo.create_bar(params) do
      # stuff...
    end
  end

  @impl true
  def handle_event("create", %{"foo" => "!bar"}, %{assigns: %{}} = socket) do
    with {_, _} <- Foo.create_notbar(params) do
      # stuff...
    end
  end
end

What's amazing about this setup is that I don't need to optimistically update local state, since the server will push the updated property state back down the hook! In practice, this cuts out half of the JS code I would normally have in my Svelte component.

One more thing

Let's review what we have so far:

Created a "cybernetic webapp" (Svelte's own words) with reactive components for a snappy browser experience
Hooked Svelte component properties to server-side state managed by Phoenix LiveView components for live re-rendering based on changes from the server
Leveraged the LiveView WebSocket to easily send changes from the client to the server as events, without messing with AJAX/REST

Pretty nice, huh!

But what if state is changing from another browser client, and that update hits a different server than I'm connected to? This is a pretty common scenario at production scale, as I'll have many servers handling browser requests. Think of any collaboration app with many users interacting with the same content.

Phoenix has another trick up its sleeve - a flexible pubsub system that is easily clustered across servers. By adding a few extra lines of code to publish an event when state is updated, all servers are notified and can update the state they are holding for their browser clients.

End-to-end reactivity can include n:n browsers and servers, not just 1:1

This is truly incredible! And because Phoenix uses a separate process for each connected browser client, this approach scales beautifully as well.

Happy coding!

- JumpWire team

Running Node.js on AWS serverless with Fargate

Ryan Cooke — Mon, 31 Oct 2022 04:49:09 +0000

Last week I had a fun project to stand up a Node.js stack for integration testing. While I've run JavaScript in a Node.js runtime on AWS Lambda many times, this testing required a backend server with more complexity, such as long-lived connections and persistent state. A great option would be to build and run the app as a docker container, but the testing environment didn't have k8s setup. Enter AWS Fargate, a container runtime that doesn't require spinning up my own infrastructure! Fargate will manage the infrastructure and the networking, and automatically pull a container from a registry such as Elastic Container Registry or Dockerhub to run with minimal configuration required.

Of course for most real-world applications, such as web servers, running a container alone is not sufficient. There is also the need to accept HTTP requests from the internet, manage DNS records, and connect to backend databases. Fortunately AWS provides options for all of these needs, with a setup called http api private integration. In a nutshell, this allows for hosts running in a private VPC to expose HTTP API endpoints to the internet through a gateway. Now the container run by Fargate can use the same isolated VPC network as other backend services and databases, without exposing them all to the public internet!

I couldn't find an up-to-date example with Terraform, so I decided to make this one! There are more than a few parts to this stack - an API gateway, application load balancer, ECR container registry, ECS cluster with Fargate, SecretsManager variables, KMS key, and Cloudwatch logs.

The headline image of this post shows the architecture of AWS services being used. Additionally, the complete code for this example can be found in this github repository subdirectory

Forest Admin

I didn't have a spare node app sitting around, so I found Forest Admin. This is actually a cool product which provides the simplicity of dashboard tools like ActiveAdmin or Retool, but preserves the privacy of the data by having you self-host the backend. The backend exposes an API that is used by the frontend client, i.e. your browser, so data doesn't need to move through Forest Admin's servers. Here's a nice graphic to visualize how this works:

Forest Admin releases their backend with a variety of frameworks - we've chosen Node.js with Express for HTTP and Sequelize for SQL ORM.

How this works

We will use Terraform to build and upload a docker container to ECR, in addition to configuring all of the AWS resources. This means I can make code or infrastructure changes, and deploy everything with a single terraform apply command.

Note: I had an existing VPC and RDS PostgreSQL database running in the testing environment, so they are excluded from this example.

I wanted to accomplish the following in this project:

Build and distribute a Node.js app as a docker container
Create production-grade architecture - use a load balancer for scaling, inject secrets securely as ENV variables, and aggregate logs for debugging
Run the container in a private network alongside other backend servers, such as RDS
Expose an HTTP API to the public Internet running in the container with node.js Express
Use Terraform to build and deploy

The key pieces to getting this all to work are:

API Gateway v2 can create a "VPC link", to forward HTTP traffic from the internet to a private host in a VPC. This allows us to keep the VPC subnets network private, i.e. not exposed to the internet.
Elastic Container Service can manage a load balancer to distribute requests to Fargate tasks. This lets us scale up and down the number of independent containers running, and eliminates the need to register task IP addresses or health checks in the load balancer.
Terraform can run a docker build command on our localhost and upload the container to Elastic Container Registry. While a mature product would have a CI/CD process set up for this, we can skip all that for smaller projects like this one.

Docker file and app.js

Created a minimal Dockerfile for this project

FROM node:16

# Create app directory
WORKDIR /usr/src/app

# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm@5+)
COPY package*.json ./
COPY yarn*.lock ./

RUN yarn install --immutable

# Bundle app source
COPY . .

EXPOSE 3000
CMD [ "node", "src/app.js" ]

app.js is a bit more interesting, we bind to 0.0.0.0 as the host to expose through the container. There is a health check endpoint configured for the root path /, so the load balancer can register a task as healthy.

require("dotenv").config();
const name = "forest-admin";
const express = require("express");

// Constants
const PORT = 3000;
// Important to bind to all hosts to run in a container
const HOST = "0.0.0.0";

// App
const app = express();
// Healthcheck route for ALB
app.get("/", (req, res) => {
  res.send("ping");
});

// Retrieve your sequelize instance
const { createAgent } = require("@forestadmin/agent");
const {
  createSequelizeDataSource,
} = require("@forestadmin/datasource-sequelize");

const { Sequelize } = require('sequelize');

const sequelize = new Sequelize(
  process.env.POSTGRESQL_URL
);
require("./models")(sequelize);

// Create your Forest Admin agent
// This must be called BEFORE all other middleware on the app
createAgent({
  ...config params...
})
  // Create your Sequelize datasource
  .addDataSource(createSequelizeDataSource(sequelize))
  // Replace "myExpressApp" by your Express application
  .mountOnExpress(app)
  .start();

...

app.listen(PORT, HOST, () => { });

Building the Docker container

We can use Terraform to run a docker build command on our localhost using a null_resource. After building, it will push the container to an ECR repository also managed up by Terraform. It can take advantage of an explicit triggers { ... } block to only rebuild when source files change. Docker and AWS cli v2 are prerequisites for this null_resource to execute correctly.

resource "aws_ecr_repository" "task_repository" {
  name         = "${var.service_name}-repository"
  force_delete = true
}

resource "null_resource" "task_ecr_image_builder" {
  triggers = {
    docker_file    = filesha256("${local.root_dir}/Dockerfile")
    package_file   = filesha256("${local.root_dir}/package.json")
    yarn_lock_file = filesha256("${local.root_dir}/yarn.lock")
    src_dir        = sha256(join("", [for f in fileset("${local.root_dir}/src", "**.js") : filesha256("${local.root_dir}/src/${f}")]))
  }

  provisioner "local-exec" {
    working_dir = local.root_dir
    interpreter = ["/bin/bash", "-c"]
    command     = <<-EOT
      aws ecr get-login-password --region ${var.region} --profile ${var.aws_profile} | docker login --username AWS --password-stdin ${local.aws_account_id}.dkr.ecr.${var.region}.amazonaws.com
      docker image build -t ${aws_ecr_repository.task_repository.repository_url}:latest .
      docker push ${aws_ecr_repository.task_repository.repository_url}:latest
    EOT
  }
}

data "aws_ecr_image" "task_image" {
  depends_on = [
    null_resource.task_ecr_image_builder
  ]

  repository_name = "${var.service_name}-repository"
  image_tag       = "latest"
}

Terraform

(Please reference the main.tf file in the repository here for a complete list of resources. Some are excluded here for brevity)

Keys, Secrets and Logs

We store sensitive variables as a JSON object in AWS SecretsManager. These variables can be referenced in the Fargate docker task defition, and will be securely injected as environment variables to the docker container at runtime. For input, we mark the terraform variables with sensitive = true to ensure they don't end up in output or state files. Also logs.

locals {
  ...
  task_secrets = {
    FOREST_ENV_SECRET  = "${var.forest_env_secret}"
    FOREST_AUTH_SECRET = "${var.forest_auth_secret}"
    POSTGRESQL_URL     = "${var.postgresql_url}"
  }
}

resource "aws_kms_key" "kms_key" {
  description             = "KMS key for encrypting secret environment variables and cloudwatch logs from the ECS cluster"
  deletion_window_in_days = 7
}

resource "aws_kms_alias" "nodejs_alias" {
  name          = "alias/ecs/${var.service_name}"
  target_key_id = aws_kms_key.kms_key.key_id
}

resource "aws_secretsmanager_secret" "task_secrets" {
  name_prefix = "/ecs/${var.service_name}"
  kms_key_id  = aws_kms_key.kms_key.id
}

resource "aws_secretsmanager_secret_version" "task_secrets" {
  secret_id     = aws_secretsmanager_secret.task_secrets.id
  secret_string = jsonencode(local.task_secrets)
}

resource "aws_cloudwatch_log_group" "nodejs_log_group" {
  name              = "/ecs/${var.service_name}-task"
  retention_in_days = 1
}

Application Load Balancer (ALB)

An ALB will connect our API Gateway to Fargate task containers running in private subnets of our VPC. The Elastic Container Service will use this load balancer to distribute requests across tasks in a cluster, as well as register/drain task containers with the ALB during deployment or scaling. Pass a list of vpc_subnet_ids corresponding to the private subnets in our VPC.

resource "aws_lb" "nodejs" {
  name            = "${var.service_name}-alb"
  internal        = true
  subnets         = var.vpc_subnet_ids
  ip_address_type = "ipv4"
}

resource "aws_lb_target_group" "nodejs" {
  name            = "${var.service_name}-alb-tar"
  target_type     = "ip"
  ip_address_type = "ipv4"
  port            = 80
  protocol        = "HTTP"
  vpc_id          = var.vpc_id
}

resource "aws_lb_listener" "nodejs" {
  load_balancer_arn = aws_lb.nodejs.arn
  protocol          = "HTTP"
  port              = 80

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.nodejs.arn
  }
}

API Gateway v2

An HTTP API Gateway will receive requests from the internet and proxy them to the ALB. aws_apigatewayv2_vpc_link is the key resource here, connecting to the ALB running in our private subnet. integration_type and protocol_type are both "HTTP", and we are forwarding all routes route_key = "ANY /{proxy+}" to the ALB.

resource "aws_apigatewayv2_api" "api" {
  name          = "${var.service_name}-api"
  protocol_type = "HTTP"
}

resource "aws_apigatewayv2_vpc_link" "nodejs" {
  name               = "${var.service_name}-api-link"
  security_group_ids = var.vpc_security_group_ids
  subnet_ids         = var.vpc_subnet_ids
}

resource "aws_apigatewayv2_integration" "nodejs" {
  api_id           = aws_apigatewayv2_api.api.id
  integration_type = "HTTP_PROXY"
  integration_uri  = aws_lb_listener.nodejs.arn

  integration_method = "ANY"
  connection_type    = "VPC_LINK"
  connection_id      = aws_apigatewayv2_vpc_link.nodejs.id
}

resource "aws_apigatewayv2_route" "nodejs" {
  api_id    = aws_apigatewayv2_api.api.id
  route_key = "ANY /{proxy+}"

  target = "integrations/${aws_apigatewayv2_integration.nodejs.id}"
}

resource "aws_apigatewayv2_stage" "default" {
  api_id      = aws_apigatewayv2_api.api.id
  name        = "$default"
  auto_deploy = true
}

Elastic Container Service

We use ECS to run our Docker container, creating a Service, Cluster and Task. The Service will use the ALB referenced above, the Cluster will use Fargate capacity providers, and the Task will have a container definition referencing the repository url and image.

The container definition also loads secrets from Secrets Manager as environment variables for the container runtime, as well as port mappings to the host. We expose port 3000 on the container, and use the same port for the load_balancer configuration on the ECS service.

The Service will manage the ALB, registering and deregistering targets in the ALB's target group for each running Task.

resource "aws_ecs_task_definition" "nodejs_task" {
  depends_on = [
    null_resource.task_ecr_image_builder
  ]

  family                   = "${var.service_name}-task"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = var.task_cpu
  memory                   = var.task_memory
  execution_role_arn       = aws_iam_role.ecs_task_execution_role.arn
  task_role_arn            = aws_iam_role.ecs_task_execution_role.arn
  container_definitions    = <<TASK_DEFINITION
[
  {
    "name": "${var.service_name}-container",
    "image": "${aws_ecr_repository.task_repository.repository_url}:${data.aws_ecr_image.task_image.image_tag}",
    "cpu": ${var.task_cpu},
    "memory": ${var.task_memory},
    "essential": true,
    "portMappings": [
      {
        "hostPort": 3000,
        "protocol": "tcp",
        "containerPort": 3000
      }
    ],
    "environment": [
      {
        "name": "NODE_ENV",
        "value": "${var.node_env}"
      },
      {
        "name": "FOREST_AGENT_URL",
        "value": "${aws_apigatewayv2_api.api.api_endpoint}"
      }
    ],
    "secrets": [
      {
        "valueFrom": "${aws_secretsmanager_secret.task_secrets.arn}:FOREST_ENV_SECRET::",
        "name": "FOREST_ENV_SECRET"
      },
      {...}
    ],
    "logConfiguration": {...}
  }
]
TASK_DEFINITION

  runtime_platform {
    operating_system_family = "LINUX"
  }
}

resource "aws_ecs_cluster" "nodejs" {
  name = "${var.service_name}-ecs-cluster"

  configuration {
    execute_command_configuration {
      kms_key_id = aws_kms_key.nodejs.key_id
      logging    = "OVERRIDE"

      log_configuration {
        cloud_watch_encryption_enabled = false
        cloud_watch_log_group_name     = aws_cloudwatch_log_group.nodejs.name
      }
    }
  }
}

resource "aws_ecs_cluster_capacity_providers" "nodejs_fargate" {
  cluster_name = aws_ecs_cluster.nodejs.name

  capacity_providers = ["FARGATE"]

  default_capacity_provider_strategy {
    base              = 1
    weight            = 100
    capacity_provider = "FARGATE"
  }
}

resource "aws_ecs_service" "nodejs" {
  depends_on = [
    aws_lb_listener.nodejs
  ]

  name          = "${var.service_name}-ecs-service"
  cluster       = aws_ecs_cluster.nodejs.id
  desired_count = 1

  # Track the latest ACTIVE revision
  task_definition        = aws_ecs_task_definition.nodejs_task.arn
  enable_execute_command = true

  network_configuration {
    subnets = var.vpc_subnet_ids
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.nodejs.arn
    container_name   = "${var.service_name}-container"
    container_port   = 3000
  }
}

And that's about it! Here's another copy of the header image to reference the services managed by Terraform:

Conclusions

Once put together, this is a pretty nice setup to run a Node.js app as serverless! The AWS services work well together, such as the API gateway providing a link into a private VPC, and the ECS service managing a load balancer to register/drain task containers automatically. But they aren't so tightly-coupled to lock you in - it's possible to bring your own load balancer or container runtime, and swap out the corresponding AWS service.

A few things I tried did not work:

At first attempt, I deployed the Fargate task into a public subnet, in hopes that it would be reachable through a public IP address. This would represent a big shortcut, potentially eliminating the gateway and load balancer. I had trouble getting the task deployed though, as it couldn't load secrets from the secretsmanager. Ultimately, this would be an inferior setup from a security perspective, as it's much better to not expose the container network to the Internet.
Using a network load balancer (NLB), instead of an application load balancer (ALB), would not see tasks as healthy when ECS with a TCP health-check. My guess is that Express doesn't respond to TCP requests in a way that the NLB recognized as healthy. After switching to an ALB, the health-check is done over HTTP, and I could see the requests hitting my health-check endpoint in Express.

But why?

You've made it to the end of this post! And you might be wondering why we run integration tests like this?

We are using Forest Admin to demonstrate how JumpWire can be plugged into an existing application without modifying any of the application's code or features. It's as simple as providing Sequelize, a popular node.js ORM used by Forest Admin, with a PostgreSQL URL for a JumpWire proxy. No other changes are necessary, we are taking the Forest Admin agent off-the-shelf and running it with JumpWire.

Happy Coding!

Elixir's DBConnection Pooling Deep Dive

Ryan Cooke — Fri, 14 Oct 2022 16:32:08 +0000

DBConnection.ConnectionError: connection not available and request was dropped from queue after 2290ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:

  1. By tracking down slow queries and making sure they are running fast enough
  2. Increasing the pool_size (albeit it increases resource consumption)
  3. Allow requests to wait longer by increasing :queue_target and :queue_interval

See DBConnection.start_link/2 for more information

It is pretty common to run unto this error as an Elixir developer working with relational databases. The suggested fixes definitely help, but personally I feel bad tweaking something I don't understand. How does changing pool_size actually affect the rest of the application? I want to keep my mysteries to Saturday morning cartoons, not production. So I decided to dig in and see how pooling works.

DBConnection is a foundation Elixir library powering many database specific adapters. Most of the time you don't need to know much about it, since the adapter libraries abstract away most of its inner workings.

While looking into some performance issues, my team realized we didn't have a shared mental modal of how pooling works with DBConnection. It is largely undocumented, and a quick glance at the code wasn't enough to let us know how it worked. So, like any rational engineer, I went way too deep in the code and traced through everything that was happening for pooled database connections.

This writeup uses Postgrex as the example adapter, but nothing about the explanation is specific to PostgreSQL. The pooling works exactly the same for any other library using DBConnection.

If you want to try the code snippets yourself, check out the Livebook version of this post.

Database setup

First comes the easy part! We'll start a database connection using a minimal set of options.

(To run the code below, you'll need postgres running locally.)

# Setup a database connection
db_opts = [
  database: "postgres",
  hostname: "localhost",
  password: "postgres",
  pool_size: 1,
  port: 5432,
  username: "postgres"
]

{:ok, conn} = Postgrex.start_link(db_opts)

What just happened?

In the configuration code above, a process is created and returned to the caller. A ton of work happened in the DBConnection library as a result - let's dive into it!

First, Postgrex.start_link/1 calls DBConnection.start_link/2 ¹

def start_link(opts) do
  ensure_deps_started!(opts)
  opts = Postgrex.Utils.default_opts(opts)
  DBConnection.start_link(Postgrex.Protocol, opts)
end

The child spec returned by DBConnection is polymorphic ². It looks for a :pool module specified in the opts and returns the result of that module's child_spec/1 function. For the default DBConnection.ConnectionPool module, this is just a wrapper around GenServer.child_spec/1 which specifies to call start_link/1 with the parameter {Postgrex.Protocol, opts}.

So the process created and returned by Postgrex.start_link/1 will be a DBConnection.ConnectionPool GenServer. Done!

No, not done.

DBConnection.ConnectionPool.init/1 creates a protected ETS table to handle queueing of queries. The table identifier is kept in the process's state, so we can look it up and read from it. The process also keeps track of whether it is :busy or :ready, initially starting as :busy. More on that later.

ConnectionPool state

# Get the ETS identifier from the process state
{_tag, queue, _codel, _timestamp} = :sys.get_state(conn)
{:busy, #Reference<0.803399952.2529296391.184107>,
 %{
   delay: 0,
   idle: #Reference<0.803399952.2529165319.184122>,
   idle_interval: 1000,
   interval: 1000,
   next: -576460751694,
   poll: #Reference<0.803399952.2529165319.184121>,
   slow: false,
   target: 50
 }, {-576460751695135000, 0}

Process tree

When the :db_connection application is first started, DBConnection.ConnectionPool.Supervisor is run as a dynamic supervisor. ³ This happens automatically when db_connection is included as a mix dependency.

ConnectionPool setup

The DBConnection.ConnectionPool process does a few things when it is initializing. Mostly it sets up timers for things like idle timeouts. It also calls DBConnection.ConnectionPool.Pool.start_supervised/3 with the ETS queue it created. ⁴

def init({mod, opts}) do
  DBConnection.register_as_pool(mod)

  queue = :ets.new(__MODULE__.Queue, [:protected, :ordered_set])
  ts = {System.monotonic_time(), 0}
  {:ok, _} = DBConnection.ConnectionPool.Pool.start_supervised(queue, mod, opts)

  # ...snip...
end

This causes the DBConnection.Watcher GenServer process (running under the application supervision tree) to create a DBConnection.ConnectionPool.Pool process as a dynamically supervised child of DBConnection.ConnectionPool.Supervisor.

Side note: normally the new process would be only be linked to its parent supervisor, but DBConnection.Watcher will monitor the calling DBConnection.ConnectionPool process and terminate the new process when the calling process stops. This effectively links the two processes together.

DBConnection.ConnectionPool.Pool is itself another supervisor. When its init/1 function is called, the :pool_size option is checked with a default of 1. This parameter is passed all the way down from the original Postgrex.start_link/1 call and determines how many children the supervisor will have. ⁵

def init({owner, tag, mod, opts}) do
  size = Keyword.get(opts, :pool_size, 1)
  children = for id <- 1..size, do: conn(owner, tag, id, mod, opts)
  sup_opts = [strategy: :one_for_one] ++ Keyword.take(opts, [:max_restarts, :max_seconds])
  Supervisor.init(children, sup_opts)
end

Each child in the pool is a DBConnection.Connection, which is an implementation of the Connection behaviour. ⁶ The adapter can specify callbacks to run after the connection is established if it chooses to.

Connection state

[{_, pool_sup, :supervisor, _}] =
  DynamicSupervisor.which_children(DBConnection.ConnectionPool.Supervisor)

[{_, pool_conn, :worker, _}] = Supervisor.which_children(pool_sup)
%Connection{mod_state: state} = :sys.get_state(pool_conn)
state
%{
  after_connect: nil,
  after_connect_timeout: 15000,
  backoff: %DBConnection.Backoff{max: 30000, min: 1000, state: {1000, 10000}, type: :rand_exp},
  client: {#Reference<0.803399952.2529296391.185395>, :pool},
  connection_listeners: [],
  mod: Postgrex.Protocol,
  opts: [
    pool_index: 1,
    types: Postgrex.DefaultTypes,
    database: "postgres",
    hostname: "localhost",
    password: "postgres",
    pool_size: 1,
    port: 5432,
    username: "postgres"
  ],
  pool: #PID<0.255.0>,
  state: %Postgrex.Protocol{
    buffer: "",
    connection_id: 88094,
    connection_key: 401165952,
    disconnect_on_error_codes: [],
    null: nil,
    parameters: #Reference<0.803399952.2529165319.184202>,
    peer: {{127, 0, 0, 1}, 5432},
    ping_timeout: 15000,
    postgres: :idle,
    queries: #Reference<0.803399952.2529296391.184199>,
    sock: {:gen_tcp, #Port<0.10>},
    timeout: 15000,
    transactions: :naive,
    types: {Postgrex.DefaultTypes, #Reference<0.803399952.2529296391.184175>}
  },
  tag: #Reference<0.803399952.2529296391.184107>,
  timer: nil
}

Each Connection process keeps track of several things in its state. The pool key contains the PID of the DBConnection.ConnectionPool process - the one that was returned from Postgrex.start_link/1. The tag key contains the table identifier for the ETS table used as a queue.

Most of the state is generic, but the nested state key is specific to the calling library - Postgrex in this case.

Adding the connection to the pool

After the connection is established, DBConnection.Holder.update/4 is called. ⁷ This creates a new public ETS table and gives ownership of it to the DBConnection.ConnectionPool process.

def update(pool, ref, mod, state) do
  holder = new(pool, ref, mod, state)

  try do
    :ets.give_away(holder, pool, {:checkin, ref, System.monotonic_time()})
    {:ok, holder}
  rescue
    ArgumentError -> :error
  end
end

When the ownership message is received, the ConnectionPool process will check its queue and try to pull off the first message. ⁸ Since the queue starts empty, this is a noop and the pool will just transition from :busy to :ready. An entry of {timestamp, holder} is inserted into the queue where holder is a reference to the ETS table created by DBConnection.Holder.

When the process is in its :ready state, the ETS queue will always contain either this holder tuple or nothing. When the process is :busy, the queue will instead have queries waiting for an available connection.

# A single entry exists in the queue that contains a reference to the holder ETS
:ets.match(queue, :'$1')
[
  [{{-576460431686553007, #Reference<0.803399952.2529296391.185475>}}]
]

Pool size

So, what happens if we change the pool size? Extra DBConnection.Connection processes are spawned under theDBConnection.ConnectionPool.Pool supervisor. They all share the same ETS queue and reference the same DBConnection.ConnectionPool process.

{:ok, conn2} = db_opts |> Keyword.put(:pool_size, 2) |> Postgrex.start_link()

# ConnectionPool.Supervisor children - one for each start_link call
Supervisor.count_children(DBConnection.ConnectionPool.Supervisor)
%{active: 2, specs: 2, supervisors: 2, workers: 0}

[_, {_, pool_sup, :supervisor, _} | _] =
  DynamicSupervisor.which_children(DBConnection.ConnectionPool.Supervisor)

# ConnectionPool.Pool children - one for each pooled connection
Supervisor.count_children(pool_sup)
%{active: 2, specs: 2, supervisors: 0, workers: 2}

The holder ETS pattern is repeated for each underlying connection in the pool. The single DBConnection.ConnectionPool process that is returned by start_link contains a single queue, and for each pooled connection an entry in the queue is inserted with holder ETS table that references a single underlying connection.

# View the queue with two pooled connections
{_tag, queue, _codel, _timestamp} = :sys.get_state(conn2)
:ets.match(queue, :'$1')
[
  [{{-576453306038052131, #Reference<0.1613371477.1188167681.181386>}}],
  [{{-576453306037963131, #Reference<0.1613371477.1188167681.181387>}}]
]

Queueing and querying

Now that we understand the processes, the remaining question is how queries get queued and sent to the underlying connections. Postgrex.query/4 takes our PID as its first argument. That gets passed to DBConnection.prepare_execute/4 ⁹ - I'll spare you the full stacktrace but eventually DBConnection.Holder.checkout_call/5 is called.

This function sends a message to the DBConnection.ConnectionPool process asking to checkout a connection. ¹⁰ One of the parameters sent is whether queueing is allowed, which is controlled by the :queue parameter and defaults to true. If the DBConnection.ConnectionPool process is in a :busy state then the request is either inserted into the ETS queue or rejected, depending on the :queue parameter. Assuming the process is :ready, the first holder tuple is deleted from the queue and ownership of that holder ETS reference is given away to the calling process. If no more holder entries are available, the process is marked as :busy.

DBConnection.Holder waits for the message from the ETS ownership transfer (within the calling process). It marks the connection contained within the holder ETS entry as locked and returns the connection.

receive do
  {:"ETS-TRANSFER", holder, pool, {^lock, ref, checkin_time}} ->
    Process.demonitor(lock, [:flush])
    {deadline, ops} = start_deadline(timeout, pool, ref, holder, start)
    :ets.update_element(holder, :conn, [{conn(:lock) + 1, lock} | ops])

    pool_ref =
      pool_ref(pool: pool, reference: ref, deadline: deadline, holder: holder, lock: lock)

    checkout_result(holder, pool_ref, checkin_time)

  # ...snip...
end

The connection is now checked out and used by DBConnection to make the query. Once the query is complete, Holder.checkin/1 is called with the connection. The holder ETS table is updated and ownership is transferred back to the DBConnection.ConnectionPool process. If the process was busy (which indicates that all connections were checkout out) then the queue is checked for any waiting queries and the steps repeat.

Summary

Let's summarize what we learned!

When start_link is called:

A DBConnection.ConnectionPool process is started. This is pretty much the only thing the user interacts with directly.
An ETS table is created by the ConnectionPool and used as queue for incoming requests, as well as tracking connections in the pool.
A DBConnection.ConnectionPool.Pool supervisor is started. This is dynamically added to the DBConnection's supervision tree and DBConnection.Watcher links it to the ConnectionPool process.
One or more DBConnection.Connection processes are started as children of the Pool supervisor. Each one represents a separate network connection to the database.
Each Connection is referenced by an ETS table created by DBConnection.Holder. Ownership of these holder tables is passed to the ConnectionPool process.

And finally when a query is sent:

The first available connection is found by looking at the ETS queue for a holder reference. The calling process gets ownership of the holder ETS table and the reference to it is removed from the queue.
If no holders are found, the query is added to the ETS queue.
When a query finishes the holder reference is passed back to the ConnectionPool process. The next queued query is pulled and run, if there are any.

Trade-offs in early company building - security edition

Ryan Cooke — Sat, 24 Sep 2022 16:11:24 +0000

In this post, we are discussing the early stages of company building, but through a lens of effort to implement security practices. Considering the harsh realities of operating in a pre-product-market-fit environment, we present a framework for pragmatic security initiatives. By adopting this framework, time is deliberately spent on activities that can scale, as an alternative to being reactive to security concerns from customers, auditors or incidents.

A trapeze of trade-offs

When starting a company, or building any new product, every decision is a balancing act that considers trade-offs of one kind over another. There are challenges of limited time, resources or information and a multitude of unknown factors that must be discovered. Goals are often moving targets, representing the best course of action at a particular moment in time, but certain to change over the course of months or weeks. Definitions of the business itself are vague and evolving.

These characteristics are naturally imbued in technical work as well - doing discovery about the nature of a problem is essential to solving it. Most startups are chasing the elusive "product-market fit", a term that is highly dependent on a cohesive vision for the product and market.

This highly dynamic environment isn't a liability for a young company - in fact it's a critical advantage to be able to react quickly to new information and incorporate it into a product. For this reason alone, startups are able to nimbly innovate around incumbents, who may have huge leads in capitalization or product in a category, but are unable to navigate change in the broader market quickly enough to stay relevant.

Nobody likes debt, except for the rich

Many technical teams face another trade-off in early product development - the ability to quickly ship features without taking on too much "technical debt". This kind of debt manifests itself as areas of the technology stack that will need to be refactored in the future due to scaling challenges or updated product features. These refactoring projects can represent major roadblocks to continuing product innovation or the ability to service a growing customer base, sometimes taking months to complete.

The ambition to avoid technical debt puts a lot of focus on the initial technology choices made by engineering teams - such as which programming languages, application frameworks, runtime infrastructure and data storage systems to use. Teams strive for an ideal of choosing the "right tool for the job", but also must consider team dynamics and competencies of individuals. Application runtimes and data storage design are particularly hard decisions to revisit, even with the advanced tooling available today. Changing these types of systems become lateral moves with respect to product capabilities, and end up representing significant opportunity cost.

Technical debt trade-offs also extend to what kind of security measures are put into place. Remediating security risks are directly correlated to the choice of programming language or runtime environment, for example, as each technology has vulnerabilities specific to them. The potential for change to these layers of the technology stack is then compounded by the need to change security controls built on top of them.

There is also the perception that extra "security" adds rigidity to a technical system - more "checks" become a (sometimes literal) drag to releasing changes, and limiting access to systems for direct updates has to be replaced by approvals from managers.

Unfortunately these factors encourage the consideration of security to be an afterthought for most early stage startups. A product can't be hacked if it doesn't exist yet, obviously?

Rabbit holes always go down

Complicating these trade-offs is that security is an incredibly wide and deep domain to comprehensively cover. There are seemingly endless optimizations to be made in the name of security, literally too many to even begin to list here. And with ambiguity from the lack of product definition, there are few guideposts to orient a "landing zone" where specific security measures might be more effective than others.

There is also an impact on velocity to consider, as most security controls will be perceived to "slow" some operations or tasks. Take access control as an example - in normal circumstances it's inconceivable that a company would grant access to customer data to every employee. But when the entire company is 6 employees, giving full access broadly across systems is typical. To have fine-grained role-based access controls for a few people is a waste of time to set up and maintain, and would come at the cost of other company-building activity. Yet strong access control is the cornerstone of any information security program.

The real risk comes when security practices don't evolve as the organization scales. Role-based access may not make sense for 6 people, but once there are 30 employees, it's reasonable to expect that teams, groups and hierarchy has found its place in the company. This is where reactive security can set in, as in the absence of internal security prioritization cedes to external forces mandating improvement, such as a key customer doing due diligence or the need for an external auditor attestation.

Scaling security - outline a roadmap & implement foundations

To avoid the fire drills that invariably accompany a reactive security posture, early stage startups should outline their security roadmap over a multi-year term. This has the benefit of allowing for agility in the near term while creating a framework for improving security over time. Taking access controls as an example again, a roadmap can highlight that a small number of people have broad access at the beginning, but as headcount increases there are corresponding plans to add roles-based authorization and SSO.

Other worthwhile initiatives to embrace early include - drafting an information security policy; defining roles and responsibilities for security in the organization (typically split between CEO and CTO); maintaining a list of vendors, sub-processors and SaaS tools in use; creating an employee handbook that outlines company values, etiquette and necessary trainings. There are public templates available for Privacy Policy and Terms of Service that are easy to amend with details about data being collected and how it is secured specific to the product. Drafts for these initiatives do not need to be exhaustive, and in some cases they might just be a few pages of materials, but they are establishing a precedent that can be revised over time and demonstrate to auditors a recorded history of good practices.

This "scaling security" approach should identify controls that can increase individual velocity as well as security over time.

Access control

Setting up a managed employee directory early on has a lot of benefits - it's a system of record for contact information, reduce risk of data loss, and provide an auditable record of individuals' authorization. Often it will include features like organization charts that capture the reporting structure of employees and managers.

This becomes a useful scaling tool for SSO and RBAC to be built out over time. It has a big impact as the company begins to grow by onboarding new employees into critical systems quickly. Roles can be refined in the directory as they evolve during restructuring, and this becomes a useful source of truth. At the time of this writing, a popular HR system called Rippling has built their business on the thesis that an employee directory is the critical system in every company.

Change management

Implementing a CI/CD pipeline for change control can immediately reduce engineering effort to publish updates to production environments. This can decrease the time it takes for new engineers to become product contributors to the product. It encourages healthy engineering practices by standardizing the review and execution of changes to non-local environments.

Over time this pipeline will be an integration point for all sorts of security controls - automated tests, static code scanning, infrastructure scanning, release tagging, and approval workflows to name a few.

Device management

Endpoint management or MDM can streamline IT help desk management, as well as ensure that employees are provided with the software they need from day one. This can take effort and cost to centralize early, so it's common to start with a BYOD policy and good documentation on how to operate workstations. It can be a manual control owned by managers to ensure that EDR, biometrics or other physical security controls are properly enabled.

Vendor due diligence and approval

Even a lightweight justification form can start to build organizational muscle around oversight for new vendors, and establish that tools shouldn't be adopted ad hoc by anyone. This diligence needs to mature every quarter, as it's a focal point for assessments, and ultimately culminates in an annual review of every vendor. It's key to start this practice early, and avoid scrambling to review hundreds of tools retroactively.

Data classification

Start with an outline of broad categories of data that is expected to be collected through websites, applications or by internal employees. Classifications can be as rough as Public/Private/Confidential and the locations of storing and processing data can name whole categories of tools, such as CRM or public cloud IaaS.

Leverage platforms, not tools

With upfront planning, it's helpful to identify platforms that can be minimally configured at the outset to not drag on velocity, but serve as a building block for improving both security and operations as the company grows. A lot of benefits of having a roadmap is being able to leverage existing systems to meet new security goals. In the examples above, adding scans for CVEs is very easy to insert into existing CI/CD pipelines; or enforcing MFA for every log-in by "flipping a switch" on SSO already active in a company directory.

An overlooked benefit of "scaling security" plans is that it helps to defend deficiencies in the control environment at the early stage. It's much easier to say "we publish alerts into Slack today and plan to install a SIEM solution in the next two quarters" than answer "N/A" when asked about security alerting.

Moving targets

Like product features and company goals, security priorities will shift as a company grows and the needs change. These kinds of changes are also easier to capture in a living roadmap, with justification documented for posterity.

The antithesis of having a plan is reactive security initiatives that compound over time into dysfunction. At some stage a dedicated security team gets hired, but they are so overwhelmed that they default to saying "no" to any request. This has a degrading effect overall as others in the organization actively avoid engaging with the security team on a regular basis.

Alternatively, having a vision for security, even one that is aspirational, can guide decisions and investment of effort in areas that can have an amplified impact on velocity, and pay dividends as the organization grows and scales.

SvelteKit Path Based Navigation

Ryan Cooke — Wed, 29 Sep 2021 20:30:22 +0000

In this post, I’ll describe how to build rich, dynamic path-based navigation using SvelteKit. It builds on SvelteKit’s routing capabilities, as well as leveraging the Svelte stores it provides to represent path and navigation state. It isn’t a trivial example, I want to show how a real-world application might work with all the code (some HTML redacted). Authentication and fetching data is left out, as topics for other posts.

Working code for this example is available at svelte-kit-navigation, which can be cloned and run locally.

Setup

We are running svelte ^3.40, @sveltejs/kit ^1.0, and a few extra libraries - @sveltejs/adapter-static, tailwindcss, postcss and others. You can see the full package list at this link.

Summary

The main moving parts for this approach leverage features of SvelteKit -- the provided load function, goto function, and $page and $navigating stores. It also uses SvelteKit’s support for dynamic paths, to encapsulate the state necessary to display the page’s intended contents. These features provide reactivity to changes in navigation and the component variables of the path, including query variables. SvelteKit also intercepts all clicks on links, allowing us to use standard HTML for navigation.

A big advantage of this approach is that it supports deep linking into your application, with rendering of each page consistent, even temporary states with modals or notifications. It also simplifies complex navigation in a SPA, without any special handling for the back button or copied links, since the page URLs are driving the details of data loading and rendering.

Detailed Overview

This example has an index page at the root path, and a page of “transactions”. Paths take a pattern of /resource/resource_id?queryParam=queryValue, and can be extended to include subpages. So a page displaying a list of transactions would match /transactions while displaying the details of a single transaction could match /transactions/000-111-000 where “000-111-000” is the transaction id. SvelteKit calls these “dynamic paths” and will extract the dynamic parts of the path as variables.

The site uses a standard SvelteKit src/routes/__layout.svelte for each page, which serves as the parent component of subsequent pages. This is a good place to initialize “global” stores with state that child components might need. There are a few states that we manage at the top level, a “loading” state while the app goes through an initial setup (such as initial user state), and authentication state to conditionally render a login prompt.

Dynamic routes

From SvelteKit's documentation:

At the heart of SvelteKit is a filesystem-based router. This means that the structure of your application is defined by the structure of your codebase — specifically, the contents of src/routes

This includes “dynamic” pages that get encoded using [brackets] in the .svelte file name. For example, the file src/routes/transactions/[...id].svelte will match paths myapp.com/transactions as well as myapp.com/transactions/00-11-00, with the latter containing an id parameter that gets parsed and passed as a prop.

Load function

This function, provided by SvelteKit, runs before each page “load”, and parses the id from the path if available, passed into the component as a prop. It’s important to note that the load function must be declared in a module script, and the variable for the prop must be exported.

In our testing, child components cannot declare additional load functions, but we’ll detail an approach that works for those below.

The load function will run each time navigation occurs, including links and the back button. You can see a full example at /transactions/[...id].svelte

<script context="module">
    // Pass the id parameter from the dynamic path slug corresponding to /transactions/[id]
    // This gets set to the exported variable transaction_id
    export async function load({ page: { params } }) {
        const { id } = params
        return { props: { transaction_id: id } }
    }
</script>

<script>
    import { goto } from '$app/navigation';
    import { writable } from 'svelte/store';
    import ...

    // This variable is set from the load function above
    export let transaction_id;

    // We use stores to reference the list of transactions as well as the transaction details
    // for the currently selected transaction.
    const transactions = writable([]);
    const selectedTxn = writable(undefined);

    // Call method reactively when transaction id changes
    $: setupPage(transaction_id, $transactions);

    //... continued below
</script>

Setup page function

In our component’s <script> section, we define a function called setupPage(). This function is responsible for setting component variables consistent with the current path. It will be reactive to changes in the path variables, invoked through reactive blocks and store subscriptions. This function should be consistent when setting state as it can be called multiple times in certain scenarios due to multiple subscriptions. As a result it’s best for this function to also be synchronous and not fetch external data (which is better done on mounting).

<script>
    // ... continuing from above
    // Main function for setting the correct state on the page.
    // This idempotent function sets the selected transaction data
    // based on the transaction id from dynamic path.
    // It identifies the selected transaction from the list of all transactions loaded
    // when the component mounts.
    function setupPage(txn_id, txns) {
        // If no transaction id is set in the path, default to the first transaction
        // This handles the path "/transactions"
        if (txn_id === '' && txns.length > 0) {
            goto(`/transactions/${txns[0].id}`)
            return
        }
        if ($selectedTxn?.id != txn_id) {
            const txn = txns.find((f) => f.id == txn_id)
            if (!txn) return
            $selectedTxn = txn
        }
    }

    // Also run the setupPage function when the list of transactions changes
    transactions.subscribe((ts) => setupPage(transaction_id, ts))
</script>

URL query parameters

We use URL query parameters to display intermediary states, such as forms or modals, that toggle on or off. In the example app, there are links to open a "create transaction" form, and a button to dismiss the form.

To show the form, we use a shorthand link to add the parameter to the current path.

<a href="?new=t">
    <!-- link contents -->
</a>

Dismissing the form takes a bit more code, as we want to only remove the parameter new without modifying the rest of the path. We can use the SvelteKit goto method to navigate without resetting the position or focus of the current page.

<button
    on:click={() => {
        // Hide form by unsetting query param new
        $page.query.delete('new')
        goto(`${$page.path}?${$page.query.toString()}`, {
            noscroll: true,
            keepfocus: true
        })
    }}
>
    Cancel
</button>

Child components and $navigating store

Since the load function is scoped to the entire component, in the case when child components need to be reactive to navigation we use subscriptions on the $page and $navigating stores. These are also used to invoke the setupPage() method.

In the example below, we have a child component displaying the details of a transaction. It also displays a form for creating a new transaction, based on a query parameter value in the URL path. The $navigating store has a few states that transition during navigation, please refer to the SvelteKit docs for full details. Here we react to the state where a to object represents the next page being loaded.

<script>
    import { page, navigating } from '$app/stores';

    let showForm = false;
    const unsubs = [];

    // Show form based on url parameters
    // Svelte-kit page store contains an instance of URLSearchParams
    // https://kit.svelte.dev/docs#loading-input-page
    function setupPage(p) {
        if (p.query.get('new') == 't') {
            showForm = true;
        } else {
            showForm = false;
        }
    }

    // Subscribe to page and navigating stores to setup page when navigation changes
    // Note that, in our testing, the Svelte-kit load function does not fire on child modules
    // This is an alternative way to detect navigation changes without a component load function
    unsubs[unsubs.length] = page.subscribe(setupPage);
    unsubs[unsubs.length] = navigating.subscribe((n) => {
        if (n?.to) {
            setupPage(n.to);
        }
    });

    // ... full component below

Put it all together

Here is the entire component. Transaction data is fetched during onMount and added to stores, and current transaction details are displayed based on the navigation. "Selecting" a transaction to view details is done through regular <a href> links or programatically using the goto method provided by SvelteKit.

Changes to navigation or state invoke the setupPage(...) method which ensures component variables are set correctly.

Also note the use of a URL query parameter ?new=t which opens (and closes) a form for "creating" a new transaction.

src/routes/transactions/[...id].svelte

<script context="module">
    // Pass the id parameter from the dynamic path slug corresponding to /transactions/[id]
    // This gets set to the exported variable transaction_id
    export async function load({ page: { params } }) {
        const { id } = params;
        return { props: { transaction_id: id } };
    }
</script>

<script>
    import { goto } from '$app/navigation';
    import { writable } from 'svelte/store';
    import { onDestroy, onMount } from 'svelte';
    import TransactionDetails from '$lib/Transaction/details.svelte';
    import { fetchTransactions } from '$lib/api';

    // This variable is set from the load function above
    export let transaction_id;

    // We use stores to reference the list of transactions as well as the transaction details
    // for the currently selected transaction.
    const transactions = writable([]);
    const selectedTxn = writable(undefined);

    // Track subscriptions to wrtable stores, to unsubscribe when the component is destroyed
    const unsubs = [];

    // Main function for setting the correct state on the page.
    // This idempotent function sets the selected transaction data
    // based on the transaction id from dynamic path.
    // It identifies the selected transaction from the list of all transactions loaded
    // when the component mounts.
    function setupPage(txn_id, txns) {
        if (txn_id === '' && txns.length > 0) {
            goto(`/transactions/${txns[0].id}`);
            return;
        }

        if ($selectedTxn?.id != txn_id) {
            const txn = txns.find((f) => f.id == txn_id);
            if (!txn) return;
            $selectedTxn = txn;
        }
    }

    // Call the setupPage method reactively when the transaction_id is changed
    $: setupPage(transaction_id, $transactions);

    // Call the setupPage method reactively when the list of all transactions is changed
    unsubs[unsubs.length] = transactions.subscribe((ts) => setupPage(transaction_id, ts));

    // Fetch all transactions when this component mounts
    onMount(() => {
        fetchTransactions().then((ts) => {
            transactions.set(ts);
        });
    });

    // Unsubscribe from all subscriptions
    onDestroy(() => unsubs.forEach((_) => _()));
</script>

<div class="flex flex-row">
    <div class="w-1/4">
        <div class="flex flex-row m-2 mt-6 justify-between">
            Transactions
            <a href="?new=t">
                <!-- SVG details omitted for conciseness -->
                <svg />
            </a>
        </div>
        <ul class="flex flex-col">
            {#each $transactions as txn (txn.id)}
                <li
                    class:active={txn.id == transaction_id}
                    class="m-2 border border-green-900 rounded-sm p-2"
                >
                    <a href={`/transactions/${txn.id}`} class="linklike">Transaction {txn.id}</a>
                </li>
            {:else}
                <li>No transactions</li>
            {/each}
        </ul>
    </div>
    <div class="w-3/4">
        {#if !$selectedTxn && $transactions?.length == 0}
            <!-- empty page element goes here -->
        {:else if $selectedTxn}
            <TransactionDetails {transaction_id} />
        {:else if transaction_id}
            <div>Transaction {transaction_id} not found</div>
        {/if}
    </div>
</div>

<style>
    li.active {
        @apply bg-gray-300 font-bold;
    }
</style>

src/lib/Transaction/details.svelte

<script>
    import { page, navigating } from '$app/stores';
    import { goto } from '$app/navigation';
    import { writable } from 'svelte/store';
    import { onDestroy } from 'svelte';

    export let transaction_id;
    let transaction = writable(undefined);
    let showForm = false;
    const unsubs = [];

    // Show form based on URL parameters
    // Svelte-kit page store contains an instance of URLSearchParams
    // https://kit.svelte.dev/docs#loading-input-page
    function setupPage(p) {
        if (p.query.get('new') == 't') {
            showForm = true;
        } else {
            showForm = false;
        }
    }

    // Subscribe to page and navigating stores to setup page when navigation changes
    // Note that, in our testing, the Svelte-kit load function does not fire on child modules
    // This is an alternative way to detect navigation changes without the component load function
    unsubs[unsubs.length] = page.subscribe(setupPage);
    unsubs[unsubs.length] = navigating.subscribe((n) => {
        if (n?.to) {
            setupPage(n.to);
        }
    });

    async function fetchTransactionDetails(txn_id) {
        if (!txn_id) return;

        // In normal circumstances, a call to an API would take place here
        // const api = fetchapi(`/api/transactions/${txn_id}`)
        // const res = await api.ready
        const res = await Promise.resolve({
            ok: true,
            json: () =>
                Promise.resolve({
                    data: {
                        id: txn_id,
                        name: `Transaction ${txn_id}`,
                        user: 'Not a person',
                        amount: '1 million dollars'
                    }
                })
        });

        if (!res.ok) throw new Error('Network error');

        const json = await res.json();
        transaction.set(json.data);
    }

    $: fetchTransactionDetails(transaction_id);

    onDestroy(() => unsubs.forEach((_) => _()));
</script>

{#if !showForm && $transaction}
    <div class="m-6 p-6 border border-gray-600 rounded">
        Details for {$transaction.name}
        <div class="grid grid-cols-2 pt-6">
            <div>Id: {$transaction.id}</div>
            <div>Name: {$transaction.name}</div>
            <div>User: {$transaction.user}</div>
            <div>Amount: {$transaction.amount}</div>
        </div>
    </div>
{/if}

{#if showForm}
    <div class="m-6 p-6 border border-gray-600 rounded">
        Create new transaction
        <form class="grid grid-cols-2">
            <label for="name">Name</label>
            <input type="text" name="name" value="" />
            <label for="user">User</label>
            <input type="text" name="user" value="" />
            <label for="amount">Amount</label>
            <input type="text" name="amount" value="" />
            <button
                name="cancel"
                class="border border-purple-800 bg-purple-100 rounded-md w-16 mt-2"
                on:click|preventDefault={() => {
                    // Hide form by unsetting query param new
                    $page.query.delete('new');
                    goto(`${$page.path}?${$page.query.toString()}`, {
                        noscroll: true,
                        keepfocus: true
                    });
                }}
            >
                Cancel
            </button>
            <button name="save" class="border border-purple-800 bg-purple-100 rounded-md w-12 mt-2"
                >Save</button
            >
        </form>
    </div>
{/if}

Here is a screenshot of the example app in action. Note the transaction id in the path, and the corresponding details selected on the page being displayed!

Svelte stores are one of the more powerful features of the Svelte framework. Importing and subscribing to state allows for component encapsulation without passing all data as props through children.

Conclusion

I’ve been working with SvelteKit for a few months now and am really enjoying the experience. There have been rare moments of coding delight as something just works in Svelte as intuited. This is in contrast with my experience in React or NextJS, where I found components, lifecycles and hooks downright befuddling at times. Svelte solves just enough problems that make reactive web page development easy, and doesn’t hide much behind magic.

Using path-based variables and parameters to set component state ties together the ease of state management in Svelte along with people's normal browsing behavior of saving links and using the back button. Additionally, driving state changes through the path drives a consistent approach to component data that simplifies the execution flow of code across a Svelte app.

We will continue to post about our use of Svelte and experience in the broader Svelte ecosystem of tools and extensions. If you found this article helpful, we would love to hear from you!

Happy coding adventures! -
The JumpWire team