DEV Community: sjdonado

Building a Fast and Compact SQLite Cache Store

sjdonado — Wed, 24 Jul 2024 06:21:00 +0000

When working on applications, caching is crucial for enhancing performance by reducing the need for repeated database fetches. Among the various SQLite libraries available, Bun's native integration is optimized for performance and takes advantage of its non-blocking I/O capabilities.

The bun:sqlite module is roughly 3-6x faster than better-sqlite3 and 8-9x faster than deno.land/x/sqlite for read queries. Each driver was benchmarked against the Northwind Traders dataset. View and run the benchmark source. Source

In addition to efficient caching, serializing JavaScript objects can be slow when using JSON. This is why it makes sense to opt for efficient binary encoding alternatives like Msgpackr or CBOR. These formats are faster to parse, support complex data types, require less CPU usage, and store data more compactly, further enhancing overall application performance.

With this in mind, let's explore how to build a cache manager using bun:sqlite along with efficient binary encoding.

A cache-manager Store

cache-manager provides a straightforward and intuitive API for caching, abstracting away the complexity of managing different cache stores and their configurations. With support for multiple stores, built-in expiration and TTL management, and robust error handling and fallback mechanisms, it ensures data integrity and freshness.

Additionally, cache-manager is highly customizable and extensible, allowing you to create custom cache stores tailored to your needs. This flexibility means you can set up and use caching with minimal code, allowing you to focus on your application's core logic.

The required interface to fulfill as a cache-manager store is as follows:

export type Store = {
    get<T>(key: string): Promise<T | undefined>;
    set<T>(key: string, data: T, ttl?: Milliseconds): Promise<void>;
    del(key: string): Promise<void>;
    reset(): Promise<void>;
    mset(arguments_: Array<[string, unknown]>, ttl?: Milliseconds): Promise<void>;
    mget(...arguments_: string[]): Promise<unknown[]>;
    mdel(...arguments_: string[]): Promise<void>;
    keys(pattern?: string): Promise<string[]>;
    ttl(key: string): Promise<number>;
};

See more here.

The Queries

Configuring SQLite for Optimal Performance

PRAGMA main.synchronous = NORMAL;: Ensures that SQLite writes are fast while still maintaining a reasonable level of data safety. It does not guarantee as much durability as FULL, but it is sufficient for many use cases.
PRAGMA main.journal_mode = WAL2;: Improves concurrency by allowing readers to access the database while a write operation is ongoing.
PRAGMA main.auto_vacuum = INCREMENTAL;: Allows SQLite to reclaim unused space incrementally, rather than all at once.

1. Creating the Cache Table

CREATE TABLE IF NOT EXISTS {table} (
    key TEXT PRIMARY KEY, 
    val BLOB, 
    created_at INTEGER, 
    expire_at INTEGER
);
CREATE INDEX IF NOT EXISTS index_expire_{table} ON {table}(expire_at);

The val column stores the cached value in a binary large object (BLOB) format, allowing it to handle various data types depending on the chosen serializer.

2. Inserting or Updating Cache Entries

INSERT OR REPLACE INTO ${name}(key, val, created_at, expire_at) VALUES (?, ?, ?, ?)

expire_at is pre-calculated based on the ttl value in milliseconds, and val is pre-checked by a function isCacheable: (value: unknown) => boolean;.

3. Retrieving Cache Entries

SELECT * FROM ${name} WHERE key = ? AND expire_at > ? LIMIT 1

Returns one record that has not expired.

Auto Purge and Batch Operations

1. purgeExpired

const purgeExpired = async () => {
  const now = Date.now();
  if (now - lastPurgeTime >= 60 * 60 * 1000) {
    const statement = db.prepare(`DELETE FROM ${name} WHERE expire_at < ?`);
    statement.run(now);
    lastPurgeTime = now;
  }
};

Keeps the cache clean and efficient by regularly removing stale entries, ensuring that the cache does not grow indefinitely and affect performance.

2. Batch Set (mset)

const mset = async (pairs: [string, unknown][], ttl?: number) => {
  const ttlValue = ttl !== undefined ? ttl * 1000 : defaultTtl;
  if (ttlValue < 0) {
    return;
  }
  const expireAt = Date.now() + ttlValue;

  const stmt = `INSERT OR REPLACE INTO ${name} (key, val, created_at, expire_at) VALUES ${pairs.map(() => '(?, ?, ?, ?)').join(', ')}`;
  const bindings = pairs.flatMap(([key, value]) => {
    if (!isCacheable(value)) {
      throw new NoCacheableError(`"${value}" is not a cacheable value`);
    }
    return [key, serializerAdapter.serialize(value), Date.now(), expireAt];
  });

  const statement = db.prepare(stmt);
  statement.run(...bindings);
};

Improves efficiency by reducing the number of individual database operations. In the same way mget executes one single query to returns valid records with the query:

SELECT * FROM ${name} WHERE key IN (${placeholders}) AND expire_at > ?

Conclusion

Bun's SQLite implementation combined with efficient binary encoding formats like Msgpackr provides a powerful solution for building fast and compact cache stores.

For the complete source code and implementation details, visit:

sjdonado / cache-manager-bun-sqlite3

Fast and compact sqlite3 cache store for Bun

Bun SQLite Store for node-cache-manager

Runs on top of bun-sqlite
Optimized mset/mget support
Multiple encoders support: msgpackr, cbor, json
Auto purge (clean expired records every hour)

Installation

bun add cache-manager-bun-sqlite3

Usage

Single store

import cacheManager from 'cache-manager';
import bunSqliteStore from 'cache-manager-bun-sqlite3';
// SQLite :memory: cache store
cache = await cacheManager.caching(bunSqliteStore, {
  serializer: 'json', // default is 'msgpackr'
  ttl: 20, // TTL in seconds
});

// On-disk cache on employees table
const cache = await cacheManager.caching(bunSqliteStore, {
  name: 'employees',
  path: '/tmp/cache.db',
});

// TTL in seconds
await cache.set('foo', { test: 'bar' }, 30);
const value = await cache.get('foo');

// TTL in seconds
await cache.set('foo'

…

View on GitHub

You can also find the npm package cache-manager-bun-sqlite3.

Happy hacking!

Validated forms with useFetcher in Remix

sjdonado — Wed, 10 Jul 2024 21:05:15 +0000

Build a custom "useFetcherForm" hook to easily handle fetcher requests.

Interacting with the server without using window.navigation significantly improves the user experience. E.g: Login forms within a dialog box or modal, optimistic UI forms or submitting multiple forms within a complex view.

If you are not familiar with Remix or the useFetcher hook, please refer to:

The FetcherForm Provider

The Remix fetcher object contains three primary attributes: fetcher.state, fetcher.data and the method fetcher.submit. To interact with them we will use React.useEffect.

This post will show how to create a provider that manages the state of a fetcher submitted form, along with a minimal custom hook:

const { onChange, submitForm, isSubmitted, error } = useFetcherForm();

Let's break down the FetcherFormProvider props one by one.

1) onChange

External inputs will notify through this method to change the internal state of the provider, it receives a FormData argument:

export default function FetcherFormProvider({
  action,
  method,
  children,
}: {
  action: string;
  method: SubmitOptions['method'];
  children: React.ReactNode;
}) {
  const [formData, setFormData] = useState<FormData>();
  [...]
  return (
    <FetcherFormContext.Provider
      value={[
        (formData: FormData) => {
          setFormData(formData);
        },
        [...]
        ]}
    >
      {children}
    </FetcherFormContext.Provider>
  );
}

2) submitForm

Since the formData was already captured by the onChange method, the request can be send by calling fetcher.submit:

export default function FetcherFormProvider({
  action,
  method,
  children,
}: {
  action: string;
  method: SubmitOptions['method'];
  children: React.ReactNode;
}) {
  const fetcher = useFetcher();
  const [formData, setFormData] = useState<FormData>();

  [...]
  return (
    <FetcherFormContext.Provider
      value={[
        (formData: FormData) => {
          setFormData(formData);
        },
        () => {
          if (formData) {
            fetcher.submit(formData, {
              method,
              action,
            });
          }
        },
        [...]
      ]}
    >
      {children}
    </FetcherFormContext.Provider>
  );
}

3) isSubmitted

Submitting the form is an asynchronous operation. There is a separate variable to listen to the form submitted event: isSubmitted. This is helpful in the following cases:

To close the Dialog or Modal when the form is successfully submitted.
To check from outside that the form was submitted and/or the request returned an OK status.

export default function FetcherFormProvider({
  action,
  method,
  children,
}: {
  action: string;
  method: SubmitOptions['method'];
  children: React.ReactNode;
}) {
  const fetcher = useFetcher();

+ const [isSubmitted, setIsSubmitted] = useState(false);
  [...]
  const [formData, setFormData] = useState<FormData>();

  useEffect(() => {
    const response = fetcher.data as { error: string } | undefined;

+   if (isSubmitted || error) return;

    if (fetcher.state === 'loading' && response) {
      [...]
+     setIsSubmitted(true);
    }
  }, [fetcher, action, formData, isSubmitted, error]);

  return (
    <FetcherFormContext.Provider
      value={[
        (formData: FormData) => {
          setFormData(formData);
        },
        () => {
          if (formData) {
            fetcher.submit(formData, {
              method,
              action,
            });
          }
        },
+       isSubmitted,
        [...]
      ]}
    >
      {children}
    </FetcherFormContext.Provider>
  );
}

4) error

One drawback of using Remix useFetcher is the lack of a straightforward error handling method. There are proposals in progress to provide a more streamlined error handling approach:

As a workaround we can rely on fetcher.state to check if the request is complete and get the message with fetcher.data by defining a common structure between the client and server actions.

The highlights

const response = fetcher.data as { error: string } | undefined; -> defines the JSON response structure to be received form the server return json({ error: new Error() }, { status: 400 } );
if (fetcher.state === 'loading' && response) { -> checks if the request is completed and a response is available.

export default function FetcherFormProvider({
  action,
  method,
  children,
}: {
  action: string;
  method: SubmitOptions['method'];
  children: React.ReactNode;
}) {
  const fetcher = useFetcher();

  const [isSubmitted, setIsSubmitted] = useState(false);
+ const [error, setError] = useState<string>();

  const [formData, setFormData] = useState<FormData>();

  useEffect(() => {
    const response = fetcher.data as { error: string } | undefined;

    if (isSubmitted || error) return;

    if (fetcher.state === 'loading' && response) {
+     if (response.error) {
+       setError(response.error);
+       return;
+     }
      setIsSubmitted(true);
    }
  }, [fetcher, action, formData, isSubmitted, error]);

  return (
    <FetcherFormContext.Provider
      value={[
        (formData: FormData) => {
          setFormData(formData);
+         setError(undefined);
        },
        () => {
          if (formData) {
            fetcher.submit(formData, {
              method,
              action,
            });
          }
        },
        isSubmitted,
+       error,
      ]}
    >
      {children}
    </FetcherFormContext.Provider>
  );
}

Optionally, the error state management can be sent as a callback. This can be received in the onSubmit function and registered as state. For example, using a registeredCallback:

- if (response.error) {
-  setError(response.error);
-  return;
- }
+ registeredCallback?.(response.error);

The useFetcherForm hook

The values sent to FetcherFormContext.Provider are defined in the FetcherFormContext

const FetcherFormContext = createContext<
  [(formData: FormData) => void, (callback?: () => void) => void, boolean, string?]
>([() => null, () => null, false]);

Then the hook exposes them in an object structure:

export const useFetcherForm = () => {
  const [onChange, submitForm, isSubmitted, error] = useContext(FetcherFormContext);
  return {
    onChange,
    submitForm,
    isSubmitted,
    error,
  };
};

An example of how it can be defined:

export function AssignmentUpdateStatusDialogButton({
  assignmentId,
  status,
}: {
  assignmentId: string;
  status: AssignmentStatus;
}) {
  const [isAttached, setIsAttached] = useState(false);
  const dialog = useRef<HTMLDialogElement>();

  useEffect(() => {
    if (isAttached) {
      dialog.current?.showModal();
    }
  }, [isAttached, dialog]);

  return (
    <ClientOnly>
      {() => (
        <>
          {isAttached &&
            createPortal(
              <FetcherFormProvider
                action={`/assignments/${assignmentId}/status`}
                method="post"
              >
                <AssignmentUpdateStatusDialog
                  ref={dialog}
                  [...]
                  setIsAttached={setIsAttached}
                />
              </FetcherFormProvider>,
              document.body
            )}
          <button
            type="button"
            className="cursor-pointer leading-none"
            onClick={() => setIsAttached(true)}
          >
            <AssignmentStatusBadge status={status} />
          </button>
        </>
      )}
    </ClientOnly>
  );
}

Demo

Real-world example available on Github: https://github.com/sjdonado/remix-dashboard/blob/da9445646392626cea065442f7758230b3d8d1fa/app/components/dialog/AssignmentUpdateStatusDialog.tsx#L32C56-L32C70

Fast, Intuitive, Smart: Restaurant search engine with Cloudflare AI

sjdonado — Sat, 13 Apr 2024 16:50:29 +0000

This is a submission for the Cloudflare AI Challenge.

What We Built

Craving your favorite meal? Our AI-powered search engine makes it easy to find the top 3 restaurants serving it near you. Just enter the dish you're looking for and your location, and let our intelligent algorithm do the rest. Whether it's a local specialty or a global delicacy, our free app will guide you to the perfect spot.

Demo

The Source Code

sjdonado / culefilo

Discover the top 3 restaurants serving your favorite food near you. Just enter your craving and location in our free AI-powered app, and start your culinary adventure today!

Cule Filo - AI-powered restaurant search engine

Discover the top 3 restaurants serving your favorite food near you. Just enter your craving and location in our free AI-powered app, and start your culinary adventure today!. Our submittion to the Cloudflare AI Challenge.

Demo: https://cule-filo.pages.dev

Screen.Recording.2024-04-13.at.18.09.12.mov

Team: @sjdonado @gjhernandez @krthr

Features

Search your favorite meal
Real time search logs
See and share your results

Design

graph TD
    A[Start] --> B{Job state?}
    B -->|Created| C[Update job state to Running]
    B -->|Running or Finished| D[Return encoded message]
    C --> E[Search for places with original query]
    E --> F{Number of places found?}
    F -->|Less than 3| G[Generate suggestions list - llama-2-13b-chat-awq]
    F -->|3 or more| H[Enhancing results]
    G --> I{Number of suggestions?}
    I -->|Greater than 0| J[Search for places with suggestions]
    I -->|0| K[Log error]
    J --> L{Number of places found?}
    L -->|Less than 3| G
    L -->|3 or more| H
    H --> M[Fetch

…

View on GitHub

Journey

Our journey in building the "Cule filo" AI-powered restaurant search engine has been an exciting and challenging one. We started by brainstorming ideas on how to leverage Cloudflare's AI capabilities to create a unique and valuable application. After exploring various domains, we settled on the idea of helping users find the best restaurants serving their favorite meals nearby.

To achieve this, we designed a system that combines multiple AI models and services. We used the llama-2-13b-chat-awq model for generating suggestions when the initial search yielded less than 3 results, ensuring that users always receive relevant recommendations. The same model was also employed for selecting appropriate thumbnails for the search results, enhancing the visual appeal and contextual relevance of the displayed images.

For generating informative and concise descriptions of restaurants based on user reviews, we utilized the bart-large-cnn model. This allowed us to provide users with a quick overview of each restaurant's characteristics and customer experiences. To further enrich the search results, we used the uform-gen2-qwen-500m model to generate textual descriptions of restaurant photos, giving users a better sense of the ambiance and offerings of each place.

Throughout the development process, we focused on creating an intuitive and responsive user interface that prioritizes simplicity and ease of use. We implemented real-time search logs to provide transparency and keep users engaged during the search process. Additionally, we integrated search history functionality, allowing users to easily access and revisit their previous searches.

Multiple Models and/or Triple Task Types

Our project, "Cule filo", utilized multiple models per task and incorporated three distinct task types, qualifying it for the additional prize categories.

1) Multiple Models per Task:

For the task of generating suggestions when the initial search yields less than 3 results, we employed the llama-2-13b-chat-awq model. This model was also used for selecting appropriate thumbnails for the search results.
To generate informative and concise descriptions of restaurants based on user reviews, we utilized the bart-large-cnn model.
For generating textual descriptions of restaurant photos, we used the uform-gen2-qwen-500m model.

2) Triple Task Types: Our project incorporated the following three task types:

Text Generation: We used the llama-2-13b-chat-awq model to generate suggestions for alternative search queries when the initial search yielded insufficient results. This model was also used to generate captions for the selected thumbnails.
Text Summarization: The bart-large-cnn model was employed to summarize user reviews and generate concise descriptions of each restaurant, providing users with a quick overview of the place's characteristics and customer experiences.
Image-to-Text: We utilized the uform-gen2-qwen-500m model to convert restaurant photos into textual descriptions, enhancing the richness and contextual relevance of the search results.

What we are proud of

We are particularly proud of the seamless integration of multiple AI models and the overall user experience we have created. Seeing the application come to life and witnessing its ability to help users discover new and exciting dining options has been incredibly rewarding.

Looking ahead, we hope to expand the capabilities of "Cule filo" by incorporating additional features such as personalized recommendations based on user preferences, integration with reservation systems, and support for multiple languages.

The Team

sjdonadoFollow

M.Sc CS - Software Engineer

José Gregorio Hernández GonzálezFollow

/gjhernandez

Wilson TovarFollow

Software Engineer @ Amazon. Applying MLOps to power state of the art ML model building.

HTMX with Bun: A Real World App

sjdonado — Tue, 05 Mar 2024 05:47:49 +0000

Let's build a real-world app with HTMX + Tailwind CSS + Bun. If you are already familiar with these tools, feel free to skip the context part.

Context

HTMX, HTMZ, HTMY...

All we need to achieve decent smooth reactivity is already out there, built-in by the major browsers. As an example, here is one interesting front-end framework:

<iframe hidden name=htmz onload="setTimeout(()=>document.querySelector(contentWindow.location.hash||null)?.replaceWith(...contentDocument.body.childNodes))"></iframe>

That's it, just an iframe, try it out yourself on https://leanrada.com/htmz/.

The philosophy behind this is the same as HTMX and the same one that drives this post: Don't reinvent rendering, just use HTML.

The blazingly fast JS server

With HTMX in mind, the next step is to get up and running a a simple web server.

Bun has gained a lot of popularity in the last few months. It is fast, and there is a framework called ElysiaJS that resembles Express.js and promises to be 22 times faster than Express.

Additionally, Bun understands TypeScript and JSX out of the box, so we can structure our app with components, server-side render them, and rely on HTMX for client-side actions.

Tailwind CSS
Unlike traditional CSS frameworks that come with predefined components, Tailwind CSS emphasizes atomic classes that represent individual CSS properties. The bundle size could be lighter in comparison to other tools, but the strength here, IMHO, is rapid prototyping.

A Real World App

The problem: I'm an Apple Music user. Someone shares a Spotify link with me, and I don't have a way to know what's inside or listen to it.

The solution: I Don't Have Spotify scrapes the links (songs/albums/artists etc) from all your favorite streaming services, and it even provides a listening preview if you just want a quick look.

The approach

Given a Spotify link, we need to extract its metadata: Fetch request + HTML parsing.
Given the metadata, we need to search for the resource on each supported streaming service: Multiple API/HTML requests + parsing (adapters).
Given the results of (1) and (2), we need to render and return the data to the client.
Additionally, we list a JSON endpoint for a cool Raycast extension.

There are two major design decisions:

The folder structure / separation of concerns
Tooling + build process, how to live reload the JSX components?

For the first one, since we have multiple streaming services, we'll inevitably encounter shared logic between them. Drawing inspiration from the adapter pattern but without the boilerplate, I organized the adapters folder with pure functions. These functions all receive the same arguments: query: string and metadata: SpotifyMetadata, and they return a SpotifyContentLink.

The adapters are called by the spotifySearch function in the search service, which is also responsible for caching and updating the statistics. Why caching? In short: these calls are quite time-consuming, and we can easily hit a rate limit if we send many of them at the same time.

For the second aspect, running Bun is very straightforward: bun run --watch www/bin.ts. However, as a real-world app, we need some JavaScript on the client side, apart from sending AJAX requests. There is an audio player that has to be rendered on the client side (in order to append listeners to the DOM). Additionally, we would like to access the Clipboard API to improve the user experience when searching.

To bundle and minify the required JavaScript, I used Vite with the rollup-plugin-copy plugin:

import path from 'path';

import { defineConfig } from 'vite';
import copy from 'rollup-plugin-copy';

export default defineConfig({
  plugins: [
    copy({
      targets: [
        {
          src: 'dist/*',
          dest: 'public',
          copyOnce: false,
        },
      ],
      hook: 'writeBundle',
    }),
  ],
  resolve: {
    alias: {
      '~/config/constants': path.resolve(__dirname, './src/config/constants.ts'),
    },
  },
  build: {
    outDir: './dist',
    target: 'esnext',
    rollupOptions: {
      input: {
        'assets/js/audio-preview': './src/views/js/audio-preview.js',
        'assets/js/search-bar': './src/views/js/search-bar.js',
        'assets/css/index': './src/views/css/index.css',
      },
      output: {
        entryFileNames: '[name].min.js',
        chunkFileNames: `[name].min.js`,
        assetFileNames: `[name].min.[ext]`,
      },
    },
  },
});

Showcase

The search bar component (where HTMX shines)

export default function SearchBar() {
  return (
    <>
      <form
        id="search-form"
        hx-post="/search"
        hx-target="#search-results"
        hx-swap="innerHTML"
        hx-indicator="#loading-indicator"
        hx-request='\"timeout\":24000'
        class="flex w-full max-w-3xl items-center justify-center"
      >
        <label for="song-link" class="sr-only">
          Search
        </label>
        <input
          type="text"
          id="song-link"
          name="spotifyLink"
          class="flex-1 rounded-lg border bg-white p-2.5 text-sm font-normal text-black placeholder:text-gray-400 lg:text-base"
          placeholder="https://open.spotify.com/track/7A8MwSsu9efJXP6xvZfRN3?si=d4f1e2eb324c43df"
          pattern={SPOTIFY_LINK_REGEX.source}
        />
        <button
          type="submit"
          class="ml-2 rounded-lg border border-green-500 bg-green-500 p-2.5 text-sm font-medium text-white focus:outline-none focus:ring-1 focus:ring-white"
        >
          <i class="fas fa-search p-1 text-black" />
          <span class="sr-only">Search</span>
        </button>
      </form>
      <div class="my-4">
        <div id="search-results"></div>
      </div>
    </>
  );
}

The web server

import { Elysia } from 'elysia';
import { html } from '@elysiajs/html';
import { staticPlugin } from '@elysiajs/static';

import { logger } from './utils/logger';

import { apiRouter } from './routes/api';
import { pageRouter } from './routes/page';

export const app = new Elysia()
  .use(html())
  .use(
    staticPlugin({
      prefix: '',
      headers: {
        'Cache-Control': 'public, max-age=86400',
      },
    })
  )
  .on('beforeHandle', async ({ request }) => {
    logger.info(
      `${request.method} ${request.url} - ${request.headers.get('user-agent')}`
    );
  })
  .use(apiRouter)
  .use(pageRouter);

Testing

In our case, testing involves mocks and more mocks. They run on push changes to master, thanks to GitHub Actions:

The setup relies on bun:test and AxiosMockAdapter. It is separated into integration and unit tests, and the requests are injected using two helpers:

export const JSONRequest = (endpoint: string, body: object) => {
  return new Request(endpoint, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(body),
  });
};

export const formDataRequest = (endpoint: string, body: object) => {
  const formData = new FormData();

  Object.entries(body).forEach(([key, value]) => {
    formData.append(key, value);
  });

  return new Request(endpoint, {
    method: 'POST',
    body: formData,
  });
};

Incorporating e2e shouldn't be complex. An example of running the dev server with Playwright:

...
  webServer: {
    command: 'bun run dev',
    port: 3333,
    reuseExistingServer: !process.env.CI,
  }

Wrapping up

The source code is available on GitHub: https://github.com/sjdonado/idonthavespotify.
The app is up and running thanks to Dokku: https://idonthavespotify.donado.co.
The Raycast extension will be addressed in a future post, stay tuned: https://www.raycast.com/sjdonado/idonthavespotify.

Tower of Hanoi in P5.js + WASM

sjdonado — Mon, 08 May 2023 06:08:39 +0000

Implementation of the Tower of Hanoi problem using P5.js for animation and Rust compiled to WebAssembly (WASM).

In 2018, a professor at the Uni asked me to build a series of computer science games; he was looking for a very visual way to show students how the solution algorithms of NP problems (like the Knapsack Problem, Tower of Hanoi, Tic-Tac-Toe, etc.) behave and grow with their inputs.

Tower of Hanoi is a good exercise when students are getting started with recursion and vectors. The pegs are usually fixed at 3, so it is easy to define the base case, and the only variable is the number of disks. It is worth noting that there are other more efficient approaches, such as The Binary solution or the Gray-code solution, but this was a Complexity course and we wanted to show how exponential time can break your computer while keeping it as minimal as possible.

Vanilla JS Implementation

There was only one design requirement: to show the animated solutions of the problems in a web browser. At that time I was familiar with HTML5 but not with any modern UI frameworks. Frankly, I couldn't imagine myself rendering objects from scratch in a plain HTML canvas. There had to be something out there that had already invented that wheel, and there was.

I found P5.js, it is sort of the JS version of Processing and it has everything I was looking for, a complete API for rendering, painting images and text, calling DOM elements and much more.

The main core of the animation system was the draw function provided by P5. It works as an infinite loop, so if we want to show the disks moving from x1, y1 to x2, y2, we have to update the current position x += speed y += speed where 0 < speed < 120. There are two types of motion, vertical and horizontal. An example of the former is:

Note that the refreshCanvas function refreshes the canvas and redraws the towers filled by the disks, but not the one being animated.

You may wonder how the disks know where to move, do they move at the same time as the Hanoi calculations? That would be cool, but unfortunately they don't. We are limited by the recursive algorithm, so we have to wait until all calculations are done (until the call stack is empty).

Therefore, the draw function constantly checks if the array of moves [towerFrom:towerTo, ...] generated by recursiveHanoi is not empty. If it is, the animation starts.

recursiveHanoi did the job, but it was very inefficient: it couldn't handle more than 14 disks (16K steps). The root cause was not in the algorithm itself, but in its execution in the V8 interpreter. Anyway, the same recursive function in C++ could run perfectly for more than 20 disks (aprox 16M of steps). So, running compiled bytecode in the browser was clearly the next step.

wasm-pack

After four years, I found some time to pay that deb-tech (yes, quite a long time, eh). To make it fun I rewrote everything from scratch in SolidJS, which went smooth thanks to this amazing library p5js-wrapper. For WASM, C++ is still a good choice, but what about Rust? I did some research and found wasm-pack. A few lines in the cargo.toml file and we were ready to generate compiled + ready to import bytecode!

Let's dive into the configuration:

The crate-type is set to indicate that the crate will be compiled as a dynamic library: wasm-bindgen.
gloo-utils and serde are included separately to avoid circular dependencies (history).

Once the configuration is ready, we run wasm-pack build -d wasm --no-typescript to compile. There is no advantage in generating ts files because we won't use those definitions since the get_moves function is gonna be called using a Web Worker.

Then, to make the wasm folder accessible to the solid app, we need to update the vite.config.js file following the recommended setup from vite-plugin-wasm:

The compiled output is now accessible as follows:

But wait a second, we haven't talked about this function yet, and why do we need a Web Worker? Let's move on to the final implementation.

Hanoi Algorithm in Rust

The tricky part of writing WASM code is how to interact with the outside world (I/O operations). Fortunately, our function is quite small, its conditions are mapped as follows:

The function takes a Number -> i32 argument: The number of disks.
The function returns an array of strings serialised by the serde_json crate (which allows you to convert a Rust data structure that implements the serde::Serialize trait to a serde_json::Value type, in this case a JsValue).

And that's it, inside the public fn, we can use regular Rust types like Vect.

Web Worker

As mentioned above, the number of disks is exponentially related to the number of steps required in the solution. 20 disks are about 1M, but 24 are more than 16M (which is quite a lot 😰). In fact, the execution time of the recursive function exceeds the Doherty threshold (300ms). So, it becomes mandatory to execute it inside a non-blocking thread, for which we can use the Web Workers API.

A new .js file must be created for the worker anywhere in the src folder.

Then, since the worker instance has nothing to do with the solid component lifecycle, we can instance it outside.

Finally, when the user clicks on the play button, the async function is called:

In Summary

NP problems are highly CPU-intensive, and wasm-pack makes it easier to reduce the execution time. In addition, wrapping these operations in a Web Worker improves the user experience.

A picture is worth a thousand words: for 20 disks the Vanilla JS implementation takes more than 312000ms, while Rust does it in under 200ms (tested on a Macbook Air M2).

Vanilla JS	WASM (Rust)

Source code available on Github:

sjdonado / cs-games

Computer Science Games powered by P5.js, Rust + wasm-pack and SolidJS

Article

https://dev.to/sjdonado/tower-of-hanoi-in-p5js-wasm-4kn0

Tic Tac Toe

Tower of Hanoi

View on GitHub

That's all I have for you today. Check out my website https://sjdonado.de.

Feedback is more than welcome, drop me a line in the comments section 🙂.

Self-Hosted Password Manager with Dokku

sjdonado — Wed, 19 Apr 2023 05:46:07 +0000

Why?

I relied on 1Password for the last few years, it is robust, and very user-friendly. However, these types of services are prime targets for attacks, one popular example is the Lastpass case. It is definitely not cool to have your sensitive info spread in another data breach. Also, privacy is precious nowadays, whoever has the data, has the power.

While exploring Open Source alternatives, I found vaultwarden, an unofficial Bitwarden-compatible server written in Rust.

Migrating data manually is a tedious process, so I first checked how it would be like with Bitwarden. Surprise, surprise, they have it all! You don't have to do it by hand, it already supports the most popular password managers.

Web - Import Data Page

PS: It was a relief to discover a well-done user interface, here's a spoiler:

macOS app

To make it cost effective, I decided to use my current Dokku setup, hosted on Google Cloud and powered by Ubuntu 20.04. If you haven't heard of Dokku, this is your chance to try it out https://dokku.com/docs/getting-started/installation/.

So, having in mind the advantages and the infra required, here's how I did it.

Prerequisites

Before we get started, check that you already have the following:

A server with Dokku installed.
A domain name (letsencrypt will check it).
Local machine with Docker, Git and any SSH client.

Step by Step

1. Create Dokku App

Set your app name and link to the domain (the Dokku Proxy plugin is built in since version 0.5.0)

dokku apps:create bitwarden

# example: dokku domains:set bitwarden mypasswords.sjdonado.de
dokku domains:set bitwarden yourdomain.com

2. Set up TLS certificates

The Let's Encrypt plugin automates the generation and renewal of our certificates


# plugin installation requires root, hence the user change
sudo dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git

dokku letsencrypt:enable bitwarden
# enable the automatic renewal of certificates 
dokku letsencrypt:cron-job --add

# replace proxy port, default: 5000
dokku config:set bitwarden DOKKU_PROXY_PORT_MAP="http:80:80 https:443:80"

3. Set up database

We'll use the PostgreSQL plugin

# setup plugin
sudo dokku plugin:install https://github.com/dokku/dokku-postgres.git

# create a new database for our app
dokku postgres:create bitwarden

# link the postgres service to the app, DATABASE_URL will be attached to the ENV variables automatically. 
dokku postgres:link bitwarden bitwarden

4. Set up persistent storage

We can use the built-in Dokku storage plugin

dokku storage:ensure-directory bitwarden

dokku storage:mount bitwarden /var/lib/dokku/data/storage/bitwarden:/data

5. Set up ENV variables

By default, Bitwarden is open to public signups, although we can restrict it to invitation-only by setting SIGNUPS_ALLOWED to false and providing a valid ADMIN_TOKEN.

We're gonna use argon2 to generate a PHC string for the ADMIN_TOKEN ENV variable

# skip this line if you are a linux user
docker run --rm -it ubuntu

$ apt update && apt install openssl argon2
$ echo -n "MySecretPassword" | argon2 "$(openssl rand -base64 32)" -e -id -k 65540 -t 3 -p 4
# output: $argon2id$v=19$m=65540,t=3,p=4$bXBGMENBZUVzT3VUSFErTzQzK25Jck1BN2Z0amFuWjdSdVlIQVZqYzAzYz0$T9m73OdD2mz9+aJKLuOAdbvoARdaKxtOZ+jZcSL9/N0

For the SMTP_ variables, feel free to use your own SMTP server or comercial solutions with free-tier such as sendgrid

dokku config:set bitwarden \
  DOMAIN=https://mypasswords.sjdonado.de \
  SIGNUPS_ALLOWED=false \
  ADMIN_TOKEN='$argon2id...' \
  SMTP_HOST=smtp.sjdonado.de \
  SMTP_FROM=vaultwarden@sjdonado.de \
  SMTP_PORT=587 \
  SMTP_SECURITY=starttls \
  SMTP_USERNAME= \
  SMTP_PASSWORD=

6. Create the Dockerfile

Let's create a new folder for the project, the Dockerfile and initialise git

mkdir yourfolder && cd yourfolder
git init
touch Dockerfile

yourfolder/Dockerfile

FROM vaultwarden/server:latest

EXPOSE 80

ENV DB=postgresql

7. Deploy

We are almost there! Before to push the changes, we have to set the dokku remote url

git remote add dokku-bitwarden dokku@yourdomain.com:bitwarden

And finally

git add -A && git commit -m "My bitwarden Dockerfile"
git push dokku master

# output:
git push using:  dokku master
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (6/6), 551 bytes | 551.00 KiB/s, done.
Total 6 (delta 0), reused 0 (delta 0), pack-reused 0
-----> Cleaning up...
-----> Building bitwarden from Dockerfile
remote: #1 [internal] load build definition from Dockerfile
remote: #1 transferring dockerfile: 65B done
remote: #1 DONE 0.0s
remote: 
remote: #2 [internal] load .dockerignore
remote: #2 transferring context: 2B done
remote: #2 DONE 0.1s
...

8. That's it

Now you can access the admin panel via https://yourdomain.com/admin with your MySecretPassword (from step 5) and send invitations by email 😎.

NOTE: Be sure to add your domain when installing the official Bitwarden apps/extensions.

Chrome Extension

Wrapping up

We've explored the process of setting up a self-hosted password manager using Dokku. This can be easily replicated on any IaaS provider.

Now, you can have peace of mind knowing that your passwords are securely stored and easily accessible whenever and wherever you need them.

Source code available on Github.

Go check my website https://sjdonado.de, drop me a line. Suggestions are more than welcome in the comments box.

Happy hacking!