DEV Community: NDREAN

On-the-fly zstandard (de)compression in Elixir

NDREAN — Wed, 15 Oct 2025 08:42:45 +0000

Notes on building Elixir bindings for the zstandard C library.

We use Zig/Zigler to wrap the zstandard primitives and integrate into Elixir.

Why Zig?: We have the amazing library Zigler that automatically generates NIFs for us from Zig code, and Zig seamlessly integrates with C libraries.

Why Zstd?: zstandard offers excellent compression ratios and very fast decompression. Its overall performance is better than zlib and is the default compression algorithm used on some Linux systems. Furthermore, it supports streaming (de)compression, which is our focus here.

What are we doing?: As said, we focus on the streaming implementation as we want to deal with potentially large files. We will:

stream files with on-the-fly (de)compression
(de)compress on-the-fly HTTP streams

Streaming implementation Trade-offs and dirty schedulers

Zstd Compression: computational costs are strategy-dependent, with a simple implementation even with streams.
For example, it takes 6.6ms to compress a 100kB PNG image with compression level 22, and 0.7ms with compression level 9, and 0.07ms for the lowest compression level 1 (with the fastest algorithm).
For this reason, a compression NIF needs to be set with dirty_cpu as this setting is compile_time.
This is acceptable when you run a one-shot compression.
When you use streams, you will run plenty of compression tasks. The good news as that (all)most of them will run quickly, meaning in less than 1ms. This means that in practice, you will not exhaust the number of dirty schedulers that the Beam can provide, thus not slow down the Beam.

Zstd decompression: significantly faster but streaming requires careful implementation. The reason is the misalignment between chunks and frames explained after.

In terms of speed, the decompression process is significantly faster than the compression process. it takes 100µs to decompress the previous compressed PNG image (90kB-> 100kB).

Whilst decompressing chunks of 100kB stays well beyond 1ms - which is our focus here-, a one-shot decompression of a 9MB file can take up to 19ms. It is still reasonably fast, but can't be run safely as a NIF. You may need to set dirty_cpu for one-shot decompression whilst chunks decompression can safely be run synchronously.

The Chunk/Frame misalignment problem

We use zstd's recommended chunk size (~128KB) obtained programmatically.

Chunk boundaries never align with zstd frame boundaries — you might read half a frame or multiple frames in one chunk.

The drawings below explains this.

▶️ ITERATION 1: Read first chunk

Step	Description
Read	128 kB from HTTP/file
Input	decompress_stream(data = 128 kB)

Result:

Frame 1	Frame 2	Frame 3
(whole): 45 kB	(whole): 60 kB	(start): 23 kB

output: 200KB decompressed
consumed: 105KB ((Frame 1 + Frame 2)
remaining: 23 kB => [Buffer for next iteration]

▶️ ITERATION 2: concatenate remaining data to the next chunk

Step	Description
Read	128 kB from HTTP/file
Input	decompress_stream(data = 23 kB + 128 kB)

Result:

Frame 3	Frame 4	Frame 5	Frame 6
23 kB + (rest: 24 kB)	(whole): 55 kB	(whole): 40 kB	(start): 9 kB

output: 200 kB
consumed: 142KB (Frame 3 + Frame 4 + Frame 5)
remaining: 9 kB => [Buffer for next iteration]

▶️ ... continues until EOF ...

Build Elixir bindings using `Zig` via `Zigler`.

We assume you have zstandard installed on your machine so it will be declared as a system dependency.

We use the C_ABI interop.

The following Elixir module is where we configure Zigler to process our Zig code. Then Zigler will build NIFs for us.
Naturally, the Zig code is adapted to use Zigler that wraps the Erlang NIF API (check it at the end).

We firstly declare:

our dependencies (the system zstd library),
how to compile the Zig code (the release mode)
what are our NIFs (we used the placeeholder ... which means we let Zigler do the work for us).
resources if any.
our Zig source: whether inlined (via ~Z) or not

Notice that the compress_stream NIF is declared as dirty_cpu. As explained before, this is because it can potentially take more than 1ms to run.
On the other hand, the decompress_stream is significantly and consistently faster to process chunks and typically processes chunks of 100 kB in less than 400µs, it can be safely run synchronously by the BEAM.

We also declare resources. These are long living processes that we want to reuse between runs: our (de)compression zstd configuration.

We declared above that we located the Zig module in the file lib/lib.zig to take advantage of the Zig language server.

defmodule ExZstandard do
  @check_leak Mix.env() in [:test, :dev]
  @release if Mix.env() == :prod, do: :fast, else: :debug

  use Zig,
    otp_app: :ex_zstandard,
    c: [link_lib: {:system, "zstd"}],
    zig_code_path: "lib.zig",
    release_mode: @release,
    leak_check: @check_leak,
    nifs: [
      ...,
      compress_stream: [:dirty_cpu]
    ],
    resources: [:ZstdCResource, :ZstdDResource]

Stream a file with real-time compression

We essentially use Stream.resource/3 where the step function is the compressor, and stream into a file. The accumulator is a tuple {file_pid, true|false} where the boolean is set to true when we reach :eof, in which case we use :end_frame for the compressor, and use : flush otherwise.

def compress_file(input_path, output_path, cctx) do
    # use a ZSTD primitive to determine a chunk size
    chunk_size = recommended_c_in_size()

    Stream.resource(
      # start_fun: state= {file_pid, true|false}
      fn ->
        {File.open!(input_path, [:read, :binary]), false}
      end,

      # read_fun: the "element" is emitted and consumed by the next stream (Stream.File here)
      # (state = {pid, false}) -> {[element], state} 
      # (state = {pid, true})   -> {:halt, state}

      fn
        {file_pid, true} ->
          {:halt, file_pid}

        {file_pid, false} ->
          case IO.binread(file_pid, chunk_size) do
            :eof ->
              {:ok, {final, _, _}} = compress_stream(cctx, <<>>, :end_frame)
              {[final], {file_pid, true}}

            {:error, reason} ->
              raise "Failed to read file: #{inspect(reason)}"

            data ->
              {:ok, {compressed, _, _}} = compress_stream(cctx, data, :flush)
              {[compressed], {file_pid, false}}
          end
      end,

      # after_fun
      fn file_pid -> File.close(file_pid) end
    )
    |> Stream.into(File.stream!(output_path, [:append]))
    |> Stream.run()

    :ok
  end
end

Stream HTTP download with real-time compression

We use Req, one of the best Elixir libraries. We send the response body to the field :into where a function - that respects Req's specs - is waiting to process the received streams.

def compress_file(url, output_path, cctx)
  compressed_pid = File.open!(output_path, [:write, :binary])

  Req.get!(url,
   into: fn
     {:data, chunk}, {req, resp} ->  
       {:ok, {compressed, _, _}} = 
          ExZstandard.compress_stream(cctx, chunk, :flush)
       :ok = IO.binwrite(compressed_pid, compressed)
       {:cont, {req, resp}}
    end
 )

 :ok = File.close(compressed_pid)
end

Stream a file with real-time decompression

Again, we use Stream.resource/3 where the step function is the decompressor, and stream into a file. The accumulator is the tuple {file_pid, buffer}.

The loop is:

We process data in the function decompress_stream at each "next-step". It will return a decompressed chunk and the unconsumed data size.
We extract the leftover with binary_part/3 and return it.
At each "next step", we can concatenation of the remaining data and the new chunk and process it.

When it receives the :eof, there's usually unconsumed data (partial frame).

The "drain" helper function will recursively decompress_stream the buffer.
Once the buffer is empty, it will return :done so that the stream can halt properly and proceed to the "after" step.

def decompress_file(input_path, output_path, dctx) do
  Stream.resource(
    # start_fun: () -> state = {pid, unconsumed_buffer = <<>>}
    fn -> {File.open!(input_path, [:read, :binary]), <<>>} end,

    # read_fun : element "decompressed" is emitted to next stream step (Stream.write)
    # state = {pid, unconsumed_buffer} -> {[decompressed], {pid, unconsumed_buffer}}
    # state = {:done, pid}             -> {:halt, state}
    fn
      # Already processed EOF and emitted final data, now halt and pass the pid to the "after_step" to conclude
      {:done, file_pid} ->
        {:halt, file_pid}

      {file_pid, buffer} ->
        case IO.binread(file_pid, chunk_size) do
           :eof when buffer == <<>> ->
              # No more data to process
              {:halt, file_pid}

           :eof ->
                # Process all remaining buffered data
                # One decompress_stream call may not consume everything, so loop until empty
                decompressed_chunks = drain_buffer(dctx, buffer, [])
                # Emit chunks to the stream and mark as done (will halt on next call)
                {Enum.reverse(decompressed_chunks), {:done, file_pid}}

          {:error, reason} ->
                raise "Failed to read file: #{inspect(reason)}"

          chunk -> 
                data = buffer <> chunk # append chunk to buffer
                {:ok, {decompressed, bytes_consumed}} = ExZstandard.decompress_stream(dctx, data)

                # Keep unconsumed bytes for next iteration
                remaining = binary_part(data, bytes_consumed, byte_size(data) - bytes_consumed)

                {[decompressed], {file_pid, remaining}}
          end
    end,

    # after_fun: (acc) -> ()
    fn file_pid -> File.close(file_pid) end
  )
  |> Stream.into(File.stream!(output_path, [:append]))
  |> Stream.run()
end

and we "drain" the residual buffer:

defp drain_buffer(_dctx, <<>>, acc), do: acc

defp drain_buffer(dctx, buffer, acc) do
  {:ok, {decompressed, bytes_consumed}} = decompress_stream(dctx, buffer)

  if bytes_consumed == 0 do
    # Can't make progress - corrupted data
    raise "Decompression stalled with #{byte_size(buffer)} bytes remaining"
  end

  remaining = binary_part(buffer, bytes_consumed, byte_size(buffer) - bytes_consumed)

  drain_buffer(dctx, remaining, [decompressed | acc])
end

Stream HTTP download with real-time decompression

The Req.Response has a builtin :private key that can be used to save the unconsumed data as a temporary state. We can then retrieve it, consume it and update it every time we receive data.

def download_decompress(url, path, dctx) do
    decompressed_pid = File.open!(path, [:write, :binary])

    # Download and decompress chunks as they arrive
    result =
      Req.get!(url,
        into: fn
          {:data, chunk}, {req, resp} ->
            buffer = Req.Response.get_private(resp, :buffer, <<>>). # Get buffer from previous iteration (unconsumed bytes)

            data = buffer <> chunk. # Concatenate with new chunk

            # Decompress what we can
            {:ok, {decompressed, bytes_consumed}} = ExZstandard.decompress_stream(dctx, data)

            :ok = IO.binwrite(decompressed_pid, decompressed)

            # Keep unconsumed bytes for next iteration
            remaining = binary_part(data, bytes_consumed, byte_size(data) - bytes_consumed)

            # Store buffer in response private for next iteration
            updated_resp = Req.Response.update_private(resp, :buffer, <<>>, fn _ -> remaining end)

            {:cont, {req, updated_resp}}
        end
      )

    # Process any remaining buffered data after stream ends (handle connection close!)
    final_buffer = Req.Response.get_private(result, :buffer, <<>>)

    if byte_size(final_buffer) > 0 do
      # Drain the final buffer (like in decompress_file)
      decompressed_chunks = drain_buffer(dctx, final_buffer, [])

      Enum.each(Enum.reverse(decompressed_chunks), fn chunk ->
        :ok = IO.binwrite(decompressed_pid, chunk)
      end)
    end

    :ok = File.close(decompressed_pid)
  end

Elixir module usage

# compression
{:ok, cctx} = ExZstandard.cctx_init(3, 2) # strategy: .ZSTD_dfast

File.read!(path)
|> ExZstandard.compress_file(output_path, cctx)

url_of_data_to_compress
|> ExZstandard.download_compress(output_path, cctx)

# decompression
{:ok, dctx} = ExZstandard.dctx_init(nil) # no max_window limitation

File.read!(path)
|> ExZstandard.decompress_file(output_path, dctx)

url_of_compressed_data
|> ExZstandard.download_decompress(output_path, dctx)

Zig code

For brevity, we expose here only the "compression" code. The "decompression" is similar.

// lib.zig

const z = @cImport({
  @cInclude("zstd");
})

// this will be our resource: an optional pointer
// to the opaque zstandard context
const ZstdCCtx = struct {
    cctx: ?*z.ZSTD_CCtx,
};

/// compression context resource
pub const ZstdCResource = beam.Resource(
    *ZstdCCtx,
    @import("root"),
    .{ .Callbacks = ZstdCCtxCallback },
);

/// NIF callback to free the ZstdCCtx when the resource
/// is garbage collected
pub const ZstdCCtxCallback = struct {
    pub fn dtor(handle: **ZstdCCtx) void {
        _ = z.ZSTD_freeCCtx(handle.*.cctx);
        beam.allocator.destroy(handle.*);
        if (@import("builtin").mode == .Debug) std.debug.print("CDOTR called\n", .{});
    }
};

/// Instantiate the reusable compression context. 
/// Args: compression level and compression algorithm in the ZSTD_CCtx.
/// Returns {:ok, cctx} or {:error, reason}
pub fn cctx_init(level: i32, strategy: i16) ZstdError!beam.term {
    if (level < z.ZSTD_minCLevel() or level > z.ZSTD_maxCLevel()) {
        return beam.make_error_pair(ZstdError.InvalidCompressionLevel, .{});
    }

    // Create the ZstdCCtx struct
    const ctx = beam.allocator.create(ZstdCCtx) catch {
        return beam.make_error_pair(ZstdError.OutOfMemory, .{});
    };
    // zigler will free it in the resource callback
    // but if any error, we free it.
    errdefer beam.allocator.destroy(ctx);

    // Create the libzstd compression context
    ctx.cctx = z.ZSTD_createCCtx() orelse {
        return beam.make_error_pair(ZstdError.OutOfMemory, .{});
    };
    errdefer _ = z.ZSTD_freeCCtx(ctx.cctx);

    // Set compression level
    var result = z.ZSTD_CCtx_setParameter(
        ctx.cctx,
        z.ZSTD_c_compressionLevel,
        level,
    );
    if (z.ZSTD_isError(result) != 0) {
        const err_name = std.mem.span(z.ZSTD_getErrorName(result));
        std.log.err("Failed to set compression level: {s}", .{err_name});
        return beam.make_error_pair(ZstdError.InvalidCompressionLevel, .{});
    }

    // Set strategy from recipe or use default
    const strategy = if (config.strategy) |recipe|
        recipe.getStrategy()
    else
        ZSTD_strategy.ZSTD_dfast;

    result = z.ZSTD_CCtx_setParameter(
        ctx.cctx,
        z.ZSTD_c_strategy,
        @intFromEnum(strategy),
    );
    if (z.ZSTD_isError(result) != 0) {
        const err_name = std.mem.span(z.ZSTD_getErrorName(result));
        std.log.err("Failed to set strategy: {s}", .{err_name});
        return beam.make_error_pair(ZstdError.InvalidCompressionLevel, .{});
    }

    // Wrap in resource
    const resource = ZstdCResource.create(ctx, .{}) catch {
        return beam.make_error_pair(ZstdError.OutOfMemory, .{});
    };
    return beam.make(.{ .ok, resource }, .{});
}

pub fn compress_stream(ctx: ZstdCResource, input: []const u8, end_op: EndOp) ZstdError!beam.term {
    const cctx = ctx.unpack().*.cctx.?;

    // Allocate output buffer using recommended size
    const out_buf_size = z.ZSTD_CStreamOutSize();
    const output_data = beam.allocator.alloc(u8, out_buf_size) catch {
        return beam.make_error_pair(ZstdError.OutOfMemory, .{});
    };
    errdefer beam.allocator.free(output_data);

    // Setup buffers
    var in_buf = z.ZSTD_inBuffer_s{
        .src = input.ptr,
        .size = input.len,
        .pos = 0,
    };

    var out_buf = z.ZSTD_outBuffer_s{
        .dst = output_data.ptr,
        .size = out_buf_size,
        .pos = 0,
    };

    // Compress
    const remaining = z.ZSTD_compressStream2(
        cctx,
        &out_buf,
        &in_buf,
        end_op.toZstd(),
    );

    if (z.ZSTD_isError(remaining) != 0) {
        const err_name = std.mem.span(z.ZSTD_getErrorName(remaining));
        std.log.err("Stream compression failed: {s}", .{err_name});
        return beam.make_error_pair(ZstdError.StreamCompressionFailed, .{});
    }

    // Resize output to actual size
    const compressed = beam.allocator.realloc(output_data, out_buf.pos) catch {
        return beam.make_error_pair(ZstdError.OutOfMemory, .{});
    };
    defer beam.allocator.free(compressed);

    return beam.make(
        .{ .ok, .{ compressed, in_buf.pos, remaining } },
        .{},
    );
}

If you may want to measure the execution time, you can add the lines below in the function you want to monitor:

const start_time = std.time.nanoTimestamp();
defer {
  const elapsed_ns = std.time.nanoTimestamp() - start_time;
  const elapsed_ms = @as(f64, @floatFromInt(elapsed_ns)) / 1_000_000.0;

  std.debug.print("[NIF duration]: {d:.3} ms, for: {} bytes)\n", .{ elapsed_ms });
};

Phoenix & Vite dev setup

NDREAN — Fri, 27 Jun 2025 22:18:13 +0000

A configuration to use Vite with Phoenix LiveView and pnpm.

mix phx.new --no-assets
mix vite.install

where the Mix.Task accepts the flags dep or dev-dep, and css with the value "heroicons" or "daisyui" (copy of Phoenix 1.8 install).

⚠️ It is however recommended to run pnpm add (-D) ... --prefix assets as dependencies might need to be compiled such as native Node.js addons, in which case pnpm will warn you.

The install does:

setup pnpm-workspace.yaml
setup package.json and the dev dependencies
install the client packages
creates the Vite.ex helper
creates a config :env giving config_env()
injects the Vite watcher in dev.exs,
creates a new root.html.heex with config env dependencies
creates tow new folders "/assets/seo" and "/assets/icons" populated with placeholders
modifies MyAppWeb.static_paths/0 and adds the folder "icons"
creates vite.config.js

It warns you to use Vite.path("path-to-my-static-file"), which works in DEV and PROD mode.

You can then start the Phoenix server:

In DEV mode, you should see (at least) two WebSocket:

ws://localhost:4000/phoenix/live_reload/socket/websocket?vsn=2.0.0
ws://localhost:5173/?token=yFGCVgkhJxQg

and

app.css -> http://localhost:5173/css/app.css
app.js  -> http://localhost:5173/js/app.js

In PROD mode, the Dockerfile should run:

pnpm vite build --mode production --config vite.config.js

❗️ Leftover. The file app.css (and daisyui and heroicons) is not setup unless you pass the option css.

How? The documentation: https://vite.dev/guide/backend-integration.html

Why? You can easily bring in plugins such as VitePWA with Workbox, or ZSTD compression and more.

What? In DEV mode, you will be running a Vite dev server on port 5173 and Phoenix on port 4000.

Static assets

All your static assets should be organised in the "/assets" folder with the structure:

/assets/{js,css,seo,fonts,icons,images,wasm,...}

Do not add anything in the "/priv/static" folder as it will be pruned but instead place in the "/assets" folder.

In DEV mode, the vite.config.js settings will copy the non-fingerprinted files into "/priv/static". All the other assets (fingerprinted) will remain in the "/assets/{js,images,...}" folders and served by Vite.

For example, you have non-fingerprinted assets such as robots.txt and sitemap.xml. You place them in "/assets/seo" and these files will by copied in the "priv/static" folder and will be served by Phoenix.
For example, all your icons, eg favicons, should be placed in the folder "assets/icons" and will be copied in "priv/static/icons".

These files are served by Phoenix as declared in static_paths/0:

def static_paths, do: ~w(
      assets
      icons
      robots.txt
      sw.js
      manifest.webmanifest
      sitemap.xml)

The other - fingerprinted - static assets should use the Elixir module Vite.path/1.
In DEV mode, it will prepend http://localhost:5173 to the file name.

For example, set src={Vite.path("js/app.js")} so Vite will serve it at http://localhost:5173/js/app.js.

Another example; suppose you have a Phoenix.Component named techs.ex where you display some - fingerprinted - images:

<img src={Vite.path("images/my.svg"} alt="an-svg" loading="lazy" />

These images are placed in the folder "assets/images".

For non-fingerprpinted assets, such as those placed in "/assets/icons", eg your favorite favicon.ico, then just reference it in the layout root.html.heex with:

<link rel="icon" href="icons/favicon.ico" type="image/png" sizes="48x48" />

In PROD mode, these "app.js" or "my.svg" files will have a hashed name.
The Vite.path/1 will look into the .vite/manifest.json file generated by Vite.
At compile time, it will bundle the files with the correct fingerprinted name into the folder "/priv/static/assets" so Phoenix will serve them.

Note that this also means you do not need mix phx.digest anymore in the build stage.

Phoenix dev.exs config
Root layout
Vite Config: server and build options
- Build options
- Server options
- Run a separate Vite dev server in DEBUG mode
Package.json
- Using workspace with pnpm
- without workspace
Tailwind, daisyui and heroicons
An Elixir file path resolving module
Vite.config.js
Dockerfile
Mix Install Task

Phoenix dev.exs config

Define a config "env" variable:

# config.exs

config :my_app, :env, config_env()

# dev.exs

config :my_app, MyAppWeb.Endpoint,
  http: [ip: {127, 0, 0, 1}, port: 4000],
  [...],
  code_reloader: true,
  live_reload: [
    web_console_logger: true,
    patterns: [
      ~r"lib/my_app_web/(controllers|live|components|channels)/.*(ex|heex)$",
      ~r"lib/my_app/.*(ex)$"
    ]
  ],
  watchers: [
    pnpm: [
      "vite",
      "serve",
      "--mode",
      "development",
      "--config",
      "vite.config.js",
      cd: Path.expand("../assets", __DIR__)
    ]
  ]

Root layout

Pass the assign @env in the LiveView (or controller), or use Application.get_env(:my_app, :env):

|> assign(:env, Application.fetch_env!(:my_app, :env))

Add the following to "root.html.heex":

# root.html.heex

<link
 :if={Application.get_env(:my_app, :env) === :prod}
 rel="stylesheet"
 href={Vite.path("css/app.css")}
/>

<script 
  :if={@env === :dev}
  type="module"
  src="http://localhost:5173/@vite/client"
>
</script>

<script
 defer
 type="module"
 src={Vite.path("js/app.js")}
>
</script>

When you run the app, you can inspect the "network" tab and should get (at least) the two WebSocket connections:

ws://localhost:4000/phoenix/live_reload/socket/websocket?vsn=2.0.0
ws://localhost:5173/?token=yFGCVgkhJxQg

and

app.css -> http://localhost:5173/css/app.css
app.js  -> http://localhost:5173/js/app.js

Vite Config, server and build options

Build options

const staticDir = "../priv/static";

const buildOps = (mode) => ({
  target: ["esnext"],
  // the directory to nest generated assets under (relative to build.outDir)
  outDir: staticDir,
  rollupOptions: {
    input: mode == "production" ? getEntryPoints() : ["./js/app.js"],
    output: mode === "production" && {
      assetFileNames: "assets/[name]-[hash][extname]",
      chunkFileNames: "assets/[name]-[hash].js",
      entryFileNames: "assets/[name]-[hash].js",
    },
  },
  // generate a manifest file that contains a mapping
  // of non-hashed asset filenames in PROD mode
  manifest: mode === "production",
  path: ".vite/manifest.json",
  minify: mode === "production",
  emptyOutDir: true, // Remove old assets
  sourcemap: mode === "development" ? "inline" : true,
  reportCompressedSize: true,
  assetsInlineLimit: 0,
});

The getEntryPoints() function is detailed in the vite.config.js part. It is a list of all your files that will be fingerprinted.

The other static assets should by copied with the plugin viteStaticCopy to which we pass a list of objects (source, destination). These are eg SEO files (robots.txt, sitemap.xml), and your icons, fonts ...

Server options

// vite.config.js

const devServer = {
  cors: { origin: "http://localhost:4000" },
  allowedHosts: ["localhost"],
  strictPort: true,
  origin: "http://localhost:5173", // Vite dev server origin
  port: 5173, // Vite dev server port
  host: "localhost", // Vite dev server host
};

The vite.config.js module will export:

export default defineConfig = ({command, mode}) => {
  if (command == 'serve') {
    process.stdin.on('close', () => process.exit(0));
    copyStaticAssetsDev(); //<- see below
    process.stdin.resume();
  }

  return {
     server: mode === 'development' && devServer,
     build: buildOps(mode),
     publicDir: false,
     plugins: [
       tailwindcss(), 
       viteStaticCopy(...) //<- see below
     ],
     [...]
  }
})

Run a separate Vite dev server in DEBUG mode

You can also run the dev server in a separate terminal in DEBUG mode.

In this case, remove the watcher above, and run:

DEBUG=vite:* pnpm vite serve

The DEBUG=vite:* option gives extra informations that can be useful even if it may seem verbose.

Package.json

Using workspace with pnpm

You can use pnpm with workspaces. In the root folder, define a "pnpm-workspace.yaml" file (❗️not "yml") and reference your "assets" folder and the "deps" folder (for Phoenix.js):

# /pnpm-workspace.yaml

packages:
  - assets
  - deps/phoenix
  - deps/phoenix_html
  - deps/phoenix_live_view

ignoreBuildDependencies:
  - esbuild

onlyBuiltDependencies:
  - '@tailwindcss/oxide'

In the "assets" folder, run:

/assets> pnpm init

and populate your newly created package.json with your favourite client dependencies:

/assets> pnpm add -D tailwindcss @tailwindcss/vite daisyui vite-plugin-static-copy fast-glob lightningcss

▶️ Set "type": "module"
▶️ Set "name": "assets"
▶️ use "workspace" to reference Phoenix dependencies

# /assets/package.json

{
  "type": "module",
  "name": "assets",
  "dependencies": {
    "phoenix": "workspace:*",
    "phoenix_html": "workspace:*",
    "phoenix_live_view": "workspace:*",
    "topbar": "^3.0.0"
  },
  "devDependencies": {
    "@tailwindcss/vite": "^4.1.11",
    "daisyui": "^5.0.43",
    "tailwindcss": "^4.1.11",
    "vite": "^7.0.0",
    "vite-plugin-static-copy": "^2.3.1",
    "fast-glob": "^3.3.3",
    "lightningcss": "^1.30.1"
  }
}

In the root folder, install everything with:

/> pnpm install

without workspace

Alternatively, if you don't use workspace, then reference directly the relative location for the Phoenix dependencies.

{
  "type": "module",
  "dependencies": {
    "phoenix": "file:../deps/phoenix",
    "phoenix_html": "file:../deps/phoenix_html",
    "phoenix_live_view": "file:../deps/phoenix_live_view",
  }
  ...
}

Tailwind, daisyui and heroicons

▶️ cf Phoenix 1.8: disable automatic source detection and instead specify sources explicitely.

# /assets/css/app.css

@import 'tailwindcss' source(none);
@source "../css";
@source "../**/.*{js, jsx}";
@source "../../lib/my_app_web/";

@plugin "daisyui";

@plugin "../vendor/heroicons.js";

where the "assets/vendor/heroicons.js" file is (from phoenix 1.8):

An Elixir file path resolving module

This is needed to resolve the file path in dev or in prod mode.

❗️You need to change your application name in this module

The mix task uses the file below as a template and injects:

app_name = Mix.Project.config()[:app]

-------- Vite.ex --------

defmodule Vite do
  @moduledoc """
  Helper for Vite assets paths in development and production.
  """

  def path(asset) do
    case Application.get_env(:my_app, :env) do
      :dev -> "http://localhost:5173/" <> asset
      _ -> get_production_path(asset)
    end
  end

  defp get_production_path(asset) do
    manifest = get_manifest(:my_app)

    case Path.extname(asset) do
      ".css" -> get_main_css_in(manifest)
      _ -> get_asset_path(manifest, asset)
    end
  end

  defp get_manifest(app_name) do
    manifest_path = Path.join(:code.priv_dir(app_name), "static/.vite/manifest.json")

    with {:ok, content} <- File.read(manifest_path),
        {:ok, decoded} <- Jason.decode(content) do
      decoded
    else
      _ -> raise "Could not read Vite manifest at #{manifest_path}"
    end
  end

  defp get_main_css_in(manifest) do
    manifest
    |> Enum.flat_map(fn {_key, entry} -> Map.get(entry, "css", []) end)
    |> Enum.find(&String.contains?(&1, "app"))
    |> case do
      nil -> raise "Main CSS file not found in manifest"
      file -> "/#{file}"
    end
  end

  defp get_asset_path(manifest, asset) do
    case manifest[asset] do
      %{"file" => file} -> "/#{file}"
      _ -> raise "Asset #{asset} not found in manifest"
    end
  end
end

Vite.config.js

Your "vite.config.js" file is placed in the "assets" folder.

You locate all your assets in this "assets" folder, with a structure like:

assets/{js, css, icons, images, wasm, fonts, seo}

This Vite file will copy and build the necessary files for you (given the structure above):

-------- vite.config.js --------

// /assets/vite.config.js

import { defineConfig } from "vite";
import fs from "fs"; // for file system operations
import path from "path";
import fg from "fast-glob"; // for recursive file scanning
import tailwindcss from "@tailwindcss/vite";
import { viteStaticCopy } from "vite-plugin-static-copy";

const rootDir = path.resolve(import.meta.dirname);
const cssDir = path.resolve(rootDir, "css");
const jsDir = path.resolve(rootDir, "js");
const seoDir = path.resolve(rootDir, "seo");
const iconsDir = path.resolve(rootDir, "icons");
const srcImgDir = path.resolve(rootDir, "images");
const staticDir = path.resolve(rootDir, "../priv/static");

/* 
PROD mode: list of fingerprinted files to pass to RollUp(/Down)
*/
const getEntryPoints = () => {
  const entries = [];
  fg.sync([`${jsDir}/**/*.{js,jsx,ts,tsx}`]).forEach((file) => {
    if (/\.(js|jsx|ts|tsx)$/.test(file)) {
      entries.push(path.resolve(rootDir, file));
    }
  });

  fg.sync([`${srcImgDir}/**/*.*`]).forEach((file) => {
    if (/\.(jpg|png|svg|webp)$/.test(file)) {
      entries.push(path.resolve(rootDir, file));
    }
  });

  return entries;
};


const buildOps = (mode) => ({
  target: ["esnext"],
  outDir: staticDir,
  rollupOptions: {
    input:
      mode == "production" ? getEntryPoints() : ["./js/app.js"],
    // hash only in production mode
    output: mode === "production" && {
      assetFileNames: "assets/[name]-[hash][extname]",
      chunkFileNames: "assets/[name]-[hash].js",
      entryFileNames: "assets/[name]-[hash].js",
    },
  },
  manifest: mode === 'production',
  path: ".vite/manifest.json",
  minify: mode === "production",
  emptyOutDir: true, // Remove old assets
  sourcemap: mode === "development" ? "inline" : true,
});


/* 
Static assets served by Phoenix via the plugin `viteStaticCopy`
=> add other folders like assets/fonts...if needed
*/

// -- DEV mode: copy non-fingerprinted files to priv/static --
function copyStaticAssetsDev() {
  console.log("[vite.config] Copying non-fingerprinted assets in dev mode...");

  const copyTargets = [
    {
      srcDir: seoDir,
      destDir: staticDir, // place directly into priv/static
    },
    {
      srcDir: iconsDir,
      destDir: path.resolve(staticDir, "icons"),
    },
  ];

  copyTargets.forEach(({ srcDir, destDir }) => {
    if (!fs.existsSync(srcDir)) {
      console.log(`[vite.config] Source dir not found: ${srcDir}`);
      return;
    }
    if (!fs.existsSync(destDir)) {
      fs.mkdirSync(destDir, { recursive: true });
    }

    fg.sync(`${srcDir}/**/*.*`).forEach((srcPath) => {
      const relPath = path.relative(srcDir, srcPath);
      const destPath = path.join(destDir, relPath);
      const destSubdir = path.dirname(destPath);
      if (!fs.existsSync(destSubdir)) {
        fs.mkdirSync(destSubdir, { recursive: true });
      }

      fs.copyFileSync(srcPath, destPath);
    });
  });
}

// -- PROD mode: config for viteStaticCopy --
const getBuildTargets = () => {
  const baseTargets = [];

  // Only add targets if source directories exist
  if (fs.existsSync(seoDir)) {
    baseTargets.push({
      src: path.resolve(seoDir, "**", "*"),
      dest: path.resolve(staticDir),
    });
  }

  if (fs.existsSync(iconsDir)) {
    baseTargets.push({
      src: path.resolve(iconsDir, "**", "*"),
      dest: path.resolve(staticDir, "icons"),
    });
  }

  const devManifestPath = path.resolve(staticDir, "manifest.webmanifest");
  if (fs.existsSync(devManifestPath)) {
    fs.writeFileSync(devManifestPath, JSON.stringify(manifestOpts, null, 2));
  };
  return baseTargets;
}; 

const resolveConfig = {
  alias: {
    "@": rootDir,
    "@js": jsDir,
    "@jsx": jsDir,
    "@css": cssDir,
    "@static": staticDir,
    "@assets": srcImgDir,
  },
  extensions: [".js", ".jsx", "png", ".css", "webp", "jpg", "svg"],
};

const devServer = {
  cors: { origin: "http://localhost:4000" },
  allowedHosts: ["localhost"],
  strictPort: true,
  origin: "http://localhost:5173", // Vite dev server origin
  port: 5173, // Vite dev server port
  host: "localhost", // Vite dev server host
  watch: {
    ignored: ["**/priv/static/**", "**/lib/**", "**/*.ex", "**/*.exs"],
  },
};

export default defineConfig(({ command, mode }) => {
  if (command == "serve") {
    console.log("[vite.config] Running in development mode");
    copyStaticAssetsDev();
    process.stdin.on("close", () => process.exit(0));
    process.stdin.resume();
  }

  return {
    base: "/",
    plugins: [
      viteStaticCopy({ targets: getBuildTargets() }),
      tailwindcss(),
    ],
    resolve: resolveConfig,
    // Disable default public dir (using Phoenix's)
    publicDir: false,
    build: buildOps(mode),
    server: mode === "development" && devServer,
  };
});

Notice that we took advantage of the resolver "@". In your code, you can do:

import { myHook} from "@js/hooks/myHook";

Dockerfile

To build for production, you will run pnpm vite build.
In the Dockerfile below, we use the "workspace" version:

-------- Dockerfile --------

# Stage 1: Build
ARG ELIXIR_VERSION=1.18.3
ARG OTP_VERSION=27.3.4
ARG DEBIAN_VERSION=bullseye-20250428-slim
ARG pnpm_VERSION=10.12.4


ARG BUILDER_IMAGE="hexpm/elixir:${ELIXIR_VERSION}-erlang-${OTP_VERSION}-debian-${DEBIAN_VERSION}"
ARG RUNNER_IMAGE="debian:${DEBIAN_VERSION}"

ARG MIX_ENV=prod
ARG NODE_ENV=production

FROM ${BUILDER_IMAGE} AS builder


RUN apt-get update -y && apt-get install -y \
  build-essential  git curl && \
  curl -sL https://deb.nodesource.com/setup_22.x | bash - && \
  apt-get install -y nodejs && \
  apt-get clean && rm -f /var/lib/apt/lists/*_*

ARG MIX_ENV
ARG NODE_ENV
ENV MIX_ENV=${MIX_ENV}
ENV NODE_ENV=${NODE_ENV}

# Install pnpm
RUN corepack enable && corepack prepare pnpm@${pnpm_VERSION} --activate

# Prepare build dir
WORKDIR /app

# Install Elixir deps
RUN mix local.hex --force && mix local.rebar --force

COPY mix.exs mix.lock pnpm-lock.yaml pnpm-workspace.yaml ./
RUN mix deps.get --only ${MIX_ENV}
RUN mkdir config

# compile Elxirr deps
COPY config/config.exs config/${MIX_ENV}.exs config/
RUN mix deps.compile

# compile Node deps
WORKDIR /app/assets
COPY assets/package.json  ./
WORKDIR /app
RUN pnpm install --frozen-lockfile

# Copy app server code before building the assets
# since the server code may contain Tailwind code.
COPY lib lib

# Copy, install & build assets--------
COPY priv priv

#  this will copy the assets/.env for the Maptiler api key loaded by Vite.loadenv
WORKDIR /app/assets
COPY assets ./ 
RUN pnpm vite build --mode ${NODE_ENV} --config vite.config.js

WORKDIR /app
# RUN mix phx.digest <-- used Vite to fingerprint assets instead
RUN mix compile

COPY config/runtime.exs config/

# Build the release-------
COPY rel rel
RUN mix release

# Stage 2: Runtime --------------------------------------------
FROM ${RUNNER_IMAGE}

RUN apt-get update -y && \
  apt-get install -y libstdc++6 openssl libncurses5 locales ca-certificates \
  && apt-get clean && rm -rf /var/lib/apt/lists/*

ENV MIX_ENV=prod

RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen
ENV LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_ALL=en_US.UTF-8

WORKDIR /app

COPY --from=builder --chown=nobody:root /app/_build/${MIX_ENV}/rel/liveview_pwa ./

# <-- needed for local testing
RUN chown -R nobody:nogroup /mnt
RUN mkdir -p /app/db && \
  chown -R nobody:nogroup /app/db && \
  chmod -R 777 /app/db && \
  chown nobody /app

USER nobody

EXPOSE 4000
CMD ["/bin/sh", "-c", "/app/bin/server"]

Mix install task

This task uses templates, the files above, to prepare the code.

Check it here: https://github.com/dwyl/phx_vite/tree/main/lib/mix/tasks

Normally, you run:

mix phx.new my_app --no-assets

and then, for example:

mix vite.install --dep lightweight-charts --dev-dep tailwind-debug-mode

The detail of the task:

defmodule Mix.Tasks.Vite.Install do
  use Mix.Task
  import Mix.Generator

  @moduledoc """
  Installs and configures Vite for Phoenix LiveView projects.

  Sets up a complete Vite-based asset pipeline with Tailwind CSS, pnpm workspace,
  and generates helper modules for development and production asset handling.

  ## Usage

      $ mix vite.install
      $ mix vite.install --dep alpinejs --dev-dep postcss

  ## Options

    * `--dep` - Add a regular dependency (can be used multiple times)
    * `--dev-dep` - Add a development dependency (can be used multiple times)

  ## Examples

      $ mix vite.install --dep react --dep lodash
      $ mix vite.install --dev-dep sass --dev-dep autoprefixer

  """
  @shortdoc "Installs and configures Vite for Phoenix projects"

  @impl Mix.Task
  def run(args) do
    case System.find_executable("pnpm") do
      nil ->
        Mix.shell().error("pnpm is not installed. Please install pnpm to continue.")
        Mix.raise("Missing dependency: pnpm")

      _ ->
        :ok
    end

    # Parse command line arguments. :keep allows multiple values
    # Note: Use hyphens in CLI arguments (--dev-dep), not underscores
    # (e.g., mix vite.install --dep topbar --dev-dep @types/node)
    {opts, _, _} =
      OptionParser.parse(args, switches: [dep: :keep, dev_dep: :keep], aliases: [d: :dep])

    extra_deps = Keyword.get_values(opts, :dep)
    extra_dev_deps = Keyword.get_values(opts, :dev_dep)

    %{app_name: app_name, app_module: app_module} = context()

    Mix.shell().info("Assets setup started for #{app_name} (#{app_module})...")

    Mix.shell().info("Extra dependencies to install: #{Enum.join(extra_deps, ", ")}")

    if extra_dev_deps != [] do
      Mix.shell().info("Extra dev dependencies to install: #{Enum.join(extra_dev_deps, ", ")}")
    end

    # Add topbar by default unless --no-topbar is specified
    extra_deps = extra_deps ++ ["topbar"]

    # Setup pnpm workspace and install all dependencies
    setup_pnpm_workspace(extra_deps, extra_dev_deps)
    setup_install_deps()

    # Create asset directories and placeholder files
    setup_asset_directories()

    # Update static_paths to include icons
    update_static_paths(app_name)

    # Add config first before generating files that depend on it
    append_to_file("config/config.exs", config_template(context()))

    create_file("lib/#{app_name}_web/vite.ex", vite_helper_template(context()))

    create_file(
      "lib/#{app_name}_web/components/layouts/root.html.heex",
      root_layout_template(context())
    )

    create_file("assets/vite.config.js", vite_config_template())

    append_to_file("config/dev.exs", vite_watcher_template(context()))

    Mix.shell().info("Assets installation completed!")
    Mix.shell().info("")
    Mix.shell().info(" What was added to your project:")
    Mix.shell().info("   • Environment config in config/config.exs")
    Mix.shell().info("   • Vite watcher configuration in config/dev.exs")
    Mix.shell().info("   • Vite configuration file at assets/vite.config.js")

    Mix.shell().info(
      "   • Updated root layout template at lib/#{app_name}_web/components/layouts/root.html.heex"
    )

    Mix.shell().info("   • Vite helper module at lib/#{app_name}_web/vite.ex")
    Mix.shell().info("   • pnpm workspace configuration at pnpm-workspace.yaml")
    Mix.shell().info("   • Package.json with Phoenix workspace dependencies")

    Mix.shell().info(
      "   • Asset directories: assets/icons/ and assets/seo/ with placeholder files"
    )

    Mix.shell().info("   • Updated static_paths in lib/#{app_name}_web.ex to include 'icons'")

    Mix.shell().info("   • Client libraries: #{Enum.join(extra_deps, ", ")}")
    Mix.shell().info("   • Dev dependencies: Tailwind CSS, Vite, DaisyUI, and build tools")
    Mix.shell().info("")
    Mix.shell().info(" Next steps:")
    Mix.shell().info("   • Check 'static_paths/0' in your endpoint config")
    Mix.shell().info("   • Use 'Vite.path/1' in your code to define the source of your assets")
    Mix.shell().info("   • Run 'mix phx.server' to start your Phoenix server")
    Mix.shell().info("   • Vite dev server will start automatically on http://localhost:5173")
  end

  defp context() do
    # Get application name from mix.exs
    app_name = Mix.Project.config()[:app]
    app_module = Mix.Project.config()[:app] |> Atom.to_string() |> Macro.camelize()

    %{
      app_name: app_name,
      app_module: app_module,
      web_module: "#{app_module}Web"
    }
  end

  defp setup_pnpm_workspace(extra_deps, extra_dev_deps) do
    {v, _} = System.cmd("pnpm", ["-v"])
    version = String.trim(v)

    workspace_content = """
    packages:
      - assets
      - deps/phoenix
      - deps/phoenix_html
      - deps/phoenix_live_view

    ignoredBuiltDependencies:
      - esbuild

    onlyBuiltDependencies:
      - '@tailwindcss/oxide'
    """

    # Build dependencies object for package.json
    base_deps = %{
      "phoenix" => "workspace:*",
      "phoenix_html" => "workspace:*",
      "phoenix_live_view" => "workspace:*"
    }

    # Add extra dependencies
    deps_map =
      Enum.reduce(extra_deps, base_deps, fn dep, acc ->
        Map.put(acc, dep, "latest")
      end)

    # Build dev dependencies
    base_dev_dependencies = [
      "@tailwindcss/oxide",
      "@tailwindcss/vite",
      "@tailwindcss/forms",
      "@tailwindcss/typography",
      "daisyui",
      "fast-glob",
      "tailwindcss",
      "vite",
      "vite-plugin-static-copy"
    ]

    all_dev_deps = base_dev_dependencies ++ extra_dev_deps

    dev_deps_map =
      Enum.reduce(all_dev_deps, %{}, fn dep, acc ->
        Map.put(acc, dep, "latest")
      end)

    # Create package.json with all dependencies
    package_json = %{
      "type" => "module",
      "dependencies" => deps_map,
      "devDependencies" => dev_deps_map,
      "packageManager" => "pnpm@#{version}"
    }

    File.write!("./pnpm-workspace.yaml", workspace_content)
    File.write!("./assets/package.json", Jason.encode!(package_json, pretty: true))

    {:ok, _} = File.rm_rf("./assets/node_modules")
    {:ok, _} = File.rm_rf("./node_modules")

    Mix.shell().info("Dependencies to install: #{length(extra_deps)} packages")
    Mix.shell().info("Dev dependencies to install: #{length(all_dev_deps)} packages")
  end

  defp setup_install_deps() do
    Mix.shell().info("Installing all dependencies with pnpm...")

    case System.cmd("pnpm", ["install"]) do
      {output, 0} ->
        Mix.shell().info("Assets installed successfully")
        Mix.shell().info(output)

      {error_output, _exit_code} ->
        Mix.shell().error("Failed to install assets: #{error_output}")
    end
  end

  defp setup_asset_directories() do
    # Create icons directory and copy favicon.ico from templates
    File.mkdir_p!("./assets/icons")
    favicon_source = Path.join([__DIR__, "templates", "favicon.ico"])
    File.cp!(favicon_source, "./assets/icons/favicon.ico")
    Mix.shell().info("Created assets/icons/ directory with favicon.ico")

    # Create SEO directory and copy robots.txt from templates, create empty sitemap.xml
    File.mkdir_p!("./assets/seo")
    robots_source = Path.join([__DIR__, "templates", "robots.txt"])
    File.cp!(robots_source, "./assets/seo/robots.txt")
    File.write!("./assets/seo/sitemap.xml", "")
    Mix.shell().info("Created assets/seo/ directory with robots.txt and sitemap.xml")
  end

  # Template functions using EEx
  defp vite_helper_template(assigns) do
    read_template("vite_helper.ex.eex")
    |> EEx.eval_string(assigns: assigns)
  end

  defp vite_watcher_template(assigns) do
    ("\n\n" <> read_template("vite_watcher.exs.eex"))
    |> EEx.eval_string(assigns: assigns)
  end

  defp config_template(assigns) do
    ("\n\n" <> read_template("config.exs.eex"))
    |> EEx.eval_string(assigns: assigns)
  end

  defp root_layout_template(assigns) do
    read_template("root_layout.html.eex")
    |> EEx.eval_string(assigns: assigns)
  end

  defp vite_config_template() do
    read_template("vite.config.js")
  end

  defp read_template(filename) do
    template_path = Path.join([__DIR__, "templates", filename])
    File.read!(template_path)
  end

  defp update_static_paths(app_name) do
    web_file_path = "lib/#{app_name}_web.ex"

    content = File.read!(web_file_path)

    if String.contains?(content, "icons") do
      Mix.shell().info("#{web_file_path} already includes 'icons' in static_paths")
    else
      updated_content = String.replace(content, ~r/~w\(/, "~w(icons ")

      if updated_content != content do
        File.write!(web_file_path, updated_content)
        Mix.shell().info("Updated #{web_file_path} to include 'icons' in static_paths")
      end
    end
  end

  defp append_to_file(path, content) do
    existing_content = File.read!(path)

    # Extract just the config line to check for (remove comments)
    config_line =
      content
      |> String.split("\n")
      |> Enum.find(&String.contains?(&1, "config :"))

    # Check if the specific config already exists
    if config_line && String.contains?(existing_content, String.trim(config_line)) do
      Mix.shell().info("#{path} already contains the configuration, skipping...")
    else
      case File.write(path, content, [:append]) do
        :ok -> Mix.shell().info("Updated #{path}")
        {:error, reason} -> Mix.shell().error("Failed to update #{path}: #{reason}")
      end
    end
  end
end

Offline first LiveView as PWA

NDREAN — Sat, 10 May 2025 23:55:50 +0000

About this demo app

An example of a real-time, collaborative multi-page web app built with Phoenix LiveView designed for offline-first ready: it is packaged as a PWA.

While the app supports full offline interaction and local persistence using CRDTs (via Yjs and y-indexeddb), the core architecture is still grounded in a server-side source of truth. The server database ultimately reconciles all updates, ensuring consistency across clients.

This design enables:

✅ Full offline functionality and interactivity

✅ Real-time collaboration between multiple users

✅ Reconciliation with a central, trusted source of truth when back online

Architecture at a glance

Client-side CRDTs (Yjs) manage local state changes (e.g. counter updates), even when offline
Server-side database (Postgres and SQLite) remains authoritative
When the client reconnects, local CRDT updates are synced with the server:
- In one page, via Postgres and Phoenix.Sync with logical replication
- In another, via SQLite using a Phoenix.Channel.
Offline first solutions naturally offloads the reactive UI logic to JavaScript. We used SolidJS.
We used Vite for bundling and bring in the vite-plugin-pwa. It registers a Service Worker to cache app shell and assets for offline usage.

How it works

Optimistic Updates with Centralised Reconciliation

Although we leverage Yjs (a CRDT library) under the hood, this isn’t a fully peer-to-peer, decentralised CRDT system. Instead, in this demo we have:

No direct client-to-client replication (not pure lazy/optimistic replication).
No concurrent writes against the same replica—all operations are serialised through the server.
Writes are serialised but actions are concurrent.

What we do have is asynchronous reconciliation with an operation-based CRDT approach:

User actions (e.g. clicking “decrement” on the counter) are applied locally to a Yjs document stored in IndexedDB.
The same operation (not the full value) is sent to the server via Phoenix (either Phoenix.Sync or Phoenix.Channel).
Phoenix broadcasts that op to all connected clients.
Upon receipt, each client applies the op to its local Yjs document—order doesn’t matter, making it commutative.
The server database (Postgres or SQLite) remains the single source of truth and persists ops in sequence.

In CRDT terms: we use an operation-based CRDT (CRDT Counter) for each shared value Ops commute (order-independent) even though they pass through a central broker.

Rendering Strategy: SSR vs. Client-Side Hooks

To keep the UI interactive both online and offline, we mix LiveView’s server-side rendering (SSR) with a client-side reactive framework:

Online (LiveView SSR or Hooks)
- The PhxSync page renders a LiveView via streams and the "click" event sends data to the client to update the local Yjs document.
- The YjsCh page renders a JS-hook which initialises a SolidJS component. In the JS-hook, the SolidJS communicates via a Channel to update the database and the local Yjs document.
Offline (Manual Rendering)
- We detect the status switch via a server polling.
- The Service Worker serves the cached HTML from the cache & JS bundle.
- We render the correct JS component.
- The component reads from and writes to the local Yjs+IndexedDB replica and remains fully interactive.

Service Worker & Asset Caching

vite-plugin-pwa generates a Service Worker that pre-caches the app shell static assets (HTML, CSS, JS) on install.

We intercept navigation with phx:navigate to store the HTML document with the Cache API. When offline, we serve the cached app shell.

This ensures the entire app loads reliably even without network connectivity.

Login and token rotation

We made a "dummy" login to set a user_id. We then use an access token (saved in the session) and a refresh token (saved in a cookie) to authenticate and rotate the custom "userSocket".

Results

Deployed on Fly.io: https://liveview-pwa.fly.dev

Repo: https://github.com/ndrean/LiveView-PWA

About the pages

The `Phoenix.Sync`+`Postgres`+ `Yjs`/`IndexedDB` page:

The `SQLite`+ `Phoenix.Channel` + `Yjs`/`IndexedDB` page:

The FlightMap page

We propose an interactive map with a form with two inputs where two users can edit collaboratively a form to display markers on the map and then draw a great circle between the two points. The state is local, ephemeral by design as we don't need persistence. We still need a state manager since it is collaborative. We also need an observer/listener on state changes: when a remote user changes an input, he broadcasts the input and this updates the local state, which renders the UI. It uses Valtio, a proxy based lightweight state manager.

Tech overview

Component	Role
Vite	Build and bundling framework
SQLite	Embedded persistent storage of latest Yjs document
Postgres	Supports logical replication
Phoenix LiveView	UI rendering, incuding hooks
Phoenix.Sync	Relays Postgres streams into LiveView
PubSub / Phoenix.Channel	Broadcast/notifies other clients of updates / conveys CRDTs binaries on a separate websocket (from te LiveSocket)
Yjs / Y.Map	Holds the CRDT state client-side (shared)
y-indexeddb	Persists state locally for offline mode
Valtio	Holds local ephemeral state
Hooks	Injects communication primitives and controls JavaScript code
Service Worker / Cache API	Enable offline UI rendering and navigation by caching HTML pages and static assets
SolidJS	renders reactive UI using signals, driven by Yjs observers
Leaflet	Map rendering
MapTiler	enable vector tiles
WebAssembly container	high-performance calculations for map "great-circle" routes use `Zig` code compiled to `WASM`

Common pitfall of combining `LiveView` with CSR components

The client-side rendered components are - when online - mounted via hooks under the tag phx-update="ignore".

These components have they own lifecycle. They can leak or stack duplicate components if you don't cleanup them properly.
The same applies to "subscriptions/observers" primitives from (any) the state manager. You must unsubscribe, otherwise you might get multiples calls and weird behaviours.

⭐️ LiveView hooks comes with a handy lifecyle and the destroyed callback is essential.

SolidJS makes this easy as it can return a cleanupSolid callback (where you take a reference to the SolidJS component in the hook).
You also need to clean subscriptions (when using a store manager).

The same applies when you navigate offline; you have to run cleanup functions, both on the components and on the subsriptions/observers from the state manager.

Service worker lifecycle

When the client code is updated, we get notified:

Once we accept the update, the new code is active:

Build tool: `Vite`

Since we need to setup a Service Worker, we used Vite and the plugin VitePWA.

We let Vite bundle all the code and can remove safely esbuild and tailwindcss. They are now imported by Vite.

No esbuild nor tailwind in the mix.exs, no mix phx.digest as files will be fingerprinted by Vite in "prod" mode.

You need to distinguish between fingerprinted and not static assets. For example, icons, robots.txt, sitemap.xml... are not fingerprinted. Since the "/priv/static" folder is pruned, we copy the non-fingerprinted files into this folder through a function defined in the vite.config.js module. These file are served by Phoenix.
We let Vite manage the others. In particular, the paths are proxied by http://localhost:5173.
In production mode, we use a plugin for this and Vite will build and hash the files.
Therefor, we use an Elixir module to proxy or discover the path.

DEV HMR mode

You will be running the vite dev server on port 5173 and the Phoenix webserver on port 4000.

You prepend the static assets with http://localhost:5173 in DEV mode, or find the hashed name in PROD mode.

# config.exs

config :liveview_app, :env, config_env()


# dev.exs

config :liveview_pwa, LiveviewPwaWeb.Endpoint,
  [...]
  code_reloader: true,
  debug_errors: true,
  secret_key_base: "aaaaaaaa.........................aaaa",
  live_reload: [
    web_console_logger: true,
    patterns: [
~r"lib/liveview_pwa_web/(controllers|live|components|router|channels)/.*\.(ex|heex)$",
~r"lib/liveview_pwa/.*\ex
    ]
  ],
  watchers: [
    pnpm: [
      "vite",
      "serve",
      "--mode",
      "development",
      "--config",
      "vite.config.js",
      cd: Path.expand("../assets", __DIR__)
    ]
  ]

The Root layout:

Pass the assign env (or the name you want) in the first live component (or controller):

|> assign(:env, Application.fetch_env!(:liveview_pwa, :env))

Add the following to "root.html.heex":

<link
  :if={@env === :prod}
  rel="stylesheet"
  href={Vite.path("css/app.css")}
/>

<script 
  :if={@env === :dev}
  type="module"
  src="http://localhost:5173/@vite/client"
>
</script>


<script
  defer
  type="module"
  src={Vite.path("js/main.js")}
>
</script>

DEV server configuration:

// vite.config.js

const devServer = {
  cors: { origin: "http://localhost:4000" },
  allowedHosts: ["localhost"],
  strictPort: true,
  origin: "http://localhost:5173", // Vite dev server origin
  port: 5173, // Vite dev server port
  host: "localhost", // Vite dev server host
};

export default defineConfig = ({command, mode}) => {
  if (command == 'serve') {
    process.stdin.on('close', () => process.exit(0));
    copyStaticAssetsDev();
    process.stdin.resume();
  }

  return {
     server: mode === 'development' && devServer
     [...]
  }
})

You will benefit from using the resolve config by using alias.
For example:

import img from "@assets/images/img.web";

// or dynamic at runtime inside a function

const img = new URL("@assets/images/img.web`", import.meta.url).href;

PROD mode

For the PROD mode, the files are versioned by Vite because we want to be able to update the app when the client code changes.

Therefor, we removed the step phx.digest from the Dockerfile.

Vite produces a dictionary between the original asset name and the fingerprinted one. You need to use an Elixir module helper and use it in "root.html.ex".

Source code of "vite.config.js"

We also need to build a Manifest file. PWABuilder is a good source.

We also need to copy the static files.

Highlight of "vite.config.js": the VitePWA plugin.

PWAConfig = {
  // Don't inject <script> to register SW (handled manually)
  // and there no client generated "index.html" by Phoenix
  injectRegister: false, // no client generated "index.html" by Phoenix

  // Let Workbox auto-generate the service worker from config
  strategies: "generateSW",

  // App manually prompts user to update SW when available
  registerType: "prompt",

  // SW lifecycle ---
  // Claim control over all uncontrolled pages as soon as the SW is activated
  clientsClaim: true,

  // Let app decide when to update; user must confirm or app logic must apply update
  skipWaiting: false,

  workbox: {
    // Disable to avoid interference with Phoenix LiveView WebSocket negotiation
    navigationPreload: false

    // ❗️ no fallback to "index.html" as it does not exist 
    navigateFallback: null

    // ‼️ tell Workbox not to split te SW as the other is fingerprinted, thus unknown to Phoenix.
    inlineWorkboxRuntime: true,

    // preload all the built static assets
    globPatterns: ["assets/**/*.*"],
  }
}

Store managers

The app uses a context store createStore powered by SolidJS.

For the PhxSync and YjsCh pages, we used Yjs client-side and Postgres and SQLite respectively.

For the FlightMap page, we use Valtio as we didn't design the state of this page to survive a network disruption.

Mandelbrot set with Numerical Elixir and Zigler

NDREAN — Fri, 08 Nov 2024 16:12:55 +0000

We build a Mandelbrot set explorer with a Livebook. It is an Elixir code runner that lets you write Elixir in the browser.

Algorithm

An explanation can be found here

Given an image with dimensions (eg w x h of 1500 x 1000), max iteration (eg 100), we want to assign to each pixel a colour which represents the "stability" of the orbit of the underlying point.

Iterate over each pixel (i,j):

map it into the 2D plane: compute its "complex coordinates"
compute the iteration number
compute a colour
Sum-up and draw.

Explorer

We have to parts:

one Livebook to visualise the orbits of a point, the iterates
one Livebook to visualise the Mandelbrot set and zoom into any point.

This is done with the Numerical Elixir library Nx. A solution is also proposed using embedded Zig within the Livebook.

The Zig code can also be compiled to WebAssembly. You can view the WeAssembly version with the link below:

https://ndrean.github.io/zig-assembly-test/

The Kino.JS.Live library makes it possible to interact with the browser.

Exploring orbits

In this Livebook, you click on a point c of the 2D-plane and it computes the succession of points, the orbit, under the iterative map z(n+1) = z(n) * z(n) + c and z(0)=c.

Code source: https://github.com/ndrean/mandelbrot/blob/main/livebook/orbits.livemd

Explore the Mandelbrot set

In this Livebook, you can click into the 2D-plane to zoom into the area of interest.

We propose two solutions:

one using only Elixir for the computations,
one using Zig for the heavy computations (you need to have Zig installed).

Although Zig gives you a huge boost, the performance of the Elixir code is good. This is because we use the numerical library Nx with the EXLA backend.

Code source: https://github.com/ndrean/mandelbrot/blob/main/livebook/mandelbrot.livemd

Notes on streaming downloads with progress in Elixir and Req

NDREAN — Thu, 02 May 2024 20:13:50 +0000

Context

We want the server to download data, ie trigger an HTTP get request to some endpoint. The response should be a stream, and we want to append the chunks into a file. We also want to display the download progress.

We will use the Req library, a superset of Finch, which is itself a superset of Mint.

A few reasons for Req:

you can stream the body response,
it takes care of reconnection when the connection fails or timeouts (not tested here).
it takes care of redirections.

[Front-end note]: an HTTP request made from the browser uses the fetch API which can consume a fetch as a stream. To get a download progress, you can use fetch and set a ReadableStream on the response body (res.body.getReader()). Check this or that. For an upload progress, you need to use XMLHttpRequest. This is the reason why Liveview used it in the uploader snippet.

Code

The code below is "livebookable" 😁 or can be run in an iex session.

========================================================

HTTP streaming with progress using Finch and Req

Mix.install([
  {:finch, "~> 0.18.0"},
  {:req, "~> 0.4.14"}
])

Finch.start_link(name: ExStream.Finch)

Test endpoints

We use two endpoints for testing. The last one includes a redirection.

vid = "https://sample-videos.com/video321/mp4/720/big_buck_bunny_720p_1mb.mp4"
img = "https://source.unsplash.com/QT-l619id6w"

Stream & progress with `Req`

We start with the Req library.

We start with a download with streams into a file. The code is pretty compact: we pass a callback in the :into option. This callback returns a File.stream to collect the chunks.

defmodule ReqWriteStream do
  def download(url, file_path) do
    Req.get!(url, raw: true, into: File.stream!(file_path, [:write]))
  end
end

We test it:

ReqWriteStream.download(img, "image2.jpg")

We now write a module that displays the progress. We grab the "content-length" with a HEAD request. The body is again streamed via a callback declared with the :into option. This callback writes the chunks into a file, and stores the progress state into the :private key of the struct %Req.Response{}.

defmodule ReqProgressStream do
  def download(url, file_path) do
    [size]  = Map.get(Req.head!(url: url).headers, "content-length", ["0"])
    size = String.to_integer(size)
    file_pid = File.open!(file_path, [:write, :binary])

    func = fn {:data, data}, {req, res} ->
      IO.binwrite(file_pid, data)
      chunk_size = byte_size(data)
      res = Req.Response.update_private(res, :progress, chunk_size, &(&1 + chunk_size))

      if size>0, do:
        {Req.Response.get_private(res, :progress) * 100 / size, chunk_size, size} |> dbg()

      {:cont, {req, res}}
    end

    Req.get!(url: url, raw: true, into: func)

    File.close(file_pid)
  end
end

We test concurrent HTTP calls with Task.async_stream because we use the same function with different arguments.

Task.async_stream(
  [[vid, "video.mp4"], [img, "image.jpg"]],
  &apply(ReqProgressStream, :download, &1),
  timeout: :infinity
)
|> Stream.run()

Stream & progress with `Finch`

We continue with the Finch library. The code below is adapted from the example coming with stream_while.

We expose a function that takeq an URL and a path. As such, it may be fragile with regards to connection errors.

defmodule FinchStream do
  def download(url, file_path) do
    IO.puts("Starting to process #{inspect(file_path)}...........")

    # Open a file to which binary chunks will be appended to.
    # this process is reset in case of redirection
    file_pid = File.open!(file_path, [:write, :binary])

    unless is_pid(file_pid), do: raise("File creation problem on disk")

    # the HTTP stream request
    Finch.build(:get, url)
    |> Finch.stream_while(ExStream.Finch, nil, fn
      # we put the status in the "acc" to handle redirections
      {:status, status}, _acc ->
        {:cont, status}

      # - when we receive 302, we put the "location" header in the "acc"
      # - when we receive a 200, we put the "content-length" and the file name in the "acc",
      {:headers, headers}, acc ->
        handle_headers(headers, acc)

      # when we receive the "location" tuple, we recurse
      # otherwise, we write the chunk into the file and print out the current progress.
      {:data, data}, acc ->
        handle_data(data, acc, file_path, file_pid)
    end)

    case File.close(file_pid) do
      :ok ->
        {:halt, {file_path, :done}}

      {:error, _reason} ->
        {:halt, :error}
    end
  end

  def handle_headers(headers, status) when status in [301, 302, 303, 307, 308] do
    IO.puts("REDIR: #{status}")

    {:cont, Enum.find(headers, &(elem(&1, 0) == "location"))}
  end

  def handle_headers(headers, 200) do
    {"content-length", size} =
      Enum.find(headers, &(elem(&1, 0) == "content-length"))

    case size do
      nil ->
        {:cont, {0, 0}}

      size  ->
        {:cont, {0, String.to_integer(size)}}
    end
  end

  def handle_headers(_, status) do  
    dbg(status)
    {:halt, :bad_status}
  end

  def handle_data(_data, {"location", location}, file_path, file_pid) do
    if Process.alive?(file_pid), do:
       :ok = File.close(file_pid)

    # recursion
    download(location, file_path)
  end

  def handle_data(data, {processed, size}, file_path, file_pid) do
    case IO.binwrite(file_pid, data) do
      :ok ->
        processed =
          if is_integer(size) and size > 0 do
            (processed + byte_size(data))
            |> tap(fn processed ->
              IO.inspect(Float.round(processed * 100 / size, 1),
                label: "Processed #{inspect(file_path)} %: "
              )
            end)
          else
            processed + byte_size(data)
          end

        {:cont, {processed, size}}

      {:error, reason} ->
        {:error, reason}
    end
  end
end

We test this:

Task.async_stream(
  [[vid, "video.mp4"], [img, "image.jpg"]],
  &apply(FinchStream, :download, &1),
  timeout: :infinity
)
|> Stream.run()

Notes on semantic search with Elixir and Bumblebee

NDREAN — Wed, 13 Dec 2023 05:26:13 +0000

Suppose you have a bunch of images and you want to find a specific image on a certain thema.
One way to solve this problem is to describe each image with a caption and perform a full-text search: find captions based on lexical match.

With Machine Learning and models, you can greatly improve this with semantic search: you look for images whose captions are close in terms of meaning to your search.

It remains to express what "close meaning" stands for and how to do this. Theses notes describes how this can be done with the Elixir language and Nx, Axon (Nx-powered Neural Network library) and Bumblebee which provides pre-trained Neural Network models.

An overview of the process:

Firstly, you upload images into a bucket. You analyse the image with a model to produce a description of it. This is a "captioning" process or Image-To-Text. You save the URL and caption ( a short descriptive text) into a DB.

Then you record an audio and run a Speech-To-Text process to produce a text transcription.

Now that we have a target text, we want to find the captions that approximates this text.
This is where embeddings come into play. We transcript a text into a well-thought vector space. We then use an approximation algorithm to find the closest neighbours.

For this, you can use HNSWLib. You build incrementally an Index struct from your captions, and then run a knn_search on this index with the audio transcription as an input.

Description of the models used and semantic search

Elixir has an already rich and growing ecosystem. We will use pre-trained models powered by Bumblebee.

Choosing the models to use is a difficult task. We simply followed the following posts:

The corresponding models used per task are:

Image-To-Text
We used the model "Salesforce/blip-image-captioning-base" with Bumblebee.Vision.image_to_text. This produces a text caption that describes the image.
Speech-To-Text
We used the model "openai/whisper-small" and Bumblebee.Audio.speech_to_text_whisper. To capture the audio, we used the MediaRecorder API. This produces a text that is a transcription of the audio.
compute embeddings
We will run a symmetric semantic search since we expect the captions and the audio trasncriptions to have the same amount of content. This leads to which pre-trained model to choose to compute vector encodings from strings
To transform a text into a vector (a so-called embedding), we used the transformer "sentence-transformers/paraphrase-MiniLM-L6-v2" and Bumblebee.Text.TextEmbedding.text_embedding. You compute the embeddings for each image caption.
Semantic search
We used the Elixir binding for HNSWLib.
All the captions embeddings are used to build the HNSWLib.Index.
You also compute the embedding of the audio transcription and use it as the input of HNSWLib.Index.knn_query to find the closest neighbour(s) of the audio transcription embedding among the set of the caption embeddings.
This process is dependant on the metric used. It returns the position(s) (indices) among the Index struct indices. This is where you need to save whether the index or the embedding to look-up for the corresponding image(s).

Some code

Firstly, the ML process are started as follows:

#Application.ex
def start(_type, _args) do
    children = [
      # Nx serving for image classifier
      {Nx.Serving, serving: App.Image2text.serving(), name: ImageClassifier},
      {Nx.Serving, serving: App.Whisper.serving(), name: Whisper},
      App.TextEmbedding,
      ...

We instantiate the HNSWLib index with a GenServer and also the tokenizing (which produces embeddings). The transformer used is a 384 dimensional vector space. Since this transformer is trained with a cosine metric, we embed the vector space of embeddings with the same distance.

#Embedding computation and HNSWLib.Index instantiation
defmodule App.TextEmbedding do
  use GenServer
  @indexes "indexes.bin"

  def start_link(_) do
    GenServer.start_link(__MODULE__, {}, name: __MODULE__)
  end

  # upload or create a new index file
  def init(_) do
    upload_dir = Application.app_dir(:app, ["priv", "static", "uploads"])
    File.mkdir_p!(upload_dir)
    path = Path.join([upload_dir, @indexes])
    space = :cosine

    {:ok, index} =
      case File.exists?(path) do
        false ->
          HNSWLib.Index.new(_space = space, _dim = 384, _max_elements = 200)

        true ->
          HNSWLib.Index.load_index(space, 384, path)
      end

    model_info = nil
    tokenizer = nil
    {:ok, {model_info, tokenizer, index}, {:continue, :load}}
  end

  def handle_continue(:load, {_, _, index}) do
    transformer = "sentence-transformers/paraphrase-MiniLM-L6-v2"

    {:ok, %{model: _model, params: _params} = model_info} =
      Bumblebee.load_model({:hf, transformer})

    {:ok, tokenizer} =
      Bumblebee.load_tokenizer({:hf, transformer})

    {:noreply, {model_info, tokenizer, index}}
  end

  # called in Liveview `mount`
  def serve() do
    GenServer.call(__MODULE__, :serve)
  end

  def handle_call(:serve, _, {model_info, tokenizer, index} = state) do
    serving = 
Bumblebee.Text.TextEmbedding.text_embedding(model_info, tokenizer)
    {:reply, {serving, index}, state}
  end
end

In a Liveview page, you can upload pictures and record audios from the browser. For this, use allow_upload twice: this will open a Channel (WS) to upload data to the server.

def mount(_,_,socket) do
  {serving, index} = App.TextEmbedding.serve()
  ...
  socket
  |> assign(serve_embedding: serving, index: index)
  |> allow_upload(:image,
     accept: ~w(image/*),
     auto_upload: true,
     progress: &handle_progress/3,
     max_entries: 1,
     chunk_size: 64_000,
     max_file_size: 5_000_000
   )
  |> allow_upload(:speech,
    accept: :any, 
    auto_upload: true,
    progress: &handle_progress/3,
    max_entries: 1
  )

In the rendered HTML, you can have a simple form like below for the image:

<form id="upload-form" phx-change="noop" phx-submit="noop" disabled={!@display}>
  <label class="cursor-pointer">
    <.live_file_input upload={@uploads.image} class="hidden" />
    <img src={"/uploads/#{@image.filename}"} />
  </label>
</form>

The data will be sent to the server to run an Image-To-Text captioning process, and also uploaded into a bucket.

For the audio, we use a "hook". The data will used by a Speech-To-Text process. We use a button to start and stop the recording process.

<audio id="audio" controls></audio>
<form phx-change="noop" class="hidden">
  <.live_file_input upload={@uploads.speech} class="hidden" />
</form>

<p>Please record a phrase. The search for matching images will run automatically</p>
<p>
<button
  id="record"
  class="bg-blue-500 hover:bg-blue-700 text-white font-bold px-4 rounded"
  type="button"
  phx-hook="Audio"
  disabled={@micro_off}
>
  <Heroicons.microphone outline class="w-6 h-6 text-white font-bold group-active:animate-pulse"/>
    <span>Record</span>
</button>

The hook to capture the audio uses the MediaRecorder API and sends to the server with the LiveviewJS primitive this.upload. For example:

#micro.js
export default {
  mounted() {
    let mediaRecorder;
    let audioChunks = [];
    const recordButton = document.getElementById("record");
    const audioElement = document.getElementById("audio");
    const blue = ["bg-blue-500", "hover:bg-blue-700"];
    const pulseGreen = ["bg-green-500", "hover:bg-green-700", "animate-pulse"];

    _this = this;

    recordButton.addEventListener("click", () => {
      if (mediaRecorder && mediaRecorder.state === "recording") {
        mediaRecorder.stop();
        recordButton.textContent = "Record";
      } else {
        navigator.mediaDevices.getUserMedia({ audio: true }).then((stream) => {
          mediaRecorder = new MediaRecorder(stream);
          mediaRecorder.start();
          recordButton.classList.remove(blue);
          recordButton.classList.add(plusGreen);

          recordButton.textContent = "Stop";

          mediaRecorder.addEventListener("dataavailable", (event) => {
            audioChunks.push(event.data);
          });

          mediaRecorder.addEventListener("stop", () => {
            const audioBlob = new Blob(audioChunks);
            console.log(audioBlob);
            audioElement.src = URL.createObjectURL(audioBlob);

            _this.upload("speech", [audioBlob]);
            audioChunks = [];
            recordButton.classList.remove(pluseGreen);
            recordButton.classList.add(blue);
          });
        });
      }
    });
  },
};

//app.js
...
import Audio from "./micro.js";
...
let liveSocket = new LiveSocket("/live", Socket, {
  params: { _csrf_token: csrfToken },
  hooks: { Audio },
});

The picture upload and the audio upload to the server are processed by a handle_progress callback.

In these handle_progresscallbacks (one for :image and one for :speech):

when you upload pictures, you run a Task in parallel to produce a caption from the image. We transform the temporary file that contains the binary of the image into a tensor with the library Vix.Vips (because we also want to transform any received image into a WEBP format to save space).

def handle_progress(:speech, entry, socket) when entry.done? do
    socket
    |> consume_uploaded_entry(entry, fn %{path: path} ->
      :ok = File.cp!(path, @tmp_wav)
      {:ok, @tmp_wav}
    end)

    audio_task =
      Task.Supervisor.async(
        App.TaskSupervisor,
        fn ->
          Nx.Serving.batched_run(Whisper, {:file, @tmp_wav})
        end
      )

    {:noreply, assign(socket, audio_ref: audio_task.ref, micro_off: true, speech_spin: true)}
end

when you receive an audio file, you run a transcription Task.

def handle_progress(:image, entry, socket) when entry.done? do

    %{
      tensor_img: tensor_img,
      upload_task: upload_task,
      filename: filename
    } =
      socket
      |> consume_uploaded_entry(entry, fn %{path: path} ->
        %{client_type: type} = entry

        file_binary =
          File.read!(path)
        ...
        upload_task =
          App.Upload.run(local_path, filename, type)

        {:ok, image} = Vimage.new_from_file(path)

        {:ok, %Vix.Tensor{data: data, shape: shape, names: names, type: tp}} =
          Vimage.write_to_tensor(image)

        tensor_img =
          Nx.from_binary(data, tp)
          |> Nx.reshape(shape, names: names)

        {:ok,
         %{
           tensor_img: tensor_img,
           upload_task: upload_task,
           filename: filename,
           type: type,
           local_path: local_path
         }}
      end)

    i2t_task =
      Task.Supervisor.async(App.TaskSupervisor, fn ->
        Nx.Serving.batched_run(ImageClassifier, tensor_img)
      end)

    {:noreply,
     socket
     |> assign(
       running: true,
       i2t_ref: i2t_task.ref,
       upload_ref: upload_task.ref,
       db_img: nil,
       display: false
     )
     |> update(:image, fn img ->
       Map.merge(img, %{
         filename: filename,
         type: type,
         tensor: tensor_img,
         local_path: local_path
       })
     end)}
  end

def handle_progress(_name, _entry, socket), do: {:noreply, socket}

These tasks will emit a response that is captured in a handle_info callback.

when you receive a response from the upload task, you save the URL into the database.

def handle_info({ref, {:ok, msg}}, %{assigns: assigns} = socket)
      when assigns.upload_ref == ref do
    Process.demonitor(ref, [:flush])
    %{body: %{location: location}} = msg
    image = Map.merge(assigns.image, %{location: location})
    %{headers: {h, w}, db_img: db_img} = assigns
    # save to db
    db_img = handle_image({h, w}, image, db_img)
    # clean the file from the server since we provide a remote path
    File.rm!(Path.join([@upload_dir, assigns.image.filename]))

    {:noreply,
     socket
     |> assign(db_img: db_img, upload_ref: nil)
     |> update(:display, &(!&1))
     |> update(:image, fn img -> Map.merge(img, %{location: location, tensor_img: nil}) end)}
end

when you receive the response from the Image-To-Text task, you compute an embedding and add it to the Index struct of HNSWLib.Index. In this case, we add an embedding once at at time into the Index struct, so we get a unique index from the Index struct so we save it to the database. We can later use this field in the database to find an image with a look up at the index.

  def handle_info({ref, %{results: [%{text: caption}]}}, %{assigns: assigns} = socket)
      when assigns.i2t_ref == ref do
    Process.demonitor(ref, [:flush])

    %{index: index, serve_embedding: serving, db_img: db_img} = assigns
    saved_index = Path.expand("priv/indexes.bin")

    with %{embedding: data} <- Nx.Serving.run(serving, caption),
         # compute an normed embedding (cosine case only) on the text result
         normed_data <- Nx.divide(data, Nx.LinAlg.norm(data)),
         :ok <- HNSWLib.Index.add_items(index, normed_data),
         {:ok, idx} <- HNSWLib.Index.get_current_count(index),
         :ok <- HNSWLib.Index.save_index(index, saved_index) do
      db_img =
        case db_img do
          nil ->
            App.Image.insert!(%App.Image{}, %{idx: idx, caption: caption})

          img ->
            App.Image.update!(img, %{idx: idx, caption: caption})
        end

      {:noreply,
       socket
       |> assign(running: false, db_img: db_img, index: index, i2t_ref: nil)
       |> update(:image, fn img -> Map.merge(img, %{caption: caption, tensor_img: nil}) end)}
  end
end

when you receive a response from the Speech-To-Text process, you also compute an embedding, but then run a HNSWLib.Index.knn_query.

def handle_info({ref, %{chunks: [%{text: text}]} = _result}, %{assigns: assigns} = socket)
      when assigns.audio_ref == ref do
    Process.demonitor(ref, [:flush])
    File.rm!(@tmp_wav)

    %{serve_embedding: serving, index: index} = assigns
    # compute an normed embedding (cosine case only) on the text result
    # and returns an App.Image{} as the result of a "knn_search"
    with %{embedding: data} <- Nx.Serving.run(serving, text),
         normed_data <- Nx.divide(data, Nx.LinAlg.norm(data)),
         %App.Image{} = result <- handle_knn(normed_data, index) do
      {:noreply,
       assign(socket,
         transcription: String.trim(text),
         micro_off: false,
         speech_spin: false,
         search_result: result,
         audio_ref: nil
       )}
    else
      # record without entries
      {:error, "no entries in index"} ->
        {:noreply,
         assign(socket,
           micro_off: false,
           search_result: nil,
           speech_spin: false,
           audio_ref: nil
         )}

      nil ->
        {:noreply,
         assign(socket,
           transcription: String.trim(text),
           micro_off: false,
           search_result: nil,
           speech_spin: false,
           audio_ref: nil
         )}
  end
end

In the knn_search below, we first check if the index is populated. The function HNSWLib.Index.knn_query(index, input, k) returns a tuple {:ok, indices, distances} where the two last elements are lists. The length of these lists is the value of the last paramter k. From this indice, we look up in the database for the image with the corresponding index.

def handle_knn(_, nil), do: {:error, "no index found"}

def handle_knn(data, index) do
  case HNSWLib.Index.get_current_count(index) do
    {:ok, 0} ->
      {:error, "no entries in index"}

    {:ok, _c} ->
      case HNSWLib.Index.knn_query(index, data, k: 1) do
        {:ok, label, _distance} ->
          label[0]
          |> Nx.to_flat_list()
          |> hd()
          |> then(&App.Repo.get_by(App.Image, %{idx: &1 + 1}))
      end
  end
end

Note that we look for the closest neighbour (with k=1). This will always give a response, the closest neighbour", but it might not be a "close" neighbour. We may need some cut-off distance to exclude unwanted responses.

New CSS-in-JS lightweight packages for React and Preact

NDREAN — Tue, 04 Jul 2023 16:50:55 +0000

It can be advantageous to write some components with some local CSS rules. This package leverages the native CSS nesting and produces a fast CSS-in-JS/Component solution: there is no parsing, just directly adding a style element to the <head> tag of the DOM.

We have produced a less than 700B package to write CSS-in-JS/Component for:

React: https://github.com/ndrean/bau-react-css
Preact: https://github.com/ndrean/bau-preat-css

These packages uses document so is not suited for SSR.

It exports the primitives: css, styled, createGlobalStyles and keyframes.

The api is very close to standard libraries.

Class

You write a template CSS string and pass it to the css function to produce a class:

const bluediv = css`
  background-color: bisque;
  color: midnightblue;
`;

You can use it:

<p className={bluediv}>A nice paragraph</p>

Styled component

You can use styled components in the form below. An example with an animation via keyframes:

const rescale = keyframes`
  0% {transform: scale(0.5)}
  100% {transform: scale(1)}
`;

const AnimSurf = (props) => styled("span", props)`
  font-size: ${props.size}em;
  animation: ${rescale} 2s ease infinite;
`;

You can use it:

<AnimSurf size={3}>🏄</AnimSurf>;

Conditional classes

You have two ways to use it. Define a function or object that returns CSS strings:

const styles = (props) => {
  base: `
    cursor: pointer;
    font-size: ${props.size ?? 1}em;
    border-radius: 0.3em;
    padding: 0.3em;
  `,
  danger: `
    color: red;
    animation: ${rescale} 1s ease infinite;
  `,
  disabled: `
    pointer-events: none;
    opacity: ${props.opacity};
  `;
}

You can write:

const Btn = (props)=> styled('button', props)`
  ${styles(props).base +
  props.danger ? styles(props).danger : ""}
`

<Btn>Base button</Btn>
<Btn danger={true} onClick={()=> alert('danger')}>Danger button</Btn>

To make life easier, the primitive styled can read the props and sets the class.

const Button = (props) => styled("button", props)`
  ${styles(props).base}
  ${styles(props)}
`;

We can use it:

<Button size={3}>Base button</Button>

<Button
  disabled
  className={css`
    box-shadow: 6px -6px teal;
  `}
  >
  Disabled with shadow
</Button>

You can use reactive styles with the style prop, extend classes for styled components, define global styles. Check the Readme and the example.

A new CSS-in-JS package for SolidJS

NDREAN — Mon, 03 Jul 2023 18:42:11 +0000

We have produced a less than 700B package to write CSS-in-JS for SolidJS: https://github.com/ndrean/solid-css/.

This package uses document so is not suited for SSR.

It exports the function css, styled, createGlobalStyles and keyframes.

You write a template CSS string and pass it to the css function to produce a class:

const bluediv = css`
  background-color: bisque;
  color: midnightblue;
`;

and use it:

<div class={bluediv}>I am blue</div>;

You can design animated components. An example with a styled component:

const rescale = keyframes`
  0% {transform: scale(0.5)}
  100% {transform: scale(1)}
`;

const AnimSurf = (props) => styled("span", props)`
  font-size: ${props.size}em;
  animation: ${rescale} 2s ease infinite;
`;

and use it:

<AnimSurf size={3}>🏄</AnimSurf>;

Conditional classes

You have two ways to use it. Define a function or object that returns CSS strings:

const styles = (props) => {
  base: `
    cursor: pointer;
    font-size: ${props.size ?? 1}em;
    border-radius: 0.3em;
    padding: 0.3em;
  `,
  danger: `
    color: red;
    animation: ${rescale} 1s ease infinite;
  `,
  disabled: `
    pointer-events: none;
    opacity: ${props.opacity};
  `;
}

You can write:

const Btn = (props)=> styled('button', props)`
  ${styles(props).base +
  props.danger ? styles(props).danger : ""}
`

<Btn>Base button</Btn>
<Btn danger={true} onClick={()=> alert('danger')}>Danger button</Btn>

To make life easier, the primitive styled can read the props and sets the class when the styles are defined as above:

const Button = (props) => styled("button", props)`
  ${styles(props).base}
  ${styles(props)}
`;

We can use it:

<Button size={3}>Base button</Button>

<Button
  disabled
  className={css`
    box-shadow: 6px -6px teal;
  `}
  >
  Disabled with shadow
</Button>

You can use reactive styles with the style prop, overwrite classes for styled components, create conditional CSS depending on the props, and define global styles. Check the Readme and the example.

Yet another micro package for reactive UI based on event-driven programming

NDREAN — Thu, 29 Jun 2023 18:31:18 +0000

The ideas of this front-end micro package are:

use event-driven programming: you write to the state and this triggers events with callbacks.
write HTML as strings: no templating, just normal Javascript interpolation,
use datasets to references listeners on elements,
declare state variables. They have properties such as:
- .val which is a setter and a getter,
- .target which sets the DOM element which will receive the event triggered by a state change,
- .resp which sets desired rendering.
diff on the data, not the DOM.

It gives a simple reactive pattern based on emitting events and limits the rendering of arrays with a diffing function on the data.

Take the example of a button to increment a counter. You have declared a state object "counter". The state is managed by the package.

<button data-click="incr" type="button">Incr counter</button>

You have set up an "onclick" listener on this element via the dataset data-click. When clicked, it will run the function you declared:

incr: ()=> counter += 1

This mutates the state. Another DOM element, say a <p id="count"> should display the updated value. You target this element for this state variable, and set counter.target= #count. The state is managed by the package and will emit a change Event that targets this "#count" element.

If you set on this element an "onchange" listener - with a dataset data-change="displayCounter" - like:

<p id="count" data-change="displayCounter"></p>

then the "onchange" listener of this element will react to the event and run the "displayCounter" function, which should generally return an HTML string element.

To finally render, we assign the output to the key counter.resp of the state. This will mutate again the "counter" state:

displayCounter: ()=> counter.resp = `<span>${counter}</span>`

The package will take this state mutation into account and continue to a rendering function that eventually does something like:

count.innerHTML = counter.resp

when the state object is an array, calculate the diff on the data, between the new data and the old, and render only the differences (limited to the whole row): a "delete", a "new", an "append", an "update".

The micro package named "biny" implements these ideas. It is very close to Vanilla JS but modest in the sense that it is short (1.4kB, 250LOC) and cannot deal with computed or derived state. This can be easily managed by another state variable. It performs almost like vanilla JS when you run the JS-framework bench test. The code to write the famous todoMVC is also quite compact.

Source: https://github.com/ndrean/binyJS

A simple counter looks like:

import B from "binyjs";

const counter = B.state({ val: 0 }),
  actions = B.Actions({
    inc: () => (counter.val += 1),
    display: () => {
      counter.target = count;
      counter.resp = `<span>binyJS state: ${counter.val}</span>`;
    },
  });

window.onload = () => actions.display();

app.innerHTML = `
  <div>
    <h1>Hello biny</h1>
    <div>
      <button type="button" data-click="inc">
      <span data-change="display" id="count"></span>
      </button>
    </div> 
  </div>
`;

Revisit Context with a useful pattern, illustrated with SolidJS and async data

NDREAN — Thu, 08 Jun 2023 11:34:15 +0000

A component is a function takes some props and renders a view. In these props, you may have some state elements. When you do multi-page SPAs, you rebuild components on every new page. You may want to use a global store to share state between them. A useful pattern for this is is the Context. You can also do this by parametrizing the function component:

(context) => function Component(props) {HTML like stuff}

Below is a simple example of a navigation between pages that renders data using the context pattern and SolidJS:

You can pass in all the stuff you may need along the road such as themes, useful functions...

With some frameworks, you can declare state outside of components. With SolidJS, you use createSignal to declare a state element; it can be encapsulated in a component, or declared outside of it. You can take advantage of this "context": you can declare it at the context level if you need to share it among components.

// context.js

import {createSignal} from from "solid-js";

const [bool, setBool] = createSignal(false);

const context = {
  bool, setBool,
  theme: {...}, 
  ...
}

In the example below, we get the state from the context, with the getter and setter.

const comp1 = (ctx) => {
  const {bool, setBool, theme} = ctx;

  return const Comp1 = (props) =>
     (
        <p>The state value is: {bool() ? "true" : "false"}</p>
        <button onClick={()=> setBool(v => !v}>Toggle</button>
        {props?.children}
    )
}

[...]

import context from "./context";
const Comp1 = comp1(context)
<Comp1>Hi</Comp1>

When you navigate to another component, you import the state and have the reactive value every time you navigate back to this component.

const comp2 = (ctx) => {
  const {bool} = ctx;

  return () =>
     (
      <div>
        <p>Comp1 changed the state: {bool()}</p>
     </div>
    )
}

[...]

import context from "./context";
const Comp2 = comp2(context)
<Comp2/>

We can have an async function that sets data from a component, and we want to use the result elsewhere. When you want to run an async call with SolidJS, it is specified in the SolidJS documentation that you should use createResource. Then, if you need ot update the state, you should avoid createEffect as this is used to mutate the DOM but rather attach a then(..) to the async call. Let's do that.

We mock an async call and update the state with the result. Furthermore, since we may want to write this function out of the scope of the component, we parametrize it with the context so the state setter will be available:

// asyncFunc.js

const asyncFunc = ({setData}) => 
  (x) =>
    new Promise(resolve => 
      setTimeout(() => resolve(x), 1000)
    )
    .then(setData)

We first add a signal in the context to share the data within components:

// context.js

import {createSignal} from from "solid-js";

const [data, setData] = createSignal("");

const context = {
  data, setData,
  ...
}

The async calls are handled with createRessource in SolidJS: it can take an argument and returns a getter function (written in the form (argOfFun, Fun)).

import { createEffect, createResource } from "solid-js";
import context from "./context";
import asyncFunc from "./asyncFunc";


const comp3 = (ctx) => {
  // import the state
  const {data, setData, theme} = ctx;
  const AsyncFun = asyncFunction(ctx)

  return function Comp1(props) {
    // run the async function; it will update the state
    const [result] = createResource(100, AsyncFun);

    return(
      <p> Async render: {result()}</p>
    )
  }
}

You can use it elsewhere by importing the state data as above and when you navigate, the component will be up-to-date.

Note that the function component will only run once, so the async function will be run only once.

What if we want to change it? Lets give an example. We use a slider to change the parameter of the async function.

We use a local state to track the value just for the display; this means it will be reset every time the component is rendered. You can choose a global state if you want persistence.

We need to implement 2 async calls: the first one will run when the component is rendered, and the second will react on changes during the life time of the component. SolidJS makes this easy as we are assured the function component is run only once.

import ...

const comp4 = (ctx) => {
  // import the state from context
  const {data, setData} = ctx;
  const AsyncFun = asyncFunction(ctx)

  return ()=> {
    // local state to display the slider values, reset on every mount
    const [slide, setSlide] = createSignal(10);

    // initial async call on navigation
    !data() && createResource(slide(), AsyncFun);

    return (
      <div>
        <input
          type="range"
          min="10"
          max="100"
          value="10"
          onchange={({target: {value}) => {
            setSlide(value);
            // dynamic async call
            createResource(value, AsyncFun)
          }}
        />
        <p>{slide()}</p>
        <p> Dynamic async update: {data()}.</p>
    </div>
  )

Any other component can access to the reactive state data.

TIL: use Maps with React radio inputs

NDREAN — Mon, 13 Mar 2023 17:38:15 +0000

An interesting usage of the object Map with React is when you use input elements of type radio. When you use a Map in your local state, it is enough to set the key as the group name, and its value can be the input value or an object (e.g. the value and the id). The Map acts as a set, so you are assured to have the image of the DOM in our Map, no further ado.

Let's take an example.

We build an input element of type "radio": the "name" attribute will group together theses inputs.

function Input({ name, value, id, handleChange }) {
  return (
    <>
      <input
        type="radio"
        name={name}
        value={value}
        id={id}
        onChange={handleChange}
      />
      <label htmlFor={id}>{id}</label>
    </>
  );
}

We setup two groups - named "gp1" and "gp2" of two radio inputs. We display the group name, the key and the value. On change, we update the state with a new Map - based on the old one - where the key is the group name, and the value is an object {value, id} here. We now have the image of the DOM in our Map.

export default function App() {
  const [radio, setRadio] = useState(new Map());

  function handleChange(e) {
    return setRadio((radio) =>
      new Map(radio).set(e.target.name, {
        val: e.target.value,
        id: e.target.id
      })
    );
  }
return (
    <div className="App">
      <Input name="gp1" value={1} id="one" handleChange={handleChange} />
      <Input name="gp1" value={2} id="two" handleChange={handleChange} /> |{" "}
      <Input name="gp2" value={3} id="three" handleChange={handleChange} />
      <Input name="gp2" value={4} id="four" handleChange={handleChange} />
      <hr />

      {[...radio.keys()].map((k, i) => (
        <p key={i}>
          group: {k}, {radio.get(k).id}: {radio.get(k).val}
        </p>
      ))}
    </div>
  );
}

TIL: query the NPM paginated Rest API with Elixir

NDREAN — Mon, 13 Mar 2023 01:22:12 +0000

These are notes on how to use Elixir to find all the NPM packages published by AWS with their download count.

The total count is 1676. We start and query the npms-api search endpoint: we give a string and it returns a list of matching Javascript packages. The endpoint is paginated. With this list, we query the NPM registry to get statistics for each package.

We send a total of approx 1750 requests. The longer one is the paginated API with a response time of approx 200-250ms per query. Then we send 1700 queries to get the stats on each package with an average response of 10-15ms. We get the result on average in around 20s (compared to a sequential 1700/25*0.200=13s + 1700*.0.010=17s = 30s).

We used Elixir to achieve this with the HTTP client Finch to stream and the Rust based JSON parser jsonrs.

We used an easy path: build the full list of found packages, and then query the details for each element of the list.

We used Stream.resource to build the list of packages. We used the total count returned by the endpoint to handle the pagination and increment a counter on each iteration.

We then use async_stream to query the second endpoint. It returns statistics on a given package. We retrieve the downloaded count during a given period.

We then Enum.sort_by to order the list of maps on a given key.

With the result, we run a side-effect as a Task: it saves the data in a file. Since we need the data unchanged for the final step, use the function tap and the data passes by. This allows us to keep the flow and run this side -effect.

We eventually prettify the data into a list of maps %{"package_name" => downloaded_count}.

Elixir can nicely chain all these streams. In a Livebook, the following code will default to the AWS packages.

Mix.install([
  {:finch, "~> 0.16.0"},
  {:jsonrs, "~> 0.3.1"}
])
Supervisor.start_link(
  [
    {Finch, name: MyFinch},
    {Task.Supervisor, name: MyTaskSup}
  ], 
  strategy: :one_for_one,
  name: MySup
)

defmodule Npm do
  require Logger

  @registry "https://api.npmjs.org"
  @search_point "https://api.npms.io/v2/search"

  @starting "2022-01-01"
  @ending "2023-01-01"
  @search "@aws-sdk/client"
  @aws_npm_packages "aws-npm-packages.json"

  def find(save? \\ false, string \\ @search, starting \\ @starting, ending \\ @ending) do

    check_response = fn response ->
      case response do
        {:ok, result} ->
          result

        {:error, reason} ->
          {:error, reason}
      end
    end

    #  the optional "save to file"
    save_to_file = fn list ->
      Logger.info(%{length: length(list)})

      Task.Supervisor.async_nolink(MyTaskSup, fn ->
        case Jsonrs.encode(list, lean: true, pretty: true) do
          {:ok, result} -> File.write!(@aws_npm_packages, result)
          {:error, reason} -> reason
        end
      end)
    end

    # the iterating function in Stream.resource
    next = fn {data, page} ->
      {response, total} = search(string, 25 * page)

      case page * 25 >= total do
        true ->
          {:halt, data}

        false ->
          {response, {data, page + 1}}
      end
    end

    try do
      Stream.resource(
        fn -> {[], 0} end,
        &next.(&1),
        fn _ -> nil end
      )
      |> Task.async_stream(&downloaded(&1, starting, ending))
      |> Stream.map(&check_response.(&1))
      |> Enum.sort_by(&Map.get(&1, "downloads"), :desc)
      |> tap(fn data -> if save?, do: save_to_file.(data) end)
      |> Enum.map(fn %{"downloads" => d, "package" => name} ->
        Map.put(%{}, name, d)
      end)

    rescue
      e ->
        Logger.warn(e)
    end
  end

  # we send a tuple {stream, total}
  def search(string, from \\ 0) do
    url = 
      URI.new!(@search_point)
      |> URI.append_query(
        URI.encode_query(%{q: string, size: 25, from: from})
        )
      |> URI.to_string()

    with {:ok, %{body: body}} <-
           Finch.build(:get, url)
           |> Finch.request(MyFinch),
         {:ok, %{"results" => results, "total" => total}} <- Jsonrs.decode(body) do
      {
        Stream.filter(results, fn package ->
          Map.has_key?(package, "flags") === false &&
            get_in(package, ["package", "name"]) |> String.contains?(string)
        end)
        |> Stream.map(&get_in(&1, ["package", "name"])),
        total
      }
    else
      {:error, reason} ->
        reason
    end

  # the second endpoint
  def downloaded(package_name, start, ending) do

    path = @registry <> "/downloads/point/" <> "#{start}" <> ":" <> "#{ending}" <> "/" <> "#{package_name}"

    with {:ok, %{body: result}} <-
           Finch.build(:get, path) |> Finch.request(MyFinch),
         {:ok, response} <- Jsonrs.decode(result) do
      response
    else
      {:error, reason} -> reason
    end
  end

The usage is:

iex> Npm.find(false, "@aws-sdk/client", "2022-01-01", "2022-03-01")

The same result with @google-cloud/:

If you have Livebook, installed, you can run a session with the button:

If not, you can easily run a Livebook from Docker. Run the image:

docker run -p 8080:8080 -p 8081:8081 --pull always -e LIVEBOOK_PASSWORD="securesecret" livebook/livebook

and then from another terminal, execute:

open http://localhost:8080/import?url=https://github.com/ndrean/gruland/blob/main/livebook.livemd

DEV Community: NDREAN

On-the-fly zstandard (de)compression in Elixir

Streaming implementation Trade-offs and dirty schedulers

The Chunk/Frame misalignment problem

Build Elixir bindings using Zig via Zigler.

Stream a file with real-time compression

Stream HTTP download with real-time compression

Stream a file with real-time decompression

Stream HTTP download with real-time decompression

Elixir module usage

Zig code

Phoenix & Vite dev setup

Static assets

Phoenix dev.exs config

Root layout

Vite Config, server and build options

Build options

Server options

Run a separate Vite dev server in DEBUG mode

Package.json

Using workspace with pnpm

without workspace

Tailwind, daisyui and heroicons

An Elixir file path resolving module

Vite.config.js

Dockerfile

Mix install task

Offline first LiveView as PWA

About this demo app

Architecture at a glance

How it works

Optimistic Updates with Centralised Reconciliation

Rendering Strategy: SSR vs. Client-Side Hooks

Service Worker & Asset Caching

Login and token rotation

Results

About the pages

The Phoenix.Sync+Postgres+ Yjs/IndexedDB page:

The SQLite+ Phoenix.Channel + Yjs/IndexedDB page:

The FlightMap page

Tech overview

Common pitfall of combining LiveView with CSR components

Service worker lifecycle

Build tool: Vite

DEV HMR mode

PROD mode

Highlight of "vite.config.js": the VitePWA plugin.

Store managers

Mandelbrot set with Numerical Elixir and Zigler

Algorithm

Explorer

Exploring orbits

Explore the Mandelbrot set

Notes on streaming downloads with progress in Elixir and Req

Context

Code

HTTP streaming with progress using Finch and Req

Test endpoints

Stream & progress with Req

Stream & progress with Finch

Notes on semantic search with Elixir and Bumblebee

An overview of the process:

Description of the models used and semantic search

Some code

New CSS-in-JS lightweight packages for React and Preact

Class

Styled component

Conditional classes

A new CSS-in-JS package for SolidJS

Conditional classes

Yet another micro package for reactive UI based on event-driven programming

Revisit Context with a useful pattern, illustrated with SolidJS and async data

TIL: use Maps with React radio inputs

TIL: query the NPM paginated Rest API with Elixir

Build Elixir bindings using `Zig` via `Zigler`.

The `Phoenix.Sync`+`Postgres`+ `Yjs`/`IndexedDB` page:

The `SQLite`+ `Phoenix.Channel` + `Yjs`/`IndexedDB` page:

Common pitfall of combining `LiveView` with CSR components

Build tool: `Vite`

Stream & progress with `Req`

Stream & progress with `Finch`