DEV Community: Sahan

How to find LM Studio plugins and MCP servers

Sahan — Mon, 14 Sep 2026 03:34:48 +0000

Originally published on sahansera.dev.

You have a model running in LM Studio. Now you want it to search the web, work with files, or trim a video. Where do you find the tool that makes that possible?

I've built Local AI Tools to help with that step. It's a free, open-source directory where you can browse native LM Studio plugins and MCP servers, search by capability, and inspect their setup information. You don't need an account to use it.

If you already know the plugin you want, a direct link is fine. This is for the stage before that: knowing what you want to do, but not what to install.

sahansera / local-ai-tools

Open-source marketplace for discovering LM Studio plugins, MCP servers, and other local AI tools.

Local AI Tools

Discover LM Studio plugins and MCP servers — with compatibility, setup, runtime, and provenance information in one place.

Browse the marketplace →

What is Local AI Tools?

Local AI Tools is an open-source discovery marketplace focused on one question:

What can I add to LM Studio?

LM Studio has its own native plugin ecosystem, while thousands of MCP servers live across the broader Model Context Protocol ecosystem. Finding the right tool — and understanding whether it can actually be used with LM Studio — is fragmented.

Local AI Tools brings those ecosystems together into a single searchable catalogue and adds an LM Studio-specific compatibility layer on top.

At a glance

Native LM Studio plugins	Discover public plugins from the LM Studio Hub and jump straight into LM Studio.
MCP servers	Discover servers from the official MCP Registry and see whether their published metadata maps safely to an

…

View on GitHub

Finding the plugin is its own problem

In this Reddit discussion about LM Studio plugins, someone asks for a central listing. They then have to clarify that they mean native plugins, because several replies point them towards MCP servers.

Both can extend a local AI setup, but they're different integration types. A useful directory needs to make that distinction visible and help you understand the setup behind a promising description.

Local AI Tools brings together public plugin listings from LM Studio Hub and server listings from the official MCP Registry. You can browse them together or select either type. Search terms such as video, memory, or web search give you a starting point without needing a package name.

Discovery: Start with the capability you need, then check the type of integration and the setup it requires.

Try it with a video tool

Here's a concrete example using Local Video Tools, a native plugin I also maintain. It exposes video inspection, trimming, conversion, and FFmpeg job management to LM Studio.

sahansera / lmstudio-local-video-tools

Native LM Studio video tools for local inspection, trimming and conversion with FFmpeg — privacy-first and hardware accelerated.

🎬 Local Video Tools for LM Studio

Reliable video processing for your local AI.

Inspect, trim, convert, and manage long-running FFmpeg jobs directly from LM Studio — locally, with hardware acceleration, and without holding one AI tool call open for the duration of a video encode.

Try the prompts · Install · How it works · Latest release · Roadmap

v0.1.2 Early Preview is out. The core workflow has been validated with real 4K HEVC/H.265 MOV video on Apple Silicon, including lossless clipping, accurate re-encoding, VideoToolbox conversion, progress tracking, and cancellation. This release also includes filesystem and persisted-job boundary hardening ahead of the public preview.

What this is: a reliable local execution layer for concrete video operations. The model decides what operation to request; FFmpeg does the actual media processing locally.

💬 Try these prompts

Attach a video in LM Studio and talk to it naturally:

What

…

View on GitHub

Open Local AI Tools and select LM Studio plugins in the type filter.
Search for video. Look for Local Video Tools by sahansera; the author matters when similarly named tools appear.
Use the arrow to open its detail page. Check the description, runtime information, and upstream link.
Read the plugin's installation instructions. This one needs FFmpeg and ffprobe installed locally, even though it has a direct LM Studio installation link.
With those requirements understood, use the + on its card or Add to LM Studio on the details page to open the installation flow in LM Studio.

The same browsing approach works for other capabilities: narrow the type, search for the task, and inspect a candidate. The directory gets you to the tool and its setup information; the upstream instructions explain how to finish configuring it.

This example also shows why “there's an install button” and “everything is already set up” aren't the same thing.

Native plugins and MCP servers have different setup paths

Native Hub plugins use LM Studio's plugin installation links. Individual plugins can still depend on software installed on your machine, as the video example does.

MCP servers expose capabilities through the Model Context Protocol. LM Studio supports both local and remote MCP servers, with configuration supplied through mcp.json or an available Add to LM Studio link. The LM Studio MCP documentation explains those options.

For MCP listings, Local AI Tools checks whether the upstream metadata is sufficient to generate an LM Studio configuration. You'll see three outcomes:

Label	What it tells you	What to do next
LM Studio Ready	The metadata can produce a configuration without additional user-supplied values.	Review the source and runtime requirements before adding it.
Setup required	Additional values are needed, such as credentials, paths, or arguments.	Open the details and check which inputs you need to supply.
Compatibility unknown	There isn't enough information to determine a configuration.	Check the upstream documentation for manual setup.

The homepage initially filters for ready entries. Change the compatibility filter if you want to include tools that need setup or have unknown compatibility. An unknown label can reflect incomplete metadata, so it doesn't by itself establish that the server is incompatible.

Compatibility: Generating a configuration and verifying a tool's behaviour are separate jobs. The directory helps with the first.

Check what will run, and where

A local model can use an integration that calls a remote service. An API key, network access, or filesystem access can therefore matter just as much as the tool's name.

The directory surfaces runtime, API-key, source, and risk information where it's available. Some metadata is inferred, and these signals aren't a security audit. I haven't tested every listed tool with every model. Check the upstream project when deciding whether to install it.

The practical distinction is between finding a tool, configuring it, and having your chosen model use it successfully. The catalogue helps with the first two. Actual behaviour still depends on the tool, its environment, and the model.

Open source, with room for better metadata

Local AI Tools is MIT-licensed and built with Astro. Discovery and enrichment happen during the build, and the site is published as static files. The source repository is linked with this article if you'd like to inspect the implementation or contribute.

Listed tools stay with their own maintainers and retain their own licenses. This is an independent community project, not affiliated with or endorsed by LM Studio.

The most useful contributions are concrete: a missing plugin, an incorrect category, or a setup requirement that a listing overlooks. Include the public tool link and what needs correcting in a GitHub issue.

Browse Local AI Tools and search for something you'd like your local model to do. You shouldn't need to know the plugin's name before you can find it.

Disclosure: This article was written with AI assistance.

When IMemoryCache Stops Scaling: Moving to Redis with .NET Aspire

Sahan — Fri, 28 Aug 2026 01:48:00 +0000

IMemoryCache is a perfectly good cache until the assumption behind it changes.

That assumption is simple: the state belongs to one process.

PulseOps starts with one API instance, so keeping recent service-health results in memory is cheap and easy. Then we run a second API instance. Suddenly each process can cache a different answer to the same question.

Nothing is wrong with IMemoryCache. The architecture changed underneath it.

This is the first post in Building PulseOps, a series where I'm getting back into modern .NET by building one system and letting real requirements pull in the next piece of infrastructure. No catalog app. No infrastructure added just because a getting-started guide says so.

In this post, scaling the API gives us the first reason to change the architecture: move the cache to Redis and bring .NET Aspire into the application.

TL;DR

IMemoryCache remains correct for state owned by one process. Once PulseOps runs multiple API replicas, each replica can cache a different service-health result. Redis gives those replicas shared cache state, while .NET Aspire makes the Redis dependency explicit, runnable, and observable.

What you'll learn

By the end, we'll have:

reproduced the inconsistency that appears when process-local caches are spread across multiple instances;
moved the shared cache state to Redis with IDistributedCache;
modeled Redis as an Aspire resource instead of hand-managing a local container and connection string;
verified that separate API instances can read the same cached value.

If you want the fundamentals first, I've already covered IMemoryCache and distributed caching with Redis separately.

This post starts where those two meet a distributed application.

Why I'm building PulseOps this way

I'm building PulseOps as the open-source companion project for this series.

The idea is simple: keep one application alive across the whole series and evolve it as the requirements get harder.

PulseOps will eventually ingest alerts, track incidents, persist operational data, expose telemetry, and grow an agentic incident assistant. But each capability has to earn its place first.

That gives the series a useful rule:

Start with the simplest thing that is correct. Make the limitation visible. Then add infrastructure because you can explain exactly which property it changes.

So this isn't going to be a sequence of disconnected Aspire demos. main will keep moving, while tagged snapshots preserve the exact version used by each article.

The rough direction is:

shared cache state with Redis;
durable persistence with PostgreSQL;
logs, traces, and metrics that make failures observable;
an incident agent built with the GitHub Copilot SDK;
giving that agent useful access to the running Aspire application.

The details may change as PulseOps grows. That's part of the point.

PulseOps starts deliberately small

The first version is intentionally boring:

The API has a small registry of services PulseOps knows about. A service looks roughly like this:

{
  "id": "payments-api",
  "name": "Payments API",
  "url": "https://example.com/health",
  "status": "unknown"
}

When the dashboard asks for a service's current status, PulseOps may need to make a remote health request. There is no point hammering the same endpoint every time someone refreshes the page, so caching the result for a short period is an obvious first optimization.

The simplest cache works fine with one process

For one API instance, IMemoryCache is a good fit.

The shape is straightforward:

A simplified implementation might look like this:

public async Task<ServiceStatus> GetStatusAsync(
    string serviceId,
    CancellationToken cancellationToken)
{
    var cacheKey = $"service-status:{serviceId}";

    if (_cache.TryGetValue(cacheKey, out ServiceStatus? cachedStatus))
    {
        return cachedStatus!;
    }

    var status = await CheckServiceAsync(serviceId, cancellationToken);

    _cache.Set(
        cacheKey,
        status,
        TimeSpan.FromSeconds(30));

    return status;
}

The first request checks the service. Requests during the next 30 seconds get the cached result.

Fast, dependency-free, and easy to understand.

The starting point: Use process-local state when the state really is local to one process. Don't distribute it before you have a reason.

A second API instance changes the correctness model

Now suppose we run two instances of PulseOps.Api.

Each process gets its own memory and therefore its own cache.

Consider this sequence:

For the next few seconds, API #1 can still report Healthy while API #2 reports Unhealthy.

Both API instances are behaving correctly according to their own cache.

PulseOps as a system is not.

A monitoring dashboard that gives you a different answer depending on which backend instance served the request is a fairly creative definition of monitoring.

This isn't an IMemoryCache bug

It's tempting to frame this as "IMemoryCache doesn't scale."

That's not quite right.

IMemoryCache is doing exactly what it promises: storing data in the memory of the current process.

Scaling out changed the ownership boundary:

The requirement changed from cache this value to let multiple processes observe the same cached value.

That second requirement needs shared state.

The real boundary: A cache stops being an implementation detail when application correctness depends on multiple processes seeing the same cached state.

Moving the cache outside the API process

Redis gives both API instances somewhere shared to store the status value:

If one instance writes:

service-status:payments-api = unhealthy

another instance can read the same entry.

This is the same basic move I covered in Distributed Caching in ASP.NET Core with Redis. The difference here is the operational side.

PulseOps is already becoming a multi-resource application. I don't want the development workflow to turn into a README containing six Docker commands, several hand-picked localhost ports, and a prayer that everyone copied the same connection string.

That's the point where Aspire becomes useful.

Model Redis in the Aspire AppHost

Before Redis, the PulseOps AppHost is roughly:

var builder = DistributedApplication.CreateBuilder(args);

var api = builder.AddProject<Projects.PulseOps_Api>("api");

builder.AddProject<Projects.PulseOps_Web>("web")
    .WithReference(api);

builder.Build().Run();

Adding Redis to the application model is small:

var builder = DistributedApplication.CreateBuilder(args);

var cache = builder.AddRedis("cache");

var api = builder.AddProject<Projects.PulseOps_Api>("api")
    .WithReference(cache);

builder.AddProject<Projects.PulseOps_Web>("web")
    .WithReference(api);

builder.Build().Run();

AddRedis("cache") adds a Redis resource to the AppHost. WithReference(cache) expresses that the API consumes it.

For local development, Aspire can run the Redis container and make its connection information available to the API. The current Redis hosting integration exposes the resource URI, host, port, and password to the consuming project when the resource is referenced.

That means the dependency is now part of the application model rather than a separate setup instruction living outside the code.

Register Redis as IDistributedCache

Inside the API, the Redis distributed-cache integration can register IDistributedCache against the resource named cache:

var builder = WebApplication.CreateBuilder(args);

builder.AddServiceDefaults();
builder.AddRedisDistributedCache("cache");

The connection name matches the AppHost resource named cache.

The application service can then depend on IDistributedCache rather than IMemoryCache:

public sealed class ServiceStatusCache(IDistributedCache cache)
{
    private readonly IDistributedCache _cache = cache;

    public async Task<ServiceStatus?> GetAsync(
        string serviceId,
        CancellationToken cancellationToken)
    {
        var key = $"service-status:{serviceId}";
        var value = await _cache.GetAsync(key, cancellationToken);

        return value is null
            ? null
            : JsonSerializer.Deserialize<ServiceStatus>(value);
    }

    public Task SetAsync(
        string serviceId,
        ServiceStatus status,
        CancellationToken cancellationToken)
    {
        var key = $"service-status:{serviceId}";
        var value = JsonSerializer.SerializeToUtf8Bytes(status);

        return _cache.SetAsync(
            key,
            value,
            new DistributedCacheEntryOptions
            {
                AbsoluteExpirationRelativeToNow = TimeSpan.FromSeconds(30)
            },
            cancellationToken);
    }
}

Depend on the shared-cache abstraction. The service no longer owns process-local cache state. It receives an IDistributedCache backed by the Aspire Redis resource.
Read the shared value. Every API instance uses the same key and reads the serialized status from Redis. A missing key remains a normal cache miss.
Prepare a shared write. The writer builds the same cache key and serializes the status so another process can read the exact value.
Store it with the same policy. Redis changes where the value lives, not the caching rule. The status still expires after 30 seconds.

The caching policy hasn't changed. PulseOps still caches a service status for 30 seconds.

What changed is ownership: the cached state now lives outside the API process in Redis, where both instances can observe it.

Aspire helps operate the dependency, Redis provides the shared state

There's an easy mental trap here.

You can look at this:

builder.AddRedis("cache");

and give Aspire too much credit.

Aspire didn't make our cache distributed. Redis did that.

Aspire helps with a different set of problems:

That distinction matters because it gives us a useful model for the rest of this series.

Aspire isn't a replacement for Redis, PostgreSQL, RabbitMQ, Kubernetes, or the application framework itself. It gives us a code-first application model that describes those resources and the relationships between them.

What Aspire does: Redis solves shared cache state. Aspire makes Redis an explicit, runnable, observable dependency of the application.

The Aspire dashboard makes the new dependency visible

At this point PulseOps has three resources we care about during development: PulseOps.Web, PulseOps.Api, and Redis.

The Aspire dashboard gives us one place to see their state and inspect the application while it is running.

This isn't hugely dramatic with three resources.

That's fine.

Later PulseOps will have multiple APIs, workers, a database, a message broker, and agent tooling. The value of having those relationships described in one place grows with the application.

For now, the useful part is much simpler: Redis is no longer a container I need to remember to start in another terminal.

Verify the property we actually care about

The important test isn't "does Redis start?"

The property we wanted to change was:

Two API instances must observe the same cached service status.

So that's what we should verify.

Run multiple API instances against the same Redis resource. Then exercise a sequence where API #1 populates the cache and API #2 serves the next request.

That verifies the architectural change we actually made, not just the fact that a container happens to be green.

When Redis is the wrong answer

This isn't an argument for replacing every IMemoryCache with Redis.

If the data is truly local to one process, a local cache remains simpler and faster. You avoid a network hop and another infrastructure dependency.

Redis becomes useful here because shared visibility is part of correctness.

There are also cache designs where small amounts of per-instance staleness are acceptable. In those systems, a local cache on every instance may be completely reasonable, especially when reducing dependency latency matters more than having one globally consistent cached value.

The question isn't "which cache is better?"

It's:

Who needs to observe this state, and how stale is it allowed to be?

For PulseOps service status, we want every API instance using the same short-lived cached observation. A shared cache matches that requirement better.

The first architectural change in PulseOps

PulseOps began with one web frontend, one API, and process-local IMemoryCache state. It ends this post with multiple API instances backed by one shared Redis cache.

The interesting part isn't that we added Redis.

It's why we added Redis.

The original solution was correct while the cache belonged to one process. Scaling the API changed the ownership boundary, so the cache had to move with it.

That's the pattern I want to keep through the rest of PulseOps: start simple, make the limitation visible, then add infrastructure because we can explain exactly which property it changes.

What's next

Redis gives the API instances shared short-lived state. It doesn't give PulseOps durable history.

If an incident is created, updated, acknowledged, or resolved, that data has to survive process restarts. Keeping it in memory would make for a wonderfully optimistic incident-management system.

So the next step is PostgreSQL.

We'll add durable persistence through Aspire, look at how the database becomes part of the application model, and deal with the first operational questions that appear with real state: startup ordering, migrations, health, and what should happen when the database isn't ready yet.

After that, we'll make the system observable enough that an agent can eventually reason about what is happening rather than guess from a prompt.

Code for this article

PulseOps is available at github.com/sahansera/PulseOps.

The series uses immutable Git tags so the code in older posts doesn't drift as the project evolves:

Starting point: pulseops-00-foundation
Finished version: pulseops-01-redis

main will keep moving. These tags are the snapshots for this article.

References

Discussion

Have you seen process-local cache state become a correctness problem after scaling an application? I'd be interested to hear how you identified the boundary and what you moved to shared storage.

Replacing Sharp with WASM: How a Hidden SVG Dependency Broke Our Rendering Pipeline

Sahan — Fri, 21 Aug 2026 01:03:00 +0000

The failure, the cause, and the fix

Every Open Graph social card on the site was returning a 404 in production, even though the same code worked locally and in CI.

The production rendering path depended on sharp, libvips, and librsvg to turn Satori's SVG output into PNG bytes.

A runtime security setting made libvips's SVG loader unavailable, so sharp failed with:

Input buffer contains unsupported image format

The existing fallback failed too.

It simplified the card content, but still used the same rendering machinery. So when the renderer failed, the fallback immediately hit the same failure.

The fix had two parts:

Replace the sharp rasterisation path with resvg WASM.
Replace the dynamic fallback with a static image read from disk.

The primary path became deterministic across environments, and the fallback stopped sharing the primary renderer's failure mode.

The important bit: Make the primary path deterministic. Make the fallback independent of the capability that just failed.

Want the full debugging story?

The fix is fairly small once you know the cause.

Getting there wasn't.

I didn't start this investigation because someone reported broken social cards. I noticed unexplained errors repeatedly appearing on a Datadog dashboard, with no obvious user-facing symptom attached to them.

I started working backwards from the timestamps.

Datadog gave me the error clusters. Splunk gave me the request-level context. Correlating the two eventually led me to the Open Graph image requests and the production-only 404s.

From there, I had to work out:

why the route was failing
why a clean reproduction worked perfectly
what was different about the production runtime
why the fallback was dying milliseconds after the primary renderer

Here's the rabbit hole.

The 404 was coming from inside the route

The application used Next.js's file-based Open Graph support with an opengraph-image.tsx route.

A failing request looked roughly like this:

GET /profiles/<user>/opengraph-image-<hash>?<build>
-> 404 text/plain 9 bytes

Nine bytes mattered.

Not Found is nine bytes.

A genuinely missing route returned the much larger Next.js HTML 404 page.

This response was ours.

That meant the route was running and something inside it was throwing.

That gave me the first useful boundary in the investigation.

Now I needed to find what was throwing.

The renderer hid an SVG stage

My mental model going into this was basically:

JSX -> PNG

That was too simple.

In that production version, the actual path looked more like this:

Satori first turns JSX into SVG.

A second stage then has to rasterise that SVG into PNG bytes.

That intermediate format ended up being the most important detail in the incident.

The renderer also selected its rasteriser at runtime. Conceptually, it did something like this:

async function getSharp() {
  if (_sharp) return _sharp;

  try {
    _sharp = (await import("sharp")).default;
  } catch {
    return undefined;
  }

  return _sharp;
}

Then later:

const svg = await satori(element, { /* ... */ });

if (sharp) {
  pngBuffer = await sharp(new TextEncoder().encode(svg))
    .resize(width)
    .png()
    .toBuffer();
} else {
  const renderer = new Resvg(svg, { /* ... */ });
}

So the useful mental model wasn't "Next.js renders JSX into an image".

It was:

JSX -> SVG -> PNG

Those stages have different dependencies and different failure modes.

Hidden stages matter: When a library appears to do A to C, find out whether production actually depends on an intermediate B.

I ruled out the obvious explanations first

Before I had the useful stack trace, there were plenty of plausible explanations.

I worked through the obvious ones:

Bad JSX or CSS: rendered fine locally.
Specific profile data: real production-like data rendered correctly.
Missing fonts: present in the standalone build.
Missing WASM binaries: present.
Incorrect generated URL: the URL was correct.
Middleware intercepting the request: the request reached the image route.
Node version differences: the major version matched.

I also managed to manufacture one completely unrelated failure myself.

The generated Open Graph URL contained a build-hash suffix. At one point I requested the bare path instead, and Next.js quite correctly returned its normal HTML 404.

For about twenty minutes I thought I'd reproduced production.

I'd reproduced my typo.

Excellent progress.

The investigation only started moving when I stopped theorising and followed the errors through the logs.

Correlating Datadog and Splunk exposed the pattern

Datadog was where I first noticed the failure, but Splunk had the request context I needed.

For each cluster of errors in Datadog, I searched the same time window in Splunk and looked at what the application was doing around the failure.

That's how I connected what initially looked like generic rendering errors to the Open Graph image requests.

It also exposed a pattern I couldn't explain yet.

Primary renders and fallback renders were failing in pairs, milliseconds apart, on the same runtime instance.

For example:

primary: 32 failures
fallback: 32 failures

Later:

primary: 93 failures
fallback: 93 failures

Failures were also spread across different avatar sources, which made an avatar-specific input problem much less likely.

The paired failures would matter later.

First, I needed to know where the renderer was dying.

The stack trace moved the failure below Next.js

The line that changed the investigation was this:

Error: Input buffer contains unsupported image format
    at Sharp.toBuffer (.../sharp/dist/output.mjs:159:17)
    at render (.../@vercel/og/index.node.js:...)

That immediately narrowed the search space.

The request was going through sharp.

More importantly, the buffer sharp couldn't identify wasn't an uploaded avatar.

It was the SVG Satori had just generated.

The failing path was now much clearer.

Satori successfully produced SVG, and the native rasterisation branch died inside sharp before PNG output existed.

That changed the question from:

What's wrong with the profile image?

to:

Why can't sharp decode a valid SVG in production?

In this path, sharp hands the work to libvips. SVG support depends on the SVG loader being available there.

If no loader claims the input, the error you get is the wonderfully specific:

Input buffer contains unsupported image format

Now I had something concrete to reproduce.

The working reproduction narrowed the problem

I built a minimal reproduction with the same important pieces:

framework version
package manager
install flags
standalone output
Node version
base architecture

It rendered perfectly, including on two architectures.

That ruled out most of the build.

The framework version wasn't inherently broken. The standalone output wasn't inherently broken. The healthy container was taking the same sharp path successfully.

The remaining difference was much narrower:

the environment the container was running in.

Negative evidence is still evidence: A reproduction that doesn't reproduce can remove an entire class of possible causes.

A runtime security setting reproduced the exact failure

libvips can block operations it considers untrusted.

In my reproduction, enabling that restriction made the SVG path fail exactly like production:

$ docker run -e VIPS_BLOCK_UNTRUSTED=1 <image> node probe.js

sharp SVG->PNG: FAILED
Input buffer contains unsupported image format

Run the same image without the setting:

$ docker run <image> node probe.js

sharp SVG->PNG: OK
31270 bytes

Same image. Same application.

One runtime setting changed the result.

I could confirm the mechanism directly too:

import sharp from "sharp";

sharp.unblock({
  operation: ["VipsForeignLoadSvg"],
});

Once the SVG loader was allowed again, the render succeeded.

That proved the failure mechanism.

The Open Graph path produced SVG, then passed it into a native image stack where SVG parsing was unavailable at runtime.

There's one caveat worth calling out.

I never established what introduced that restriction into the production environment. I couldn't find it in the application source, container image, or deployment configuration I investigated.

I could prove the mechanism.

I couldn't prove its provenance.

That uncertainty influenced the fix.

The fallback used the same broken renderer

There was already a fallback renderer.

If the main card failed, the fallback removed most of the complicated inputs:

no avatar
no custom fonts
no profile text
no dynamic content

The assumption was reasonable.

If some unexpected input broke the renderer, render something simpler.

Except the input wasn't broken.

The renderer was.

The fallback still crossed the same rendering boundary as the primary:

Suddenly the paired error counts made sense.

Every time the primary failed because SVG parsing wasn't available, the fallback generated another SVG and sent it through the same broken machinery a few milliseconds later.

The fallback had removed complexity from the input.

It hadn't removed the failing dependency.

The fallback rule: A fallback that shares the primary's failure mode isn't independent. It's the same system with fewer features.

I made the fallback deliberately boring

The replacement fallback doesn't render anything.

It reads a pre-generated image from disk:

const FALLBACK_CARD_CONTENT_TYPE = "image/jpeg";

export async function renderFallbackSocialCard(): Promise<Response> {
  const bytes = await readFile(
    join(process.cwd(), "public/card.jpg"),
  );

  return new Response(new Uint8Array(bytes), {
    headers: {
      "Content-Type": FALLBACK_CARD_CONTENT_TYPE,
    },
  });
}

That's it.

No Satori. No SVG. No sharp. No libvips.

It's boring on purpose.

If the dynamic rendering stack catches fire, the fallback shouldn't politely walk back into the fire.

Why I replaced sharp with resvg WASM

For the main renderer, I had two practical options.

Option 1: allow SVG parsing again

Mechanically, this was the quickest path.

But I still didn't know why the production environment had restricted that capability in the first place.

I didn't want an Open Graph feature to override an environment-level security control just to preserve the existing implementation, especially when I could remove that dependency from the path entirely.

Option 2: remove libvips from this path

The other option was to remove libvips from this rendering path entirely and make resvg WASM the explicit rasteriser.

That's the route I took.

The implementation was relatively small:

import { readFile } from "node:fs/promises";
import { join } from "node:path";
import { initWasm, Resvg } from "@resvg/resvg-wasm";
import satori from "satori";

const WASM_PATH =
  "node_modules/@resvg/resvg-wasm/index_bg.wasm";

let wasmReady: Promise<void> | undefined;

function initResvg(): Promise<void> {
  wasmReady ??= readFile(
    join(process.cwd(), WASM_PATH),
  ).then(initWasm);

  return wasmReady;
}

export async function renderElementToPng(
  element: ReactNode,
  { width, height, fonts }: RenderOptions,
): Promise<Buffer> {
  const svg = await satori(element, {
    width,
    height,
    fonts,
  });

  await initResvg();

  const resvg = new Resvg(svg, {
    fitTo: {
      mode: "width",
      value: width,
    },
  });

  try {
    const rendered = resvg.render();

    try {
      return Buffer.from(rendered.asPng());
    } finally {
      rendered.free();
    }
  } finally {
    resvg.free();
  }
}

The important change wasn't the syntax.

It was the runtime behaviour.

resvg became the one explicit rasteriser.

No native image loader in this path. No environment-dependent SVG decoder. No silent rasteriser selection.

The WASM fix created a packaging problem

Of course, replacing the renderer wasn't the end of it.

Reading the WASM file via require.resolve() made the bundler try to process it as a module, which failed.

Reading the binary directly by path avoided that problem, but then Next.js's dependency tracer couldn't infer that the file was required at runtime.

The standalone output didn't contain it.

I had to make that dependency explicit:

serverExternalPackages: ["@resvg/resvg-wasm"],

outputFileTracingIncludes: {
  "/**": [
    "./node_modules/@resvg/resvg-wasm/index_bg.wasm",
  ],
},

Configuration like this is exactly the sort of thing that gets removed six months later because nobody remembers why it's there.

So I added a build assertion too:

RUN test -f .next/standalone/node_modules/@resvg/resvg-wasm/index_bg.wasm \
    || { echo "resvg wasm missing from standalone output"; exit 1; }

Then I verified the assertion in both directions.

Remove the tracing configuration and the build fails.

That turns a hidden runtime dependency into something the build can enforce.

One green test had never rendered an image

The migration also exposed a test problem.

There was already a test that appeared to cover image rendering. It created an ImageResponse and asserted against it.

The problem was that ImageResponse rendered lazily.

Constructing the response didn't force Satori to execute.

The test was green because the failing operation had never happened.

Once I changed the path to render eagerly, the test failed immediately.

A test called "render image" that never renders the image is a particularly optimistic form of testing.

I verified the real renderer, not just a successful response

The static fallback introduced one final trap.

Once it shipped, every Open Graph request returned a valid-looking image again.

From the outside, the feature looked healthy.

That still didn't prove the dynamic renderer was working.

It could remain completely broken while every request quietly returned the fallback.

So I verified the responses themselves.

Before the renderer fix

Property	Result
Content type	`image/jpeg`
Size	`425,081` bytes every time
Requests	`45/45` fallback

After the renderer fix

Property	Result
Content type	`image/png`
Size	roughly `143KB` to `245KB`
Requests	`45/45` dynamically rendered

The fallback responses were byte-for-byte identical.

The dynamic cards varied with each profile.

That variation was evidence that I was exercising the real renderer again.

Verification matters: A valid response only proves that some path worked. Verify the behavioural property that distinguishes the path you actually fixed.

What I'd do earlier next time

A few things from this investigation are going straight into my debugging playbook.

Correlate telemetry before guessing

The Datadog errors weren't very useful by themselves.

Correlating their timestamps with Splunk turned an unexplained counter into an actual request path.

Look for hidden stages

"JSX to PNG" hid an SVG conversion with its own runtime dependencies.

The intermediate representation was where the incident lived.

Make implementation selection explicit

A library silently choosing between implementations is convenient until production takes a different path from your laptop.

For an important path, the chosen implementation should be observable.

Design fallbacks around failure boundaries

Removing input complexity helps when the input is the problem.

It doesn't help when the capability underneath it has disappeared.

A useful fallback should avoid the failed step wherever practical.

The lesson I kept

Technically, this incident came down to an SVG loader being unavailable inside the production runtime.

That's not the part I expect to remember.

I started with unexplained errors on a Datadog dashboard, correlated them against Splunk, connected them to broken Open Graph images, followed the failure through Satori and sharp, reproduced the runtime behaviour in isolation, and replaced the native rasterisation path with WASM.

But the most useful discovery was the fallback.

I'd assumed it gave the feature resilience because it rendered something simpler.

It didn't.

It depended on the exact capability that had just failed.

The fallback wasn't independent. It was a smaller version of the same pipeline.

Thanks for reading ✌️

References

When Kubernetes Pods Try to Run One Database Migration

Sahan — Fri, 14 Aug 2026 22:23:00 +0000

This startup command looked fine while our application had one replica:

prisma migrate deploy && node server.js

Then we scaled the Deployment to three pods.

All three pods started together. All three ran the migration command. Prisma used a PostgreSQL advisory lock to protect the migration, so the database was not corrupted, but the losing pods waited for the lock and eventually timed out. Kubernetes saw failed containers, restarted them, and filled the rollout logs with errors.

One pod was doing useful work. The other two were turning expected contention into application failures.

The clean solution is a migration Job that runs before the Deployment. Our release platform could only start one application workload, so we used a small PostgreSQL-backed coordinator instead.

This post shows the complete pattern.

The problem in one picture

A Deployment creates copies of a long-running process. A database migration is one-shot work.

The coordinator gives every pod the same startup protocol:

Try to acquire one application-specific advisory lock.
If the lock is busy, close the connection, wait with jitter, and try again.
If the lock is acquired, run prisma migrate deploy.
Release the lock and start the application.

Every pod runs the migration command after acquiring the lock. For the first pod it applies pending migrations. For later pods it is a no-op. This matters because a released lock only proves that the previous database session ended. It does not prove the previous migration succeeded.

Reconcile after waiting

A released lock does not mean the previous migration worked. Every pod that acquires the lock should reconcile the database to the expected migration state.

Use a Job when your platform supports one

Before adding coordination code, check whether the delivery system can run an ordered pre-deployment Job.

A Job is the better default because it gives the migration its own status, logs, image and database credentials. The application Deployment does not need schema-changing permissions, and a failed migration stops the rollout at the correct boundary.

The coordinator below is for the awkward case where the platform can only deploy the application workload. That was our constraint.

Stay in the loop

Enjoying this article?

Get practical engineering notes in your inbox, usually once or twice a month.

Get new posts →

Add the coordinator

The application already contained Prisma. The coordinator only needed the PostgreSQL client:

npm install pg

Create prisma/migration-coordinator.mjs:

import { spawn } from "node:child_process";
import { createHash } from "node:crypto";
import { fileURLToPath } from "node:url";
import pg from "pg";

const { Client } = pg;

const connectionString = process.env.DIRECT_DATABASE_URL;
const lockName = process.env.MIGRATION_LOCK_NAME ?? "web:public:prisma-migrations";
const lockId = createHash("sha256")
  .update(lockName)
  .digest()
  .readBigInt64BE(0)
  .toString();
const waitTimeoutMs = Number(process.env.MIGRATION_WAIT_TIMEOUT_MS ?? "300000");
const retryDelayMs = Number(process.env.MIGRATION_RETRY_DELAY_MS ?? "2000");
const prismaCli = fileURLToPath(new URL("../node_modules/.bin/prisma", import.meta.url));

if (!connectionString) {
  throw new Error("DIRECT_DATABASE_URL is required");
}

function sleep(ms) {
  return new Promise((resolve) =&gt; setTimeout(resolve, ms));
}

function runPrismaMigrateDeploy() {
  return new Promise((resolve, reject) =&gt; {
    const child = spawn(prismaCli, ["migrate", "deploy"], {
      stdio: "inherit",
      env: {
        ...process.env,
        DATABASE_URL: connectionString,
      },
    });

    child.once("error", reject);
    child.once("exit", (code) =&gt; {
      if (code === 0) {
        resolve();
        return;
      }

      reject(new Error(`prisma migrate deploy exited with code ${code}`));
    });
  });
}

const deadline = Date.now() + waitTimeoutMs;
let migrationComplete = false;

while (!migrationComplete) {
  const client = new Client({
    connectionString,
    connectionTimeoutMillis: 10_000,
  });
  let acquired = false;

  try {
    await client.connect();

    const result = await client.query(
      "SELECT pg_try_advisory_lock($1::bigint) AS acquired",
      [lockId],
    );
    acquired = result.rows[0].acquired === true;

    if (acquired) {
      console.log("migration lock acquired");
      await runPrismaMigrateDeploy();
      console.log("database migrations are current");
      migrationComplete = true;
    }
  } finally {
    if (acquired) {
      await client
        .query("SELECT pg_advisory_unlock($1::bigint)", [lockId])
        .catch((error) =&gt; console.error("failed to release migration lock", error));
    }

    await client.end().catch(() =&gt; {});
  }

  if (migrationComplete) {
    break;
  }

  if (Date.now() &gt;= deadline) {
    throw new Error("timed out waiting for the migration lock");
  }

  const jitterMs = Math.floor(Math.random() * 500);
  console.log("migration lock busy; waiting before retry");
  await sleep(retryDelayMs + jitterMs);
}

There are four details worth keeping:

pg_try_advisory_lock returns immediately instead of keeping a waiting connection open.
The same database connection stays open while the Prisma subprocess runs. Session-level advisory locks belong to the connection that acquired them.
Every lock owner runs migrate deploy. Already-applied migrations make it a no-op.
A failed migration rejects the script, so the application does not start against an unknown schema.

Use a direct PostgreSQL connection for DIRECT_DATABASE_URL. Transaction-mode poolers do not preserve session state, so they are the wrong boundary for a session-level advisory lock.

The lock name is not a secret, but it must be stable. The script hashes the readable name into the signed 64-bit integer PostgreSQL expects. Every revision of this application should use the same name for the same database and schema. Different applications sharing a database should use different names.

After adding the coordinator, expected contention becomes an ordinary waiting path:

Run it before the server

Update the container command so the server starts only after coordination succeeds:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: registry.example.com/web:abc123
          ports:
            - name: http
              containerPort: 3000
          command: ["sh", "-c"]
          args:
            - node prisma/migration-coordinator.mjs &amp;&amp; exec node server.js
          env:
            - name: DIRECT_DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: web-database
                  key: direct-url
            - name: MIGRATION_LOCK_NAME
              value: "web:public:prisma-migrations"
            - name: MIGRATION_WAIT_TIMEOUT_MS
              value: "300000"
          startupProbe:
            httpGet:
              path: /health/startup
              port: 3000
            periodSeconds: 5
            failureThreshold: 72
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 3000
            periodSeconds: 5

exec replaces the shell with the Node process after the migration finishes. That lets the application receive termination signals directly as PID 1.

The coordinator calls the Prisma binary already installed in the image. Keep the Prisma CLI in the production image rather than downloading it during startup. If your image uses a different project layout, adjust prismaCli to match it.

The startup probe allows six minutes for lock waiting, migration and application startup. Set that budget from your slowest expected migration rather than copying this number. Readiness still decides when the running pod can receive traffic.

What happens during a rollout

With three new pods, the rollout now looks like this:

pod-a migration lock acquired
pod-b migration lock busy; waiting before retry
pod-c migration lock busy; waiting before retry

pod-a database migrations are current
pod-a application starts

pod-b migration lock acquired
pod-b no pending migrations to apply
pod-b application starts

pod-c migration lock acquired
pod-c no pending migrations to apply
pod-c application starts

Expected contention is logged as waiting, not as a migration failure. If the winning pod disappears, PostgreSQL releases its session lock when the connection closes. Another pod acquires the lock and runs migrate deploy again.

This has the same shape as the shared-state problem in Scaling an Agentic Coding SDK: What Concurrency Actually Costs. Sequential execution had hidden an ownership problem. Adding replicas made it visible.

The lock does not make breaking migrations safe

The advisory lock serialises migration commands. It does not stop old pods from serving traffic while a new pod changes the schema.

Rolling deployments still need expand-and-contract migrations:

Add the new schema in a backward-compatible form.
Deploy code that works with both representations.
Backfill data if required.
Switch reads and writes.
Remove the old schema after old pods are gone.

I covered the same compatibility problem in more detail in Replacing a Critical Data Path Without a Flag Day.

Coordination is not compatibility

The lock keeps migration commands from colliding. Expand-and-contract changes keep old and new application versions compatible during the rollout.

Before shipping it

Check these six things:

A pre-deployment Job really is unavailable.
The lock uses a direct PostgreSQL session.
The lock name is stable and unique to the application and schema.
Waiting has jitter and an overall deadline.
A failed migration prevents the application from starting.
Schema changes remain compatible with old pods during the rollout.

That is the whole pattern. One pod applies the migration. The others wait, check the same desired state, and then start normally.

The code is small because PostgreSQL already owns the hard guarantee: only one session can hold the advisory lock at a time.

Thanks for reading ✌️

References

Scaling an Agentic Coding SDK: What Concurrency Actually Costs

Sahan — Wed, 12 Aug 2026 11:51:00 +0000

I previously wrote about building an AI-assisted dependency vulnerability fixer. It used the GitHub Copilot SDK to start a repository-scoped coding agent for the fixes that could not be handled reliably by a deterministic packaging change. Each agent took a known finding, made the smallest safe change it could, validated the result, and opened a pull request for the repository owner.

The .NET and Python tutorials isolate the permission boundary behind this kind of work: one agent, one repository, and explicit approval for side effects. This article looks at the next operational problem from the production fixer - what changes when many such workloads run concurrently.

The first version processed one repository at a time. A representative batch of 30 repositories took roughly 46 minutes. With a worker pool of five, it took about 12.

Replacing the sequential loop was the easy part. Before those five agents could run safely, each needed an isolated workspace, reliable cleanup, resource limits, rate limiting, and enough telemetry to show whether more concurrency helped. Multiple service replicas also needed durable job ownership and safe retries.

That work changed my mental model:

The mental model

Agent concurrency is not a larger number of requests. It is a larger number of partially trusted computers operating at once.

An agent session is a workload, not a request

When I say “agent” in this article, I mean one live Copilot SDK client and session operating against one repository. In this setup, the client started the Copilot runtime and the session drove its model and repository tools. It was not a name for one model API request.

An HTTP request usually borrows a connection, does bounded work, and returns. A Copilot session can live for minutes and contain many model and tool turns. It keeps a transcript in memory, drives a shell, mutates a checkout, creates subprocesses, uses upstream capacity, and may hold credentials that can create branches and pull requests.

This introduces familiar distributed-systems questions. What isolates jobs? Who owns each resource? What happens when a worker disappears? Can a job run twice? Which resource sets the safe concurrency limit? Did a timed-out operation fail before or after changing an external system?

The novelty is in who chooses the commands. The operational concerns are not new.

Three decisions need to remain separate:

Isolation: Each repository gets its own session and workspace, ideally inside a disposable sandbox.
Concurrency: The worker count is bounded by the first resource likely to run out, not simply CPU.
Persistence: Agent sessions can be ephemeral while job identities, attempts, and external effects remain durable.

Sequential execution had been protecting shared state

The first implementation was intentionally boring:

for (const repository of repositories) {
  await fixRepository(repository);
}

It helped validate prompts, permissions, branch strategy, and review before concurrency obscured product mistakes. It also hid unsafe assumptions.

The prompt cloned every repository into /tmp/agent-workdir. With two agents, one install could rewrite the lockfile another was preparing to commit. One cleanup could delete the other job’s files.

The service now allocates the path and passes it to the agent as job data:

import { randomUUID } from "node:crypto";

const safeName = `${repository}-${branch}`.replace(/[^a-zA-Z0-9_-]+/g, "-");
const workdir = `/tmp/agent-${safeName}-${randomUUID()}`;

The same audit applies to ports, branches, cache keys, and temporary filenames. Any literal value becomes shared state when two jobs can use it.

A unique directory prevents accidental overlap, but it is not a security boundary. Repository-controlled code still needs a disposable sandbox with bounded access to the host and network.

The service did not call an LLM endpoint directly. For each job, it started a GitHub Copilot SDK client inside the unique working directory and created one agent session. Cleanup also needed to survive failure. The original code disconnected the session only after successful work, so an exception leaked both session and runtime resources.

A simplified version of the corrected lifecycle looks like this:

import { CopilotClient, type CopilotSession } from "@github/copilot-sdk";

async function runCopilotRemediation(job: Job, workdir: string) {
  const client = new CopilotClient({ workingDirectory: workdir });
  let session: CopilotSession | undefined;

  try {
    await client.start();
    session = await client.createSession({
      systemMessage: {
        mode: "append",
        content: remediationInstructions(job),
      },
      onPermissionRequest: remediationPolicy,
    });

    return await session.sendAndWait({
      prompt: buildRemediationTask(job),
    });
  } finally {
    await session?.disconnect();
    await client.stop();
    await removeWorkspace(workdir);
  }
}

client.start() launches or connects to the Copilot runtime. createSession() gives the job its own transcript and agent loop. sendAndWait() lets that loop continue through repository reads, edits, shell commands, and validation until the session becomes idle. A worker pool of five can therefore mean five Copilot runtimes, five sessions, five mutable checkouts, and all of their subprocesses operating at once.

The component that acquires a resource owns its lifecycle. Concurrency makes violations more frequent, not more complicated.

Stay in the loop

Enjoying this article?

Get practical engineering notes in your inbox, usually once or twice a month.

Get new posts →

Bound concurrency with evidence

This is concise but unsafe:

await Promise.all(repositories.map(fixRepository));

It lets the input size set infrastructure policy. Thirty jobs may work; 300 may exhaust memory, fill disk, or trigger rate limits.

I used a small worker pool instead:

async function runPool(
  items: T[],
  limit: number,
  run: (item: T) =&gt; Promise,
) {
  let next = 0;

  async function worker() {
    while (next &lt; items.length) {
      const index = next++;

      try {
        await run(items[index]);
      } catch (error) {
        recordFailure(items[index], error);
      }
    }
  }

  await Promise.all(
    Array.from({ length: Math.min(limit, items.length) }, worker),
  );
}

The cap is explicit, one repository failure does not cancel unrelated work, and the batch drains before completion is reported.

I started with five workers. Five is not a generally safe number for coding agents. It was a conservative operating point for this workload.

The real cap is the smallest limit implied by memory, ephemeral storage, subprocesses, file descriptors, provider requests and tokens, source-control operations, network bandwidth, spend, and acceptable blast radius. For memory, the rough calculation is:

memory_cap = floor(
  (container_limit - service_baseline - safety_headroom)
  / p95_incremental_memory_per_session
)

A job-level cap does not replace API rate limiting. Five agents can still push branches or create pull requests simultaneously. The source-control client must independently honour rate-limit headers, Retry-After, and backoff.

Capacity needs evidence

“Five at a time” is not a capacity plan. “Five because memory is the first measured budget we approach” is.

Local limits stop being global limits

A process-local cap works only while there is one process. With a cap of five and four replicas, the service can create 20 live sessions:

effective_concurrency = replicas × per_replica_cap

An autoscaler can increase that number precisely when an upstream system is already under pressure.

At this point, job ownership moves to a durable queue or database table. A worker atomically claims a job for a limited period, renews the lease while running, and records the outcome before acknowledging completion. If it disappears, the lease expires and another worker can retry. A global limiter protects shared provider and credential budgets.

The agent session and workspace remain disposable. The durable state is the job identity, lease, attempt count, and record of external effects.

Retries need reconciliation. Suppose the agent opens a pull request, but the response is lost before the job records success. A retry can create a duplicate.

I gave each remediation a stable idempotency key derived from the repository and requested change. It enforces one active job, supports a stable branch name, and lets a retry find an existing branch or pull request. An atomic claim or uniqueness constraint closes the race that an existence check alone cannot.

Stateless workers are useful. Statelessness means any worker can continue the protocol, not that the system remembers nothing.

Concurrency needs gauges and guardrails

Total batch duration is not enough to tune the pool. I needed queue wait and execution time separately, active sessions, peak memory, workspace size, subprocess count, cost per job, upstream throttling, retries, cleanup failures, and orphaned sessions.

Three questions make those measurements useful:

Does active work regularly reach the cap?
Is queue time growing while constrained resources still have headroom?
Do failures, latency, throttling, or resource pressure rise with inflight work?

If the pool never fills, a higher cap will not help. If queues grow while resources remain healthy, there may be room. If failures rise with inflight work, the system has found a boundary.

Security belongs in the same discussion. The dependency fixer already treated repositories, install scripts, and tests as untrusted input. Concurrency multiplies that exposure. Each job needs bounded CPU, memory, processes, disk, time, network access, and short-lived repository-scoped credentials. It should have no ambient infrastructure credentials or permission to merge.

Audit trails also need redaction. Terminal output, environment dumps, remote URLs, and package-manager logs can contain credentials.

Why five workers were not five times faster

For one representative batch of 30 repositories:

Phase	Sequential	Pool of five
Clone and dependency setup	~10 minutes	~4 minutes
Agent inspection and editing	~22 minutes	~5 minutes
Push and pull-request creation	~12 minutes	~3 minutes
Artificial inter-job delay	~1.5 minutes	0
Total	~46 minutes	~12 minutes

That is about a 3.8-times speedup. It is an operational measurement, not a benchmark. Removing an artificial delay contributed, and the rest was limited by uneven job duration, disk and network contention, provider latency, and source-control operations.

The goal is not maximum concurrency. It is the best useful throughput inside the safety, cost, and reliability budgets.

I would scale the next service in the same order: isolate each job, make cleanup unconditional, add a bounded local pool, measure it, then introduce durable claims and reconciliation before adding replicas.

The worker pool reduced this batch from roughly 46 minutes to 12. Everything else was the cost of making that number safe.

That is what concurrency actually costs.

Thanks for reading ✌️

References

Build a Safe Repository Maintenance Agent with GitHub Copilot and Microsoft Agent Framework in Python

Sahan — Mon, 10 Aug 2026 22:30:00 +0000

In the .NET tutorial, we built a repository agent that could inspect a checkout, propose a one-line repair, run its tests, and report the result. Every write and shell command required explicit approval.

Now we will build the same agent in Python.

The outcome and safety policy stay identical on purpose - I didn’t want to quietly pick an easier demo just because Python made one more convenient. Both implementations work against the same failing fixture, get the same instructions, stop at the same approval boundaries, and have to pass the same tests. Only the host-language implementation changes.

The result is an async Python command-line application using the GitHub Copilot harness for repository work and Microsoft Agent Framework for the agent abstraction, streaming, sessions, and telemetry.

💡 Code repository: The complete Python and .NET implementations use the same fixture and are available at github.com/sahansera/safe-repository-maintenance-agent.

What we are building

The agent receives a local repository path and a maintenance task. Its system instructions require it to:

Work only inside that repository.
Read its AGENTS.md before changing anything.
Make the smallest coherent repair.
Avoid the network, package installation, commits, pushes, and pull requests.
Run focused validation.
Report the changed files, commands, and result.

The included fixture contains a small JavaScript function. The agent is written in Python, but the target repository does not have to be:

export function normalizeTitle(value) {
  return value.toLowerCase().replace(/\s+/g, "-");
}

One test expects ordinary title normalization. A second expects whitespace around the title to be ignored. The second test fails until the implementation trims the input.

Using a language-neutral target is deliberate. A Python agent can maintain a .NET, JavaScript, Go, or documentation repository. The agent host language determines how we integrate and operate the harness, not which source files the harness can understand.

Prerequisites and package setup

You need Python 3.11 or later and an active GitHub Copilot subscription. The example uses Python 3.12 as its documented baseline and was also verified with Python 3.13.

Here’s the environment I actually tested this against:

[project]
requires-python = ">=3.11"
dependencies = [
  "agent-framework-github-copilot==1.0.1",
  "github-copilot-sdk==1.0.2",
]

[project.optional-dependencies]
dev = ["pytest>=8.4,<9"]

Create an isolated environment and install the project:

python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'

Here’s what it looks like when run end to end:

The current Python SDK package includes the Copilot runtime for supported platforms. Authentication still depends on GitHub Copilot and may ask you to sign in when you run the agent for the first time.

The Agent Framework integration is stable; the GitHub Copilot SDK underneath it isn’t, and its preview label didn’t stop it from changing shape more than once while I was writing this pair of posts. Pin these versions - it’s cheap insurance against a future SDK bump silently rewriting what you’re reading.

Make permissions a small, testable policy

The agent can request several capability types, including read, write, shell, url, and mcp. We will not treat them equally:

Capability	Default
Read inside the selected repository	Approve
Write a file	Ask
Run a shell command	Ask
Fetch a URL or call MCP	Deny
Anything unknown	Deny

The policy is an ordinary function:

from enum import Enum

class PolicyDecision(Enum):
    APPROVE = "approve"
    PROMPT = "prompt"
    DENY = "deny"

def decide(permission_kind: str) -> PolicyDecision:
    if permission_kind == "read":
        return PolicyDecision.APPROVE
    if permission_kind in {"write", "shell"}:
        return PolicyDecision.PROMPT
    return PolicyDecision.DENY

The fallback denies URL access, MCP calls, new SDK permission kinds, and any malformed value. The application has to be changed deliberately before one of those capabilities becomes available.

The test suite makes that contract visible:

@pytest.mark.parametrize(
    ("kind", "expected"),
    [
        ("read", PolicyDecision.APPROVE),
        ("write", PolicyDecision.PROMPT),
        ("shell", PolicyDecision.PROMPT),
        ("url", PolicyDecision.DENY),
        ("mcp", PolicyDecision.DENY),
        ("unknown", PolicyDecision.DENY),
    ],
)
def test_decide_returns_expected_decision(kind, expected):
    assert decide(kind) is expected

These tests do not need a model, a Copilot subscription, or a repository. They test application authority rather than probabilistic behavior.

You do not need an LLM evaluation to prove that URL access is denied. Separate deterministic authorization policy from the agent and test it like any other security-sensitive function.

Convert policy into Copilot decisions

The permission handler receives a typed request and a context dictionary. It first prints enough detail for the operator to understand the action:

async def handle_permission(request, context):
    decision = decide(request.kind)

    print(f"\n[permission: {request.kind}]")
    print(describe(request))

    if decision is PolicyDecision.APPROVE:
        return PermissionHandler.approve_all(request, context)

    if decision is PolicyDecision.DENY:
        return PermissionDecisionReject(
            feedback="Blocked by the repository agent policy."
        )

    answer = (
        await asyncio.to_thread(input, "Approve once? [y/N] ")
    ).strip().lower()

    if answer == "y":
        return PermissionHandler.approve_all(request, context)

    return PermissionDecisionReject(
        feedback="The operator denied this action."
    )

input() is blocking, so asyncio.to_thread keeps it away from the event loop. That detail is easy to miss in a console sample and becomes more important when the application also streams output or handles more than one session.

For a write request, describe prints file_name and diff. For a shell request, it printsfull_command_text. URL and MCP requests are displayed before being denied, leaving an audit-friendly record of what the agent attempted.

The helper name approve_all can be misleading in this context. It constructs an approval response for the current request. Our application still decides which requests reach that line.

Configure the agent without ambient repository state

The command resolves the repository path before creating the agent:

repository = args.repository.expanduser().resolve(strict=True)

if not repository.is_dir():
    raise NotADirectoryError(repository)

The Copilot session options include that working directory and our permission callback:

options = GitHubCopilotOptions(
    working_directory=str(repository),
    enable_config_discovery=False,
    on_permission_request=handle_permission,
)

agent = GitHubCopilotAgent(
    instructions=INSTRUCTIONS,
    default_options=options,
)

Repository instructions are genuinely useful, so turning configuration discovery off seems backwards at first. I ran into the reason for it while building the .NET version of this project: the fixture sat inside another Git checkout during development, and the runtime happily walked up to that parent repo’s AGENTS.md before it ever noticed the fixture’s own. Nested repositories, monorepos, and temporary worktrees all make that boundary easy to blur without meaning to.

So the application instructions explicitly tell the agent to read AGENTS.md inside its working directory itself. That keeps the source of project guidance visible in the tool activity, instead of letting an unrelated parent directory quietly change what the agent thinks the rules are.

That does not make repository instructions trusted. A repository can contain prompt injection just as it can contain a malicious build script. The host policy remains authoritative when instructions ask for a forbidden action.

Repository instructions can improve the patch, but they must not expand the agent's authority. The host permission policy wins when project guidance asks for a forbidden side effect.

Stream the maintenance run

GitHubCopilotAgent owns an async client, so the natural Python lifecycle is an async context manager:

async with agent:
    async for update in agent.run(args.task, stream=True):
        print(update.text, end="", flush=True)

The context manager starts and stops the Copilot client even if the run raises an exception. The command entry point keeps the synchronous boundary small:

def run() -> None:
    raise SystemExit(asyncio.run(main()))

This lifecycle matters in longer-running applications. Agent sessions own processes, connections, history, and sometimes temporary files. An exception should not leave those resources attached to a worker indefinitely.

Run the same repair

Start the agent against the included fixture:

safe-repo-agent ../fixture

It reads the source and tests, then proposes the same one-line change as the .NET version:

- return value.toLowerCase().replace(/\s+/g, "-");
+ return value.trim().toLowerCase().replace(/\s+/g, "-");

The write does not happen until the operator approves it. The later npm test request has its own prompt, so approving a patch does not grant standing permission to execute arbitrary commands.

The verified run ended with:

File changed: src/normalize-title.js
Fix: Added .trim() before .toLowerCase().
Validation: Both tests pass (npm test exit 0).

Run the deterministic policy tests separately:

pytest

All six policy cases pass without starting Copilot.

Adding OpenTelemetry

The Python GitHubCopilotAgent includes Agent Framework’s telemetry layer. For local exploration, the framework can configure console exporters before the agent is created:

from agent_framework.observability import configure_otel_providers

configure_otel_providers(enable_console_exporters=True)

In a service, export through OTLP to your normal observability backend and attach the repository job identifier to the surrounding trace. Useful signals include:

End-to-end run duration
Time waiting for human approval
Tool calls by permission kind
Denied actions
Command duration and exit status
Repair attempts and validation failures
Cleanup failures

Do not enable prompt and completion capture casually. Repository paths, source code, terminal output, and environment-related errors can contain sensitive information. Telemetry should explain the run without becoming another copy of every secret the agent could see.

From console tutorial to production worker

The console application demonstrates the control points, not a complete isolation platform. Before I would let it process untrusted repositories in a service, I would add:

A fresh container or microVM for every job
A read-only base image and disposable writable workspace
asyncio-aware timeouts and cancellation for CPU, memory, disk, and wall-clock limits, not just a number typed into a config file
Network deny-by-default
No ambient developer or cloud credentials
Short-lived repository credentials without merge permission
Durable approval records instead of terminal input
A bounded repair loop and diff-size limit
Cleanup and audit recording in finally, including cancelled tasks

The same concerns apply to a deterministic tool that runs repository scripts. The agent makes the risk easier to see because it chooses commands dynamically, but the repository and its dependencies were already untrusted executable input.

What differs from .NET?

The architectural boundaries did not change. Both implementations use the same harness, working directory, permission table, task, and fixture.

The Python version expresses the lifecycle with async with, passes Copilot session settings throughGitHubCopilotOptions, and moves blocking operator input to a thread. The .NET version usesCopilotClient, SessionConfig, typed permission request subclasses, and IAsyncEnumerable for streaming.

Those are ecosystem differences, not different safety models.

Building the same agent twice made one thing obvious: the language wrapping the model barely mattered. What mattered was the authority boundary around its tools, and whether “it worked” was backed by a diff and a passing test, or just the model’s own word for it.

Thanks for reading ✌️

References

Build a Safe Repository Maintenance Agent with GitHub Copilot and Microsoft Agent Framework in .NET

Sahan — Mon, 10 Aug 2026 22:27:00 +0000

Giving an AI agent access to a repository takes only a few lines of code. Giving it useful access without silently approving every command is the real tutorial.

In this post, we will build a .NET 10 console application that uses the GitHub Copilot harness as its coding runtime and Microsoft Agent Framework as the application-facing agent abstraction. The agent will inspect a small repository, repair a failing test, ask before changing a file, ask again before running the test command, and finish with a concise maintenance report.

It will not access the network, install packages, commit, push, or create a pull request.

This is a hands-on continuation of my earlier article about building an AI-assisted dependency vulnerability fixer. That system used an agent to handle repository-specific exceptions while keeping review and merge authority with the repository owner. Here, we will build the smaller execution boundary behind that idea.

💡 Code repository: The complete .NET and Python implementations use the same fixture and are available at github.com/sahansera/safe-repository-maintenance-agent.

What the two frameworks contribute

The GitHub Copilot SDK supplies the coding harness. It owns the agent loop and provides repository-oriented capabilities such as reading files, writing files, running shell commands, fetching URLs, and calling MCP tools.

Microsoft Agent Framework wraps that runtime in the same AIAgent abstraction used by its other providers. That gives the application a consistent run interface, streaming, sessions, middleware, and OpenTelemetry integration.

The distinction matters. We are not asking Agent Framework to recreate a coding loop around a chat model. Copilot remains responsible for planning and tool execution. Agent Framework gives us the surface on which the rest of the application can depend.

The Agent Framework integration itself is stable, but the GitHub Copilot SDK underneath it is still labeled public preview. I pinned exact versions below for that reason - the lower-level APIs moved twice while I was drafting this post, and I’d rather you hit a clean build than chase a breaking change mid-tutorial.

The repository we will repair

The completed sample accepts any repository path, including a checkout such as sahansera.dev. For the write demonstration, however, it includes a disposable fixture with no dependencies:

fixture/
├── AGENTS.md
├── package.json
├── src/
│ └── normalize-title.js
└── test/
    └── normalize-title.test.js

The implementation is deliberately wrong:

export function normalizeTitle(value) {
  return value.toLowerCase().replace(/\s+/g, "-");
}

The second test expects surrounding whitespace to be ignored. Running npm test gives us one pass and one failure because the actual result is -safe-repository-agent-.

This fixture gives the agent a real task with an objectively verifiable result. It also means nobody has to grant a first experiment write access to an important repository.

Prerequisites and project setup

You need:

.NET 10
An active GitHub Copilot subscription
The GitHub Copilot and Agent Framework integration packages

Here’s what I had installed when this worked:

<ItemGroup>
  <PackageReference Include="GitHub.Copilot.SDK" Version="1.0.9" />
  <PackageReference Include="Microsoft.Agents.AI.GitHub.Copilot" Version="1.17.0" />
</ItemGroup>

The SDK bundles its compatible Copilot runtime, so a separate global CLI installation is not required by the current .NET package. You still need to authenticate and have an active subscription.

Define the permission policy before creating the agent

The quickest demo is an approval callback that returns ApproveOnce() for everything. It is also a poor default for an application that can run commands and rewrite a checkout.

Our policy separates four decisions:

Capability	Default
Read inside the selected working directory	Approve once
Write a file	Ask the operator
Run a shell command	Ask the operator
Fetch a URL or call an MCP server	Deny

Unknown permission types are denied. A new SDK capability should not become authorized merely because the application has not been updated to recognize it.

public enum PolicyDecision
{
    Approve,
    Prompt,
    Deny,
}

public static PolicyDecision Decide(string permissionKind) => permissionKind switch
{
    "read" => PolicyDecision.Approve,
    "write" or "shell" => PolicyDecision.Prompt,
    "url" or "mcp" => PolicyDecision.Deny,
    _ => PolicyDecision.Deny,
};

This function contains no agent or console dependencies, so it is easy to unit test. The sample has six cases covering every known branch and the fail-closed fallback.

Permission handling is application policy, not prompt wording. Put it in ordinary code, test every branch, and deny capabilities you do not recognize.

Turn policy decisions into operator prompts

The Copilot SDK sends a typed PermissionRequest. That means we can show the operator the actual command, filename, diff, URL, or MCP tool instead of asking them to approve an unexplained action.

public static Task<PermissionDecision> HandleAsync(
    PermissionRequest request,
    PermissionInvocation _)
{
    PolicyDecision decision = PermissionPolicy.Decide(request.Kind);

    Console.WriteLine($"\n[permission: {request.Kind}]");
    Console.WriteLine(Describe(request));

    return decision switch
    {
        PolicyDecision.Approve =>
            Task.FromResult(PermissionDecision.ApproveOnce()),
        PolicyDecision.Deny =>
            Task.FromResult(PermissionDecision.Reject(
                "Blocked by the repository agent policy.")),
        _ => Task.FromResult(Prompt()),
    };
}

For a write request, Describe prints both the path and proposed diff. For a shell request, it printsFullCommandText. Approval is always for the current action rather than the whole session.

Do not treat the displayed command as a complete security parser. Shell syntax, symlinks, subprocesses, and package scripts make static classification difficult. The prompt improves operator judgment; the real containment boundary should still be a disposable sandbox with limited credentials and network access.

Scope the Copilot runtime to one repository

The application resolves the supplied path before starting Copilot:

string repositoryPath = Path.GetFullPath(args[0]);

if (!Directory.Exists(repositoryPath))
{
    Console.Error.WriteLine(
        $"Repository directory does not exist: {repositoryPath}");
    return 2;
}

We use that path for both the client process and the session:

await using CopilotClient copilotClient = new(new CopilotClientOptions
{
    WorkingDirectory = repositoryPath,
});

await copilotClient.StartAsync();

SessionConfig sessionConfig = new()
{
    WorkingDirectory = repositoryPath,
    EnableConfigDiscovery = false,
    OnPermissionRequest = ConsolePermissionHandler.HandleAsync,
    SystemMessage = new SystemMessageConfig
    {
        Mode = SystemMessageMode.Append,
        Content = instructions,
    },
};

This revised system instruction tells the agent to read the selected repository’s AGENTS.md itself. With ambient discovery disabled, a nested checkout no longer silently inherits instructions from a parent checkout.

A working directory also limits the paths Copilot considers available by default. It is useful scope, but it is not process isolation. I would still run an agent against an untrusted repository in a container with a non-root user, a disposable filesystem, no ambient cloud credentials, and a narrow network policy.

Create the Agent Framework agent and stream the result

With the client and session configured, the integration is one call:

AIAgent agent = copilotClient.AsAIAgent(sessionConfig);

await foreach (AgentResponseUpdate update in agent.RunStreamingAsync(task))
{
    Console.Write(update);
}

The instructions constrain the task further:

Work only inside the supplied working directory.
Read AGENTS.md before making changes.
Make the smallest change that satisfies the task.
Do not access the network, install packages, commit, push, or create a pull request.
Run focused validation and report the files changed, commands run, and result.

Those instructions improve agent behavior, but they do not replace the permission callback. The network prohibition exists in both places intentionally: the prompt tells the agent not to try, and the callback denies the capability if it does.

Run the repair

Start the sample against the fixture:

dotnet run --project src/SafeRepositoryAgent -- ../fixture

The agent reads the repository instructions and tests. When it proposes adding .trim(), the application prints the exact diff and pauses:

[permission: write]
File: .../fixture/src/normalize-title.js

- return value.toLowerCase().replace(/\s+/g, "-");
+ return value.trim().toLowerCase().replace(/\s+/g, "-");

Approve once? [y/N]

After approval, it asks separately before running npm test. The verified run finished with:

Changed: src/normalize-title.js - added .trim() before lowercasing.
Command: npm test - 2/2 tests pass.

The result matters because it is supported by a small diff and a repeatable test, not because the agent described itself as successful.

A useful maintenance result contains the proposed diff, the command that was allowed, and its exit status. The final natural-language answer is only a summary of that evidence.

What would change for a production service?

An interactive console prompt is appropriate for a tutorial and a developer workstation. A service needs a durable approval protocol instead.

I would keep the same policy function, then replace Console.ReadLine() with an approval record tied to a stable job and tool-call identity. The worker would pause, persist the request, notify an authorized reviewer, and resume only after receiving a valid decision. Every decision would be part of the audit trail.

I would also add:

A disposable container or microVM per repository job
Host-enforced CPU, memory, disk, process, and wall-clock limits, driven by a CancellationTokenrather than just requested of Copilot
Short-lived repository credentials with no merge permission
Network deny-by-default with explicit destinations where required
OpenTelemetry export, wired through the worker’s own lifetime, for agent runs, permission latency, tool calls, and failures
A maximum number of repair attempts before the job fails loudly instead of looping
A final diff-size limit and required validation commands
Deterministic cleanup via await using/finally, including failed and cancelled runs

Agent Framework emits OpenTelemetry-compatible telemetry, but traces can contain prompts, paths, commands, and model output. Keep sensitive-data capture disabled unless you have an explicit reason and a suitable storage policy.

The important part is outside the model

The agent itself did the easy part - a two-line fix, the kind of thing plenty of tutorials would stop at. What actually took the iteration was everything around that decision: scoping the working directory, closing the config-discovery hole, making every write and shell command visible before it ran, and keeping the test suite - not the model’s own summary - as the judge of success.

It’s a small application. But the boundary it draws is the whole point.

The Python version of this tutorial builds the same agent and fixture using async context managers, a typed options dictionary, and pytest. Keeping the task and permission table identical makes the differences between the two SDKs much easier to see.

Thanks for reading ✌️

References

I Grew an Engineering Blog from 0 to 463,000 Pageviews - Here's What Worked

Sahan — Tue, 04 Aug 2026 01:43:00 +0000

I published the first posts on sahansera.dev in December 2019.

There was no launch campaign, existing audience, or reliable stream of visitors waiting for them. Like most new personal sites, the blog started at zero. I wrote about problems I had encountered, shared the posts where I could, and hoped somebody searching for the same answers would eventually find them.

By 1 August 2026, the blog had accumulated 463,362 recorded pageviews.

Along the way, it has reached readers in at least 184 countries and territories. Here is where readers came from last month:

Earlier geography data is not included, so the lifetime reach may be broader.

That number is personally meaningful, but it is not the most interesting part of the story. The useful part is what happened underneath it: a small collection of practical engineering articles generated most of the traffic, some posts kept helping people for years, and many things I assumed would matter barely moved the numbers at all.

This is what I learned.

The short version

Specific, practical articles generated most of the traffic.
Evergreen posts kept growing for years after publication.
The blog continued reaching readers during a long publishing break.
Traffic revealed demand, but it did not tell me whether readers returned.
Comparing posts fairly requires consistent time windows, not raw lifetime totals.

From zero to 463,362 pageviews

I calculated the lifetime total using two consecutive analytics exports with no overlapping dates.

A few posts did most of the work

The distribution of traffic surprised me more than the total. Just five articles account for roughly 46% of all the pageviews the blog has recorded since launch.

The leading articles are remarkably consistent in what they offer:

Article	Lifetime pageviews
Simple In-Memory Caching in .NET with `IMemoryCache`	86,975
Understanding WebSockets with ASP.NET	43,272
Distributed Caching in ASP.NET Core with Redis	31,911
Having Fun with Microsoft IoC Container for .NET Core	27,812
Understanding the .NET Generic Host Model	23,040

Other durable performers cover securing the Hangfire dashboard, Kubernetes commands and arguments, and gRPC across Go and .NET.

These are not broad opinion pieces. Each one helps a developer understand a specific concept or complete a specific task.

The lesson is not simply that these technologies are popular. It is that clear intent compounds. A post answering a concrete question can remain useful every day for years, even when I am not actively promoting it.

💡 The pattern: Useful, specific posts can compound for years.

Evergreen technical writing compounds slowly

My highest-traffic article is an introduction to in-memory caching that I published in January 2020. Years later, it is still the largest entry point to the site.

That changed how I think about the return on writing.

A social post has a short distribution window. It may reach many people immediately and then disappear. A useful technical article behaves differently. It may receive very little attention on its first day, but it can be discovered repeatedly through search, links, code repositories, and recommendations.

The early results can feel underwhelming because the compounding is almost invisible. The article needs to be indexed. It needs to answer the query well enough for people to stay. Other pages need to link to it. Search engines need time to understand whether it is useful.

None of my successful posts felt like a breakthrough when I pressed publish. Their value accumulated quietly.

The blog kept working while life took priority

The growth was not driven by a perfectly consistent publishing schedule.

In 2023, I became a first-time dad. My family also moved into a new home, and I changed jobs. I wrote about that season in my plans for sahansera.dev in 2024, acknowledging that blogging had taken a back seat while I focused on being the best dad I could be.

The publication dates tell the story plainly. I published two posts in 2023, none in 2024, and returned with seven posts in 2025. I had hoped to resume a regular schedule sooner, but life had a different rhythm.

I am glad I did not treat the pause as a reason to abandon the blog. While I was not publishing, the existing articles continued answering questions, appearing in search results, and bringing new readers to the site. The work I had already done kept compounding when I did not have the time or energy to add more.

That makes the lifetime total more meaningful to me. It did not come from operating a content machine or forcing myself to publish through every season of life. It came from building a useful body of work, letting it breathe, and returning when I had something worthwhile to share.

💡 The pause: The blog kept growing even when publishing had to wait.

When I started writing again in 2025, I explored streaming APIs, HTTP internals, Python environments, and home-lab Kubernetes. The break had not erased the audience. It gave me a chance to return with different experiences and better questions.

Practical specificity beats broad ambition

The best-performing titles make a small promise:

Configure in-memory caching.
Build a gRPC server or client.
Run Kafka locally for testing.
Secure a Hangfire dashboard.
Understand how WebSockets work.

The posts are narrow enough for the reader to know why they should click, but substantial enough to teach the surrounding concepts.

This balance matters. A title such as "Everything You Need to Know About Distributed Systems" sounds ambitious but does not reveal which problem it solves. "Building a gRPC Server in Go" is less grand and much more useful to the person who needs exactly that.

My better articles tend to combine three layers:

A direct answer to a practical problem.
An explanation of what is happening underneath.
A working implementation readers can adapt.

That combination has become the clearest description of what I want this blog to be.

One successful article should become a cluster

For a long time, I treated each post as an isolated piece of work. The analytics show why that leaves value on the table.

The audience for an in-memory caching tutorial is likely to care about distributed caching, Redis, invalidation, cache stampedes, testing, and production failure modes. Someone building a gRPC server may next need a client, authentication, retries, deadlines, streaming, observability, or Kubernetes deployment guidance.

💡 The opportunity: A successful article is evidence that a useful topic cluster exists.

The gRPC series already demonstrates this. Its introduction, Go server and client, .NET server and client, and deployment posts reinforce one another. The individual posts can satisfy focused searches while the series gives interested readers a natural route through the broader subject.

I want to apply the same model to three areas:

.NET caching and application reliability
Kafka and event-driven system failure modes
Kubernetes operations and production troubleshooting

This does not mean publishing minor variations of the same article. Each post still needs a distinct problem and search intent. The connection between them should help a reader progress from a basic implementation to the difficult production questions.

Traffic is not the same as an audience

The historical reports showed that most visitors left after reading a single page. That sounds alarming until the context is considered.

Many visitors arrive from search, find a code sample or explanation, solve their immediate problem, and leave. For a reference-style engineering article, that can be a successful visit rather than a rejection.

At the same time, the data exposes a real weakness: I made it easy to consume one answer but did not always make the next useful step obvious.

Chronological previous-and-next links are not enough. A reader on a caching article probably does not want the post I happened to publish immediately afterward. They want the most relevant continuation of the problem they are already solving.

The improvements I am making are straightforward:

Add contextual links to related articles within the explanation.
Show a clear next step at the end of high-traffic posts.
Organise related material into visible series and topic hubs.
Keep the email and RSS subscription options easy to find.
Link runnable examples to maintained repositories.

The goal is not to trap somebody on the site. It is to make the site more useful when they want to go deeper.

💡 The real goal: Traffic becomes an audience only when readers have a reason and a way to return.

I measured traffic but not outcomes

Another uncomfortable lesson is that I collected a lot of traffic data without defining what success should mean beyond pageviews.

My current analytics setup contains no configured conversion events. I can see that people read an article, but I cannot reliably answer questions such as:

Did the article lead someone to another useful post?
Did they subscribe by email or RSS?
Did they visit the example repository?
Did they copy a code sample?
Which landing pages create returning readers?

These questions matter more now than the raw total.

Pageviews helped me understand which subjects have demand. The next stage is to measure whether the blog creates a relationship with the reader. I plan to treat a confirmed email subscription as the primary conversion, then track supporting actions such as RSS clicks, repository visits, code copying, deep scrolling, and movement between related articles.

Not every personal blog needs a conversion funnel. But if I want to improve something, I need to be explicit about what "better" means.

Analytics data needs maintenance too

Just over 35% of the views in the recent export have the page title (not set). The file also contains fragmented title variants and at least one suspicious spam-like title.

That is a useful reminder that analytics is not automatically a source of truth just because it contains precise-looking numbers.

Collection can break. Titles can change. A migration can alter definitions. Filters can create misleading exports. Bots and spam can contaminate reports. Privacy settings can change how users and sessions are identified.

I will use page paths as the canonical dimension for content reporting, investigate why page titles are missing, filter known noise, and verify that a single page view is recorded for each navigation. I also want Google Search Console beside Analytics so I can see queries, impressions, rankings, and click-through rates - not only the visits that already happened.

Measurement is part of maintaining the site, not something completed by pasting in a tracking ID once.

New posts need a fair evaluation window

Several of my newer articles have far fewer lifetime views than posts published in 2020 or 2021. That does not necessarily mean they failed.

An article published last month should not be compared directly with one that has accumulated search traffic for six years. Lifetime totals reward age.

For new work, I am moving toward a smaller set of time-normalised measures:

Search impressions and clicks during the first 28 and 90 days
Views per 30 days since publication
Engagement and code-copy actions
Movement to another related article
Email or RSS subscription actions
Whether traffic continues growing after the initial promotion window

This should make it easier to distinguish a promising article that needs time from one whose topic, title, or distribution genuinely missed the mark.

What 463,000 pageviews means to me

Four hundred and sixty-three thousand is not a huge number on the scale of the internet. It is huge compared with the zero visitors I had in December 2019.

More importantly, it represents individual moments when somebody had a problem and something I wrote may have helped them move forward.

The experience has changed my view of successful technical writing:

You do not need a large initial audience.
Useful, specific posts can compound for years.
A handful of durable articles may matter more than a constant publishing schedule.
Updating and connecting existing work can be more valuable than always starting from zero.
Honest measurement is more useful than the largest possible headline.

If you are starting an engineering blog with no visitors, that is normal. Write down the problem you just solved. Explain enough of the underlying system that the solution remains useful. Include the details you wish had been available when you were searching.

Then publish it and give it time.

That is more or less how this blog went from zero to 463,000 pageviews - one specific problem at a time.

If you write technical articles, what has surprised you most about the posts that keep finding readers? I would love to compare notes in the comments.

How We Replaced a Critical Data Path Without a Flag Day

Sahan — Mon, 03 Aug 2026 12:55:00 +0000

Replacing an API call is easy. Replacing the source of truth behind an automated decision is not.

I was reminded of this while migrating a critical workflow from a legacy event feed to a canonical state API. Both systems appeared to answer the same question: should the workflow act on this record now? But they had different schemas, different update timings, and, more importantly, slightly different models of the same lifecycle.

This was not a path where we could deploy the new code on Friday and watch the error rate. A false negative could leave work undone. A false positive could trigger an irreversible action from stale data. The HTTP request succeeding told us almost nothing about whether the new path was making the right decision.

So we did not treat it as a normal code replacement. We treated it as a controlled transfer of authority.

The migration used a query-only mode, shadow comparisons, discrepancy alerts, and a deliberately boring cutover. None of those techniques are particularly novel. What mattered was how we combined them, what we chose to compare, and how we decided the old path was finally safe to delete.

This post walks through that process.

The dangerous assumption: same data, newer API

The legacy path consumed a source built specifically around one class of lifecycle event. The replacement exposed a broader canonical record containing current state, relevant dates, and other attributes used by several workflows.

On paper, the migration looked like this:

That diagram hides the risky part. The two sources did not merely encode the same fact using different field names.

A purpose-built event feed tends to answer an event-shaped question: which transitions were recorded? A canonical API tends to answer a state-shaped question: what is true about this entity now?

Those questions overlap, but they are not identical.

An entity can have an old end-state event and later return to an active state. A record can contain several dates, each valid in its own business context. One source may update immediately while another catches up later. Missing data can mean “not applicable,” “not received yet,” or “something is broken.”

If we had translated fields one-for-one and switched traffic, the code would have looked correct while preserving none of those semantics.

The first useful decision was therefore to stop calling this an API migration. We were migrating a business decision from one model of the world to another.

Start by writing down the invariants

Before building the new path, we wrote down what the workflow must continue to guarantee.

The important invariants were roughly:

An active entity must never be processed because of a stale historical event.
A legitimate lifecycle transition must not be missed because one optional field is absent.
The effective date must have the same business meaning before and after migration.
Ambiguous or contradictory data must fail safely and become visible.
Reprocessing the same record must not produce duplicate downstream actions.

This step sounds obvious, but it changed the review entirely. Instead of asking whether the new client correctly parsed a payload, we could ask whether it preserved the rules the system existed to enforce.

It also exposed a subtle problem with “the old system is the source of truth.” If the legacy path had known defects, perfect agreement would reproduce them. The old output was a baseline, not an oracle.

That meant every mismatch needed investigation, but it did not mean the new path was automatically wrong.

Phase one: make the new path incapable of causing damage

The first version of the new integration was deliberately incomplete. It could query the canonical state API, normalize the response, and calculate what action it would take. It could not perform that action.

I think of this as query-only mode :

decision = evaluate_canonical_record(record)

if query_only:
    return decision

execute(decision)

The real implementation had more safeguards than this, but the boundary was just as explicit. Read and decide on one side; mutate on the other.

Query-only mode gave us a few useful properties:

We could run the code with realistic production data.
We could inspect decisions without creating tickets, sending notifications, or changing access.
We could debug authentication, pagination, missing fields, and schema assumptions separately from cutover risk.
We had a reusable operational tool for investigating individual records.

This last point was unexpectedly valuable. Migration controls are often treated as temporary scaffolding, but a safe read-only execution mode is also a good diagnostic interface. It lets an engineer ask, “What would the system do with this input right now?” without having to reproduce the entire workflow locally.

There was one rule we kept firm: query-only could not mean “mostly read-only.” If a code path still emitted an event or called a downstream service before checking the flag, the control was cosmetic. The no-side-effect guarantee had to sit at the boundary where side effects began.

Phase two: one writer, two decision-makers

Once the new path could evaluate real records safely, we ran it alongside the legacy implementation.

Only the legacy path was allowed to perform actions. The new path observed the same logical input and produced a candidate decision. We then compared the two.

This is usually called a shadow migration or dark launch. The new code sees production-shaped traffic but does not own production effects.

The obvious implementation compares two Boolean values:

legacy_should_process == new_should_process

That is useful, but not enough. When the values differ, a Boolean tells you that the migration is unsafe and nothing about why.

We made the decision explain itself:

Decision(
    should_process=False,
    effective_date=None,
    reason="currently_active",
    source_record_id="...",
)

Now the shadow result could compare:

Whether each path would act.
Which effective date it would use.
Why it reached that conclusion.
Which source record contributed to the decision.
Whether either path considered the input incomplete or ambiguous.

This is the part of the migration I would reuse everywhere: compare normalized decisions, not raw responses.

Raw payload comparison is noisy. One API may use null where another omits a field. Dates may use different time zones. Identifiers may refer to different resources. A hundred harmless representation differences can hide the one semantic difference that matters.

The normalized decision is the contract your users actually experience.

A mismatch is a finding, not just an error

As soon as the comparison ran against real data, discrepancies appeared. That was the point.

It is tempting to turn every mismatch into a page. I would avoid that. Early shadow traffic can be noisy, and training people to ignore an alert stream is a poor way to launch a critical system.

Instead, we recorded every mismatch with enough context to investigate it and grouped them into a small taxonomy:

Expected timing differences — one source had updated before the other.
Representation differences — the sources agreed, but normalization was incomplete.
Semantic differences — both records were valid, but the business interpretation differed.
Data-quality problems — records were missing, stale, or internally contradictory.
Implementation defects — our new code had selected the wrong field or applied the rule incorrectly.

This classification matters because each category leads to a different response.

A timing difference may need a grace period or a later recheck. A representation problem belongs in the adapter. A semantic difference needs a product or domain decision. Bad source data needs a defensive rule and an escalation path. An implementation bug needs code and a regression test.

Without a taxonomy, teams tend to “fix the diff” until the graphs become green. That can accidentally teach the new system to mimic legacy behaviour without understanding it.

The goal was not zero differences at any cost. It was zero unexplained differences.

The edge cases were the migration

The happy path agreed quickly. That did not make the migration nearly complete.

Two edge cases forced us to refine the new model.

The first involved a historical end-state event for an entity whose current state had since changed. If we looked only for the existence of that event, we could produce a false positive. The canonical record gave us another signal: the entity’s current status. We added that verification before allowing the workflow to proceed.

The second involved choosing the effective date. The new source exposed more than one plausible date, and the most conveniently named field was not necessarily the date the downstream process expected. We had to trace the business meaning through the old path and deliberately select the corresponding value.

Neither bug was difficult to fix once understood. The hard part was creating a migration that allowed us to see them before they became actions.

That is why I do not judge shadow migrations by the amount of traffic replayed. A million ordinary records can give more confidence than they deserve. One reactivation, one delayed update, or one contradictory date can tell you much more about whether the new model is correct.

Coverage should be measured across business scenarios , not only request counts.

For this kind of workflow I want an explicit scenario set:

Normal lifecycle transition.
Future-dated transition.
Reactivation or status reversal.
Missing optional attributes.
Conflicting dates.
Duplicate input.
Source timeout or partial response.
A record that changes while being processed.

Some scenarios will occur naturally during the shadow period. Rare but dangerous ones should be exercised with fixtures or controlled replay rather than waiting for production to provide them.

Deciding when to cut over

“The dashboards look fine” is not a cutover criterion.

Before transferring authority to the new path, we wanted evidence in several dimensions:

Correctness: no unexplained decision or effective-date mismatches.
Scenario coverage: important lifecycle transitions had been observed or tested.
Stability: the comparison stayed clean across a meaningful observation window, not just one quiet day.
Failure behaviour: timeouts, missing records, and contradictory data failed safely.
Observability: we could tell whether the new path queried, decided, skipped, failed, or acted.
Rollback: restoring the old authority was understood and quick.

There is no universal percentage or number of days that makes a migration safe. The right window depends on the frequency and consequence of the events you are trying to observe.

If an important edge case happens once a month, a clean afternoon tells you nothing about it. If the cost of a false positive is high, the threshold should reflect that asymmetry.

The cutover itself was intentionally uneventful. We changed which path was authoritative while retaining the ability to compare and roll back. We did not bundle unrelated cleanup into the same release. Boring is a feature when transferring control of a critical workflow.

Do not leave the old path “just in case”

After the new path had operated successfully, we removed the legacy implementation and the shadow comparison.

This can feel premature. Keeping the old path around appears to preserve a fallback. In reality, a dormant fallback decays quickly:

Its credentials and dependencies still need maintenance.
Engineers must continue reasoning about two implementations.
Future changes may update one path but not the other.
Someone can accidentally reactivate code that has not been exercised in months.
The temporary feature flag becomes permanent architecture.

A rollback path is valuable during migration. A second production system with no clear retirement date is technical debt.

We treated deletion as a planned migration phase rather than optional cleanup:

The order matters. Deleting before the soak period removes your quickest recovery option. Never deleting leaves you paying for the migration forever.

The comparison infrastructure should also be removed or deliberately repurposed. Shadow code often doubles reads, emits high-cardinality logs, and contains branching that the main workflow no longer needs. Once its question has been answered, it should not quietly become part of the permanent request path.

What I would carry into the next migration

The practical pattern is straightforward:

Define the invariants before translating fields.
Separate reads and decisions from side effects.
Give the new path a genuine query-only mode.
Keep exactly one writer while both paths evaluate.
Compare normalized decisions and their reasons.
Investigate and classify every meaningful mismatch.
Measure coverage across business scenarios, not only traffic volume.
Set evidence-based cutover and rollback criteria.
Keep the cutover small.
Delete the legacy path after a defined soak period.

The broader lesson is that migrations like this are not primarily data plumbing exercises. They are exercises in transferring trust.

The old path has years of accumulated behaviour, including behaviour nobody documented because the code made it seem obvious. The new source may be cleaner and more canonical, but that does not make your interpretation of it correct. Confidence comes from forcing both systems to make their decisions in the open, then explaining every place they disagree.

That takes longer than changing an endpoint. It is still much cheaper than discovering after cutover that a green deployment was making the wrong decision perfectly.

Thanks for reading ✌️

When "no healthy upstream" isn't about the upstream you think

Sahan — Thu, 23 Jul 2026 00:00:00 +0000

no healthy upstream is the kind of error that makes you expect wreckage.

Then you open the dashboards and find… almost nothing. CPU is low. No pods have crashed. The last deployment was hours ago. By the time you refresh the page, the service has recovered by itself.

That was the scene a few weeks ago when I started chasing an intermittent failure in a search backend. The eventual fix was only a few lines. The interesting part was getting there.

We already had a confident root-cause analysis (RCA), complete with a tidy explanation and a one-line remedy. It was also pointing at the wrong subsystem. This post is about the gap between a plausible story and the evidence, and about a common failure mode in which a mostly healthy fleet slowly removes itself from service.

The shape of the failure

The setup was ordinary: a search backend behind a load balancer, with a fixed pool of worker processes on each instance.

Every so often, with no obvious schedule, a burst of requests failed. The browser showed a bare no healthy upstream, and a minute or two later everything worked again.

One clue appeared every time. Backend p99 latency climbed to almost exactly the load balancer timeout, stayed flat, and then dropped back to normal:

That shape matters. Organic slowdowns tend to be uneven. This was a cliff, a flat top, and a recovery. Requests were not gradually becoming slower; they were running into a deadline and being cut off.

The theory I inherited was CPU throttling. A heavy periodic job supposedly pegged the pod’s CPU, the scheduler throttled it, request handling starved, and the load balancer eventually evicted the instance.

It was coherent. Better still, it came with a one-line fix: raise the CPU limit. That is an attractive combination during an incident. But a root-cause theory makes predictions, and these predictions did not survive contact with the evidence.

Treating the RCA as a hypothesis, not a conclusion

Instead of treating the existing RCA as a conclusion, I treated it as a hypothesis: if CPU throttling caused the incidents, what else should I be able to observe?

Three checks came back wrong.

There was no CPU-bound work in the serving path. The compute-heavy batch indexers ran on separate machines and reached the datastore over the network. They never ran inside the pods serving requests. A process outside the pod’s cgroup cannot cause that pod to be CPU-throttled. The graphs agreed: container throttling counters stayed flat during every event.

The timing did not fit a scheduled trigger. The batch jobs ran on a coarse schedule and produced one sustained utilisation bump. The incidents arrived at arbitrary minutes and happened far more often than the jobs ran. If a timer were responsible, the failures should have followed the timer. They did not.

The failures ignored instance and version boundaries. The same symptom appeared across different pods and deployment SHAs. A regression usually follows a version. A shared external event usually hits the fleet together. These failures did neither. The pattern looked more like each instance was doing something to itself, triggered by its own traffic.

To keep the CPU theory alive, I would have had to explain away the topology, the timing, and the distribution of failures. At that point the theory was creating more questions than it answered, so I dropped it and went back to the logs.

One line in the logs, and why it changes the model

The event window contained one recurring error:

ConnectionTimeout: Connection timed out

The stack ended in the datastore client, blocked on a socket read that never returned.

That one line flipped the model.

A CPU-throttled worker is ready to run but cannot get enough scheduler time. A worker blocked on I/O is off-CPU, waiting on the network while still occupying its worker slot. From the outside, both look like “latency went up, then requests timed out.” Underneath, they are opposites.

Adding CPU to an I/O stall does not unblock the socket. At best, it gives you more workers to park behind the same slow dependency. This is why edge symptoms are a dangerous thing to tune against: resource saturation and dependency blocking can produce the same fever while needing completely different treatment.

Why one slow dependency saturates the whole instance

The mechanism came down to two ordinary client settings that were dangerous in combination:

No timeout on individual datastore calls. One call could occupy a worker long after the user had given up.
Retries on connection timeouts. After waiting too long once, the worker would wait again, with backoff in between.

Nothing exotic. That is partly what makes this failure mode easy to miss.

Little’s Law shows why it is fatal to a fixed worker pool. Concurrency is L = λW: the arrival rate (λ) multiplied by the average time each request spends in the system (W).

Suppose the service receives 200 requests per second and normally responds in 40 ms:

L = 200 req/s × 0.04 s = 8 concurrent requests

Eight concurrent requests are easy for the pool to absorb. But if the datastore slows down and requests wait for tens of seconds, W increases by three orders of magnitude. Retries stretch it further. The required concurrency quickly exceeds the number of workers available.

The pool then looks something like this:

This creates head-of-line blocking. Workers parked on datastore calls cannot serve the fast requests behind them, so latency rises for everything, not only for requests that reached the slow dependency.

Health checks are caught in the same queue. They time out, the load balancer removes the instance from rotation, and the remaining instances receive more traffic. Once enough instances fail their health checks, the load balancer has nowhere to send the next request. That is when the user sees no healthy upstream.

There was one more twist. The client’s total retry time could exceed the load balancer’s deadline:

The load balancer returned an error while the worker continued retrying a result nobody could receive. Every millisecond after the outer deadline was wasted work, and the wasted work held a scarce worker slot.

This is a deadline-propagation failure. Inner operations must finish inside the outer request deadline. Better still, pass the outer deadline through the call chain so every layer knows when its result has become useless.

Why it recovered by itself

The recovery was initially reassuring. In hindsight, it was the worrying part.

The datastore blip triggered retries. Those retries added load to the datastore while it was already struggling, which caused more timeouts and therefore more retries:

That is the start of a metastable failure : a brief trigger knocks the system out of its healthy state, then a feedback loop keeps it unhealthy after the original trigger has passed.

We got lucky. The datastore blips were short enough that traffic fell below the tipping point before the retry loop became self-sustaining. A slightly longer blip could have kept the loop alive until we restarted the fleet or shed enough traffic to escape it.

Self-recovery was not proof of resilience. It was a warning shot.

The fix, and why we kept it small

The instinct during an availability incident is to add headroom: raise CPU limits, increase the worker pool, add replicas. That can help with genuine capacity problems. Here, it would only give the retry loop more workers to occupy.

The useful fix was to put a hard bound on the cost of one request.

First, add an explicit timeout to every downstream call. Choose it from the dependency’s healthy latency distribution rather than picking a pleasing round number. It should sit comfortably above healthy p99.9, but well below the load balancer’s deadline. A call that can wait forever is a worker you can lose forever.

Second, bound retries as a fraction of normal traffic, not as an unconditional count on every request. A rule such as “retry three times” allows every client to multiply traffic precisely when the dependency is least able to handle it. A token-bucket retry budget keeps the added load bounded; the Google SRE guidance uses 10% as an example. Add jitter too, or clients can wake up and retry in synchronised waves.

Third, be especially reluctant to retry connection timeouts. During saturation, a timeout often means the dependency is already over its limit. Another immediate attempt is unlikely to help. Retries make the most sense for independent, transient failures and only for idempotent operations. Search requests were idempotent, at least, so duplicate side effects were not another problem waiting for us.

The decision for each failed call becomes straightforward:

Circuit breakers, load shedding, and bulkheads could all strengthen this design:

A circuit breaker stops callers repeatedly rediscovering the same outage.
Load shedding rejects excess work early enough to keep the service responsive.
A bulkhead gives the dependency its own bounded concurrency pool, preventing it from occupying every worker.

All three add code, tuning, and operational state. We did not have evidence that we needed them yet. Once a small timeout and a bounded retry policy removed the amplifier, adding more machinery would have solved a hypothetical problem rather than the incident in front of us.

“Won’t failing fast just move the errors to the caller?”

This was the immediate pushback: if the backend gives up sooner, won’t users simply see more errors?

Only if we compare failing fast with a world in which every request eventually succeeds. That was not the world we had.

With the worker pool full of hung calls, every request eventually failed. The entire search feature became a no healthy upstream page, including requests that never needed the slow dependency in the first place.

Failing fast keeps the instances responsive and in rotation. It turns one correlated, fleet-wide outage into a smaller number of independent failures: the requests that actually hit the bad path. Those are failures a caller can absorb with a cached result, an empty state, or a retry button.

A nonessential component should not be able to take down the whole page. Give it its own deadline and a graceful fallback, and a slow dependency degrades one part of the experience instead of replacing the entire document with an error page.

Verifying the bound, not asserting it

I did not want to ship a fix that worked only on a whiteboard, so the validation was intentionally mechanical.

With a healthy dependency , normal traffic should remain normal: the same latency distribution and no new errors. If the timeout clips healthy p99.9 requests, it is too aggressive and will create the very failures it is meant to prevent.

With an unreachable dependency , requests should fail near the configured bound and before the outer load balancer deadline. More importantly, worker occupancy and in-flight request counts should remain flat instead of climbing.

That flat line under induced failure is the real acceptance test. It shows that the amplifier is gone.

The saturating case needs its own load test. The happy path will never prove that a service behaves well when every downstream call is stuck.

What I actually took from it

The technical lesson is easy to audit for: an unbounded downstream call plus eager retries is a latent outage with a feedback loop attached. It can sit quietly for months, green on every dashboard, until a dependency has a bad thirty seconds. Then it turns that blip into a fleet-wide event.

It is worth searching your own services for downstream calls without deadlines and retries without a shared budget. Those two settings deserve to be reviewed together, because together they can change the shape of a failure.

The lesson that stayed with me, though, was about diagnosis.

The CPU theory was clean. It was mechanistic. It came with a satisfying one-line fix. And it survived three pieces of contradictory evidence because we had started treating it as an answer instead of a claim.

A useful root cause makes predictions you can check. When the topology, timing, and distribution of failures all disagree with the story, elegance stops counting. In this case, one dull line in a log file told us more than the tidy explanation we had already grown attached to.

That was the expensive part of the incident: not the eventual three-line fix, but learning to let the evidence ruin a good story.

Thanks for reading ✌️

References

The Acknowledgment Gap - How Event-Driven Systems Lose Messages Without Errors

Sahan — Thu, 16 Jul 2026 01:05:00 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

If you have built event-driven systems for any length of time, you have probably internalized the mantra of at-least-once delivery: keep retrying until the work is done, and design everything downstream to be idempotent. It’s good advice. But there’s a subtle failure mode that hides right underneath it - one where your system faithfully reports success, commits its progress, and quietly drops work on the floor. No exception, no alert, no dead letter. Just a message that was supposed to do something, and didn’t.

I ran into this recently while debugging why a small percentage of events were mysteriously never being processed. Everything looked healthy. The producer got a 2xx. The consumer committed its offset. The dashboards were green. And yet the work never happened. This post is about that gap - the space between “the system accepted my request” and “the system actually did the work” - and why it’s one of the more dangerous places for a distributed system to lose data.

Motivation

Most write-ups about message-processing reliability focus on the well-known hazards: duplicate delivery, out-of-order messages, poison pills, consumer lag. Those are real, and there’s plenty written about them. What I found much less discussed is the case where the acknowledgment itself is a lie - where a component reports success for an operation that has only been accepted, not completed, and a second component treats that acknowledgment as permission to throw the original message away.

This is a design smell that shows up across all sorts of stacks: a queue consumer that calls an async API, a workflow engine that enqueues a job, a service that hands off to a background worker. Any time you have a handoff across an asynchronous boundary , you have the potential for this gap. So I want to walk through the anatomy of the bug from first principles, then talk about how to close it properly.

Background: two kinds of “yes”

Before we get to the bug, let’s be precise about acknowledgments, because the whole problem lives in some sloppy vocabulary.

When a system replies to your request, it can mean one of two very different things:

“I have accepted your request.” - I’ve durably recorded your intent, and I promise to try to do the work. Think HTTP 202 Accepted. The work hasn’t happened yet.
“I have completed your request.” - The work is done, and its effects are durable. Think 200 OK with a result body.

These are worlds apart, and conflating them is the root of a lot of pain. The trouble is that many APIs return the same status code for “accepted” whether or not the work will eventually succeed. A 202 (or a 204 No Content, which is even more ambiguous) tells you the request was received. It tells you nothing about whether the work will run.

Now layer on the consumer side. A huge number of event-driven systems are built on brokers that use offset-based consumer groups - Apache Kafka being the canonical example. If you want a primer, I wrote an introduction to Apache Kafka a while back. The mental model is simple:

Messages in a partition have monotonically increasing offsets.
Your consumer reads a message, does some work, and then commits (or “marks”) the offset to say “I’m done with everything up to here.”
If the consumer crashes before committing, the broker redelivers from the last committed offset. That’s what gives you at-least-once semantics.

💡 The offset commit is a promise about the past. When you commit offset N, you are telling the broker “every message up to and including N has been fully handled, and you never need to give them to me again.” If that statement isn’t actually true, you have manufactured data loss with your own hands.

Hold onto those two ideas - the ambiguous “yes” and the offset-as-promise - because the bug is what happens when they collide.

Anatomy of the pipeline

Let me describe a deliberately generic pipeline. Strip away the specific technologies and almost every async system looks like this:

The consumer’s job is to read an event and translate it into an action by calling some downstream control plane - an async job API, a workflow trigger, a task queue. The control plane accepts the request and, at some later point, a worker actually executes it.

The consumer’s loop, in pseudocode, looked roughly like this:

for {
    msg := broker.Read()

    err := dispatchJob(msg) // calls the async control plane
    if err != nil {
        retryWithBackoff(dispatchJob, msg)
    }

    // Always commit so we never get stuck reprocessing a bad message.
    broker.Commit(msg)
}

At a glance this looks reasonable, even defensive. There’s a retry with backoff. There’s a comment explaining that we always commit to avoid getting wedged on a poison message. Someone clearly thought about failure here.

And that is exactly what makes the bug so insidious.

The bug: the acknowledgment gap

Here’s the sequence that loses data.

The consumer reads a message and calls dispatchJob.
The control plane returns 204 No Content - “request accepted, a job has been created.” From the consumer’s point of view, this is success. err is nil.
Milliseconds later, and entirely outside the consumer’s view , the control plane rejects the job before it executes. Maybe a concurrency quota was exceeded. Maybe an admission controller said no. Maybe the queue was full. The job transitions straight to a terminal “rejected” state without a single line of business logic ever running.
Back in the consumer, dispatchJob returned nil, so the retry loop never fires - there was nothing to retry, as far as it knows.
The consumer commits the offset.

The offset moves forward. The broker will never redeliver that message. The job never ran. And nobody was told.

This is the acknowledgment gap. The failure happened in the window between “accepted” and “executed” , and our success signal was wired to the wrong end of that window. We treated “a job was created” as if it meant “a job will run,” and those are not the same statement.

💡 The most dangerous bugs aren’t the ones that throw. They’re the ones that return nil. An exception is a gift - it’s the system telling you something is wrong. Silent, structurally-invisible loss gives you nothing to catch.

What made it worse is that the “always commit” decision - added defensively to avoid an infinite reprocessing loop - turned a recoverable failure into an unrecoverable one. The one safety mechanism that could have saved us (letting the broker redeliver) was disabled precisely when we needed it.

Why the usual instincts don’t save you

When engineers first see this, they reach for familiar fixes. Most of them don’t actually close the gap:

“Just check the status code.” We did. It was 204. The status code describes the acceptance, not the outcome. The information we needed didn’t exist yet at the moment we got the response.
“Add a retry.” There was one. It only triggers on a failed dispatch, not a failed execution. You can’t retry something you don’t know failed.
“Make it idempotent.” Idempotency is necessary but not sufficient here. Idempotency protects you from doing the work twice; it does nothing to protect you from doing it zero times.
“Use exactly-once semantics.” Setting aside the long debate about whether exactly-once is even a coherent goal across independent systems - the transactional guarantees of your broker do not extend into a third-party control plane you’re calling over HTTP. The moment you cross that boundary, you’re back to coordinating two independent systems with no shared transaction.

The real issue is architectural: we committed our durable progress based on a signal that didn’t actually confirm the work was durable. No amount of tuning the individual pieces fixes that. You have to move the acknowledgment.

Fixing it: verify before you ack

The core principle is a single sentence:

💡 Never acknowledge a message until you have confirmed the work it represents has actually started (or completed) - not merely been accepted.

Everything else is mechanics. Let’s walk through them, because the mechanics are where the interesting distributed-systems problems hide.

1. Close the loop: confirm execution, don’t assume it

Instead of trusting the 204, the consumer now verifies that the dispatched job reached a real running (or terminal-success) state before committing. In practice that means polling the control plane’s read API after dispatch:

started, err := dispatchAndVerify(msg)
if started {
    broker.Commit(msg) // safe: the work is genuinely underway
} else {
    // do NOT commit - let redelivery give us another shot
    alert(msg, err)
}

dispatchAndVerify dispatches, then polls: did a job actually enter a non-rejected state? If it sees the tell-tale “rejected before executing” terminal state, or it can’t find the job at all within a bounded window, it treats that as a failure - which is the thing our original code could never see.

This is really just applying read-after-write thinking to a control plane. Don’t trust the write acknowledgment; go read the state back and confirm reality matches your intent.

2. The correlation problem

Here’s a genuinely tricky sub-problem that this exposes, and it’s a great example of why distributed systems are hard: the dispatch API often doesn’t tell you the ID of the thing it just created. You fire a request, you get back 204 No Content - literally no content - and now you need to find “the job I just created” among all the jobs.

You’re left correlating on secondary signals:

A creation timestamp window (“a job created after time T”), which is racy under concurrency - two near-simultaneous dispatches can be ambiguous.
A business key embedded into the job’s metadata at creation time, if the API lets you set something like a name or a label.

The robust fix is to make the work self-identifying : stamp a correlation key you already own (the entity ID, a request UUID) into the job at dispatch time, so that when you read the state back you can match on it exactly rather than guessing by time. If your control plane supports naming or tagging the work, use it. This is the async equivalent of propagating a trace ID, and it pays for itself the first time you have to debug a race.

💡 Any time you hand work across an async boundary, ask: “When this comes back, how will I know it’s mine?” If the answer is “by timestamp,” you have a race waiting to happen.

3. Bounded redelivery, or: don’t trade loss for a hot loop

The moment you say “don’t commit on failure so the broker redelivers,” someone will rightly point out the opposite failure mode: what if the work keeps failing? Now you’ve built an infinite reprocessing loop, and you’re hammering a control plane that’s already unhappy. This is the eternal tension:

Commit too eagerly → you lose messages (the original bug).
Never commit on failure → you can wedge the consumer forever.

The answer is bounded retries with escalation. Track how many times a given message has been through the wringer - keyed by its stable identity (partition + offset, or a business key) - and:

Retry with exponential backoff while attempts remain, so a transient quota exhaustion gets a chance to clear.
Once you’ve exhausted the budget, stop, escalate loudly, and then commit so a single doomed message can’t block the whole partition behind it.

That final commit is not “giving up silently” - it’s a deliberate, observable decision to route the message to a human (or a dead-letter queue) instead of blocking the stream. The difference between this and the original bug is everything: the original dropped work with zero signal; this drops it only after N visible, alarmed attempts.

4. Separate transient failures from terminal ones

A subtle but important refinement: when you poll to verify, a failure to read the state is not the same as the work being rejected. If your verification call itself hits a network blip or a 503, and you treat that as “the job failed,” you’ll re-dispatch and potentially create duplicate work - trading a lost-message bug for a double-processing bug.

So the verification loop needs to distinguish:

“I confirmed the job was rejected” → terminal, re-dispatch is warranted.
“I couldn’t reach the control plane to check” → transient, just retry the read, don’t re-dispatch.

Only definitive answers should drive irreversible decisions. Everything else is a retryable read. This is the same discipline as not making state transitions on ambiguous signals - you wait until you actually know.

5. Make the invisible visible

The reason this bug survived in production is that it was structurally unobservable. So the last piece is observability, and it’s not optional:

Emit a metric/event whenever a dispatched unit of work fails to start.
Alert when the retry budget is exhausted and a message is dropped.
Log the correlation key, the attempt count, and a link to the rejected work.

If your system is going to make a hard decision like “I’m dropping this after five tries,” that decision must be the loudest thing in the room, not a silent commit. A good rule of thumb: every place your code can decide to discard work should be capable of paging a human. You may choose not to page - but the capability being there forces you to consciously design the failure path instead of falling into one.

Stepping back: the general principle

If you zoom out from the specific mechanics, this whole class of bug reduces to a few reusable principles that are worth carrying into any distributed system you build:

Distinguish “accepted” from “completed” - always. Treat them as different events with different names, different metrics, and different handling. Never let one masquerade as the other.
Anchor your durable acknowledgment to the durable outcome. Commit your offset (or delete your message, or mark your row done) based on confirmation of the effect you care about, not on a transport-level receipt.
A nil error is a claim, not a fact. Verify claims that cross trust boundaries, especially async ones. Read-after-write is cheap insurance.
Bound every retry, and escalate at the boundary. Unbounded retries and silent drops are two sides of the same coin; the cure for both is a visible, finite budget with a loud exit.
Only act irreversibly on unambiguous signals. Transient “I don’t know” should never trigger a decision that assumes “no.”

None of these are novel on their own. What’s interesting is how a single missing distinction - accepted vs. executed - cascades into silent data loss when it meets an offset commit. It’s a good reminder that in distributed systems, the bugs rarely live inside a component. They live in the seams between components , where two reasonable local decisions add up to one unreasonable global one.

Tradeoffs

Nothing here is free, and I’d be doing you a disservice to pretend otherwise.

What you gain

No more silent loss - the failure mode that’s hardest to detect and most damaging to trust.
A verifiable, observable processing pipeline where “done” actually means done.

What it costs

Extra latency and load. Verifying execution means additional reads against the control plane per message. Poll intervals and budgets need tuning.
More moving parts. Correlation keys, attempt tracking, and escalation paths are code you now own and test.
You must embrace at-least-once for real. Verify-and-redeliver will occasionally produce duplicates (e.g., if a job actually started but your confirmation read failed). Idempotency downstream stops being optional - but that was always true; this just makes it honest.

For low-value, high-volume telemetry you might happily accept silent loss and skip all of this. For work where every single message must result in an action, the cost is obviously worth it. As always, the right answer depends on what the data is worth.

Conclusion

The most memorable bugs are the ones that teach you to distrust a word you’d been using carelessly. For me, that word was “success.” A 2xx is not success. A committed offset is not success. Success is the effect you actually wanted, confirmed to be durable. Everything else is just a system being polite.

If you take one thing away: go look at your event-driven pipelines and ask where you commit progress based on an acknowledgment rather than a confirmation. If those two things are wired together, you probably have an acknowledgment gap hiding in there too - quietly green on every dashboard, right up until someone asks where their data went.

Thanks for reading ✌️

References

Introducing gh-weekly-updates - Automate Your Weekly GitHub Impact Summaries

Sahan — Sat, 21 Mar 2026 22:22:00 +0000

If you are anything like me, you’ve probably spent a Friday afternoon trying to remember everything you did that week. Maybe it’s for a standup, a 1:1 with your manager, or just to keep track of your own progress. You end up clicking through PRs, issues, and Slack threads, trying to piece together a coherent story. It’s tedious, and honestly, it’s time you could spend doing actual work.

That’s why I built gh-weekly-updates - a CLI tool that automatically collects your GitHub activity and generates a structured weekly summary using AI.

💡 GitHub repo : github.com/sahansera/gh-weekly-updates. It’s open source and available on PyPI!

The Problem

As engineers, we’re constantly shipping code, reviewing PRs, filing issues, and jumping into discussions. But when it comes time to reflect on the week, all that context is scattered across repos. I wanted something that could:

Pull all my GitHub activity into one place
Summarise it in a way that highlights what actually matters
Run on a schedule so I don’t have to think about it

I couldn’t find anything that did exactly this, so I built it.

What It Does

gh-weekly-updates connects to the GitHub API, collects your activity for a given period, and sends it to an AI model (via GitHub Models) to produce a structured Markdown summary.

Here’s what it picks up:

Pull requests you authored (with merge status, additions/deletions, changed files)
Pull requests you reviewed
Issues you created
Issue comments you left
Discussions you started or participated in

The output is grouped by project or theme and structured into sections like Wins, Challenges, and What’s Next. You can also customise the prompt to match whatever format your team uses.

Getting Started

It’s a Python CLI tool, so you can install it with pip:

pip install gh-weekly-updates

Then just run it:

# If you're already logged in with the GitHub CLI
gh auth login
gh-weekly-updates

That’s it. It will auto-discover repos you contributed to in the past week and generate a summary.

You can also point it at specific repos and date ranges:

gh-weekly-updates --since 2026-02-09 --until 2026-02-16 --repos my-org/my-repo

Configuration

For more control, you can create a config.yaml:

org: my-org

repos:
  - my-org/api-service
  - my-org/web-app

model: openai/gpt-4.1

# Automatically push the summary to a repo
push_repo: my-user/my-weekly-updates

# Customise the AI prompt
prompt_file: my-prompt.txt

The config supports everything from repo lists to custom prompts. You can even swap out the AI model if you have a preference.

Running on a Schedule with GitHub Actions

This is where it gets really useful. You can set up a GitHub Actions workflow to run it every Monday morning and push the summary to a repo automatically:

name: Weekly Summary

on:
  schedule:
    - cron: '0 9 * * 1' # Every Monday at 9am UTC
  workflow_dispatch:

jobs:
  summarise:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - run: pip install gh-weekly-updates

      - name: Generate summary
        env:
          GITHUB_TOKEN: ${{ secrets.GH_PAT }} # must be named GITHUB_TOKEN
        run: gh-weekly-updates --config config.yaml

Now every Monday, you get a fresh summary committed to your repo. No manual effort required.

Custom Prompts

The default prompt produces a summary with Wins, Challenges, and What’s Next sections. But you can tailor it to your needs. For example, if your team does impact-style updates, you might want sections like Strategic Influence or Next Steps.

Just create a text file with your prompt and reference it in your config:

prompt_file: my-custom-prompt.txt

The prompt receives all your raw activity data as context, so you can shape the output however you like.

Why Open Source?

I initially built this for myself to automate my own weekly updates at work. But I figured other engineers probably have the same problem, so I cleaned it up and open-sourced it. The tool is intentionally simple - it does one thing and tries to do it well.

If you find it useful, give it a ⭐ on GitHub. And if you have ideas for improvements, PRs and issues are always welcome!

What’s Next

A few things I’m thinking about for future releases:

More activity sources : Picking up commit messages, release notes, and code review comments
Multiple output formats : Slack messages, email digests, Notion pages
Team summaries : Aggregate activity across a whole team, not just one person

If any of these sound interesting to you, feel free to open an issue or start a discussion on the repo.