DEV Community: Sahan

Replacing Sharp with WASM: How a Hidden SVG Dependency Broke Our Rendering Pipeline

Sahan — Fri, 21 Aug 2026 01:03:00 +0000

The failure, the cause, and the fix

Every Open Graph social card on the site was returning a 404 in production, even though the same code worked locally and in CI.

The production rendering path depended on sharp, libvips, and librsvg to turn Satori's SVG output into PNG bytes.

A runtime security setting made libvips's SVG loader unavailable, so sharp failed with:

Input buffer contains unsupported image format

The existing fallback failed too.

It simplified the card content, but still used the same rendering machinery. So when the renderer failed, the fallback immediately hit the same failure.

The fix had two parts:

Replace the sharp rasterisation path with resvg WASM.
Replace the dynamic fallback with a static image read from disk.

The primary path became deterministic across environments, and the fallback stopped sharing the primary renderer's failure mode.

The important bit: Make the primary path deterministic. Make the fallback independent of the capability that just failed.

Want the full debugging story?

The fix is fairly small once you know the cause.

Getting there wasn't.

I didn't start this investigation because someone reported broken social cards. I noticed unexplained errors repeatedly appearing on a Datadog dashboard, with no obvious user-facing symptom attached to them.

I started working backwards from the timestamps.

Datadog gave me the error clusters. Splunk gave me the request-level context. Correlating the two eventually led me to the Open Graph image requests and the production-only 404s.

From there, I had to work out:

why the route was failing
why a clean reproduction worked perfectly
what was different about the production runtime
why the fallback was dying milliseconds after the primary renderer

Here's the rabbit hole.

The 404 was coming from inside the route

The application used Next.js's file-based Open Graph support with an opengraph-image.tsx route.

A failing request looked roughly like this:

GET /profiles/<user>/opengraph-image-<hash>?<build>
-> 404 text/plain 9 bytes

Nine bytes mattered.

Not Found is nine bytes.

A genuinely missing route returned the much larger Next.js HTML 404 page.

This response was ours.

That meant the route was running and something inside it was throwing.

That gave me the first useful boundary in the investigation.

Now I needed to find what was throwing.

The renderer hid an SVG stage

My mental model going into this was basically:

JSX -> PNG

That was too simple.

In that production version, the actual path looked more like this:

Satori first turns JSX into SVG.

A second stage then has to rasterise that SVG into PNG bytes.

That intermediate format ended up being the most important detail in the incident.

The renderer also selected its rasteriser at runtime. Conceptually, it did something like this:

async function getSharp() {
  if (_sharp) return _sharp;

  try {
    _sharp = (await import("sharp")).default;
  } catch {
    return undefined;
  }

  return _sharp;
}

Then later:

const svg = await satori(element, { /* ... */ });

if (sharp) {
  pngBuffer = await sharp(new TextEncoder().encode(svg))
    .resize(width)
    .png()
    .toBuffer();
} else {
  const renderer = new Resvg(svg, { /* ... */ });
}

So the useful mental model wasn't "Next.js renders JSX into an image".

It was:

JSX -> SVG -> PNG

Those stages have different dependencies and different failure modes.

Hidden stages matter: When a library appears to do A to C, find out whether production actually depends on an intermediate B.

I ruled out the obvious explanations first

Before I had the useful stack trace, there were plenty of plausible explanations.

I worked through the obvious ones:

Bad JSX or CSS: rendered fine locally.
Specific profile data: real production-like data rendered correctly.
Missing fonts: present in the standalone build.
Missing WASM binaries: present.
Incorrect generated URL: the URL was correct.
Middleware intercepting the request: the request reached the image route.
Node version differences: the major version matched.

I also managed to manufacture one completely unrelated failure myself.

The generated Open Graph URL contained a build-hash suffix. At one point I requested the bare path instead, and Next.js quite correctly returned its normal HTML 404.

For about twenty minutes I thought I'd reproduced production.

I'd reproduced my typo.

Excellent progress.

The investigation only started moving when I stopped theorising and followed the errors through the logs.

Correlating Datadog and Splunk exposed the pattern

Datadog was where I first noticed the failure, but Splunk had the request context I needed.

For each cluster of errors in Datadog, I searched the same time window in Splunk and looked at what the application was doing around the failure.

That's how I connected what initially looked like generic rendering errors to the Open Graph image requests.

It also exposed a pattern I couldn't explain yet.

Primary renders and fallback renders were failing in pairs, milliseconds apart, on the same runtime instance.

For example:

primary: 32 failures
fallback: 32 failures

Later:

primary: 93 failures
fallback: 93 failures

Failures were also spread across different avatar sources, which made an avatar-specific input problem much less likely.

The paired failures would matter later.

First, I needed to know where the renderer was dying.

The stack trace moved the failure below Next.js

The line that changed the investigation was this:

Error: Input buffer contains unsupported image format
    at Sharp.toBuffer (.../sharp/dist/output.mjs:159:17)
    at render (.../@vercel/og/index.node.js:...)

That immediately narrowed the search space.

The request was going through sharp.

More importantly, the buffer sharp couldn't identify wasn't an uploaded avatar.

It was the SVG Satori had just generated.

The failing path was now much clearer.

Satori successfully produced SVG, and the native rasterisation branch died inside sharp before PNG output existed.

That changed the question from:

What's wrong with the profile image?

to:

Why can't sharp decode a valid SVG in production?

In this path, sharp hands the work to libvips. SVG support depends on the SVG loader being available there.

If no loader claims the input, the error you get is the wonderfully specific:

Input buffer contains unsupported image format

Now I had something concrete to reproduce.

The working reproduction narrowed the problem

I built a minimal reproduction with the same important pieces:

framework version
package manager
install flags
standalone output
Node version
base architecture

It rendered perfectly, including on two architectures.

That ruled out most of the build.

The framework version wasn't inherently broken. The standalone output wasn't inherently broken. The healthy container was taking the same sharp path successfully.

The remaining difference was much narrower:

the environment the container was running in.

Negative evidence is still evidence: A reproduction that doesn't reproduce can remove an entire class of possible causes.

A runtime security setting reproduced the exact failure

libvips can block operations it considers untrusted.

In my reproduction, enabling that restriction made the SVG path fail exactly like production:

$ docker run -e VIPS_BLOCK_UNTRUSTED=1 <image> node probe.js

sharp SVG->PNG: FAILED
Input buffer contains unsupported image format

Run the same image without the setting:

$ docker run <image> node probe.js

sharp SVG->PNG: OK
31270 bytes

Same image. Same application.

One runtime setting changed the result.

I could confirm the mechanism directly too:

import sharp from "sharp";

sharp.unblock({
  operation: ["VipsForeignLoadSvg"],
});

Once the SVG loader was allowed again, the render succeeded.

That proved the failure mechanism.

The Open Graph path produced SVG, then passed it into a native image stack where SVG parsing was unavailable at runtime.

There's one caveat worth calling out.

I never established what introduced that restriction into the production environment. I couldn't find it in the application source, container image, or deployment configuration I investigated.

I could prove the mechanism.

I couldn't prove its provenance.

That uncertainty influenced the fix.

The fallback used the same broken renderer

There was already a fallback renderer.

If the main card failed, the fallback removed most of the complicated inputs:

no avatar
no custom fonts
no profile text
no dynamic content

The assumption was reasonable.

If some unexpected input broke the renderer, render something simpler.

Except the input wasn't broken.

The renderer was.

The fallback still crossed the same rendering boundary as the primary:

Suddenly the paired error counts made sense.

Every time the primary failed because SVG parsing wasn't available, the fallback generated another SVG and sent it through the same broken machinery a few milliseconds later.

The fallback had removed complexity from the input.

It hadn't removed the failing dependency.

The fallback rule: A fallback that shares the primary's failure mode isn't independent. It's the same system with fewer features.

I made the fallback deliberately boring

The replacement fallback doesn't render anything.

It reads a pre-generated image from disk:

const FALLBACK_CARD_CONTENT_TYPE = "image/jpeg";

export async function renderFallbackSocialCard(): Promise<Response> {
  const bytes = await readFile(
    join(process.cwd(), "public/card.jpg"),
  );

  return new Response(new Uint8Array(bytes), {
    headers: {
      "Content-Type": FALLBACK_CARD_CONTENT_TYPE,
    },
  });
}

That's it.

No Satori. No SVG. No sharp. No libvips.

It's boring on purpose.

If the dynamic rendering stack catches fire, the fallback shouldn't politely walk back into the fire.

Why I replaced sharp with resvg WASM

For the main renderer, I had two practical options.

Option 1: allow SVG parsing again

Mechanically, this was the quickest path.

But I still didn't know why the production environment had restricted that capability in the first place.

I didn't want an Open Graph feature to override an environment-level security control just to preserve the existing implementation, especially when I could remove that dependency from the path entirely.

Option 2: remove libvips from this path

The other option was to remove libvips from this rendering path entirely and make resvg WASM the explicit rasteriser.

That's the route I took.

The implementation was relatively small:

import { readFile } from "node:fs/promises";
import { join } from "node:path";
import { initWasm, Resvg } from "@resvg/resvg-wasm";
import satori from "satori";

const WASM_PATH =
  "node_modules/@resvg/resvg-wasm/index_bg.wasm";

let wasmReady: Promise<void> | undefined;

function initResvg(): Promise<void> {
  wasmReady ??= readFile(
    join(process.cwd(), WASM_PATH),
  ).then(initWasm);

  return wasmReady;
}

export async function renderElementToPng(
  element: ReactNode,
  { width, height, fonts }: RenderOptions,
): Promise<Buffer> {
  const svg = await satori(element, {
    width,
    height,
    fonts,
  });

  await initResvg();

  const resvg = new Resvg(svg, {
    fitTo: {
      mode: "width",
      value: width,
    },
  });

  try {
    const rendered = resvg.render();

    try {
      return Buffer.from(rendered.asPng());
    } finally {
      rendered.free();
    }
  } finally {
    resvg.free();
  }
}

The important change wasn't the syntax.

It was the runtime behaviour.

resvg became the one explicit rasteriser.

No native image loader in this path. No environment-dependent SVG decoder. No silent rasteriser selection.

The WASM fix created a packaging problem

Of course, replacing the renderer wasn't the end of it.

Reading the WASM file via require.resolve() made the bundler try to process it as a module, which failed.

Reading the binary directly by path avoided that problem, but then Next.js's dependency tracer couldn't infer that the file was required at runtime.

The standalone output didn't contain it.

I had to make that dependency explicit:

serverExternalPackages: ["@resvg/resvg-wasm"],

outputFileTracingIncludes: {
  "/**": [
    "./node_modules/@resvg/resvg-wasm/index_bg.wasm",
  ],
},

Configuration like this is exactly the sort of thing that gets removed six months later because nobody remembers why it's there.

So I added a build assertion too:

RUN test -f .next/standalone/node_modules/@resvg/resvg-wasm/index_bg.wasm \
    || { echo "resvg wasm missing from standalone output"; exit 1; }

Then I verified the assertion in both directions.

Remove the tracing configuration and the build fails.

That turns a hidden runtime dependency into something the build can enforce.

One green test had never rendered an image

The migration also exposed a test problem.

There was already a test that appeared to cover image rendering. It created an ImageResponse and asserted against it.

The problem was that ImageResponse rendered lazily.

Constructing the response didn't force Satori to execute.

The test was green because the failing operation had never happened.

Once I changed the path to render eagerly, the test failed immediately.

A test called "render image" that never renders the image is a particularly optimistic form of testing.

I verified the real renderer, not just a successful response

The static fallback introduced one final trap.

Once it shipped, every Open Graph request returned a valid-looking image again.

From the outside, the feature looked healthy.

That still didn't prove the dynamic renderer was working.

It could remain completely broken while every request quietly returned the fallback.

So I verified the responses themselves.

Before the renderer fix

Property	Result
Content type	`image/jpeg`
Size	`425,081` bytes every time
Requests	`45/45` fallback

After the renderer fix

Property	Result
Content type	`image/png`
Size	roughly `143KB` to `245KB`
Requests	`45/45` dynamically rendered

The fallback responses were byte-for-byte identical.

The dynamic cards varied with each profile.

That variation was evidence that I was exercising the real renderer again.

Verification matters: A valid response only proves that some path worked. Verify the behavioural property that distinguishes the path you actually fixed.

What I'd do earlier next time

A few things from this investigation are going straight into my debugging playbook.

Correlate telemetry before guessing

The Datadog errors weren't very useful by themselves.

Correlating their timestamps with Splunk turned an unexplained counter into an actual request path.

Look for hidden stages

"JSX to PNG" hid an SVG conversion with its own runtime dependencies.

The intermediate representation was where the incident lived.

Make implementation selection explicit

A library silently choosing between implementations is convenient until production takes a different path from your laptop.

For an important path, the chosen implementation should be observable.

Design fallbacks around failure boundaries

Removing input complexity helps when the input is the problem.

It doesn't help when the capability underneath it has disappeared.

A useful fallback should avoid the failed step wherever practical.

The lesson I kept

Technically, this incident came down to an SVG loader being unavailable inside the production runtime.

That's not the part I expect to remember.

I started with unexplained errors on a Datadog dashboard, correlated them against Splunk, connected them to broken Open Graph images, followed the failure through Satori and sharp, reproduced the runtime behaviour in isolation, and replaced the native rasterisation path with WASM.

But the most useful discovery was the fallback.

I'd assumed it gave the feature resilience because it rendered something simpler.

It didn't.

It depended on the exact capability that had just failed.

The fallback wasn't independent. It was a smaller version of the same pipeline.

Thanks for reading ✌️

References

When Kubernetes Pods Try to Run One Database Migration

Sahan — Fri, 14 Aug 2026 22:23:00 +0000

This startup command looked fine while our application had one replica:

prisma migrate deploy && node server.js

Then we scaled the Deployment to three pods.

All three pods started together. All three ran the migration command. Prisma used a PostgreSQL advisory lock to protect the migration, so the database was not corrupted, but the losing pods waited for the lock and eventually timed out. Kubernetes saw failed containers, restarted them, and filled the rollout logs with errors.

One pod was doing useful work. The other two were turning expected contention into application failures.

The clean solution is a migration Job that runs before the Deployment. Our release platform could only start one application workload, so we used a small PostgreSQL-backed coordinator instead.

This post shows the complete pattern.

The problem in one picture

A Deployment creates copies of a long-running process. A database migration is one-shot work.

The coordinator gives every pod the same startup protocol:

Try to acquire one application-specific advisory lock.
If the lock is busy, close the connection, wait with jitter, and try again.
If the lock is acquired, run prisma migrate deploy.
Release the lock and start the application.

Every pod runs the migration command after acquiring the lock. For the first pod it applies pending migrations. For later pods it is a no-op. This matters because a released lock only proves that the previous database session ended. It does not prove the previous migration succeeded.

Reconcile after waiting

A released lock does not mean the previous migration worked. Every pod that acquires the lock should reconcile the database to the expected migration state.

Use a Job when your platform supports one

Before adding coordination code, check whether the delivery system can run an ordered pre-deployment Job.

A Job is the better default because it gives the migration its own status, logs, image and database credentials. The application Deployment does not need schema-changing permissions, and a failed migration stops the rollout at the correct boundary.

The coordinator below is for the awkward case where the platform can only deploy the application workload. That was our constraint.

Stay in the loop

Enjoying this article?

Get practical engineering notes in your inbox, usually once or twice a month.

Get new posts →

Add the coordinator

The application already contained Prisma. The coordinator only needed the PostgreSQL client:

npm install pg

Create prisma/migration-coordinator.mjs:

import { spawn } from "node:child_process";
import { createHash } from "node:crypto";
import { fileURLToPath } from "node:url";
import pg from "pg";

const { Client } = pg;

const connectionString = process.env.DIRECT_DATABASE_URL;
const lockName = process.env.MIGRATION_LOCK_NAME ?? "web:public:prisma-migrations";
const lockId = createHash("sha256")
  .update(lockName)
  .digest()
  .readBigInt64BE(0)
  .toString();
const waitTimeoutMs = Number(process.env.MIGRATION_WAIT_TIMEOUT_MS ?? "300000");
const retryDelayMs = Number(process.env.MIGRATION_RETRY_DELAY_MS ?? "2000");
const prismaCli = fileURLToPath(new URL("../node_modules/.bin/prisma", import.meta.url));

if (!connectionString) {
  throw new Error("DIRECT_DATABASE_URL is required");
}

function sleep(ms) {
  return new Promise((resolve) =&gt; setTimeout(resolve, ms));
}

function runPrismaMigrateDeploy() {
  return new Promise((resolve, reject) =&gt; {
    const child = spawn(prismaCli, ["migrate", "deploy"], {
      stdio: "inherit",
      env: {
        ...process.env,
        DATABASE_URL: connectionString,
      },
    });

    child.once("error", reject);
    child.once("exit", (code) =&gt; {
      if (code === 0) {
        resolve();
        return;
      }

      reject(new Error(`prisma migrate deploy exited with code ${code}`));
    });
  });
}

const deadline = Date.now() + waitTimeoutMs;
let migrationComplete = false;

while (!migrationComplete) {
  const client = new Client({
    connectionString,
    connectionTimeoutMillis: 10_000,
  });
  let acquired = false;

  try {
    await client.connect();

    const result = await client.query(
      "SELECT pg_try_advisory_lock($1::bigint) AS acquired",
      [lockId],
    );
    acquired = result.rows[0].acquired === true;

    if (acquired) {
      console.log("migration lock acquired");
      await runPrismaMigrateDeploy();
      console.log("database migrations are current");
      migrationComplete = true;
    }
  } finally {
    if (acquired) {
      await client
        .query("SELECT pg_advisory_unlock($1::bigint)", [lockId])
        .catch((error) =&gt; console.error("failed to release migration lock", error));
    }

    await client.end().catch(() =&gt; {});
  }

  if (migrationComplete) {
    break;
  }

  if (Date.now() &gt;= deadline) {
    throw new Error("timed out waiting for the migration lock");
  }

  const jitterMs = Math.floor(Math.random() * 500);
  console.log("migration lock busy; waiting before retry");
  await sleep(retryDelayMs + jitterMs);
}

There are four details worth keeping:

pg_try_advisory_lock returns immediately instead of keeping a waiting connection open.
The same database connection stays open while the Prisma subprocess runs. Session-level advisory locks belong to the connection that acquired them.
Every lock owner runs migrate deploy. Already-applied migrations make it a no-op.
A failed migration rejects the script, so the application does not start against an unknown schema.

Use a direct PostgreSQL connection for DIRECT_DATABASE_URL. Transaction-mode poolers do not preserve session state, so they are the wrong boundary for a session-level advisory lock.

The lock name is not a secret, but it must be stable. The script hashes the readable name into the signed 64-bit integer PostgreSQL expects. Every revision of this application should use the same name for the same database and schema. Different applications sharing a database should use different names.

After adding the coordinator, expected contention becomes an ordinary waiting path:

Run it before the server

Update the container command so the server starts only after coordination succeeds:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: registry.example.com/web:abc123
          ports:
            - name: http
              containerPort: 3000
          command: ["sh", "-c"]
          args:
            - node prisma/migration-coordinator.mjs &amp;&amp; exec node server.js
          env:
            - name: DIRECT_DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: web-database
                  key: direct-url
            - name: MIGRATION_LOCK_NAME
              value: "web:public:prisma-migrations"
            - name: MIGRATION_WAIT_TIMEOUT_MS
              value: "300000"
          startupProbe:
            httpGet:
              path: /health/startup
              port: 3000
            periodSeconds: 5
            failureThreshold: 72
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 3000
            periodSeconds: 5

exec replaces the shell with the Node process after the migration finishes. That lets the application receive termination signals directly as PID 1.

The coordinator calls the Prisma binary already installed in the image. Keep the Prisma CLI in the production image rather than downloading it during startup. If your image uses a different project layout, adjust prismaCli to match it.

The startup probe allows six minutes for lock waiting, migration and application startup. Set that budget from your slowest expected migration rather than copying this number. Readiness still decides when the running pod can receive traffic.

What happens during a rollout

With three new pods, the rollout now looks like this:

pod-a migration lock acquired
pod-b migration lock busy; waiting before retry
pod-c migration lock busy; waiting before retry

pod-a database migrations are current
pod-a application starts

pod-b migration lock acquired
pod-b no pending migrations to apply
pod-b application starts

pod-c migration lock acquired
pod-c no pending migrations to apply
pod-c application starts

Expected contention is logged as waiting, not as a migration failure. If the winning pod disappears, PostgreSQL releases its session lock when the connection closes. Another pod acquires the lock and runs migrate deploy again.

This has the same shape as the shared-state problem in Scaling an Agentic Coding SDK: What Concurrency Actually Costs. Sequential execution had hidden an ownership problem. Adding replicas made it visible.

The lock does not make breaking migrations safe

The advisory lock serialises migration commands. It does not stop old pods from serving traffic while a new pod changes the schema.

Rolling deployments still need expand-and-contract migrations:

Add the new schema in a backward-compatible form.
Deploy code that works with both representations.
Backfill data if required.
Switch reads and writes.
Remove the old schema after old pods are gone.

I covered the same compatibility problem in more detail in Replacing a Critical Data Path Without a Flag Day.

Coordination is not compatibility

The lock keeps migration commands from colliding. Expand-and-contract changes keep old and new application versions compatible during the rollout.

Before shipping it

Check these six things:

A pre-deployment Job really is unavailable.
The lock uses a direct PostgreSQL session.
The lock name is stable and unique to the application and schema.
Waiting has jitter and an overall deadline.
A failed migration prevents the application from starting.
Schema changes remain compatible with old pods during the rollout.

That is the whole pattern. One pod applies the migration. The others wait, check the same desired state, and then start normally.

The code is small because PostgreSQL already owns the hard guarantee: only one session can hold the advisory lock at a time.

Thanks for reading ✌️

References

Scaling an Agentic Coding SDK: What Concurrency Actually Costs

Sahan — Wed, 12 Aug 2026 11:51:00 +0000

I previously wrote about building an AI-assisted dependency vulnerability fixer. It used the GitHub Copilot SDK to start a repository-scoped coding agent for the fixes that could not be handled reliably by a deterministic packaging change. Each agent took a known finding, made the smallest safe change it could, validated the result, and opened a pull request for the repository owner.

The .NET and Python tutorials isolate the permission boundary behind this kind of work: one agent, one repository, and explicit approval for side effects. This article looks at the next operational problem from the production fixer - what changes when many such workloads run concurrently.

The first version processed one repository at a time. A representative batch of 30 repositories took roughly 46 minutes. With a worker pool of five, it took about 12.

Replacing the sequential loop was the easy part. Before those five agents could run safely, each needed an isolated workspace, reliable cleanup, resource limits, rate limiting, and enough telemetry to show whether more concurrency helped. Multiple service replicas also needed durable job ownership and safe retries.

That work changed my mental model:

The mental model

Agent concurrency is not a larger number of requests. It is a larger number of partially trusted computers operating at once.

An agent session is a workload, not a request

When I say “agent” in this article, I mean one live Copilot SDK client and session operating against one repository. In this setup, the client started the Copilot runtime and the session drove its model and repository tools. It was not a name for one model API request.

An HTTP request usually borrows a connection, does bounded work, and returns. A Copilot session can live for minutes and contain many model and tool turns. It keeps a transcript in memory, drives a shell, mutates a checkout, creates subprocesses, uses upstream capacity, and may hold credentials that can create branches and pull requests.

This introduces familiar distributed-systems questions. What isolates jobs? Who owns each resource? What happens when a worker disappears? Can a job run twice? Which resource sets the safe concurrency limit? Did a timed-out operation fail before or after changing an external system?

The novelty is in who chooses the commands. The operational concerns are not new.

Three decisions need to remain separate:

Isolation: Each repository gets its own session and workspace, ideally inside a disposable sandbox.
Concurrency: The worker count is bounded by the first resource likely to run out, not simply CPU.
Persistence: Agent sessions can be ephemeral while job identities, attempts, and external effects remain durable.

Sequential execution had been protecting shared state

The first implementation was intentionally boring:

for (const repository of repositories) {
  await fixRepository(repository);
}

It helped validate prompts, permissions, branch strategy, and review before concurrency obscured product mistakes. It also hid unsafe assumptions.

The prompt cloned every repository into /tmp/agent-workdir. With two agents, one install could rewrite the lockfile another was preparing to commit. One cleanup could delete the other job’s files.

The service now allocates the path and passes it to the agent as job data:

import { randomUUID } from "node:crypto";

const safeName = `${repository}-${branch}`.replace(/[^a-zA-Z0-9_-]+/g, "-");
const workdir = `/tmp/agent-${safeName}-${randomUUID()}`;

The same audit applies to ports, branches, cache keys, and temporary filenames. Any literal value becomes shared state when two jobs can use it.

A unique directory prevents accidental overlap, but it is not a security boundary. Repository-controlled code still needs a disposable sandbox with bounded access to the host and network.

The service did not call an LLM endpoint directly. For each job, it started a GitHub Copilot SDK client inside the unique working directory and created one agent session. Cleanup also needed to survive failure. The original code disconnected the session only after successful work, so an exception leaked both session and runtime resources.

A simplified version of the corrected lifecycle looks like this:

import { CopilotClient, type CopilotSession } from "@github/copilot-sdk";

async function runCopilotRemediation(job: Job, workdir: string) {
  const client = new CopilotClient({ workingDirectory: workdir });
  let session: CopilotSession | undefined;

  try {
    await client.start();
    session = await client.createSession({
      systemMessage: {
        mode: "append",
        content: remediationInstructions(job),
      },
      onPermissionRequest: remediationPolicy,
    });

    return await session.sendAndWait({
      prompt: buildRemediationTask(job),
    });
  } finally {
    await session?.disconnect();
    await client.stop();
    await removeWorkspace(workdir);
  }
}

client.start() launches or connects to the Copilot runtime. createSession() gives the job its own transcript and agent loop. sendAndWait() lets that loop continue through repository reads, edits, shell commands, and validation until the session becomes idle. A worker pool of five can therefore mean five Copilot runtimes, five sessions, five mutable checkouts, and all of their subprocesses operating at once.

The component that acquires a resource owns its lifecycle. Concurrency makes violations more frequent, not more complicated.

Stay in the loop

Enjoying this article?

Get practical engineering notes in your inbox, usually once or twice a month.

Get new posts →

Bound concurrency with evidence

This is concise but unsafe:

await Promise.all(repositories.map(fixRepository));

It lets the input size set infrastructure policy. Thirty jobs may work; 300 may exhaust memory, fill disk, or trigger rate limits.

I used a small worker pool instead:

async function runPool(
  items: T[],
  limit: number,
  run: (item: T) =&gt; Promise,
) {
  let next = 0;

  async function worker() {
    while (next &lt; items.length) {
      const index = next++;

      try {
        await run(items[index]);
      } catch (error) {
        recordFailure(items[index], error);
      }
    }
  }

  await Promise.all(
    Array.from({ length: Math.min(limit, items.length) }, worker),
  );
}

The cap is explicit, one repository failure does not cancel unrelated work, and the batch drains before completion is reported.

I started with five workers. Five is not a generally safe number for coding agents. It was a conservative operating point for this workload.

The real cap is the smallest limit implied by memory, ephemeral storage, subprocesses, file descriptors, provider requests and tokens, source-control operations, network bandwidth, spend, and acceptable blast radius. For memory, the rough calculation is:

memory_cap = floor(
  (container_limit - service_baseline - safety_headroom)
  / p95_incremental_memory_per_session
)

A job-level cap does not replace API rate limiting. Five agents can still push branches or create pull requests simultaneously. The source-control client must independently honour rate-limit headers, Retry-After, and backoff.

Capacity needs evidence

“Five at a time” is not a capacity plan. “Five because memory is the first measured budget we approach” is.

Local limits stop being global limits

A process-local cap works only while there is one process. With a cap of five and four replicas, the service can create 20 live sessions:

effective_concurrency = replicas × per_replica_cap

An autoscaler can increase that number precisely when an upstream system is already under pressure.

At this point, job ownership moves to a durable queue or database table. A worker atomically claims a job for a limited period, renews the lease while running, and records the outcome before acknowledging completion. If it disappears, the lease expires and another worker can retry. A global limiter protects shared provider and credential budgets.

The agent session and workspace remain disposable. The durable state is the job identity, lease, attempt count, and record of external effects.

Retries need reconciliation. Suppose the agent opens a pull request, but the response is lost before the job records success. A retry can create a duplicate.

I gave each remediation a stable idempotency key derived from the repository and requested change. It enforces one active job, supports a stable branch name, and lets a retry find an existing branch or pull request. An atomic claim or uniqueness constraint closes the race that an existence check alone cannot.

Stateless workers are useful. Statelessness means any worker can continue the protocol, not that the system remembers nothing.

Concurrency needs gauges and guardrails

Total batch duration is not enough to tune the pool. I needed queue wait and execution time separately, active sessions, peak memory, workspace size, subprocess count, cost per job, upstream throttling, retries, cleanup failures, and orphaned sessions.

Three questions make those measurements useful:

Does active work regularly reach the cap?
Is queue time growing while constrained resources still have headroom?
Do failures, latency, throttling, or resource pressure rise with inflight work?

If the pool never fills, a higher cap will not help. If queues grow while resources remain healthy, there may be room. If failures rise with inflight work, the system has found a boundary.

Security belongs in the same discussion. The dependency fixer already treated repositories, install scripts, and tests as untrusted input. Concurrency multiplies that exposure. Each job needs bounded CPU, memory, processes, disk, time, network access, and short-lived repository-scoped credentials. It should have no ambient infrastructure credentials or permission to merge.

Audit trails also need redaction. Terminal output, environment dumps, remote URLs, and package-manager logs can contain credentials.

Why five workers were not five times faster

For one representative batch of 30 repositories:

Phase	Sequential	Pool of five
Clone and dependency setup	~10 minutes	~4 minutes
Agent inspection and editing	~22 minutes	~5 minutes
Push and pull-request creation	~12 minutes	~3 minutes
Artificial inter-job delay	~1.5 minutes	0
Total	~46 minutes	~12 minutes

That is about a 3.8-times speedup. It is an operational measurement, not a benchmark. Removing an artificial delay contributed, and the rest was limited by uneven job duration, disk and network contention, provider latency, and source-control operations.

The goal is not maximum concurrency. It is the best useful throughput inside the safety, cost, and reliability budgets.

I would scale the next service in the same order: isolate each job, make cleanup unconditional, add a bounded local pool, measure it, then introduce durable claims and reconciliation before adding replicas.

The worker pool reduced this batch from roughly 46 minutes to 12. Everything else was the cost of making that number safe.

That is what concurrency actually costs.

Thanks for reading ✌️

References

Build a Safe Repository Maintenance Agent with GitHub Copilot and Microsoft Agent Framework in Python

Sahan — Mon, 10 Aug 2026 22:30:00 +0000

In the .NET tutorial, we built a repository agent that could inspect a checkout, propose a one-line repair, run its tests, and report the result. Every write and shell command required explicit approval.

Now we will build the same agent in Python.

The outcome and safety policy stay identical on purpose - I didn’t want to quietly pick an easier demo just because Python made one more convenient. Both implementations work against the same failing fixture, get the same instructions, stop at the same approval boundaries, and have to pass the same tests. Only the host-language implementation changes.

The result is an async Python command-line application using the GitHub Copilot harness for repository work and Microsoft Agent Framework for the agent abstraction, streaming, sessions, and telemetry.

💡 Code repository: The complete Python and .NET implementations use the same fixture and are available at github.com/sahansera/safe-repository-maintenance-agent.

What we are building

The agent receives a local repository path and a maintenance task. Its system instructions require it to:

Work only inside that repository.
Read its AGENTS.md before changing anything.
Make the smallest coherent repair.
Avoid the network, package installation, commits, pushes, and pull requests.
Run focused validation.
Report the changed files, commands, and result.

The included fixture contains a small JavaScript function. The agent is written in Python, but the target repository does not have to be:

export function normalizeTitle(value) {
  return value.toLowerCase().replace(/\s+/g, "-");
}

One test expects ordinary title normalization. A second expects whitespace around the title to be ignored. The second test fails until the implementation trims the input.

Using a language-neutral target is deliberate. A Python agent can maintain a .NET, JavaScript, Go, or documentation repository. The agent host language determines how we integrate and operate the harness, not which source files the harness can understand.

Prerequisites and package setup

You need Python 3.11 or later and an active GitHub Copilot subscription. The example uses Python 3.12 as its documented baseline and was also verified with Python 3.13.

Here’s the environment I actually tested this against:

[project]
requires-python = ">=3.11"
dependencies = [
  "agent-framework-github-copilot==1.0.1",
  "github-copilot-sdk==1.0.2",
]

[project.optional-dependencies]
dev = ["pytest>=8.4,<9"]

Create an isolated environment and install the project:

python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'

Here’s what it looks like when run end to end:

The current Python SDK package includes the Copilot runtime for supported platforms. Authentication still depends on GitHub Copilot and may ask you to sign in when you run the agent for the first time.

The Agent Framework integration is stable; the GitHub Copilot SDK underneath it isn’t, and its preview label didn’t stop it from changing shape more than once while I was writing this pair of posts. Pin these versions - it’s cheap insurance against a future SDK bump silently rewriting what you’re reading.

Make permissions a small, testable policy

The agent can request several capability types, including read, write, shell, url, and mcp. We will not treat them equally:

Capability	Default
Read inside the selected repository	Approve
Write a file	Ask
Run a shell command	Ask
Fetch a URL or call MCP	Deny
Anything unknown	Deny

The policy is an ordinary function:

from enum import Enum

class PolicyDecision(Enum):
    APPROVE = "approve"
    PROMPT = "prompt"
    DENY = "deny"

def decide(permission_kind: str) -> PolicyDecision:
    if permission_kind == "read":
        return PolicyDecision.APPROVE
    if permission_kind in {"write", "shell"}:
        return PolicyDecision.PROMPT
    return PolicyDecision.DENY

The fallback denies URL access, MCP calls, new SDK permission kinds, and any malformed value. The application has to be changed deliberately before one of those capabilities becomes available.

The test suite makes that contract visible:

@pytest.mark.parametrize(
    ("kind", "expected"),
    [
        ("read", PolicyDecision.APPROVE),
        ("write", PolicyDecision.PROMPT),
        ("shell", PolicyDecision.PROMPT),
        ("url", PolicyDecision.DENY),
        ("mcp", PolicyDecision.DENY),
        ("unknown", PolicyDecision.DENY),
    ],
)
def test_decide_returns_expected_decision(kind, expected):
    assert decide(kind) is expected

These tests do not need a model, a Copilot subscription, or a repository. They test application authority rather than probabilistic behavior.

You do not need an LLM evaluation to prove that URL access is denied. Separate deterministic authorization policy from the agent and test it like any other security-sensitive function.

Convert policy into Copilot decisions

The permission handler receives a typed request and a context dictionary. It first prints enough detail for the operator to understand the action:

async def handle_permission(request, context):
    decision = decide(request.kind)

    print(f"\n[permission: {request.kind}]")
    print(describe(request))

    if decision is PolicyDecision.APPROVE:
        return PermissionHandler.approve_all(request, context)

    if decision is PolicyDecision.DENY:
        return PermissionDecisionReject(
            feedback="Blocked by the repository agent policy."
        )

    answer = (
        await asyncio.to_thread(input, "Approve once? [y/N] ")
    ).strip().lower()

    if answer == "y":
        return PermissionHandler.approve_all(request, context)

    return PermissionDecisionReject(
        feedback="The operator denied this action."
    )

input() is blocking, so asyncio.to_thread keeps it away from the event loop. That detail is easy to miss in a console sample and becomes more important when the application also streams output or handles more than one session.

For a write request, describe prints file_name and diff. For a shell request, it printsfull_command_text. URL and MCP requests are displayed before being denied, leaving an audit-friendly record of what the agent attempted.

The helper name approve_all can be misleading in this context. It constructs an approval response for the current request. Our application still decides which requests reach that line.

Configure the agent without ambient repository state

The command resolves the repository path before creating the agent:

repository = args.repository.expanduser().resolve(strict=True)

if not repository.is_dir():
    raise NotADirectoryError(repository)

The Copilot session options include that working directory and our permission callback:

options = GitHubCopilotOptions(
    working_directory=str(repository),
    enable_config_discovery=False,
    on_permission_request=handle_permission,
)

agent = GitHubCopilotAgent(
    instructions=INSTRUCTIONS,
    default_options=options,
)

Repository instructions are genuinely useful, so turning configuration discovery off seems backwards at first. I ran into the reason for it while building the .NET version of this project: the fixture sat inside another Git checkout during development, and the runtime happily walked up to that parent repo’s AGENTS.md before it ever noticed the fixture’s own. Nested repositories, monorepos, and temporary worktrees all make that boundary easy to blur without meaning to.

So the application instructions explicitly tell the agent to read AGENTS.md inside its working directory itself. That keeps the source of project guidance visible in the tool activity, instead of letting an unrelated parent directory quietly change what the agent thinks the rules are.

That does not make repository instructions trusted. A repository can contain prompt injection just as it can contain a malicious build script. The host policy remains authoritative when instructions ask for a forbidden action.

Repository instructions can improve the patch, but they must not expand the agent's authority. The host permission policy wins when project guidance asks for a forbidden side effect.

Stream the maintenance run

GitHubCopilotAgent owns an async client, so the natural Python lifecycle is an async context manager:

async with agent:
    async for update in agent.run(args.task, stream=True):
        print(update.text, end="", flush=True)

The context manager starts and stops the Copilot client even if the run raises an exception. The command entry point keeps the synchronous boundary small:

def run() -> None:
    raise SystemExit(asyncio.run(main()))

This lifecycle matters in longer-running applications. Agent sessions own processes, connections, history, and sometimes temporary files. An exception should not leave those resources attached to a worker indefinitely.

Run the same repair

Start the agent against the included fixture:

safe-repo-agent ../fixture

It reads the source and tests, then proposes the same one-line change as the .NET version:

- return value.toLowerCase().replace(/\s+/g, "-");
+ return value.trim().toLowerCase().replace(/\s+/g, "-");

The write does not happen until the operator approves it. The later npm test request has its own prompt, so approving a patch does not grant standing permission to execute arbitrary commands.

The verified run ended with:

File changed: src/normalize-title.js
Fix: Added .trim() before .toLowerCase().
Validation: Both tests pass (npm test exit 0).

Run the deterministic policy tests separately:

pytest

All six policy cases pass without starting Copilot.

Adding OpenTelemetry

The Python GitHubCopilotAgent includes Agent Framework’s telemetry layer. For local exploration, the framework can configure console exporters before the agent is created:

from agent_framework.observability import configure_otel_providers

configure_otel_providers(enable_console_exporters=True)

In a service, export through OTLP to your normal observability backend and attach the repository job identifier to the surrounding trace. Useful signals include:

End-to-end run duration
Time waiting for human approval
Tool calls by permission kind
Denied actions
Command duration and exit status
Repair attempts and validation failures
Cleanup failures

Do not enable prompt and completion capture casually. Repository paths, source code, terminal output, and environment-related errors can contain sensitive information. Telemetry should explain the run without becoming another copy of every secret the agent could see.

From console tutorial to production worker

The console application demonstrates the control points, not a complete isolation platform. Before I would let it process untrusted repositories in a service, I would add:

A fresh container or microVM for every job
A read-only base image and disposable writable workspace
asyncio-aware timeouts and cancellation for CPU, memory, disk, and wall-clock limits, not just a number typed into a config file
Network deny-by-default
No ambient developer or cloud credentials
Short-lived repository credentials without merge permission
Durable approval records instead of terminal input
A bounded repair loop and diff-size limit
Cleanup and audit recording in finally, including cancelled tasks

The same concerns apply to a deterministic tool that runs repository scripts. The agent makes the risk easier to see because it chooses commands dynamically, but the repository and its dependencies were already untrusted executable input.

What differs from .NET?

The architectural boundaries did not change. Both implementations use the same harness, working directory, permission table, task, and fixture.

The Python version expresses the lifecycle with async with, passes Copilot session settings throughGitHubCopilotOptions, and moves blocking operator input to a thread. The .NET version usesCopilotClient, SessionConfig, typed permission request subclasses, and IAsyncEnumerable for streaming.

Those are ecosystem differences, not different safety models.

Building the same agent twice made one thing obvious: the language wrapping the model barely mattered. What mattered was the authority boundary around its tools, and whether “it worked” was backed by a diff and a passing test, or just the model’s own word for it.

Thanks for reading ✌️

References

Build a Safe Repository Maintenance Agent with GitHub Copilot and Microsoft Agent Framework in .NET

Sahan — Mon, 10 Aug 2026 22:27:00 +0000

Giving an AI agent access to a repository takes only a few lines of code. Giving it useful access without silently approving every command is the real tutorial.

In this post, we will build a .NET 10 console application that uses the GitHub Copilot harness as its coding runtime and Microsoft Agent Framework as the application-facing agent abstraction. The agent will inspect a small repository, repair a failing test, ask before changing a file, ask again before running the test command, and finish with a concise maintenance report.

It will not access the network, install packages, commit, push, or create a pull request.

This is a hands-on continuation of my earlier article about building an AI-assisted dependency vulnerability fixer. That system used an agent to handle repository-specific exceptions while keeping review and merge authority with the repository owner. Here, we will build the smaller execution boundary behind that idea.

💡 Code repository: The complete .NET and Python implementations use the same fixture and are available at github.com/sahansera/safe-repository-maintenance-agent.

What the two frameworks contribute

The GitHub Copilot SDK supplies the coding harness. It owns the agent loop and provides repository-oriented capabilities such as reading files, writing files, running shell commands, fetching URLs, and calling MCP tools.

Microsoft Agent Framework wraps that runtime in the same AIAgent abstraction used by its other providers. That gives the application a consistent run interface, streaming, sessions, middleware, and OpenTelemetry integration.

The distinction matters. We are not asking Agent Framework to recreate a coding loop around a chat model. Copilot remains responsible for planning and tool execution. Agent Framework gives us the surface on which the rest of the application can depend.

The Agent Framework integration itself is stable, but the GitHub Copilot SDK underneath it is still labeled public preview. I pinned exact versions below for that reason - the lower-level APIs moved twice while I was drafting this post, and I’d rather you hit a clean build than chase a breaking change mid-tutorial.

The repository we will repair

The completed sample accepts any repository path, including a checkout such as sahansera.dev. For the write demonstration, however, it includes a disposable fixture with no dependencies:

fixture/
├── AGENTS.md
├── package.json
├── src/
│ └── normalize-title.js
└── test/
    └── normalize-title.test.js

The implementation is deliberately wrong:

export function normalizeTitle(value) {
  return value.toLowerCase().replace(/\s+/g, "-");
}

The second test expects surrounding whitespace to be ignored. Running npm test gives us one pass and one failure because the actual result is -safe-repository-agent-.

This fixture gives the agent a real task with an objectively verifiable result. It also means nobody has to grant a first experiment write access to an important repository.

Prerequisites and project setup

You need:

.NET 10
An active GitHub Copilot subscription
The GitHub Copilot and Agent Framework integration packages

Here’s what I had installed when this worked:

<ItemGroup>
  <PackageReference Include="GitHub.Copilot.SDK" Version="1.0.9" />
  <PackageReference Include="Microsoft.Agents.AI.GitHub.Copilot" Version="1.17.0" />
</ItemGroup>

The SDK bundles its compatible Copilot runtime, so a separate global CLI installation is not required by the current .NET package. You still need to authenticate and have an active subscription.

Define the permission policy before creating the agent

The quickest demo is an approval callback that returns ApproveOnce() for everything. It is also a poor default for an application that can run commands and rewrite a checkout.

Our policy separates four decisions:

Capability	Default
Read inside the selected working directory	Approve once
Write a file	Ask the operator
Run a shell command	Ask the operator
Fetch a URL or call an MCP server	Deny

Unknown permission types are denied. A new SDK capability should not become authorized merely because the application has not been updated to recognize it.

public enum PolicyDecision
{
    Approve,
    Prompt,
    Deny,
}

public static PolicyDecision Decide(string permissionKind) => permissionKind switch
{
    "read" => PolicyDecision.Approve,
    "write" or "shell" => PolicyDecision.Prompt,
    "url" or "mcp" => PolicyDecision.Deny,
    _ => PolicyDecision.Deny,
};

This function contains no agent or console dependencies, so it is easy to unit test. The sample has six cases covering every known branch and the fail-closed fallback.

Permission handling is application policy, not prompt wording. Put it in ordinary code, test every branch, and deny capabilities you do not recognize.

Turn policy decisions into operator prompts

The Copilot SDK sends a typed PermissionRequest. That means we can show the operator the actual command, filename, diff, URL, or MCP tool instead of asking them to approve an unexplained action.

public static Task<PermissionDecision> HandleAsync(
    PermissionRequest request,
    PermissionInvocation _)
{
    PolicyDecision decision = PermissionPolicy.Decide(request.Kind);

    Console.WriteLine($"\n[permission: {request.Kind}]");
    Console.WriteLine(Describe(request));

    return decision switch
    {
        PolicyDecision.Approve =>
            Task.FromResult(PermissionDecision.ApproveOnce()),
        PolicyDecision.Deny =>
            Task.FromResult(PermissionDecision.Reject(
                "Blocked by the repository agent policy.")),
        _ => Task.FromResult(Prompt()),
    };
}

For a write request, Describe prints both the path and proposed diff. For a shell request, it printsFullCommandText. Approval is always for the current action rather than the whole session.

Do not treat the displayed command as a complete security parser. Shell syntax, symlinks, subprocesses, and package scripts make static classification difficult. The prompt improves operator judgment; the real containment boundary should still be a disposable sandbox with limited credentials and network access.

Scope the Copilot runtime to one repository

The application resolves the supplied path before starting Copilot:

string repositoryPath = Path.GetFullPath(args[0]);

if (!Directory.Exists(repositoryPath))
{
    Console.Error.WriteLine(
        $"Repository directory does not exist: {repositoryPath}");
    return 2;
}

We use that path for both the client process and the session:

await using CopilotClient copilotClient = new(new CopilotClientOptions
{
    WorkingDirectory = repositoryPath,
});

await copilotClient.StartAsync();

SessionConfig sessionConfig = new()
{
    WorkingDirectory = repositoryPath,
    EnableConfigDiscovery = false,
    OnPermissionRequest = ConsolePermissionHandler.HandleAsync,
    SystemMessage = new SystemMessageConfig
    {
        Mode = SystemMessageMode.Append,
        Content = instructions,
    },
};

This revised system instruction tells the agent to read the selected repository’s AGENTS.md itself. With ambient discovery disabled, a nested checkout no longer silently inherits instructions from a parent checkout.

A working directory also limits the paths Copilot considers available by default. It is useful scope, but it is not process isolation. I would still run an agent against an untrusted repository in a container with a non-root user, a disposable filesystem, no ambient cloud credentials, and a narrow network policy.

Create the Agent Framework agent and stream the result

With the client and session configured, the integration is one call:

AIAgent agent = copilotClient.AsAIAgent(sessionConfig);

await foreach (AgentResponseUpdate update in agent.RunStreamingAsync(task))
{
    Console.Write(update);
}

The instructions constrain the task further:

Work only inside the supplied working directory.
Read AGENTS.md before making changes.
Make the smallest change that satisfies the task.
Do not access the network, install packages, commit, push, or create a pull request.
Run focused validation and report the files changed, commands run, and result.

Those instructions improve agent behavior, but they do not replace the permission callback. The network prohibition exists in both places intentionally: the prompt tells the agent not to try, and the callback denies the capability if it does.

Run the repair

Start the sample against the fixture:

dotnet run --project src/SafeRepositoryAgent -- ../fixture

The agent reads the repository instructions and tests. When it proposes adding .trim(), the application prints the exact diff and pauses:

[permission: write]
File: .../fixture/src/normalize-title.js

- return value.toLowerCase().replace(/\s+/g, "-");
+ return value.trim().toLowerCase().replace(/\s+/g, "-");

Approve once? [y/N]

After approval, it asks separately before running npm test. The verified run finished with:

Changed: src/normalize-title.js - added .trim() before lowercasing.
Command: npm test - 2/2 tests pass.

The result matters because it is supported by a small diff and a repeatable test, not because the agent described itself as successful.

A useful maintenance result contains the proposed diff, the command that was allowed, and its exit status. The final natural-language answer is only a summary of that evidence.

What would change for a production service?

An interactive console prompt is appropriate for a tutorial and a developer workstation. A service needs a durable approval protocol instead.

I would keep the same policy function, then replace Console.ReadLine() with an approval record tied to a stable job and tool-call identity. The worker would pause, persist the request, notify an authorized reviewer, and resume only after receiving a valid decision. Every decision would be part of the audit trail.

I would also add:

A disposable container or microVM per repository job
Host-enforced CPU, memory, disk, process, and wall-clock limits, driven by a CancellationTokenrather than just requested of Copilot
Short-lived repository credentials with no merge permission
Network deny-by-default with explicit destinations where required
OpenTelemetry export, wired through the worker’s own lifetime, for agent runs, permission latency, tool calls, and failures
A maximum number of repair attempts before the job fails loudly instead of looping
A final diff-size limit and required validation commands
Deterministic cleanup via await using/finally, including failed and cancelled runs

Agent Framework emits OpenTelemetry-compatible telemetry, but traces can contain prompts, paths, commands, and model output. Keep sensitive-data capture disabled unless you have an explicit reason and a suitable storage policy.

The important part is outside the model

The agent itself did the easy part - a two-line fix, the kind of thing plenty of tutorials would stop at. What actually took the iteration was everything around that decision: scoping the working directory, closing the config-discovery hole, making every write and shell command visible before it ran, and keeping the test suite - not the model’s own summary - as the judge of success.

It’s a small application. But the boundary it draws is the whole point.

The Python version of this tutorial builds the same agent and fixture using async context managers, a typed options dictionary, and pytest. Keeping the task and permission table identical makes the differences between the two SDKs much easier to see.

Thanks for reading ✌️

References

I Grew an Engineering Blog from 0 to 463,000 Pageviews - Here's What Worked

Sahan — Tue, 04 Aug 2026 01:43:00 +0000

I published the first posts on sahansera.dev in December 2019.

There was no launch campaign, existing audience, or reliable stream of visitors waiting for them. Like most new personal sites, the blog started at zero. I wrote about problems I had encountered, shared the posts where I could, and hoped somebody searching for the same answers would eventually find them.

By 1 August 2026, the blog had accumulated 463,362 recorded pageviews.

Along the way, it has reached readers in at least 184 countries and territories. Here is where readers came from last month:

Earlier geography data is not included, so the lifetime reach may be broader.

That number is personally meaningful, but it is not the most interesting part of the story. The useful part is what happened underneath it: a small collection of practical engineering articles generated most of the traffic, some posts kept helping people for years, and many things I assumed would matter barely moved the numbers at all.

This is what I learned.

The short version

Specific, practical articles generated most of the traffic.
Evergreen posts kept growing for years after publication.
The blog continued reaching readers during a long publishing break.
Traffic revealed demand, but it did not tell me whether readers returned.
Comparing posts fairly requires consistent time windows, not raw lifetime totals.

From zero to 463,362 pageviews

I calculated the lifetime total using two consecutive analytics exports with no overlapping dates.

A few posts did most of the work

The distribution of traffic surprised me more than the total. Just five articles account for roughly 46% of all the pageviews the blog has recorded since launch.

The leading articles are remarkably consistent in what they offer:

Article	Lifetime pageviews
Simple In-Memory Caching in .NET with `IMemoryCache`	86,975
Understanding WebSockets with ASP.NET	43,272
Distributed Caching in ASP.NET Core with Redis	31,911
Having Fun with Microsoft IoC Container for .NET Core	27,812
Understanding the .NET Generic Host Model	23,040

Other durable performers cover securing the Hangfire dashboard, Kubernetes commands and arguments, and gRPC across Go and .NET.

These are not broad opinion pieces. Each one helps a developer understand a specific concept or complete a specific task.

The lesson is not simply that these technologies are popular. It is that clear intent compounds. A post answering a concrete question can remain useful every day for years, even when I am not actively promoting it.

💡 The pattern: Useful, specific posts can compound for years.

Evergreen technical writing compounds slowly

My highest-traffic article is an introduction to in-memory caching that I published in January 2020. Years later, it is still the largest entry point to the site.

That changed how I think about the return on writing.

A social post has a short distribution window. It may reach many people immediately and then disappear. A useful technical article behaves differently. It may receive very little attention on its first day, but it can be discovered repeatedly through search, links, code repositories, and recommendations.

The early results can feel underwhelming because the compounding is almost invisible. The article needs to be indexed. It needs to answer the query well enough for people to stay. Other pages need to link to it. Search engines need time to understand whether it is useful.

None of my successful posts felt like a breakthrough when I pressed publish. Their value accumulated quietly.

The blog kept working while life took priority

The growth was not driven by a perfectly consistent publishing schedule.

In 2023, I became a first-time dad. My family also moved into a new home, and I changed jobs. I wrote about that season in my plans for sahansera.dev in 2024, acknowledging that blogging had taken a back seat while I focused on being the best dad I could be.

The publication dates tell the story plainly. I published two posts in 2023, none in 2024, and returned with seven posts in 2025. I had hoped to resume a regular schedule sooner, but life had a different rhythm.

I am glad I did not treat the pause as a reason to abandon the blog. While I was not publishing, the existing articles continued answering questions, appearing in search results, and bringing new readers to the site. The work I had already done kept compounding when I did not have the time or energy to add more.

That makes the lifetime total more meaningful to me. It did not come from operating a content machine or forcing myself to publish through every season of life. It came from building a useful body of work, letting it breathe, and returning when I had something worthwhile to share.

💡 The pause: The blog kept growing even when publishing had to wait.

When I started writing again in 2025, I explored streaming APIs, HTTP internals, Python environments, and home-lab Kubernetes. The break had not erased the audience. It gave me a chance to return with different experiences and better questions.

Practical specificity beats broad ambition

The best-performing titles make a small promise:

Configure in-memory caching.
Build a gRPC server or client.
Run Kafka locally for testing.
Secure a Hangfire dashboard.
Understand how WebSockets work.

The posts are narrow enough for the reader to know why they should click, but substantial enough to teach the surrounding concepts.

This balance matters. A title such as "Everything You Need to Know About Distributed Systems" sounds ambitious but does not reveal which problem it solves. "Building a gRPC Server in Go" is less grand and much more useful to the person who needs exactly that.

My better articles tend to combine three layers:

A direct answer to a practical problem.
An explanation of what is happening underneath.
A working implementation readers can adapt.

That combination has become the clearest description of what I want this blog to be.

One successful article should become a cluster

For a long time, I treated each post as an isolated piece of work. The analytics show why that leaves value on the table.

The audience for an in-memory caching tutorial is likely to care about distributed caching, Redis, invalidation, cache stampedes, testing, and production failure modes. Someone building a gRPC server may next need a client, authentication, retries, deadlines, streaming, observability, or Kubernetes deployment guidance.

💡 The opportunity: A successful article is evidence that a useful topic cluster exists.

The gRPC series already demonstrates this. Its introduction, Go server and client, .NET server and client, and deployment posts reinforce one another. The individual posts can satisfy focused searches while the series gives interested readers a natural route through the broader subject.

I want to apply the same model to three areas:

.NET caching and application reliability
Kafka and event-driven system failure modes
Kubernetes operations and production troubleshooting

This does not mean publishing minor variations of the same article. Each post still needs a distinct problem and search intent. The connection between them should help a reader progress from a basic implementation to the difficult production questions.

Traffic is not the same as an audience

The historical reports showed that most visitors left after reading a single page. That sounds alarming until the context is considered.

Many visitors arrive from search, find a code sample or explanation, solve their immediate problem, and leave. For a reference-style engineering article, that can be a successful visit rather than a rejection.

At the same time, the data exposes a real weakness: I made it easy to consume one answer but did not always make the next useful step obvious.

Chronological previous-and-next links are not enough. A reader on a caching article probably does not want the post I happened to publish immediately afterward. They want the most relevant continuation of the problem they are already solving.

The improvements I am making are straightforward:

Add contextual links to related articles within the explanation.
Show a clear next step at the end of high-traffic posts.
Organise related material into visible series and topic hubs.
Keep the email and RSS subscription options easy to find.
Link runnable examples to maintained repositories.

The goal is not to trap somebody on the site. It is to make the site more useful when they want to go deeper.

💡 The real goal: Traffic becomes an audience only when readers have a reason and a way to return.

I measured traffic but not outcomes

Another uncomfortable lesson is that I collected a lot of traffic data without defining what success should mean beyond pageviews.

My current analytics setup contains no configured conversion events. I can see that people read an article, but I cannot reliably answer questions such as:

Did the article lead someone to another useful post?
Did they subscribe by email or RSS?
Did they visit the example repository?
Did they copy a code sample?
Which landing pages create returning readers?

These questions matter more now than the raw total.

Pageviews helped me understand which subjects have demand. The next stage is to measure whether the blog creates a relationship with the reader. I plan to treat a confirmed email subscription as the primary conversion, then track supporting actions such as RSS clicks, repository visits, code copying, deep scrolling, and movement between related articles.

Not every personal blog needs a conversion funnel. But if I want to improve something, I need to be explicit about what "better" means.

Analytics data needs maintenance too

Just over 35% of the views in the recent export have the page title (not set). The file also contains fragmented title variants and at least one suspicious spam-like title.

That is a useful reminder that analytics is not automatically a source of truth just because it contains precise-looking numbers.

Collection can break. Titles can change. A migration can alter definitions. Filters can create misleading exports. Bots and spam can contaminate reports. Privacy settings can change how users and sessions are identified.

I will use page paths as the canonical dimension for content reporting, investigate why page titles are missing, filter known noise, and verify that a single page view is recorded for each navigation. I also want Google Search Console beside Analytics so I can see queries, impressions, rankings, and click-through rates - not only the visits that already happened.

Measurement is part of maintaining the site, not something completed by pasting in a tracking ID once.

New posts need a fair evaluation window

Several of my newer articles have far fewer lifetime views than posts published in 2020 or 2021. That does not necessarily mean they failed.

An article published last month should not be compared directly with one that has accumulated search traffic for six years. Lifetime totals reward age.

For new work, I am moving toward a smaller set of time-normalised measures:

Search impressions and clicks during the first 28 and 90 days
Views per 30 days since publication
Engagement and code-copy actions
Movement to another related article
Email or RSS subscription actions
Whether traffic continues growing after the initial promotion window

This should make it easier to distinguish a promising article that needs time from one whose topic, title, or distribution genuinely missed the mark.

What 463,000 pageviews means to me

Four hundred and sixty-three thousand is not a huge number on the scale of the internet. It is huge compared with the zero visitors I had in December 2019.

More importantly, it represents individual moments when somebody had a problem and something I wrote may have helped them move forward.

The experience has changed my view of successful technical writing:

You do not need a large initial audience.
Useful, specific posts can compound for years.
A handful of durable articles may matter more than a constant publishing schedule.
Updating and connecting existing work can be more valuable than always starting from zero.
Honest measurement is more useful than the largest possible headline.

If you are starting an engineering blog with no visitors, that is normal. Write down the problem you just solved. Explain enough of the underlying system that the solution remains useful. Include the details you wish had been available when you were searching.

Then publish it and give it time.

That is more or less how this blog went from zero to 463,000 pageviews - one specific problem at a time.

If you write technical articles, what has surprised you most about the posts that keep finding readers? I would love to compare notes in the comments.

How We Replaced a Critical Data Path Without a Flag Day

Sahan — Mon, 03 Aug 2026 12:55:00 +0000

Replacing an API call is easy. Replacing the source of truth behind an automated decision is not.

I was reminded of this while migrating a critical workflow from a legacy event feed to a canonical state API. Both systems appeared to answer the same question: should the workflow act on this record now? But they had different schemas, different update timings, and, more importantly, slightly different models of the same lifecycle.

This was not a path where we could deploy the new code on Friday and watch the error rate. A false negative could leave work undone. A false positive could trigger an irreversible action from stale data. The HTTP request succeeding told us almost nothing about whether the new path was making the right decision.

So we did not treat it as a normal code replacement. We treated it as a controlled transfer of authority.

The migration used a query-only mode, shadow comparisons, discrepancy alerts, and a deliberately boring cutover. None of those techniques are particularly novel. What mattered was how we combined them, what we chose to compare, and how we decided the old path was finally safe to delete.

This post walks through that process.

The dangerous assumption: same data, newer API

The legacy path consumed a source built specifically around one class of lifecycle event. The replacement exposed a broader canonical record containing current state, relevant dates, and other attributes used by several workflows.

On paper, the migration looked like this:

That diagram hides the risky part. The two sources did not merely encode the same fact using different field names.

A purpose-built event feed tends to answer an event-shaped question: which transitions were recorded? A canonical API tends to answer a state-shaped question: what is true about this entity now?

Those questions overlap, but they are not identical.

An entity can have an old end-state event and later return to an active state. A record can contain several dates, each valid in its own business context. One source may update immediately while another catches up later. Missing data can mean “not applicable,” “not received yet,” or “something is broken.”

If we had translated fields one-for-one and switched traffic, the code would have looked correct while preserving none of those semantics.

The first useful decision was therefore to stop calling this an API migration. We were migrating a business decision from one model of the world to another.

Start by writing down the invariants

Before building the new path, we wrote down what the workflow must continue to guarantee.

The important invariants were roughly:

An active entity must never be processed because of a stale historical event.
A legitimate lifecycle transition must not be missed because one optional field is absent.
The effective date must have the same business meaning before and after migration.
Ambiguous or contradictory data must fail safely and become visible.
Reprocessing the same record must not produce duplicate downstream actions.

This step sounds obvious, but it changed the review entirely. Instead of asking whether the new client correctly parsed a payload, we could ask whether it preserved the rules the system existed to enforce.

It also exposed a subtle problem with “the old system is the source of truth.” If the legacy path had known defects, perfect agreement would reproduce them. The old output was a baseline, not an oracle.

That meant every mismatch needed investigation, but it did not mean the new path was automatically wrong.

Phase one: make the new path incapable of causing damage

The first version of the new integration was deliberately incomplete. It could query the canonical state API, normalize the response, and calculate what action it would take. It could not perform that action.

I think of this as query-only mode :

decision = evaluate_canonical_record(record)

if query_only:
    return decision

execute(decision)

The real implementation had more safeguards than this, but the boundary was just as explicit. Read and decide on one side; mutate on the other.

Query-only mode gave us a few useful properties:

We could run the code with realistic production data.
We could inspect decisions without creating tickets, sending notifications, or changing access.
We could debug authentication, pagination, missing fields, and schema assumptions separately from cutover risk.
We had a reusable operational tool for investigating individual records.

This last point was unexpectedly valuable. Migration controls are often treated as temporary scaffolding, but a safe read-only execution mode is also a good diagnostic interface. It lets an engineer ask, “What would the system do with this input right now?” without having to reproduce the entire workflow locally.

There was one rule we kept firm: query-only could not mean “mostly read-only.” If a code path still emitted an event or called a downstream service before checking the flag, the control was cosmetic. The no-side-effect guarantee had to sit at the boundary where side effects began.

Phase two: one writer, two decision-makers

Once the new path could evaluate real records safely, we ran it alongside the legacy implementation.

Only the legacy path was allowed to perform actions. The new path observed the same logical input and produced a candidate decision. We then compared the two.

This is usually called a shadow migration or dark launch. The new code sees production-shaped traffic but does not own production effects.

The obvious implementation compares two Boolean values:

legacy_should_process == new_should_process

That is useful, but not enough. When the values differ, a Boolean tells you that the migration is unsafe and nothing about why.

We made the decision explain itself:

Decision(
    should_process=False,
    effective_date=None,
    reason="currently_active",
    source_record_id="...",
)

Now the shadow result could compare:

Whether each path would act.
Which effective date it would use.
Why it reached that conclusion.
Which source record contributed to the decision.
Whether either path considered the input incomplete or ambiguous.

This is the part of the migration I would reuse everywhere: compare normalized decisions, not raw responses.

Raw payload comparison is noisy. One API may use null where another omits a field. Dates may use different time zones. Identifiers may refer to different resources. A hundred harmless representation differences can hide the one semantic difference that matters.

The normalized decision is the contract your users actually experience.

A mismatch is a finding, not just an error

As soon as the comparison ran against real data, discrepancies appeared. That was the point.

It is tempting to turn every mismatch into a page. I would avoid that. Early shadow traffic can be noisy, and training people to ignore an alert stream is a poor way to launch a critical system.

Instead, we recorded every mismatch with enough context to investigate it and grouped them into a small taxonomy:

Expected timing differences — one source had updated before the other.
Representation differences — the sources agreed, but normalization was incomplete.
Semantic differences — both records were valid, but the business interpretation differed.
Data-quality problems — records were missing, stale, or internally contradictory.
Implementation defects — our new code had selected the wrong field or applied the rule incorrectly.

This classification matters because each category leads to a different response.

A timing difference may need a grace period or a later recheck. A representation problem belongs in the adapter. A semantic difference needs a product or domain decision. Bad source data needs a defensive rule and an escalation path. An implementation bug needs code and a regression test.

Without a taxonomy, teams tend to “fix the diff” until the graphs become green. That can accidentally teach the new system to mimic legacy behaviour without understanding it.

The goal was not zero differences at any cost. It was zero unexplained differences.

The edge cases were the migration

The happy path agreed quickly. That did not make the migration nearly complete.

Two edge cases forced us to refine the new model.

The first involved a historical end-state event for an entity whose current state had since changed. If we looked only for the existence of that event, we could produce a false positive. The canonical record gave us another signal: the entity’s current status. We added that verification before allowing the workflow to proceed.

The second involved choosing the effective date. The new source exposed more than one plausible date, and the most conveniently named field was not necessarily the date the downstream process expected. We had to trace the business meaning through the old path and deliberately select the corresponding value.

Neither bug was difficult to fix once understood. The hard part was creating a migration that allowed us to see them before they became actions.

That is why I do not judge shadow migrations by the amount of traffic replayed. A million ordinary records can give more confidence than they deserve. One reactivation, one delayed update, or one contradictory date can tell you much more about whether the new model is correct.

Coverage should be measured across business scenarios , not only request counts.

For this kind of workflow I want an explicit scenario set:

Normal lifecycle transition.
Future-dated transition.
Reactivation or status reversal.
Missing optional attributes.
Conflicting dates.
Duplicate input.
Source timeout or partial response.
A record that changes while being processed.

Some scenarios will occur naturally during the shadow period. Rare but dangerous ones should be exercised with fixtures or controlled replay rather than waiting for production to provide them.

Deciding when to cut over

“The dashboards look fine” is not a cutover criterion.

Before transferring authority to the new path, we wanted evidence in several dimensions:

Correctness: no unexplained decision or effective-date mismatches.
Scenario coverage: important lifecycle transitions had been observed or tested.
Stability: the comparison stayed clean across a meaningful observation window, not just one quiet day.
Failure behaviour: timeouts, missing records, and contradictory data failed safely.
Observability: we could tell whether the new path queried, decided, skipped, failed, or acted.
Rollback: restoring the old authority was understood and quick.

There is no universal percentage or number of days that makes a migration safe. The right window depends on the frequency and consequence of the events you are trying to observe.

If an important edge case happens once a month, a clean afternoon tells you nothing about it. If the cost of a false positive is high, the threshold should reflect that asymmetry.

The cutover itself was intentionally uneventful. We changed which path was authoritative while retaining the ability to compare and roll back. We did not bundle unrelated cleanup into the same release. Boring is a feature when transferring control of a critical workflow.

Do not leave the old path “just in case”

After the new path had operated successfully, we removed the legacy implementation and the shadow comparison.

This can feel premature. Keeping the old path around appears to preserve a fallback. In reality, a dormant fallback decays quickly:

Its credentials and dependencies still need maintenance.
Engineers must continue reasoning about two implementations.
Future changes may update one path but not the other.
Someone can accidentally reactivate code that has not been exercised in months.
The temporary feature flag becomes permanent architecture.

A rollback path is valuable during migration. A second production system with no clear retirement date is technical debt.

We treated deletion as a planned migration phase rather than optional cleanup:

The order matters. Deleting before the soak period removes your quickest recovery option. Never deleting leaves you paying for the migration forever.

The comparison infrastructure should also be removed or deliberately repurposed. Shadow code often doubles reads, emits high-cardinality logs, and contains branching that the main workflow no longer needs. Once its question has been answered, it should not quietly become part of the permanent request path.

What I would carry into the next migration

The practical pattern is straightforward:

Define the invariants before translating fields.
Separate reads and decisions from side effects.
Give the new path a genuine query-only mode.
Keep exactly one writer while both paths evaluate.
Compare normalized decisions and their reasons.
Investigate and classify every meaningful mismatch.
Measure coverage across business scenarios, not only traffic volume.
Set evidence-based cutover and rollback criteria.
Keep the cutover small.
Delete the legacy path after a defined soak period.

The broader lesson is that migrations like this are not primarily data plumbing exercises. They are exercises in transferring trust.

The old path has years of accumulated behaviour, including behaviour nobody documented because the code made it seem obvious. The new source may be cleaner and more canonical, but that does not make your interpretation of it correct. Confidence comes from forcing both systems to make their decisions in the open, then explaining every place they disagree.

That takes longer than changing an endpoint. It is still much cheaper than discovering after cutover that a green deployment was making the wrong decision perfectly.

Thanks for reading ✌️

When "no healthy upstream" isn't about the upstream you think

Sahan — Thu, 23 Jul 2026 00:00:00 +0000

no healthy upstream is the kind of error that makes you expect wreckage.

Then you open the dashboards and find… almost nothing. CPU is low. No pods have crashed. The last deployment was hours ago. By the time you refresh the page, the service has recovered by itself.

That was the scene a few weeks ago when I started chasing an intermittent failure in a search backend. The eventual fix was only a few lines. The interesting part was getting there.

We already had a confident root-cause analysis (RCA), complete with a tidy explanation and a one-line remedy. It was also pointing at the wrong subsystem. This post is about the gap between a plausible story and the evidence, and about a common failure mode in which a mostly healthy fleet slowly removes itself from service.

The shape of the failure

The setup was ordinary: a search backend behind a load balancer, with a fixed pool of worker processes on each instance.

Every so often, with no obvious schedule, a burst of requests failed. The browser showed a bare no healthy upstream, and a minute or two later everything worked again.

One clue appeared every time. Backend p99 latency climbed to almost exactly the load balancer timeout, stayed flat, and then dropped back to normal:

That shape matters. Organic slowdowns tend to be uneven. This was a cliff, a flat top, and a recovery. Requests were not gradually becoming slower; they were running into a deadline and being cut off.

The theory I inherited was CPU throttling. A heavy periodic job supposedly pegged the pod’s CPU, the scheduler throttled it, request handling starved, and the load balancer eventually evicted the instance.

It was coherent. Better still, it came with a one-line fix: raise the CPU limit. That is an attractive combination during an incident. But a root-cause theory makes predictions, and these predictions did not survive contact with the evidence.

Treating the RCA as a hypothesis, not a conclusion

Instead of treating the existing RCA as a conclusion, I treated it as a hypothesis: if CPU throttling caused the incidents, what else should I be able to observe?

Three checks came back wrong.

There was no CPU-bound work in the serving path. The compute-heavy batch indexers ran on separate machines and reached the datastore over the network. They never ran inside the pods serving requests. A process outside the pod’s cgroup cannot cause that pod to be CPU-throttled. The graphs agreed: container throttling counters stayed flat during every event.

The timing did not fit a scheduled trigger. The batch jobs ran on a coarse schedule and produced one sustained utilisation bump. The incidents arrived at arbitrary minutes and happened far more often than the jobs ran. If a timer were responsible, the failures should have followed the timer. They did not.

The failures ignored instance and version boundaries. The same symptom appeared across different pods and deployment SHAs. A regression usually follows a version. A shared external event usually hits the fleet together. These failures did neither. The pattern looked more like each instance was doing something to itself, triggered by its own traffic.

To keep the CPU theory alive, I would have had to explain away the topology, the timing, and the distribution of failures. At that point the theory was creating more questions than it answered, so I dropped it and went back to the logs.

One line in the logs, and why it changes the model

The event window contained one recurring error:

ConnectionTimeout: Connection timed out

The stack ended in the datastore client, blocked on a socket read that never returned.

That one line flipped the model.

A CPU-throttled worker is ready to run but cannot get enough scheduler time. A worker blocked on I/O is off-CPU, waiting on the network while still occupying its worker slot. From the outside, both look like “latency went up, then requests timed out.” Underneath, they are opposites.

Adding CPU to an I/O stall does not unblock the socket. At best, it gives you more workers to park behind the same slow dependency. This is why edge symptoms are a dangerous thing to tune against: resource saturation and dependency blocking can produce the same fever while needing completely different treatment.

Why one slow dependency saturates the whole instance

The mechanism came down to two ordinary client settings that were dangerous in combination:

No timeout on individual datastore calls. One call could occupy a worker long after the user had given up.
Retries on connection timeouts. After waiting too long once, the worker would wait again, with backoff in between.

Nothing exotic. That is partly what makes this failure mode easy to miss.

Little’s Law shows why it is fatal to a fixed worker pool. Concurrency is L = λW: the arrival rate (λ) multiplied by the average time each request spends in the system (W).

Suppose the service receives 200 requests per second and normally responds in 40 ms:

L = 200 req/s × 0.04 s = 8 concurrent requests

Eight concurrent requests are easy for the pool to absorb. But if the datastore slows down and requests wait for tens of seconds, W increases by three orders of magnitude. Retries stretch it further. The required concurrency quickly exceeds the number of workers available.

The pool then looks something like this:

This creates head-of-line blocking. Workers parked on datastore calls cannot serve the fast requests behind them, so latency rises for everything, not only for requests that reached the slow dependency.

Health checks are caught in the same queue. They time out, the load balancer removes the instance from rotation, and the remaining instances receive more traffic. Once enough instances fail their health checks, the load balancer has nowhere to send the next request. That is when the user sees no healthy upstream.

There was one more twist. The client’s total retry time could exceed the load balancer’s deadline:

The load balancer returned an error while the worker continued retrying a result nobody could receive. Every millisecond after the outer deadline was wasted work, and the wasted work held a scarce worker slot.

This is a deadline-propagation failure. Inner operations must finish inside the outer request deadline. Better still, pass the outer deadline through the call chain so every layer knows when its result has become useless.

Why it recovered by itself

The recovery was initially reassuring. In hindsight, it was the worrying part.

The datastore blip triggered retries. Those retries added load to the datastore while it was already struggling, which caused more timeouts and therefore more retries:

That is the start of a metastable failure : a brief trigger knocks the system out of its healthy state, then a feedback loop keeps it unhealthy after the original trigger has passed.

We got lucky. The datastore blips were short enough that traffic fell below the tipping point before the retry loop became self-sustaining. A slightly longer blip could have kept the loop alive until we restarted the fleet or shed enough traffic to escape it.

Self-recovery was not proof of resilience. It was a warning shot.

The fix, and why we kept it small

The instinct during an availability incident is to add headroom: raise CPU limits, increase the worker pool, add replicas. That can help with genuine capacity problems. Here, it would only give the retry loop more workers to occupy.

The useful fix was to put a hard bound on the cost of one request.

First, add an explicit timeout to every downstream call. Choose it from the dependency’s healthy latency distribution rather than picking a pleasing round number. It should sit comfortably above healthy p99.9, but well below the load balancer’s deadline. A call that can wait forever is a worker you can lose forever.

Second, bound retries as a fraction of normal traffic, not as an unconditional count on every request. A rule such as “retry three times” allows every client to multiply traffic precisely when the dependency is least able to handle it. A token-bucket retry budget keeps the added load bounded; the Google SRE guidance uses 10% as an example. Add jitter too, or clients can wake up and retry in synchronised waves.

Third, be especially reluctant to retry connection timeouts. During saturation, a timeout often means the dependency is already over its limit. Another immediate attempt is unlikely to help. Retries make the most sense for independent, transient failures and only for idempotent operations. Search requests were idempotent, at least, so duplicate side effects were not another problem waiting for us.

The decision for each failed call becomes straightforward:

Circuit breakers, load shedding, and bulkheads could all strengthen this design:

A circuit breaker stops callers repeatedly rediscovering the same outage.
Load shedding rejects excess work early enough to keep the service responsive.
A bulkhead gives the dependency its own bounded concurrency pool, preventing it from occupying every worker.

All three add code, tuning, and operational state. We did not have evidence that we needed them yet. Once a small timeout and a bounded retry policy removed the amplifier, adding more machinery would have solved a hypothetical problem rather than the incident in front of us.

“Won’t failing fast just move the errors to the caller?”

This was the immediate pushback: if the backend gives up sooner, won’t users simply see more errors?

Only if we compare failing fast with a world in which every request eventually succeeds. That was not the world we had.

With the worker pool full of hung calls, every request eventually failed. The entire search feature became a no healthy upstream page, including requests that never needed the slow dependency in the first place.

Failing fast keeps the instances responsive and in rotation. It turns one correlated, fleet-wide outage into a smaller number of independent failures: the requests that actually hit the bad path. Those are failures a caller can absorb with a cached result, an empty state, or a retry button.

A nonessential component should not be able to take down the whole page. Give it its own deadline and a graceful fallback, and a slow dependency degrades one part of the experience instead of replacing the entire document with an error page.

Verifying the bound, not asserting it

I did not want to ship a fix that worked only on a whiteboard, so the validation was intentionally mechanical.

With a healthy dependency , normal traffic should remain normal: the same latency distribution and no new errors. If the timeout clips healthy p99.9 requests, it is too aggressive and will create the very failures it is meant to prevent.

With an unreachable dependency , requests should fail near the configured bound and before the outer load balancer deadline. More importantly, worker occupancy and in-flight request counts should remain flat instead of climbing.

That flat line under induced failure is the real acceptance test. It shows that the amplifier is gone.

The saturating case needs its own load test. The happy path will never prove that a service behaves well when every downstream call is stuck.

What I actually took from it

The technical lesson is easy to audit for: an unbounded downstream call plus eager retries is a latent outage with a feedback loop attached. It can sit quietly for months, green on every dashboard, until a dependency has a bad thirty seconds. Then it turns that blip into a fleet-wide event.

It is worth searching your own services for downstream calls without deadlines and retries without a shared budget. Those two settings deserve to be reviewed together, because together they can change the shape of a failure.

The lesson that stayed with me, though, was about diagnosis.

The CPU theory was clean. It was mechanistic. It came with a satisfying one-line fix. And it survived three pieces of contradictory evidence because we had started treating it as an answer instead of a claim.

A useful root cause makes predictions you can check. When the topology, timing, and distribution of failures all disagree with the story, elegance stops counting. In this case, one dull line in a log file told us more than the tidy explanation we had already grown attached to.

That was the expensive part of the incident: not the eventual three-line fix, but learning to let the evidence ruin a good story.

Thanks for reading ✌️

References

The Acknowledgment Gap - How Event-Driven Systems Lose Messages Without Errors

Sahan — Thu, 16 Jul 2026 01:05:00 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

If you have built event-driven systems for any length of time, you have probably internalized the mantra of at-least-once delivery: keep retrying until the work is done, and design everything downstream to be idempotent. It’s good advice. But there’s a subtle failure mode that hides right underneath it - one where your system faithfully reports success, commits its progress, and quietly drops work on the floor. No exception, no alert, no dead letter. Just a message that was supposed to do something, and didn’t.

I ran into this recently while debugging why a small percentage of events were mysteriously never being processed. Everything looked healthy. The producer got a 2xx. The consumer committed its offset. The dashboards were green. And yet the work never happened. This post is about that gap - the space between “the system accepted my request” and “the system actually did the work” - and why it’s one of the more dangerous places for a distributed system to lose data.

Motivation

Most write-ups about message-processing reliability focus on the well-known hazards: duplicate delivery, out-of-order messages, poison pills, consumer lag. Those are real, and there’s plenty written about them. What I found much less discussed is the case where the acknowledgment itself is a lie - where a component reports success for an operation that has only been accepted, not completed, and a second component treats that acknowledgment as permission to throw the original message away.

This is a design smell that shows up across all sorts of stacks: a queue consumer that calls an async API, a workflow engine that enqueues a job, a service that hands off to a background worker. Any time you have a handoff across an asynchronous boundary , you have the potential for this gap. So I want to walk through the anatomy of the bug from first principles, then talk about how to close it properly.

Background: two kinds of “yes”

Before we get to the bug, let’s be precise about acknowledgments, because the whole problem lives in some sloppy vocabulary.

When a system replies to your request, it can mean one of two very different things:

“I have accepted your request.” - I’ve durably recorded your intent, and I promise to try to do the work. Think HTTP 202 Accepted. The work hasn’t happened yet.
“I have completed your request.” - The work is done, and its effects are durable. Think 200 OK with a result body.

These are worlds apart, and conflating them is the root of a lot of pain. The trouble is that many APIs return the same status code for “accepted” whether or not the work will eventually succeed. A 202 (or a 204 No Content, which is even more ambiguous) tells you the request was received. It tells you nothing about whether the work will run.

Now layer on the consumer side. A huge number of event-driven systems are built on brokers that use offset-based consumer groups - Apache Kafka being the canonical example. If you want a primer, I wrote an introduction to Apache Kafka a while back. The mental model is simple:

Messages in a partition have monotonically increasing offsets.
Your consumer reads a message, does some work, and then commits (or “marks”) the offset to say “I’m done with everything up to here.”
If the consumer crashes before committing, the broker redelivers from the last committed offset. That’s what gives you at-least-once semantics.

💡 The offset commit is a promise about the past. When you commit offset N, you are telling the broker “every message up to and including N has been fully handled, and you never need to give them to me again.” If that statement isn’t actually true, you have manufactured data loss with your own hands.

Hold onto those two ideas - the ambiguous “yes” and the offset-as-promise - because the bug is what happens when they collide.

Anatomy of the pipeline

Let me describe a deliberately generic pipeline. Strip away the specific technologies and almost every async system looks like this:

The consumer’s job is to read an event and translate it into an action by calling some downstream control plane - an async job API, a workflow trigger, a task queue. The control plane accepts the request and, at some later point, a worker actually executes it.

The consumer’s loop, in pseudocode, looked roughly like this:

for {
    msg := broker.Read()

    err := dispatchJob(msg) // calls the async control plane
    if err != nil {
        retryWithBackoff(dispatchJob, msg)
    }

    // Always commit so we never get stuck reprocessing a bad message.
    broker.Commit(msg)
}

At a glance this looks reasonable, even defensive. There’s a retry with backoff. There’s a comment explaining that we always commit to avoid getting wedged on a poison message. Someone clearly thought about failure here.

And that is exactly what makes the bug so insidious.

The bug: the acknowledgment gap

Here’s the sequence that loses data.

The consumer reads a message and calls dispatchJob.
The control plane returns 204 No Content - “request accepted, a job has been created.” From the consumer’s point of view, this is success. err is nil.
Milliseconds later, and entirely outside the consumer’s view , the control plane rejects the job before it executes. Maybe a concurrency quota was exceeded. Maybe an admission controller said no. Maybe the queue was full. The job transitions straight to a terminal “rejected” state without a single line of business logic ever running.
Back in the consumer, dispatchJob returned nil, so the retry loop never fires - there was nothing to retry, as far as it knows.
The consumer commits the offset.

The offset moves forward. The broker will never redeliver that message. The job never ran. And nobody was told.

This is the acknowledgment gap. The failure happened in the window between “accepted” and “executed” , and our success signal was wired to the wrong end of that window. We treated “a job was created” as if it meant “a job will run,” and those are not the same statement.

💡 The most dangerous bugs aren’t the ones that throw. They’re the ones that return nil. An exception is a gift - it’s the system telling you something is wrong. Silent, structurally-invisible loss gives you nothing to catch.

What made it worse is that the “always commit” decision - added defensively to avoid an infinite reprocessing loop - turned a recoverable failure into an unrecoverable one. The one safety mechanism that could have saved us (letting the broker redeliver) was disabled precisely when we needed it.

Why the usual instincts don’t save you

When engineers first see this, they reach for familiar fixes. Most of them don’t actually close the gap:

“Just check the status code.” We did. It was 204. The status code describes the acceptance, not the outcome. The information we needed didn’t exist yet at the moment we got the response.
“Add a retry.” There was one. It only triggers on a failed dispatch, not a failed execution. You can’t retry something you don’t know failed.
“Make it idempotent.” Idempotency is necessary but not sufficient here. Idempotency protects you from doing the work twice; it does nothing to protect you from doing it zero times.
“Use exactly-once semantics.” Setting aside the long debate about whether exactly-once is even a coherent goal across independent systems - the transactional guarantees of your broker do not extend into a third-party control plane you’re calling over HTTP. The moment you cross that boundary, you’re back to coordinating two independent systems with no shared transaction.

The real issue is architectural: we committed our durable progress based on a signal that didn’t actually confirm the work was durable. No amount of tuning the individual pieces fixes that. You have to move the acknowledgment.

Fixing it: verify before you ack

The core principle is a single sentence:

💡 Never acknowledge a message until you have confirmed the work it represents has actually started (or completed) - not merely been accepted.

Everything else is mechanics. Let’s walk through them, because the mechanics are where the interesting distributed-systems problems hide.

1. Close the loop: confirm execution, don’t assume it

Instead of trusting the 204, the consumer now verifies that the dispatched job reached a real running (or terminal-success) state before committing. In practice that means polling the control plane’s read API after dispatch:

started, err := dispatchAndVerify(msg)
if started {
    broker.Commit(msg) // safe: the work is genuinely underway
} else {
    // do NOT commit - let redelivery give us another shot
    alert(msg, err)
}

dispatchAndVerify dispatches, then polls: did a job actually enter a non-rejected state? If it sees the tell-tale “rejected before executing” terminal state, or it can’t find the job at all within a bounded window, it treats that as a failure - which is the thing our original code could never see.

This is really just applying read-after-write thinking to a control plane. Don’t trust the write acknowledgment; go read the state back and confirm reality matches your intent.

2. The correlation problem

Here’s a genuinely tricky sub-problem that this exposes, and it’s a great example of why distributed systems are hard: the dispatch API often doesn’t tell you the ID of the thing it just created. You fire a request, you get back 204 No Content - literally no content - and now you need to find “the job I just created” among all the jobs.

You’re left correlating on secondary signals:

A creation timestamp window (“a job created after time T”), which is racy under concurrency - two near-simultaneous dispatches can be ambiguous.
A business key embedded into the job’s metadata at creation time, if the API lets you set something like a name or a label.

The robust fix is to make the work self-identifying : stamp a correlation key you already own (the entity ID, a request UUID) into the job at dispatch time, so that when you read the state back you can match on it exactly rather than guessing by time. If your control plane supports naming or tagging the work, use it. This is the async equivalent of propagating a trace ID, and it pays for itself the first time you have to debug a race.

💡 Any time you hand work across an async boundary, ask: “When this comes back, how will I know it’s mine?” If the answer is “by timestamp,” you have a race waiting to happen.

3. Bounded redelivery, or: don’t trade loss for a hot loop

The moment you say “don’t commit on failure so the broker redelivers,” someone will rightly point out the opposite failure mode: what if the work keeps failing? Now you’ve built an infinite reprocessing loop, and you’re hammering a control plane that’s already unhappy. This is the eternal tension:

Commit too eagerly → you lose messages (the original bug).
Never commit on failure → you can wedge the consumer forever.

The answer is bounded retries with escalation. Track how many times a given message has been through the wringer - keyed by its stable identity (partition + offset, or a business key) - and:

Retry with exponential backoff while attempts remain, so a transient quota exhaustion gets a chance to clear.
Once you’ve exhausted the budget, stop, escalate loudly, and then commit so a single doomed message can’t block the whole partition behind it.

That final commit is not “giving up silently” - it’s a deliberate, observable decision to route the message to a human (or a dead-letter queue) instead of blocking the stream. The difference between this and the original bug is everything: the original dropped work with zero signal; this drops it only after N visible, alarmed attempts.

4. Separate transient failures from terminal ones

A subtle but important refinement: when you poll to verify, a failure to read the state is not the same as the work being rejected. If your verification call itself hits a network blip or a 503, and you treat that as “the job failed,” you’ll re-dispatch and potentially create duplicate work - trading a lost-message bug for a double-processing bug.

So the verification loop needs to distinguish:

“I confirmed the job was rejected” → terminal, re-dispatch is warranted.
“I couldn’t reach the control plane to check” → transient, just retry the read, don’t re-dispatch.

Only definitive answers should drive irreversible decisions. Everything else is a retryable read. This is the same discipline as not making state transitions on ambiguous signals - you wait until you actually know.

5. Make the invisible visible

The reason this bug survived in production is that it was structurally unobservable. So the last piece is observability, and it’s not optional:

Emit a metric/event whenever a dispatched unit of work fails to start.
Alert when the retry budget is exhausted and a message is dropped.
Log the correlation key, the attempt count, and a link to the rejected work.

If your system is going to make a hard decision like “I’m dropping this after five tries,” that decision must be the loudest thing in the room, not a silent commit. A good rule of thumb: every place your code can decide to discard work should be capable of paging a human. You may choose not to page - but the capability being there forces you to consciously design the failure path instead of falling into one.

Stepping back: the general principle

If you zoom out from the specific mechanics, this whole class of bug reduces to a few reusable principles that are worth carrying into any distributed system you build:

Distinguish “accepted” from “completed” - always. Treat them as different events with different names, different metrics, and different handling. Never let one masquerade as the other.
Anchor your durable acknowledgment to the durable outcome. Commit your offset (or delete your message, or mark your row done) based on confirmation of the effect you care about, not on a transport-level receipt.
A nil error is a claim, not a fact. Verify claims that cross trust boundaries, especially async ones. Read-after-write is cheap insurance.
Bound every retry, and escalate at the boundary. Unbounded retries and silent drops are two sides of the same coin; the cure for both is a visible, finite budget with a loud exit.
Only act irreversibly on unambiguous signals. Transient “I don’t know” should never trigger a decision that assumes “no.”

None of these are novel on their own. What’s interesting is how a single missing distinction - accepted vs. executed - cascades into silent data loss when it meets an offset commit. It’s a good reminder that in distributed systems, the bugs rarely live inside a component. They live in the seams between components , where two reasonable local decisions add up to one unreasonable global one.

Tradeoffs

Nothing here is free, and I’d be doing you a disservice to pretend otherwise.

What you gain

No more silent loss - the failure mode that’s hardest to detect and most damaging to trust.
A verifiable, observable processing pipeline where “done” actually means done.

What it costs

Extra latency and load. Verifying execution means additional reads against the control plane per message. Poll intervals and budgets need tuning.
More moving parts. Correlation keys, attempt tracking, and escalation paths are code you now own and test.
You must embrace at-least-once for real. Verify-and-redeliver will occasionally produce duplicates (e.g., if a job actually started but your confirmation read failed). Idempotency downstream stops being optional - but that was always true; this just makes it honest.

For low-value, high-volume telemetry you might happily accept silent loss and skip all of this. For work where every single message must result in an action, the cost is obviously worth it. As always, the right answer depends on what the data is worth.

Conclusion

The most memorable bugs are the ones that teach you to distrust a word you’d been using carelessly. For me, that word was “success.” A 2xx is not success. A committed offset is not success. Success is the effect you actually wanted, confirmed to be durable. Everything else is just a system being polite.

If you take one thing away: go look at your event-driven pipelines and ask where you commit progress based on an acknowledgment rather than a confirmation. If those two things are wired together, you probably have an acknowledgment gap hiding in there too - quietly green on every dashboard, right up until someone asks where their data went.

Thanks for reading ✌️

References

Introducing gh-weekly-updates - Automate Your Weekly GitHub Impact Summaries

Sahan — Sat, 21 Mar 2026 22:22:00 +0000

If you are anything like me, you’ve probably spent a Friday afternoon trying to remember everything you did that week. Maybe it’s for a standup, a 1:1 with your manager, or just to keep track of your own progress. You end up clicking through PRs, issues, and Slack threads, trying to piece together a coherent story. It’s tedious, and honestly, it’s time you could spend doing actual work.

That’s why I built gh-weekly-updates - a CLI tool that automatically collects your GitHub activity and generates a structured weekly summary using AI.

💡 GitHub repo : github.com/sahansera/gh-weekly-updates. It’s open source and available on PyPI!

The Problem

As engineers, we’re constantly shipping code, reviewing PRs, filing issues, and jumping into discussions. But when it comes time to reflect on the week, all that context is scattered across repos. I wanted something that could:

Pull all my GitHub activity into one place
Summarise it in a way that highlights what actually matters
Run on a schedule so I don’t have to think about it

I couldn’t find anything that did exactly this, so I built it.

What It Does

gh-weekly-updates connects to the GitHub API, collects your activity for a given period, and sends it to an AI model (via GitHub Models) to produce a structured Markdown summary.

Here’s what it picks up:

Pull requests you authored (with merge status, additions/deletions, changed files)
Pull requests you reviewed
Issues you created
Issue comments you left
Discussions you started or participated in

The output is grouped by project or theme and structured into sections like Wins, Challenges, and What’s Next. You can also customise the prompt to match whatever format your team uses.

Getting Started

It’s a Python CLI tool, so you can install it with pip:

pip install gh-weekly-updates

Then just run it:

# If you're already logged in with the GitHub CLI
gh auth login
gh-weekly-updates

That’s it. It will auto-discover repos you contributed to in the past week and generate a summary.

You can also point it at specific repos and date ranges:

gh-weekly-updates --since 2026-02-09 --until 2026-02-16 --repos my-org/my-repo

Configuration

For more control, you can create a config.yaml:

org: my-org

repos:
  - my-org/api-service
  - my-org/web-app

model: openai/gpt-4.1

# Automatically push the summary to a repo
push_repo: my-user/my-weekly-updates

# Customise the AI prompt
prompt_file: my-prompt.txt

The config supports everything from repo lists to custom prompts. You can even swap out the AI model if you have a preference.

Running on a Schedule with GitHub Actions

This is where it gets really useful. You can set up a GitHub Actions workflow to run it every Monday morning and push the summary to a repo automatically:

name: Weekly Summary

on:
  schedule:
    - cron: '0 9 * * 1' # Every Monday at 9am UTC
  workflow_dispatch:

jobs:
  summarise:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - run: pip install gh-weekly-updates

      - name: Generate summary
        env:
          GITHUB_TOKEN: ${{ secrets.GH_PAT }} # must be named GITHUB_TOKEN
        run: gh-weekly-updates --config config.yaml

Now every Monday, you get a fresh summary committed to your repo. No manual effort required.

Custom Prompts

The default prompt produces a summary with Wins, Challenges, and What’s Next sections. But you can tailor it to your needs. For example, if your team does impact-style updates, you might want sections like Strategic Influence or Next Steps.

Just create a text file with your prompt and reference it in your config:

prompt_file: my-custom-prompt.txt

The prompt receives all your raw activity data as context, so you can shape the output however you like.

Why Open Source?

I initially built this for myself to automate my own weekly updates at work. But I figured other engineers probably have the same problem, so I cleaned it up and open-sourced it. The tool is intentionally simple - it does one thing and tries to do it well.

If you find it useful, give it a ⭐ on GitHub. And if you have ideas for improvements, PRs and issues are always welcome!

What’s Next

A few things I’m thinking about for future releases:

More activity sources : Picking up commit messages, release notes, and code review comments
Multiple output formats : Slack messages, email digests, Notion pages
Team summaries : Aggregate activity across a whole team, not just one person

If any of these sound interesting to you, feel free to open an issue or start a discussion on the repo.

Deploying GitHub Self-Hosted Runners on Your Home Kubernetes Cluster with ARC

Sahan — Tue, 05 Aug 2025 12:28:00 +0000

If you followed my last post, Building a Home Lab Kubernetes Cluster with Old Hardware and k3s, you now have a proper x86 Kubernetes cluster humming away on your old laptops. So, what’s next? Time to put that cluster to work—let’s run GitHub Actions jobs on your own hardware!

Why? Because if you have got decent hardware - self-hosting your runners might be faster, gives you full control (no GitHub minutes limit!), and lets you run bigger jobs (CI, builds, ML, you name it) on your home infra. And with Actions Runner Controller - ARC, managing runners at scale on Kubernetes is surprisingly easy.

Here’s how to set it all up.

What is ARC and Why Should You Care?

ARC (Actions Runner Controller) is an open-source Kubernetes operator from GitHub. It spins up and manages GitHub Actions runners as Kubernetes pods—no more manually registering runners, no more pets, just cattle. Runners auto-scale up and down as jobs arrive. It’s perfect for CI/CD, especially on clusters you own.

Here’s a high-level view of how it works under the hood

Prerequisites

Working Kubernetes cluster (see previous post)
kubectl and helm installed on your machine
A GitHub Personal Access Token (PAT) with repo and admin:org scopes

1️⃣ Pre-Setup: Quick Checks

Make sure you have what you need:

which helm
kubectl version --client
helm list -A

If those commands work, you’re good to go.

2️⃣ Install ARC Controller

Let’s install the ARC controller into your control plane namespace:

NAMESPACE="actions-runner-controller"
helm install arc \
  --namespace "${NAMESPACE}" \
  --create-namespace \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller

This deploys the controller which manages your runners.

3️⃣ Deploy a Runner Scale Set

Time to create the runners that will actually do the work. Replace the example GitHub URL and PAT with your details:

INSTALLATION_NAME="arc-runner-set"
NAMESPACE="arc-runners"
GITHUB_CONFIG_URL="https://github.com/youruser/yourrepo"
GITHUB_PAT="ghp_123456..."

helm install "${INSTALLATION_NAME}" \
  --namespace "${NAMESPACE}" \
  --create-namespace \
  --set githubConfigUrl="${GITHUB_CONFIG_URL}" \
  --set githubConfigSecret.github_token="${GITHUB_PAT}" \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

GITHUB_CONFIG_URL: The repo or org you want to run jobs for.
GITHUB_PAT: Your Personal Access Token.

4️⃣ Check That It’s Working

Verify the controller and runner pods are up:

# Controller
kubectl get pods -n actions-runner-controller

# Runners
kubectl get pods -n arc-runners

# Runner set status
kubectl get AutoscalingRunnerSet -A
kubectl describe AutoscalingRunnerSet arc-runner-set -n arc-runners

You should see your runners show up as pods. If you trigger a workflow in your GitHub repo, you’ll see a pod spin up, do the job, and then shut down—magic.

5️⃣ Testing It Out

Here’s the fun part. Create a simple GitHub Actions workflow in your repo to test the runners:

name: Test ARC Runners
on: [push]

jobs:
  build:
    runs-on: arc-runners # This tells GitHub to use your self-hosted runners. Use the NAMESPACE name you defined in step 3.
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Run a script
        run: echo "Hello from ARC Runner!"
      - name: List files
        run: ls -la

Here’s ci.yaml from my githubstats repo if you need a working example: ci.yaml

Make a commit to trigger the workflow. You should see the runner pod spin up, execute the job, and then terminate.

Here's the workflow running on my ARC self-hosted runner

6️⃣ Monitoring (Bonus: Grafana)

Want to geek out and monitor your runners? If you’ve set up Prometheus/Grafana (see my upcoming post if not!), you can:

Check pod CPU/memory usage
Track how many runners are running
See logs for each pod

Handy queries for Grafana dashboards:

# Number of ARC runner pods
kube_pod_status_phase{namespace="arc-runners", phase="Running"}

# Pod CPU usage
rate(container_cpu_usage_seconds_total{namespace="arc-runners"}[5m])

Here's mine:

Here's what the workflow looks like in action

6️⃣ Useful Commands for Day-to-Day Ops

# Watch runner scaling in real-time
kubectl get AutoscalingRunnerSet -n arc-runners -w

# See events and troubleshoot
kubectl get events -n arc-runners --sort-by=.metadata.creationTimestamp

# Pod logs (for a specific runner)
kubectl logs -n arc-runners <pod-name>

Troubleshooting Tips

Runner not connecting? Double-check your PAT and network access.
Pods stuck or crash-looping? Check logs for clues and make sure your cluster has enough resources.
Can’t see runners in GitHub? Make sure the config URL matches your repo/org and the PAT has correct scopes.

That’s It! You’re Running GitHub Actions on Your Own Cluster

You now have GitHub Actions jobs running at home on your cluster, scaling up and down automatically. No more slow or limited runners. Your home lab just levelled up—CI/CD, builds, ML, you name it.

Stay tuned for my next post where I’ll show you how to get beautiful observability dashboards and set up alerting for your home cluster.

Happy automating!

Questions? Want to show off your setup? Ping me on X (Twitter) or drop a comment below!

Building a Home Lab Kubernetes Cluster with Old Hardware and k3s

Sahan — Fri, 01 Aug 2025 10:30:00 +0000

If you've read my previous post about building a Raspberry Pi k3s cluster, you know I'm a huge fan of home labs. There's something uniquely satisfying about getting distributed systems running on a bunch of hardware you already own. This time, though, I wanted something a bit more powerful-a cluster that could handle not just learning and tinkering, but also heavier dev, CI/CD, and even ML workloads.

And as it turns out, there's a ton you can do with a handful of old laptops, a simple switch, and Ubuntu Server. If you're thinking about upgrading your home cluster or want to avoid vendor lock-in, this one's for you.

Why Move From Raspberry Pi to x86?

Don't get me wrong-RPi clusters are awesome for learning, hacking, and even some light home automation. But if you've ever tried running real dev pipelines, CI/CD, or any data-heavy ML stuff, you'll hit those limits fast. Plus, WiFi can get a bit flaky when you're trying to keep nodes connected under load.

I had a few spare laptops sitting around, and it made perfect sense to give them a second life and push them to their limits. Bonus: with Ethernet and a decent switch, you get much more reliable connectivity.

Network Setup: Keep It Simple (and Reliable)

For this build, I kept things straightforward: all nodes are wired to a simple network switch. This gives much better performance and reliability compared to WiFi, but honestly, you could still pull this off over wireless if needed.

My network switch - how it all started 🛜

A quick tip:

Reserve static IPs for each node using your router's DHCP reservations.
This makes node management, SSH, and Kubernetes networking so much easier.
Check your DHCP table and make sure every machine has a unique, predictable IP address.

Getting Ubuntu Server Set Up on Each Node

Here's a high level diagram of what my setup looks like:

Let's talk about prepping each machine. My old laptops had a new lease on life running Ubuntu Server. I recommend Ubuntu Server LTS for its simplicity and compatibility. If you're using VMs, just make sure to set the NIC to bridged mode so each VM acts as a full member of your home LAN.

Here's my go-to checklist for each node:

Update the system and install SSH:

   sudo apt update && sudo apt upgrade -y
   sudo apt install openssh-server -y
   sudo systemctl enable ssh
   sudo systemctl start ssh

Set a memorable hostname:

   sudo hostnamectl set-hostname master   # or worker-1, etc.

(Optional) Update /etc/hosts for local resolution:

   sudo nano /etc/hosts

Change any lines like 127.0.1.1 old-hostname to your new hostname.

Find your node's IP address:

   ip a

Use this IP for your DHCP reservation so it always stays the same.

Test SSH from another machine:

   ssh <username>@<node-ip>

Installing k3s: Lightweight, Powerful, Familiar

This process feels pretty magical the first time you see it all come together. I stuck with k3s-lightweight and perfect for home or edge clusters.

On the Control Plane Node ("master")

Install k3s:

   curl -sfL https://get.k3s.io | sh -

Check that k3s is running:

   sudo systemctl status k3s

Get your cluster join token:

   sudo cat /var/lib/rancher/k3s/server/node-token

Copy this somewhere safe-you'll need it for your worker nodes.

Get the node's LAN IP:

   ip a

Let's say your master node IP is 192.168.0.200.

Check your cluster status:

   kubectl get nodes

If you ever want to reset/reinstall k3s, just run:

sudo /usr/local/bin/k3s-uninstall.sh

On Each Worker Node ("worker-1", "worker-2", etc.)

Join the cluster:

   curl -sfL https://get.k3s.io | K3S_URL=https://192.168.0.200:6443 K3S_TOKEN=<your-token-here> INSTALL_K3S_EXEC="--node-ip=<worker-ip>" sh -

Replace 192.168.0.200 with your control plane's IP.

Replace <your-token-here> with the token from your master node.

Replace <worker-ip> with this worker's IP, e.g., 192.168.0.201.

That's it! The node will auto-register with your cluster. No manual kubeconfig needed.
Check the cluster again from the master:

   kubectl get nodes

You should see both master and your worker(s) as Ready!

Troubleshooting and Gotchas

As with any home lab project, there are always a few snags-here's what I ran into and how to fix it:

Node not joining?
- Double-check your token (no spaces, copy the whole thing).
- Make sure you can ping the control plane from your worker node.
- Firewalls can get in the way. Ensure port 6443 is open between nodes.
- If you get hostname conflicts, just change the hostname and restart the k3s agent (sudo systemctl restart k3s-agent).
Cluster state looks weird after hostname change?
- You might see both the old and new hostnames in kubectl get nodes. Just delete the old one:
```
kubectl delete node <old-node-name>
```
IP address keeps changing?
- Make a DHCP reservation for each node's MAC address in your router.
WiFi unreliable?
- Ethernet is always the best bet for clusters, especially for heavy workloads.

What's Next?

With the basics in place, you're ready to:

Run and test real-world apps and dev environments
Build your own CI/CD pipelines
Experiment with ML workloads on dedicated nodes

This way, you get all the power of a real Kubernetes lab, but full control and no monthly surprises from a cloud provider.

Lessons Learned

RPi clusters are great for edge computing and learning, but once you need real muscle, x86 hardware makes a huge difference.
WiFi is convenient, but Ethernet is still king for stable clusters-especially if you're doing anything performance-sensitive.
Old laptops/desktops are a goldmine for home lab builds. Don't let them collect dust!

Final Thoughts

If you have old machines and a bit of curiosity, you can build a cluster that's surprisingly capable-no Raspberry Pis required.

And if you've already done it with RPis, this is the perfect upgrade path.

Keep an eye out for my next post where I'll deep-dive into adding observability and tooling!

Happy clustering! 🫡