DEV Community: yureki_lab

How I Let Claude Code Write My Load Tests and Caught 3 Bottlenecks Pre-Launch

yureki_lab — Sun, 27 Sep 2026 14:32:21 +0000

TL;DR

I had never written a real load test before a launch. I let Claude Code build a k6 suite for my Node.js API in an afternoon, and it surfaced three bottlenecks that would have taken the service down on day one. This post walks through the exact prompts, the k6 setup, what broke, and the five lessons I'd apply to any perf-testing work with an AI coding agent. ⚡

The Problem

Two weeks before launching a customer-facing API (Node.js 22.x, Fastify, Postgres 16, Redis 7), I realized I had a pretty embarrassing gap: zero load testing.

We had 340 unit tests. We had integration tests. We had a CI pipeline that ran in under ten minutes. What we did not have was any idea what happened when 200 people hit the checkout endpoint at the same time.

I knew the theory. Ramp up virtual users, watch p95 latency, look for the cliff. But every time I sat down to write a k6 script, I hit the same wall: the API had 38 endpoints, most of them needed auth tokens and realistic payloads, and I did not want to hand-write 38 scenario files with fake data that looked nothing like production traffic.

The constraint that made this interesting: I had about one working day to spend on it. Anything longer and it would eat into the launch buffer.

So I decided to run an experiment. I would give Claude Code (v2.1, running in the terminal) the OpenAPI spec and the route handlers, and let it own the load-testing work end to end. My job would be to review, run, and interpret.

How I Solved It

Step 1: Feed it the shape of the traffic, not just the spec

My first attempt was lazy. I pointed Claude Code at openapi.yaml and said "write k6 load tests for every endpoint." It produced 38 files that each hammered one endpoint with a fixed payload. Technically correct, practically useless. Real users don't call GET /orders/{id} in isolation; they log in, browse, add items, and check out.

So I backed up and gave it context about the actual user journeys. Here is the prompt that worked:

Here are the 3 user journeys that matter for launch, in order of business impact:

1. Browse → view product → add to cart → checkout (60% of traffic)
2. Login → view order history → view single order (30%)
3. Admin: list orders with filters, paginated (10%)

Write a k6 test suite that:
- Models these as 3 scenarios with the traffic split above
- Reuses the auth token per virtual user (login once, not per request)
- Generates realistic payloads from the Zod schemas in src/schemas/
- Ramps from 0 to 300 VUs over 5 minutes, holds 10 minutes, ramps down
- Fails if p95 > 500ms or error rate > 1% on any scenario

Don't write one file per endpoint. Organize by journey.

The difference between "here's the spec" and "here's who uses this and how" was enormous. The second version produced a suite I actually wanted to run.

Step 2: The k6 structure it generated

It landed on a layout I have kept since:

loadtest/
├── config.js          # thresholds, stages, base URL
├── lib/
│   ├── auth.js        # login once per VU, cache token
│   └── factories.js   # payload generators derived from Zod schemas
├── scenarios/
│   ├── shopper.js
│   ├── returning-customer.js
│   └── admin.js
└── main.js            # wires scenarios + traffic split

The load-bearing part is main.js. Here is the trimmed version:

import { shopper } from './scenarios/shopper.js';
import { returningCustomer } from './scenarios/returning-customer.js';
import { admin } from './scenarios/admin.js';

export const options = {
  scenarios: {
    shopper: {
      executor: 'ramping-vus',
      exec: 'shopper',
      startVUs: 0,
      stages: [
        { duration: '5m', target: 180 },
        { duration: '10m', target: 180 },
        { duration: '2m', target: 0 },
      ],
    },
    returning: {
      executor: 'ramping-vus',
      exec: 'returningCustomer',
      startVUs: 0,
      stages: [
        { duration: '5m', target: 90 },
        { duration: '10m', target: 90 },
        { duration: '2m', target: 0 },
      ],
    },
    admin: {
      executor: 'constant-vus',
      exec: 'admin',
      vus: 30,
      duration: '17m',
    },
  },
  thresholds: {
    'http_req_duration{scenario:shopper}': ['p(95)<500'],
    'http_req_duration{scenario:returning}': ['p(95)<500'],
    'http_req_duration{scenario:admin}': ['p(95)<800'],
    'http_req_failed': ['rate<0.01'],
  },
};

export { shopper, returningCustomer, admin };

Two things I would not have thought to do on my own:

Per-scenario thresholds using tags, so a slow admin endpoint doesn't mask a fast checkout path (or vice versa)
Splitting the 300 VUs across scenarios to match the 60/30/10 traffic mix, instead of one big pool that picks randomly

The factories file was the other clever bit. Instead of hardcoding fake data, it imported the same Zod schemas the API uses for validation and generated payloads that would always pass. When I later added a required field to the checkout schema, the load test picked it up with zero changes.

Step 3: Run it against staging and watch things break

I ran it with k6 v1.1 against a staging environment sized identically to production (2 API replicas, 1 Postgres instance, 1 Redis).

k6 run loadtest/main.js --out json=results.json

First run: failed at 140 VUs. Not 300. Not even half.

This is where the workflow got interesting. I did not want to guess at the cause, so I pasted the k6 summary output and the API logs from the same window into Claude Code and asked it to correlate. Here is what it found, in the order it found them.

Bottleneck 1: The database pool was sized for a laptop

graph LR
  A[300 VUs] --> B[2 API replicas]
  B --> C[Pool: 10 conns each]
  C --> D[(Postgres: max 100)]
  style C fill:#f96,stroke:#333

The Postgres client pool was set to its default of 10 connections per replica. With two replicas, that is 20 concurrent queries max. Once more than about 20 requests needed the DB at the same time, everything else queued. The p95 went from 80ms at 100 VUs to 2,400ms at 140 VUs. Classic cliff.

Claude Code spotted it from the log pattern: timeout exceeded when trying to connect spiking exactly when latency did. Fix was a one-liner in the pool config, bumped to 40 per replica, still comfortably under the Postgres limit. That alone got us to 220 VUs.

Bottleneck 2: An N+1 hiding on one path

At 220 VUs the returning-customer scenario started failing thresholds while shopper stayed fine. That asymmetry was the clue.

The order history endpoint used a batching layer to load line items. The single-order endpoint did not. It looked like this:

// Before: one query per line item 😬
const order = await db.orders.findById(id);
for (const item of order.itemIds) {
  item.product = await db.products.findById(item.productId);
}

One order with 12 line items meant 13 queries. Under load, that path alone was generating more DB traffic than the entire shopper journey. Claude Code proposed the fix and wrote it:

// After: one query, keyed lookup
const order = await db.orders.findById(id);
const products = await db.products.findByIds(order.itemIds.map(i => i.productId));
const byId = new Map(products.map(p => [p.id, p]));
order.items = order.itemIds.map(i => ({ ...i, product: byId.get(i.productId) }));

I had reviewed that file before. I had tests for that file. None of that caught it, because a 13-query endpoint is fast when one person calls it.

Bottleneck 3: Logging was eating CPU

The last one was the sneakiest. At 280 VUs, p95 crept up again, but the DB was fine and Redis was fine. CPU on the API replicas was pinned at 100%.

Claude Code asked me to run a 30-second CPU profile during load. The top frame was JSON.stringify inside the request logger. Someone (me, six months ago) had added a log line that serialized the full request body at info level for "debugging." On a checkout request with a 40-item cart, that was serializing several kilobytes per request, 200 times a second.

// Before
logger.info({ body: req.body }, 'incoming request');

// After
logger.info({ path: req.url, bodyBytes: req.headers['content-length'] }, 'incoming request');

Removing it dropped CPU to 60% at the same load and bought us the last 20 VUs of headroom.

The final run

After all three fixes:

Metric	Before	After
Max VUs before threshold failure	140	300+ (target hit)
p95 latency at 300 VUs	n/a (failed)	310ms
Error rate at 300 VUs	11%	0.2%
Wall-clock time spent	0	~6 hours

Six hours, including the runs themselves. Three bugs that would have surfaced on launch day in front of real customers.

Lessons Learned

1. The agent is only as good as your traffic model

"Write load tests for this API" gets you noise. "Here are the three journeys and their traffic split" gets you a real test. The single most important input was not the spec; it was me writing down who uses the system and in what order. If you can't describe your traffic in three bullet points, do that before touching an agent.

2. Derive payloads from your validation schemas, not from fixtures

Letting the factories import the Zod schemas meant the load test could never drift out of sync with the API. This is one of those things an agent will suggest if you ask "how do we keep this from rotting," and it is worth asking every time.

3. Per-scenario thresholds beat global ones

If I had used a single global p95 threshold, the N+1 on the order-detail path would have been averaged away by the fast shopper traffic. Tag your scenarios and set thresholds per tag. This was the difference between finding bug 2 and shipping it.

4. Let the agent correlate logs and metrics, but you run the profiler

Claude Code was excellent at reading a k6 summary next to a log excerpt and saying "these two spikes line up, here is the likely cause." It could not run the CPU profile for me on staging, and I would not want it to. The division of labor that worked: the agent does the pattern matching and proposes the fix, I execute anything that touches a live environment.

5. Load tests find the bugs your unit tests are structurally blind to

All three bottlenecks were invisible at concurrency 1. Pool sizing, N+1s, and hot-path serialization cost only show up under load. If your test pyramid has no load layer, you have a category of bugs you are guaranteed to find in production. An agent makes that layer cheap enough that "no time" stops being an excuse.

What's Next

Wire it into CI on a nightly schedule against staging, with results posted to a channel. A daily p95 trend line catches regressions before anyone notices.
Add a soak test (2 hours at 60% of peak) to hunt memory leaks. The 17-minute run is too short to catch slow growth.
Teach the agent to read the flame graph. Right now I run the profiler and paste the top frames. I want to try feeding it the raw profile output and see if it can find the hot path without my summary.

Wrap-up

If you have a launch coming and no load tests, block one afternoon, write your three traffic journeys on paper, and hand them to Claude Code with your schemas. You will probably find something. I found three somethings.

If this was useful, follow me here on Dev.to for more build logs like this one. I write about running AI coding agents on real production work, including the parts that go wrong. And if you have found a bottleneck under load that no unit test could have caught, I would love to hear about it in the comments. 🚀

How I Built a Regression Suite for My AI Coding Agent's Prompts: 5 Lessons

yureki_lab — Sat, 26 Sep 2026 14:32:35 +0000

TL;DR

I changed one line in my coding agent's spec file and, two weeks later, it quietly started skipping tests on every PR. Nobody noticed because nothing "broke." So I built a regression suite for the prompts themselves: frozen repo snapshots, canned tasks, deterministic graders, and an LLM judge, all gated in CI. Here's how it works and the 5 lessons that cost me the most to learn.

The Problem

I run a fully autonomous implementation system built on Claude Code (v2.x at the time of writing). It picks up tasks, implements them, runs tests, opens PRs, and fixes its own CI failures. It has been doing this 24/7 for months across several repos.

The system's behavior is controlled by prompts: a project spec file (CLAUDE.md), a handful of skill definitions, and an orchestrator prompt that decides what to work on next. These files are code. They change the system's behavior just as much as a Python function does.

But I was treating them like config. I'd tweak a sentence, eyeball one run, and merge. Zero tests.

Here is the incident that changed my mind. I added this line to the spec file to reduce noise in PR descriptions:

Keep PR descriptions short. Do not list every command you ran.

Reasonable, right? Two weeks later I noticed the agent's PRs had stopped including test output. Then I noticed it had stopped running the full test suite for small changes. The agent had generalized "do not list every command" into "running fewer commands is preferred." That's a stretch, but it was consistent, and it took 14 days and one broken deploy to catch.

The core problem: prompt changes have non-local effects, and I had no way to measure them before merging. ⚠️

I needed the equivalent of a test suite for prompts.

How I Solved It

The design borrows shamelessly from normal software testing. A regression suite needs three things: fixtures, a runner, and assertions.

flowchart LR
    A[Prompt change PR] --> B[CI: eval job]
    B --> C[Restore fixture repos]
    C --> D[Run agent headless per task]
    D --> E[Deterministic graders]
    D --> F[LLM judge]
    E --> G[Score report]
    F --> G
    G --> H{Score >= baseline?}
    H -->|yes| I[Merge allowed]
    H -->|no| J[Block + diff report]

1. Fixtures: frozen repos plus canned tasks

Every time the agent does something interesting in production, good or bad, I snapshot the state into a fixture. A fixture is a directory with:

A small git repo (usually 20 to 200 files, trimmed from the real one)
A task prompt, exactly as the orchestrator would phrase it
An expectations file

The expectations file looks like this:

# fixtures/skips-tests-on-small-change/expect.yaml
task: "Fix the off-by-one error in pagination in src/api/list.py"
must:
  - ran_command: "pytest"
  - touched_files_only: ["src/api/list.py", "tests/test_list.py"]
  - added_or_modified_test: true
  - pr_description_mentions: ["pytest", "passed"]
must_not:
  - touched_files: ["src/api/auth.py"]
  - ran_command: "git push --force"
judge:
  - "The PR description explains WHY the bug happened, not just what changed."
  - "The fix does not introduce a new edge case for empty lists."
max_turns: 40

I have about 60 of these now. Roughly a third came straight from incidents. The rest are "golden path" tasks I want to keep working.

2. Runner: headless, sandboxed, reproducible

Each fixture runs in a throwaway container with the fixture repo restored from a tarball. The agent runs in headless mode with the candidate spec file mounted in:

#!/usr/bin/env bash
set -euo pipefail
fixture="$1"
candidate_spec="$2"

workdir=$(mktemp -d)
tar -xzf "fixtures/${fixture}/repo.tar.gz" -C "$workdir"
cp "$candidate_spec" "$workdir/CLAUDE.md"

task=$(yq '.task' "fixtures/${fixture}/expect.yaml")
max_turns=$(yq '.max_turns // 40' "fixtures/${fixture}/expect.yaml")

# Headless run; everything the agent does is captured as JSONL
( cd "$workdir" && claude -p "$task" \
    --output-format stream-json \
    --max-turns "$max_turns" \
    --permission-mode acceptEdits \
  ) > "runs/${fixture}.jsonl"

# Also capture the resulting diff and the git log
( cd "$workdir" && git diff HEAD > "runs/${fixture}.diff" \
    && git log --oneline -n 20 > "runs/${fixture}.log" )

The important part: the run produces artifacts, not just a pass/fail. The JSONL stream has every tool call. The diff has every file change. The graders work on those artifacts, never on the live agent.

3. Graders: deterministic first, LLM second

Most of my assertions don't need an LLM at all. Did it run pytest? Grep the tool calls. Did it touch a forbidden file? Parse the diff. These graders are fast, free, and never flaky.

# graders/deterministic.py
import json, re, sys
from pathlib import Path

def tool_calls(jsonl_path: Path):
    for line in jsonl_path.read_text().splitlines():
        ev = json.loads(line)
        if ev.get("type") == "assistant":
            for block in ev["message"].get("content", []):
                if block.get("type") == "tool_use":
                    yield block["name"], block.get("input", {})

def ran_command(calls, pattern: str) -> bool:
    rx = re.compile(pattern)
    return any(
        name == "Bash" and rx.search(inp.get("command", ""))
        for name, inp in calls
    )

def touched_files(diff_path: Path) -> set[str]:
    files = set()
    for line in diff_path.read_text().splitlines():
        if line.startswith("+++ b/"):
            files.add(line[6:])
    return files

def grade(fixture: str, expect: dict) -> list[str]:
    calls = list(tool_calls(Path(f"runs/{fixture}.jsonl")))
    files = touched_files(Path(f"runs/{fixture}.diff"))
    failures = []
    for rule in expect.get("must", []):
        if "ran_command" in rule and not ran_command(calls, rule["ran_command"]):
            failures.append(f"MUST ran_command {rule['ran_command']!r} not found")
        if "touched_files_only" in rule and not files <= set(rule["touched_files_only"]):
            failures.append(f"touched unexpected files: {files - set(rule['touched_files_only'])}")
    for rule in expect.get("must_not", []):
        if "ran_command" in rule and ran_command(calls, rule["ran_command"]):
            failures.append(f"MUST NOT ran_command {rule['ran_command']!r} was executed")
        if "touched_files" in rule and files & set(rule["touched_files"]):
            failures.append(f"touched forbidden files: {files & set(rule['touched_files'])}")
    return failures

Then, for the fuzzy stuff ("does the PR description explain why?"), an LLM judge reads the diff plus the PR body and answers each judge: question with a yes/no and a one-sentence reason. I use a smaller, cheaper model for judging than the one doing the work. The judge gets the rubric, the artifacts, and nothing else. It never sees the candidate spec file, so it can't be biased by it.

# graders/judge.py (trimmed)
RUBRIC = """You are grading an AI coding agent's output.
Answer each question with exactly YES or NO on its own line,
followed by one sentence of justification.
Questions:
{questions}

--- DIFF ---
{diff}
--- PR DESCRIPTION ---
{pr_body}
"""

4. Scoring and the CI gate

Each fixture yields a score: deterministic failures are hard fails, judge questions are weighted 1 point each. The suite score is the mean across fixtures. CI compares it against the baseline from main.

A prompt PR is blocked if:

Any fixture that passed on main now hard-fails (regression)
The suite score drops more than 3 points

The CI job posts a diff report as a PR comment: which fixtures flipped, and the judge's one-sentence reasons for any changed answer. That comment is honestly the most useful artifact in the whole system. It turns "I think this wording is better" into "this wording made fixture #23 stop writing tests, here's the evidence." 💡

Running the full suite costs me about $4 and 25 minutes. I run a 15-fixture smoke subset on every push and the full 60 on merge to main and nightly.

Lessons Learned

1. Fixtures from incidents are worth 10x fixtures you invent

Every time the agent does something dumb in production, I freeze it into a fixture the same day. Those fixtures catch real regressions. The "golden path" fixtures I wrote from imagination almost never fail. If you only do one thing from this post, do this.

2. Assert on behavior, not on output text

My first version asserted on strings in the final diff. It was flaky within a week, because a correct fix can be written ten different ways. Asserting on what the agent did (which tools it called, which files it touched, whether it ran tests) is stable across runs. Save the fuzzy judgments for the LLM judge.

3. Non-determinism is a feature you have to budget for

The same fixture with the same prompt will not produce identical runs. I run each fixture 3 times and take the median score. That tripled the cost and cut false alarms to nearly zero. A single-run eval suite will train you to ignore it, which is worse than having none.

4. The judge needs to be dumber than the worker

When I used the same large model as both worker and judge, the judge was too forgiving. It would rationalize the worker's choices. Switching to a smaller model with a tight rubric made the judge stricter and 5x cheaper. Counterintuitive, but it held up across months of runs.

5. Prompt diffs need code review, and now they get it

Once the suite existed, prompt changes started going through actual review. Reviewers ask "which fixture covers this?" the same way they'd ask "where's the test?" That cultural shift mattered more than any single caught regression. The spec file went from a scratchpad to a maintained artifact with a changelog.

What's Next

Two things I'm working on:

Mutation testing for prompts. Automatically delete each rule from the spec file one at a time and check that at least one fixture fails. Rules that nothing depends on are dead weight (and there are more of them than I'd like to admit).
Fixture aging. Fixtures capture a moment in the codebase. Some are now testing behavior on code that no longer looks like production. I want an automated "is this fixture still representative?" check.

I'll write both up once they've survived a month of real use.

Wrap-up

If your agent's behavior lives in prompt files, those files are code, and code without tests will drift. You don't need anything fancy to start. One fixture from your last incident, one grep-based grader, and a CI job that runs it. Build from there.

If this was useful, follow me here on Dev.to 🚀. I write about running autonomous coding agents in production, including the parts that break. And if you've built something similar, drop a comment. I'd love to compare notes on graders. ✅

How I Made My Autonomous Coding Agent Survive Context Resets: 5 Lessons

yureki_lab — Fri, 25 Sep 2026 14:32:49 +0000

TL;DR

My fully autonomous implementation system runs for days at a time, and every few hours the agent's context window fills up and gets wiped. Early on, each reset meant the agent forgot what it was working on, redid finished tasks, or quietly reversed decisions it had made an hour earlier. I fixed it with a three-file state handoff protocol (a state memo, a task file, and a decisions log), strict precedence rules, and a pre-commit hook that refuses to let the agent commit without checkpointing. Here's the design, the code, and the five lessons from six months of running it. ✅

The Problem

I run a fully autonomous implementation system on a Mac mini in my closet. It's built on Claude Code (the 2.x CLI line as of September 2026) and it works through a task backlog across several repos without me in the loop. An orchestrator module hands work to parallel implementation agents, a self-healing agent cleans up after them, and a remote control dashboard lets me peek in from my phone.

The system is great at doing work. It was terrible at remembering it.

Here's why. A single agent session has a finite context window. On a long task, the window fills up with tool output, diffs, and test logs. When it's full, the harness compacts the conversation into a summary and continues. That summary is lossy. It keeps what happened reasonably well, but it drops why and what's next with alarming frequency.

Across an overnight run, that happens four or five times. The failure modes I saw in the first month:

Amnesia loops. One night the agent investigated the same failing database migration test three separate times in 11 hours. Each time it reached the same conclusion (a timezone mismatch in a fixture), each time it lost the conclusion in a reset, each time it started over. That night burned roughly $40 in tokens and shipped nothing.
Decision reversal. The agent evaluated two HTTP client libraries, picked one for a good reason, then after a reset picked the other one because it only saw "chose a client library" in the summary and re-ran the evaluation with slightly different weights. Two PRs in the same week used different clients.
Ghost tasks. A task that had been finished and merged was still described as "in progress" in the summary. The agent reopened it, rewrote the same feature on a new branch, and hit merge conflicts with its own earlier work.

My first instinct was "just use git history." Git is a great record of what changed. It says nothing about what was tried and rejected, what's half-finished, or what the agent should pick up next. I needed something the agent writes for its future self.

How I Solved It

The fix is boring, which is why I trust it: a small set of plain Markdown files that the agent must read on boot and must update at specific moments. No vector database, no fancy memory service. Just files in the repo, with rules.

The three files

project-root/
├── CLAUDE.md          # project spec file: goals, rules, read order (rarely changes)
├── state/
│   ├── current.md     # state memo: where we are, what's next (overwritten)
│   ├── backlog.md     # task file: checklist with status (edited in place)
│   └── decisions.md   # decisions log: append-only, numbered, with rationale

Each file has a different job and, crucially, a different write mode:

File	Contains	Write mode	Size cap
State memo	Current position, next action, last 3–5 decisions	Overwrite	~150 lines
Task file	Every task with `pending` / `doing` / `done` / `blocked`	Edit in place	One line per task
Decisions log	Numbered entries: context, options, choice, why	Append only	Unbounded

The state memo is the one the agent reads first after a reset. It's deliberately short. If it grows past about 150 lines, the agent is required to compress it, moving old decisions into the log and dropping resolved context.

Here's the actual template the state memo follows:

# State memo (single source of truth)

## Where we are (as of 2026-09-24 03:12 JST)
Working on: task #47 — rate limiter for the public API
Branch: feat/rate-limiter
Status: implementation done, 2 of 9 integration tests failing

## Next action
Fix the two failing tests in the burst-window case. They fail because the
fake clock isn't advanced between requests. Do NOT touch the limiter logic.

## Recent decisions (full detail in state/decisions.md)
- D-031: token bucket over sliding window (memory footprint at 10k clients)
- D-030: limits live in config, not env vars (need per-tenant overrides)

## Do not redo
- Already evaluated leaky bucket. Rejected. See D-031.
- Migration 0042 is applied in staging. Don't regenerate it.

That "Do not redo" section is the single highest-value thing in the whole system. It exists purely because of the amnesia loops.

Read order and precedence

Files alone don't help if the agent reads them in a random order or, worse, trusts a stale one over a fresh one. So the project spec file has a hard rule that runs at every boot, including every post-reset boot:

## On every session start (mandatory, in this order)
1. Read CLAUDE.md (this file)
2. Read state/current.md
3. Read state/backlog.md
4. Skim the last 10 entries of state/decisions.md

Do not start work before completing all four.

## When files contradict each other
state/current.md  >  state/backlog.md  >  CLAUDE.md  >  state/decisions.md

The state memo is always the latest truth. If the decisions log says X
and the memo says Y, Y wins. Update the older file, do not argue with it.

That precedence line killed the decision-reversal problem almost overnight. Before it, the agent would find an old decision entry, a newer memo note, and "resolve" the conflict by re-deriving the answer from scratch. Now it has a tiebreaker and moves on.

Here's the loop as a diagram:

flowchart TD
    A[Session boot] --> B[Read spec → memo → tasks → decisions]
    B --> C[Pick next action from memo]
    C --> D[Do work]
    D --> E{Unit of work done?}
    E -- no --> D
    E -- yes --> F[Checkpoint: update memo + tasks, append decision if any]
    F --> G[Commit]
    G --> H{Context full?}
    H -- no --> C
    H -- yes --> I[Compaction / reset]
    I --> A

The key insight in that diagram: the checkpoint happens before the commit, and the commit happens before the reset can hurt you. If the reset lands mid-task, the worst case is losing the in-flight edits since the last commit, and the memo already says what the agent was about to do.

The checkpoint rule (and the hook that enforces it)

Telling an agent "update the state files regularly" doesn't work. "Regularly" gets interpreted as "when I remember," and the agent remembers less as the context fills. So I made the rule mechanical and enforced it with a hook.

The rule is write on completion, not on a timer. Specifically, the agent must update the state memo:

After finishing a unit of work (a task reaches done or blocked)
Before any operation that's hard to undo (migration, force push, deleting a branch)
Immediately after making any decision that has a rationale

And the enforcement is a plain Git pre-commit hook:

#!/usr/bin/env bash
# .git/hooks/pre-commit
# Refuse to commit if the state memo hasn't been updated recently.
# The agent runs for hours; a memo older than the last 45 minutes of
# commits means it's drifting and the next reset will hurt.

set -euo pipefail

MEMO="state/current.md"
MAX_AGE_MIN=45

if [[ ! -f "$MEMO" ]]; then
  echo "❌ $MEMO is missing. Create it before committing." >&2
  exit 1
fi

# Skip the check if the memo itself is part of this commit.
if git diff --cached --name-only | grep -qx "$MEMO"; then
  exit 0
fi

memo_age_min=$(( ( $(date +%s) - $(stat -f %m "$MEMO") ) / 60 ))

if (( memo_age_min > MAX_AGE_MIN )); then
  echo "❌ $MEMO was last updated ${memo_age_min} min ago (limit: ${MAX_AGE_MIN})." >&2
  echo "   Update 'Where we are' and 'Next action', then commit again." >&2
  exit 1
fi

(That stat -f %m is macOS. On Linux it's stat -c %Y.)

When the hook fires, the agent sees the error in its tool output, updates the memo, and retries the commit. It took zero prompt engineering to get this behavior. The agent already knows how to read a failed command and fix the cause. I just had to make the cause explicit.

What NOT to persist

The first version of the memo got polluted fast. The agent dumped test output into it, pasted whole stack traces, and once wrote a 900-line "current understanding of the codebase" section. A 900-line memo is as useless as no memo, because the agent skims it and misses the two lines that matter.

So the spec file now has an explicit blocklist:

## Never write to state/current.md
- Raw tool output (test logs, stack traces, diffs). Link to the file instead.
- Speculation ("might be caused by..."). Only write what you verified.
- Anything you can derive from git (what changed, who changed it).
- Secrets, tokens, credentials, or absolute paths on this machine.

That last line matters more than it looks. The state files get committed. If the agent copies an API key into the memo "for reference," it's in history forever.

The resume test

You can't improve what you don't measure, so I built a small test for the handoff. Once a day, a script kills the running session mid-task, boots a fresh one, and asks a single question:

"Without doing any work: what task are you on, what's the next concrete action, and what should you not redo?"

I diff the answer against the memo and against what actually happened in git. It's scored by a second, cheap model run on three yes/no questions: correct task, correct next action, no contradiction with the log.

	Month 1 (summary only)	Month 6 (three-file protocol)
Resume accuracy	~40%	~95%
Amnesia loops per week	4–6	0–1
Wasted overnight token spend	~$120/week	<$15/week

Those numbers are from my own logs on my own system, so treat them as one data point, not a benchmark. But the direction is not subtle.

Lessons Learned

Compaction is not memory. Every harness will eventually summarize your context, and every summary drops the "why." If you're running agents for longer than a single sitting, you need an explicit, agent-written handoff. Don't wait for the amnesia loop to teach you this at $40 a night.
One file has to be the boss. The single biggest win wasn't the files themselves, it was the precedence rule. When two records disagree, the agent needs a tiebreaker it doesn't have to think about. "Memo wins, update the other one" removed an entire class of re-litigated decisions.
Write on completion, never on a timer. "Every 30 minutes" produces memos written mid-thought that describe a state that no longer exists. "After every finished unit of work" produces memos that describe a real, committed checkpoint. And enforce it with a hook, because the agent's memory of the rule degrades exactly when the context is fullest.
Small enough to read in one gulp. The memo has a hard size cap and the agent is responsible for enforcing it. The moment it becomes a dumping ground, the agent skims it and you're back to square one. A "Do not redo" section of five lines beats a "Full context" section of five hundred.
Match the write mode to the data. State gets overwritten. Decisions get appended. Tasks get edited in place. When I had everything in one file with one write mode, the agent would overwrite decisions or append state, and both were wrong. Three files, three write modes, zero ambiguity.

What's Next

Three things on my list:

Structured front matter. The memo is prose right now. I'm adding a YAML block at the top (current task ID, branch, status enum) so the remote control dashboard can render it without parsing Markdown.
A staleness detector in the observability layer. The pre-commit hook catches drift at commit time. I want a background check that flags a memo whose "Where we are" hasn't changed in two hours while commits keep landing.
Cross-repo handoff. When the orchestrator module moves an agent from one repo to another, the memo in repo A doesn't help in repo B. I'm experimenting with a lightweight "last seen elsewhere" pointer.

Wrap-up

If you take one thing from this: the moment your agent runs longer than a context window, its memory is your problem, not the model's. Give it a place to write for its future self, tell it exactly when to write there, and make the rule mechanical.

I'm writing up the fully autonomous implementation system one piece at a time here on Dev.to: the orchestrator, the parallel agents, the self-healing loop, the remote dashboard. Follow me to catch the next one. 🚀

And I'd genuinely like to know: how do you handle context resets on long-running agents? Files, a database, a memory MCP server, something else? Drop it in the comments. 💬

How I Added OpenTelemetry Tracing to 47 Services With Claude Code in 9 Days

yureki_lab — Thu, 24 Sep 2026 14:32:24 +0000

TL;DR

I had 47 backend services that emitted logs but no traces, so every incident turned into a cross-repo grep party. I used Claude Code to instrument all of them with OpenTelemetry in 9 days — not by asking it to "add tracing," but by writing one reference implementation by hand and turning our conventions into an executable validator the agent had to pass. Here's the loop, the code, and the five things I'd do differently.

The Problem

Our backend is 47 services: mostly Node.js 22.x with Express or Fastify, plus a handful of Python 3.13 FastAPI services. Every one of them had structured JSON logs. None of them had distributed tracing.

That's a fine setup right up until a request crosses four service boundaries and gets slow somewhere. Then the debugging session looks like this:

Find the request ID in the edge service logs.
Hope the next service logged the same request ID.
Discover it called the header x-request-id instead of x-correlation-id.
Give up and add a console.log.

Our median incident spent 40+ minutes just on "which service is actually slow." I timeboxed a manual fix and estimated it honestly: 47 services × roughly half a day each for setup, span conventions, config, and a smoke test, comes out around six weeks of focused work. Nobody was going to fund six weeks of plumbing.

The constraint that made this interesting: tracing instrumentation is repetitive but not identical. Auto-instrumentation gets you 70% of the way, and then every service has its own bespoke 30% — a custom HTTP client, a queue consumer, a cron entrypoint that isn't a request at all. That mix is exactly where an AI coding agent is strong and exactly where it will quietly invent conventions if you let it.

How I Solved It

Step 1: Write the reference implementation by hand

My first attempt was a 2,000-word CONVENTIONS.md describing how spans should be named, which attributes were required, and how to wire the SDK. The agent followed maybe 70% of it and improvised the rest — different attribute names in half the services, startSpan in some places and startActiveSpan in others.

So I threw the prose away and instrumented one service by hand, carefully, and made that service the spec:

// tracing.js — the reference every other service was told to mirror.
// Imported for side effects before anything else in the process.
import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'
import { resourceFromAttributes } from '@opentelemetry/resources'
import {
  ATTR_SERVICE_NAME,
  ATTR_SERVICE_VERSION,
} from '@opentelemetry/semantic-conventions'

const sdk = new NodeSDK({
  resource: resourceFromAttributes({
    [ATTR_SERVICE_NAME]: process.env.SERVICE_NAME,
    [ATTR_SERVICE_VERSION]: process.env.GIT_SHA ?? 'dev',
    'deployment.environment.name': process.env.APP_ENV ?? 'local',
  }),
  traceExporter: new OTLPTraceExporter({
    url: `${process.env.OTEL_COLLECTOR_URL}/v1/traces`,
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      // Noisy, zero-signal, and it doubles span volume.
      '@opentelemetry/instrumentation-fs': { enabled: false },
    }),
  ],
})

sdk.start()
process.on('SIGTERM', () => sdk.shutdown().finally(() => process.exit(0)))

Then the one piece of manual instrumentation that auto-instrumentation can't do for you — wrapping a domain operation:

import { trace, SpanStatusCode } from '@opentelemetry/api'

const tracer = trace.getTracer('billing')

export async function chargeInvoice(invoiceId, amountCents) {
  // Span name = "<domain>.<operation>", low cardinality, never includes an ID.
  return tracer.startActiveSpan('billing.charge_invoice', async (span) => {
    try {
      span.setAttribute('billing.invoice_id', invoiceId)
      span.setAttribute('billing.amount_cents', amountCents)
      const result = await gateway.charge(invoiceId, amountCents)
      span.setAttribute('billing.gateway_status', result.status)
      return result
    } catch (err) {
      span.recordException(err)
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message })
      throw err
    } finally {
      span.end()
    }
  })
}

Roughly 90 lines of real, working, opinionated code. Handing the agent that file and saying "do this, for this service" worked dramatically better than any description of it. Agents pattern-match. Give them a pattern.

Step 2: Turn the conventions into a script that fails

Prose conventions are unenforceable, and "looks right to me" doesn't scale across 47 pull requests. So I spent half a day writing a validator and told the agent it was not done until the validator passed:

# check_tracing.py — run per service in CI. Exit 1 blocks the PR.
import re, sys, pathlib

FORBIDDEN_ATTR = re.compile(
    r"setAttribute\(\s*['\"](?:.*\b(?:email|user_id|token|url|path|query)\b.*)['\"]"
)
SPAN_NAME = re.compile(r"startActiveSpan\(\s*['\"]([a-z0-9_]+\.[a-z0-9_]+)['\"]")
TEMPLATED_SPAN_NAME = re.compile(r"startActiveSpan\(\s*[`'\"].*\$\{")

def check(service: pathlib.Path) -> list[str]:
    errors = []
    if not (service / "tracing.js").exists():
        errors.append("missing tracing.js bootstrap")
    for f in service.rglob("*.js"):
        src = f.read_text()
        if TEMPLATED_SPAN_NAME.search(src):
            errors.append(f"{f}: interpolated span name (cardinality bomb)")
        for m in FORBIDDEN_ATTR.finditer(src):
            errors.append(f"{f}: high-cardinality or PII attribute: {m.group(0)}")
        for m in SPAN_NAME.finditer(src):
            if m.group(1).split(".")[0] != service.name.replace("-", "_"):
                errors.append(f"{f}: span namespace != service name: {m.group(1)}")
    return errors

if __name__ == "__main__":
    problems = check(pathlib.Path(sys.argv[1]))
    print("\n".join(problems) or "tracing conventions OK")
    sys.exit(1 if problems else 0)

This single file changed the economics of the whole project. Before it, I reviewed taste. After it, I reviewed judgment calls only — the validator caught every mechanical violation before I ever opened the diff.

Step 3: Prove the spans actually arrive

A service can pass every static check and still export nothing, because the SDK started after the HTTP framework was imported, or the collector URL was wrong. So each service also got a smoke test that boots the process against a local collector and asserts on real exported spans:

// tracing.smoke.test.js
test('http request produces a parented db span', async () => {
  await request(app).get('/invoices/42').expect(200)
  await flushSpans()

  const spans = collector.finished()
  const http = spans.find((s) => s.name === 'GET /invoices/:id')
  const db = spans.find((s) => s.name.startsWith('pg.query'))

  expect(http).toBeDefined()
  expect(db?.parentSpanContext?.spanId).toBe(http.spanContext().spanId)
  // The route is templated, so no invoice IDs leak into span names.
  expect(http.name).not.toMatch(/42/)
})

That parent-child assertion caught the single most common real failure: context propagation silently broken, producing 47 disconnected single-span "traces" that look fine in a list view and are worthless in a waterfall.

Step 4: Run the loop

flowchart LR
    A[Pick next service<br/>grouped by framework] --> B[Agent instruments<br/>against reference impl]
    B --> C{check_tracing.py}
    C -- fail --> B
    C -- pass --> D{smoke test<br/>vs local collector}
    D -- fail --> B
    D -- pass --> E[Human review:<br/>judgment calls only]
    E --> F[PR merged]
    F --> A

The two validators ran inside the agent's own loop, so most iteration happened without me. My job was the last box: does this service's custom queue consumer actually deserve a span here, and is this attribute worth its storage cost?

Nine days, 47 services, about 90 minutes a day of my attention. The first six services took four of those days. The remaining 41 took five.

Lessons Learned

1. A working reference implementation beats any style guide

My 2,000-word convention doc produced ~70% compliance. A 90-line reference file produced near-perfect structural compliance immediately. If you find yourself writing a long prose description of how code should look, stop and write the code instead — then point at it.

2. Make the convention executable or it doesn't exist

The validator was the highest-leverage half-day of the project. Anything you'd write as "please always..." in a prompt should be a script that exits non-zero. Prompts are advice; exit codes are physics. It also means the convention survives you: the next person to add a service gets the same enforcement without reading a single doc.

3. Cardinality is where an agent will hurt you

Left alone, the agent wrote genuinely reasonable-looking code like startActiveSpan(`charge invoice ${invoiceId}`) and span.setAttribute('user.email', email). Both are defensible as "descriptive." Both are catastrophic — unbounded span names wreck your backend's indexing and cost, and PII in attributes is a compliance incident that lives in your telemetry store for the retention period.

The agent isn't being careless; it's optimizing for readability because nothing told it to optimize for cardinality. That tradeoff is invisible in the diff and expensive in production, which makes it exactly the kind of rule that belongs in a linter rather than a review comment.

4. Adding instrumentation is mechanical; removing the old logging isn't

My original plan included "delete the now-redundant logging." I cut that from the agent's scope on day two. Deciding whether a log line is redundant with a span requires knowing who greps for it at 3am — that's tribal knowledge the codebase doesn't contain, and an agent confidently deleting an on-call runbook's load-bearing log line is a bad trade against the time it saves.

Split the work by information availability, not by difficulty. The mechanical 90% went to the agent. The 10% that needs context living in someone's head stayed with people.

5. Batch by framework, not alphabetically

I started in repo order and burned tokens re-establishing context every single service. Grouping by stack — all Fastify, then all Express, then all FastAPI — meant each batch reused the same mental model, the same gotchas, and the same recently-solved problems. Same work, noticeably fewer tokens and far fewer wrong turns.

Adjacent tasks are cheaper than shuffled tasks. Order your backlog by similarity, not by convenience.

What's Next

Three things I'm working on now:

Sampling policy. We're at head-based 100% sampling in staging and it's already loud. Tail-based sampling that keeps every error trace and 5% of the boring ones is the obvious next move.
Trace-driven perf work. Now that waterfalls exist, I want to feed real slow traces back to the agent as the starting point for optimization instead of vague "this endpoint feels slow" prompts.
Generated service dependency graphs. The traces already encode the real call graph, including the three calls nobody remembered existed. Rendering that automatically beats maintaining an architecture diagram by hand.

The pattern generalizes well beyond tracing. Any sweeping, repetitive, convention-heavy migration — feature flag SDKs, error reporting, auth middleware — fits the same shape: one hand-written reference, one executable validator, one smoke test that proves it's live, then let the agent grind.

Wrap-Up / CTA

If you're sitting on a "someday" observability migration because it's six weeks of boring work, it probably isn't six weeks anymore. But the leverage isn't in the prompt — it's in the reference implementation and the validator you write before you start.

If you try this, I'd genuinely like to know what your validator ends up catching. Mine caught 31 cardinality violations I would have merged without noticing.

💬 Comment below with the worst high-cardinality span name you've shipped. I'll go first: GET /users/8f21c...
🚀 Follow me here on Dev.to — I write up these AI-agent engineering experiments as I run them.
💡 Try it yourself with Claude Code on one service this week. One service is enough to find out whether your conventions are real or imaginary.

How I Sandboxed My AI Coding Agent's Shell Access Without Slowing It Down

yureki_lab — Wed, 23 Sep 2026 14:32:16 +0000

TL;DR

I gave my autonomous coding agent real shell access, then spent two weeks building guardrails around it after a rm -rf near-miss. The result is a three-tier command classifier that parses shell syntax instead of regex-matching it, plus filesystem and network confinement and an append-only audit log. Interruption rate dropped from 31% of commands to 4%, and I stopped watching the terminal like a hawk.

The Problem

An agent that can't run shell commands is a very expensive autocomplete.

The whole point of a fully autonomous implementation system is that it closes the loop by itself: write code, run the tests, read the failure, fix it, run the tests again. Take away Bash and every one of those steps needs me. I tried the read-only-agent approach for about a week. It produced code that looked correct and was never once verified.

So I turned shell access on. And then I watched it do this:

# the agent, "cleaning up" a test fixture directory
rm -rf $BUILD_DIR/artifacts

BUILD_DIR was unset in that subprocess. That expands to rm -rf /artifacts. On my machine that particular path didn't exist, so nothing happened, and I only noticed because I was reading the transcript for an unrelated reason.

That's the real problem with agent shell access. It isn't that the agent is malicious — it obviously isn't. It's that:

The agent doesn't know what it doesn't know. Unset variables, a different working directory than it assumed, a git remote that isn't the one it thinks it is.
Failures are silent and asymmetric. 400 successful npm test runs don't earn back one git push --force to main.
The obvious fix makes the agent useless. My first attempt was "confirm every command." I approved 180 prompts in one afternoon, 174 of which were ls, cat, and pytest. I started rubber-stamping by hour two, which is strictly worse than no gate at all — I had the feeling of oversight without the substance.

What I actually wanted: the agent runs the boring 95% at full speed, and I get a hard stop on the 5% that can ruin my day.

How I Solved It

Three layers. Each one catches a different class of mistake.

flowchart TD
    A[Agent proposes command] --> B[Layer 1: Parse and classify]
    B -->|deny| C[Blocked, reason returned to agent]
    B -->|ask| D[Human confirm]
    B -->|allow| E[Layer 2: Confined execution]
    D -->|approved| E
    D -->|rejected| C
    E --> F[Layer 3: Append-only audit log]
    F --> G[Output back to agent]

Layer 1: Classify by parsing, not by regex

This is the part I got wrong first, so it's the part worth the most words.

My original rules were regex against the raw command string. ^rm\s+-rf\s+/ and friends. It took the agent about three days to walk straight through one, not because it was trying to, but because it wrote a perfectly ordinary command that my pattern didn't anticipate:

cd "$(git rev-parse --show-toplevel)/tmp" && rm -rf ./*

No leading slash. No match. My regex was looking at text; the danger was in the semantics.

The fix was to stop pattern-matching strings and start parsing shell into a syntax tree. Python's shlex gets you tokenization; bashlex gets you actual structure — pipelines, redirects, command substitutions, && chains. Once you have the tree you can walk every simple command inside a compound one and classify each independently.

import bashlex

DENY = {
    ("rm", "-rf", "/"),        # root deletion, any form
    ("git", "push", "--force"),
    ("chmod", "777"),
}
ASK = {"git push", "git reset --hard", "docker", "brew", "pip install", "npm publish"}
ALLOW = {"ls", "cat", "grep", "rg", "find", "git status", "git diff",
         "git log", "pytest", "npm test", "make", "head", "tail", "wc"}

def simple_commands(src: str):
    """Yield every simple command inside a compound command."""
    for tree in bashlex.parse(src):
        for node in _walk(tree):
            if node.kind == "command":
                yield [p.word for p in node.parts if p.kind == "word"]

def classify(src: str) -> tuple[str, str]:
    try:
        commands = list(simple_commands(src))
    except bashlex.errors.ParsingError:
        return "ask", "unparseable shell; falling back to human review"

    verdicts = []
    for words in commands:
        if not words:
            continue
        verdicts.append(_classify_one(words))

    # The whole pipeline is only as safe as its most dangerous link.
    for level in ("deny", "ask", "allow"):
        for verdict, reason in verdicts:
            if verdict == level:
                return verdict, reason
    return "ask", "no rule matched"

Two design decisions in there that I'd defend to anyone:

Unparseable means ask, never allow. If my parser can't understand the command, I have zero information about it, and zero information is not the same as "safe." Early on I had this fall through to allow because parse failures were annoying. They were annoying because they were mostly heredocs and process substitution — exactly the constructs where interesting things hide.

The most dangerous link decides. npm test && rm -rf ./dist is not an allow just because it starts with one. Walking every simple command in the tree and taking the worst verdict is the only version of this that holds up.

Layer 2: Confine what "allow" can reach

Classification decides whether to run. Confinement decides how much it can touch when you get the classification wrong — and you will get it wrong.

Three cheap constraints, none of which required a container:

import os, subprocess

def run_confined(cmd: str, workdir: str, timeout: int = 300):
    env = {
        "PATH": "/usr/bin:/bin:/usr/local/bin",
        "HOME": workdir,             # keep stray writes inside the work tree
        "LANG": "en_US.UTF-8",
        # Note what is NOT here: every API token in my real shell.
    }
    return subprocess.run(
        ["/bin/bash", "-o", "pipefail", "-c", cmd],
        cwd=workdir,
        env=env,                     # allowlist, not os.environ.copy()
        capture_output=True,
        text=True,
        timeout=timeout,             # no unbounded hangs
    )

The env allowlist is the highest-value line in this entire post. The default instinct is os.environ.copy(), which hands the agent every credential you've ever exported — and those credentials end up in command output, which ends up in the model's context, which ends up in logs. Starting from an empty dict and adding back only what's needed took twenty minutes and removed an entire category of problem.

For network I run the agent's shell under a user account whose outbound traffic is filtered to a host allowlist (package registries, my git remote, and nothing else). The agent can npm install. It cannot curl an arbitrary host and pipe the result to bash.

Layer 3: Log everything, append-only

Every command, verdict, exit code, and duration goes to a JSONL file that the agent can write to but not read or rewrite:

import json, time

def audit(cmd: str, verdict: str, reason: str, exit_code: int | None, ms: int):
    with open(AUDIT_PATH, "a", encoding="utf-8") as f:
        f.write(json.dumps({
            "ts": time.time(), "cmd": cmd, "verdict": verdict,
            "reason": reason, "exit_code": exit_code, "duration_ms": ms,
        }, ensure_ascii=False) + "\n")

This started as a debugging aid and turned into the tuning instrument for the whole system. Once a week I run:

jq -r 'select(.verdict=="ask") | .cmd' audit.jsonl \
  | awk '{print $1, $2}' | sort | uniq -c | sort -rn | head -20

Anything at the top of that list that I approved every single time is a rule that's costing me attention and buying me nothing. It gets promoted to allow. That one query is how the interruption rate went from 31% to 4% without loosening anything that mattered.

Lessons Learned

1. A gate you rubber-stamp is worse than no gate. Confirming everything trained me in about ninety minutes to approve without reading. The security value of a prompt is exactly the attention you still bring to it, and attention is a budget that depletes. Spend it on the 4%.

2. Regex over shell strings is security theater. Shell has command substitution, variable expansion, quoting, and chaining. Any of those turns a dangerous command into text your pattern doesn't match. Parse it or don't pretend to check it.

3. Fail toward the human, never toward execution. Every ambiguous case — parse error, unknown binary, no matching rule — resolves to ask. This produces more prompts early, which is uncomfortable, and then the audit log tells you exactly which ones to relax.

4. The environment is the attack surface nobody looks at. I spent days on command classification while every API key I own was being handed to every subprocess. An env allowlist is twenty minutes of work and it's probably the single highest-leverage thing in this post.

5. Measure the interruption rate or you're guessing. "Does this feel safe" is not a metric. Commands-per-prompt is. Track it, and guardrail tuning becomes an ordinary optimization problem instead of a vibes argument with yourself.

What's Next

Two things I haven't solved:

Per-command network policy. Right now the allowlist is process-wide. I'd like npm install to reach the registry and nothing else to reach anything, which probably means a real sandbox rather than a filtered user account.
Learned classification. The rule set is hand-written and it's at 140 lines. I'd like to feed the audit log back in and have consistently-approved patterns propose their own promotion — with me approving the promotion, not the individual commands.

Stack this was built and tested on, since all of it rots: Claude Code (2026-09 release), Python 3.13, bashlex 0.18, macOS 15 and Ubuntu 24.04.

Wrap-up

If you're giving an agent shell access — and you should, a verified loop is worth far more than a supervised one — start with the env allowlist. It's an afternoon at most. Then add the audit log, run for a week, and let the data tell you where the real gate belongs. Don't start with 200 rules; you'll write the wrong ones.

If you've built something similar, I'd genuinely like to know how you handled the network layer — it's the part I'm least happy with. Drop it in the comments.

And if this was useful, follow me here on Dev.to — I write up what I learn building autonomous coding systems, war stories and dead ends included. 🚀

5 Logging Habits That Made My AI Coding Agent Actually Good at Debugging

yureki_lab — Tue, 22 Sep 2026 14:33:14 +0000

TL;DR

My AI coding agent wasn't bad at debugging — my logs were bad at being read. Five changes to how my services log (correlation IDs, structured JSON, errors that carry state, honest log levels, and a query command instead of a log file) took its median time-to-root-cause from ~40 minutes of flailing down to under 5. None of these habits are new. What's new is that a machine is now the primary reader of your logs.

The Problem

A checkout endpoint started returning 500s for about 2% of requests. Classic intermittent bug: not reproducible locally, no obvious pattern, and the kind of thing that eats an afternoon.

So I did what I'd been doing all year — handed it to my coding agent with production log access and went to make coffee.

Twenty minutes later it had:

grepped error across a 2.3 GB log file and found 4,100 matches
picked a Redis connection reset line that turned out to be unrelated noise from a nightly job
added a retry wrapper around the Redis client
told me, confidently, that this "should resolve the intermittent 500s"

It didn't. The actual bug was a race between a coupon-validation call and a cart-refresh call, and the evidence was sitting right there in the logs — just spread across four lines that had nothing linking them together.

That's when it clicked. The agent hadn't failed at reasoning. It had failed at evidence gathering, because my logs made evidence gathering nearly impossible. I'd spent years writing logs for a reader who already knew the system: me. Lines like:

[2026-09-04 11:42:07] processing cart
[2026-09-04 11:42:07] validating
[2026-09-04 11:42:08] failed, retrying

I can read that. I know "validating" means the coupon service and "failed, retrying" is the HTTP client's third attempt. An agent reads those lines the way a new hire does — except a new hire asks you a question, and an agent makes a plausible guess and writes code based on it.

The constraint that made this interesting: I couldn't make the agent smarter, but I could completely control what it had to read. So I spent a week rewriting logging across three services with exactly one design goal — a reader with zero prior context should be able to reconstruct one request end-to-end.

How I Solved It

Here's the loop I was optimizing for. The failure mode is always the same: the agent can't narrow, so it guesses.

flowchart TD
    A[Bug report] --> B{Can the agent isolate<br/>one failing request?}
    B -->|No| C[Grep for 'error']
    C --> D[4,100 matches]
    D --> E[Picks a plausible line]
    E --> F[Fixes the wrong thing]
    B -->|Yes| G[Pull full trace by ID]
    G --> H[Read ordered, typed events]
    H --> I[Compare to a passing request]
    I --> J[Root cause]

Everything below is about moving from the left branch to the right branch. Stack: Python 3.13 with structlog, Node.js 22 LTS with pino 9.x, and Claude Code CLI (v2.x line, as of September 2026) as the agent.

Habit 1: One ID that survives the whole request

This is the single highest-leverage change, and it's boring. Generate an ID at the edge, stuff it in a context variable, and attach it to every single log line — including the ones inside your background workers.

# Python 3.13 + structlog
import contextvars, uuid, structlog

request_id_var = contextvars.ContextVar("request_id", default=None)

def bind_request_id(logger, method_name, event_dict):
    rid = request_id_var.get()
    if rid:
        event_dict["request_id"] = rid
    return event_dict

structlog.configure(
    processors=[
        bind_request_id,
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.JSONRenderer(),
    ]
)

# FastAPI middleware
@app.middleware("http")
async def attach_request_id(request, call_next):
    rid = request.headers.get("x-request-id") or str(uuid.uuid4())
    request_id_var.set(rid)
    response = await call_next(request)
    response.headers["x-request-id"] = rid
    return response

The payoff isn't the ID itself — it's that "show me everything about the failing request" becomes a single filter instead of a research project. The first time I gave my agent a codebase with this in place, it stopped grepping for keywords entirely and started pulling traces.

If you adopt exactly one thing from this post, adopt this one. It's about 30 lines per service and it changes what questions are answerable.

Habit 2: Structured JSON, not prose

Prose logs force the agent to write a regex to parse your sentence structure, which it will do badly, and then reason on top of that bad parse. Compounding errors.

# Before — needs a parser, and the parser will be wrong
log.info(f"Coupon {code} rejected for user {uid} after {ms}ms")

# After — needs no parser
log.info("coupon_rejected", coupon=code, user_id=uid, duration_ms=ms, reason="expired")

Two rules that matter more than the format itself:

The event name is a stable identifier, not a sentence. coupon_rejected, not "Coupon was rejected". Stable names make cross-request comparison trivial: count coupon_rejected in failing vs. passing traces and the anomaly jumps out.
Values go in fields, never interpolated into the message. The moment a user ID lives inside a string, filtering by user ID becomes substring matching.

Habit 3: Errors that carry state, not just a stack trace

A stack trace tells you where it broke. It almost never tells you why. An agent staring at TypeError: Cannot read properties of undefined (reading 'total') at line 214 will go read line 214 and start theorizing about null checks — when the real story is that an upstream call returned a 204 with an empty body.

// Node.js 22 LTS + pino 9
try {
  return await applyCoupon(cart, code);
} catch (err) {
  logger.error({
    err,                          // pino serializes stack + message
    event: "coupon_apply_failed",
    coupon: code,
    cart_id: cart.id,
    cart_item_count: cart.items.length,
    cart_version: cart.version,   // the field that actually solved it
    upstream_status: err.cause?.status ?? null,
  }, "coupon application failed");
  throw err;
}

cart_version was the whole ballgame in my race condition. Two requests were mutating the same cart, and the version numbers in the log made it obvious within seconds — to the agent, not to me. I'd been staring at those logs for an hour.

The rule I now apply when writing an error log: what would I need to know to reproduce this without asking anyone? Log that. If the answer is "the state of the object", log the state of the object.

Habit 4: Log levels that mean something

My levels had rotted into decoration. Half of ERROR was retryable noise; real failures were hiding at WARN because someone didn't want to page anybody. An agent takes your levels literally — that's why mine latched onto a Redis connection reset that a human would have skipped on sight.

The definitions I settled on and wrote into the project spec file:

Level	Meaning	Agent's reading
`ERROR`	A user-visible operation failed and will not succeed	Start here
`WARN`	Something degraded but recovered (retry succeeded, fallback used)	Context, not cause
`INFO`	A business event happened (order placed, coupon applied)	Timeline material
`DEBUG`	Internal step detail	Only when tracing one request

Then I did the unglamorous part: swept the codebase and re-leveled roughly 300 call sites to match. Unsurprisingly, this is exactly the kind of mechanical-but-judgment-heavy sweep an agent is great at — I reviewed the diff in chunks rather than writing it by hand.

Habit 5: Give the agent a query command, not a log file

Pointing an agent at raw logs is how you burn 50k tokens to learn nothing. Pointing it at a command that returns exactly one trace is how you burn 2k tokens and get an answer.

#!/usr/bin/env bash
# logq — trace a single request across services
# usage: logq trace <request_id> [--since 24h]
#        logq errors [--since 1h] [--top 20]

case "$1" in
  trace)
    jq -c --arg rid "$2" 'select(.request_id == $rid)' "$LOG_PATH" \
      | jq -s 'sort_by(.timestamp)' ;;
  errors)
    jq -c 'select(.level == "error")' "$LOG_PATH" \
      | jq -r '.event' | sort | uniq -c | sort -rn | head -"${3:-20}" ;;
esac

Then I documented it in CLAUDE.md so the agent reaches for it unprompted:

## Debugging production issues

Never grep the raw log file — it's multi-GB and unstructured at the edges.

1. `logq errors --since 1h` to find which event names are spiking
2. `logq trace <request_id>` to pull one full request timeline
3. Always pull a *passing* trace too and diff the two before theorizing

That last line did more than the script did. "Compare a failing trace to a passing trace" is the single instruction that most reliably stops an agent from pattern-matching on the first scary-looking line it sees.

Lessons Learned

1. Your logs are now a machine-readable API. Version them like one. Once an agent (and a logq script, and a dashboard) depends on the event name coupon_rejected, renaming it is a breaking change. I treat event names with the same care as route paths now.

2. Agents fail at evidence gathering far more than at reasoning. Every "the AI wrote a dumb fix" story I've hit this year traced back to the agent reasoning correctly over bad or partial evidence. Fixing inputs beat every prompt-engineering trick I tried.

3. Optimize for "reconstruct one request", not "search everything". Search-everything is a human affordance — we're good at skimming and discarding. An agent's context window is small and expensive, and irrelevant lines don't just waste tokens, they actively mislead. Narrow beats complete.

4. Tools beat instructions when the task is mechanical. I spent two days writing increasingly elaborate prompt instructions about how to search logs carefully. A 20-line shell script made all of them unnecessary. If you find yourself writing a paragraph explaining how to gather information, you should probably be writing a command instead.

5. This is just good observability, and that's the point. Every habit here is something an SRE would have told you in 2018. The AI angle didn't change the advice — it changed the ROI. Sloppy logs used to cost me an afternoon occasionally. Now they cost me every single automated debugging run, and the improvements compound daily.

What's Next

Two things I'm working on:

Sampling DEBUG in production for a small slice of traffic. Right now DEBUG is off in prod, which means the richest data disappears exactly when I need it. Head-based sampling at ~1% with forced capture on error looks like the right trade.
Wiring OpenTelemetry spans into the same query command, so logq trace returns logs and timing spans on one timeline. Half of my remaining hard bugs are latency-shaped, and logs alone under-serve those.

I'd also like to stop maintaining logq as a shell script and expose it as a proper tool the agent calls directly — the bash-wrapper stage is clearly temporary.

Wrap-up

If your AI agent keeps "fixing" the wrong thing, before you rewrite your prompts, go read your logs the way it has to: with no prior knowledge, one line at a time, unable to ask a question. It's an uncomfortable exercise and it'll show you exactly what to fix.

Start with correlation IDs. It's an afternoon of work and it's the one that unlocks everything else.

Your turn: what's the one logging change that paid for itself fastest in your codebase? Drop it in the comments — I'm still collecting these. 👇

If this was useful, follow me here on Dev.to — I write about building and living with autonomous coding agents, usually with the embarrassing parts left in. 🚀

How I Built a Research Agent That Reads the Docs Before My Coding Agent Ships

yureki_lab — Mon, 21 Sep 2026 14:32:22 +0000

TL;DR

My autonomous coding agent kept shipping code against library APIs that didn't exist, because it wrote first and read the docs never. I fixed it by adding a read-only research sub-agent that runs before implementation, returns a short structured brief, and blocks the implementer until that brief exists. Hallucinated-API bugs dropped from roughly one in five tasks to about one in forty over three months, and the extra cost was under 10% of tokens. Here's how it works and what I got wrong along the way. 🚀

The Problem

I run a fully autonomous implementation system on a Mac mini. An orchestrator picks up tasks, hands them to parallel implementation agents built on Claude Code, and a self-healing agent cleans up whatever fails. It ships real code to real projects, mostly unattended.

For the first few months, the single most annoying failure mode was this:

The implementer confidently calls client.batchUpsert(). The library has never had a batchUpsert(). It has upsertMany(), added in v4.2, with a different argument shape.

Tests fail, the self-healing agent tries three fixes, burns 40k tokens, and finally gives up and marks the task blocked. I'd wake up to a pile of "blocked" tasks that all shared one root cause: the agent guessed an API instead of reading it.

When I audited two weeks of failed tasks, the numbers were ugly:

Failure cause	Share of failed tasks
Hallucinated or outdated library API	38%
Misread existing internal code	22%
Genuinely hard bug	19%
Everything else	21%

Sixty percent of failures were "didn't look before writing." That's not an intelligence problem. That's a process problem.

What made it interesting: the implementer could read docs. It had web fetch and file tools. It just didn't, because the moment you give a capable model a task, its instinct is to start producing code. Prompting "please read the docs first" helped for about two days, then the instruction got crowded out by everything else in the context window.

How I Solved It

The fix was structural, not prompt-based. I split "figure out how this works" from "build it" into two different agents with two different tool sets.

The shape of it

flowchart LR
    O[Orchestrator] -->|task + question list| R[Research agent<br/>read-only]
    R -->|research brief| O
    O -->|task + brief| I[Implementation agent]
    I -->|diff| V[Verifier]
    V -->|pass / fail| O
    O -.->|brief missing or low confidence| R

Three rules make this work:

The research agent cannot write. It gets Read, Grep, Glob, web search, and web fetch. No Edit, no Write, no Bash. It physically cannot "just fix it real quick."
The implementer cannot start without a brief. The orchestrator refuses to dispatch an implementation task unless a research brief file for that task exists and has a confidence score.
The brief is short and structured. Not a doc dump. A contract.

The research brief contract

Every brief is a small Markdown file with a fixed shape. Here's a real one, lightly anonymized:

# Research brief: task-0417

## Questions
1. How do we bulk-insert rows with the ORM at the pinned version?
2. Does the existing repo already wrap this anywhere?

## Findings
### Q1
- Pinned version is 4.3.1 (from lockfile).
- Bulk insert is `Model.bulkCreate(rows, { updateOnDuplicate: [...] })`.
- `upsertMany` does NOT exist in this version. It was a community plugin.
- Source: node_modules/<orm>/lib/model.js lines 2210-2290, and the 4.x changelog.

### Q2
- Yes. `src/db/batch.ts` exports `insertBatch()` which already handles chunking at 500 rows.
- Two callers use it. Reuse it; don't add a second path.

## Verdict
Use the existing `insertBatch()` helper. Do not touch the ORM directly.

## Confidence: 0.9
## Sources read: 4 files, 1 changelog, 0 web pages
## Staleness risk: low (pinned version, checked lockfile)

The four fields that matter are Questions, Verdict, Confidence, and Staleness risk. Everything else is supporting evidence. The implementer reads the verdict first. The orchestrator reads the confidence.

The research agent definition

I define sub-agents as Markdown files with frontmatter. The research one looks roughly like this:

---
name: researcher
description: Read-only investigation. Answers "how does X actually work"
  by reading source, lockfiles, changelogs, and official docs. Returns a
  research brief, never code.
tools: Read, Grep, Glob, WebSearch, WebFetch
---

You answer questions about how things actually work. You never propose
an implementation.

Rules:
- Check the pinned version in the lockfile BEFORE reading any docs.
  Docs for the wrong major version are worse than no docs.
- Prefer reading node_modules / site-packages source over web docs.
  Source can't be out of date for the installed version.
- Search the repo for existing wrappers before answering "how do I call X".
- If two sources disagree, report both and lower your confidence.
- Output ONLY the brief format. No preamble.

That "check the lockfile first" line was added after a painful week. More on that below.

How the orchestrator gates on it

The gate itself is boring shell. That's the point. It runs before every dispatch:

brief="state/briefs/${task_id}.md"

if [[ ! -f "$brief" ]]; then
  dispatch_agent researcher "$task_id"
  exit 0   # come back next tick
fi

confidence=$(grep -E '^## Confidence:' "$brief" | awk '{print $3}')
if (( $(echo "$confidence < 0.6" | bc -l) )); then
  # Re-research with the low-confidence sections as new questions
  dispatch_agent researcher "$task_id" --refine
  exit 0
fi

dispatch_agent implementer "$task_id" --brief "$brief"

Two things I like about this:

Deterministic gating. I don't ask the model whether it feels ready. A file exists with a number in it, or it doesn't.
Refinement is a loop with a cap. Low confidence triggers a second research pass focused on the weak questions. After two refinements, the task is marked "needs human" instead of looping forever.

What the implementer sees

The implementer's prompt gets the brief injected verbatim at the top, followed by one line:

The brief above is the source of truth for library APIs and existing helpers. If you believe it is wrong, stop and write why to the task file. Do not work around it.

That last sentence matters. Before I added it, the implementer would occasionally read the brief, disagree silently, and do its own thing. Now disagreement is a visible event I can grep for.

Results after three months

Across about 1,100 tasks on four projects:

Metric	Before	After
Tasks failed on hallucinated API	~20%	~2.5%
Self-healing retries per task (avg)	1.8	0.6
Tokens per completed task	baseline	+8%
Tasks marked "needs human"	11%	6%

The 8% token overhead surprised me. I expected 25% or more. It turned out that the retries I eliminated were far more expensive than the research pass I added. A failed implementation plus three self-healing attempts costs more than reading four files.

Lessons Learned

1. Separate "understand" from "build" with tools, not prompts

Telling a single agent "research first, then code" decays. It works until the context fills up, then the instruction loses to the task. Giving the research agent no write tools makes the separation physical. It can't drift into implementation because implementation isn't possible.

This generalizes: if you want a behavior to be reliable, remove the ability to do the alternative rather than asking nicely.

2. The lockfile is the first document, not the last

My worst week with this system was when the research agent read the latest web docs for a library, wrote a confident brief, and the implementer built against a major version we didn't have installed. Confidence was 0.95. The brief was completely correct for the wrong version.

Now the agent's first action is always reading the lockfile or pip freeze output, and the brief has a Staleness risk field. Source in node_modules beats docs on the web every time, because installed source cannot be out of date for the installed version.

3. Confidence scores are only useful if something acts on them

I initially added the confidence field for my own reading. It was decorative. It became useful the day the orchestrator started gating on it. The moment a number has a consumer, the agent producing it gets noticeably more careful about calibration, because low confidence triggers visible re-work.

Same lesson applies to any structured output from an agent. If nothing reads the field, delete it or wire it up.

4. Make disagreement a logged event

"If you think the brief is wrong, stop and write why" turned a silent failure mode into data. In three months the implementer disagreed with a brief 23 times. It was right 9 times. Those 9 cases became test cases for the research agent's instructions. The 14 wrong disagreements were almost always the implementer wanting to use a newer API it "remembered" from training.

5. Skip research when the task is small, and be explicit about the threshold

Not everything needs a brief. Renaming a variable, adjusting a CSS margin, bumping a copyright year. Running research on those wasted tokens and, worse, made the system feel slow for trivial work.

I added a size heuristic: if the task touches fewer than 2 files and mentions no external library, the orchestrator writes a one-line auto-brief with confidence 1.0 and moves on. The threshold is dumb. Dumb thresholds that exist beat smart thresholds I never got around to building.

What's Next

Two things I'm working on:

Brief reuse across tasks. Right now every task gets fresh research even if the last task answered the same question. I'm adding a simple index keyed on library plus version so briefs get reused for 7 days, then expire.
Research as a first-class MCP server. I want other tools to be able to ask "how does X work in this repo at this version" and get the same brief format back, without going through the orchestrator.

I also want to try the same split for debugging: a read-only "diagnose" agent that produces a hypothesis brief before any fix agent touches the code. Early signs are promising, but I don't have enough data to write about it yet.

Wrap-up

If your coding agent keeps inventing APIs, the fix probably isn't a better model or a longer prompt. It's a process boundary. Give one agent eyes and no hands, give the other hands and a contract, and make the orchestrator refuse to skip the first step.

Versions for anyone reproducing this: Claude Code on the 2.x line as of September 2026, Node.js 22.x, Python 3.13, orchestrator in plain Bash on macOS.

If this was useful, follow me here on Dev.to for more build logs from running a fully autonomous implementation system 24/7. And if you've built a different kind of research or planning gate for your agents, I'd genuinely like to hear how it behaves in the comments. 💡

How I Built a Task Spec Contract Between My Planner and Implementer Agents

yureki_lab — Sun, 20 Sep 2026 14:32:42 +0000

TL;DR

My fully autonomous implementation system splits work between a planner agent and a fleet of implementer sub-agents that start with zero context. For months, roughly one in three implementer runs solved the wrong problem, even though the planner's prose instructions looked fine to me. The fix was not a better prompt. It was a task spec contract: a small, structured schema the planner must emit and the implementer must echo back before touching code. Retries dropped from 31% to 7%, and I got a bonus: the same spec became the input for my verifier agent. Here is the schema, the war story that forced it, and five lessons about agent-to-agent handoffs.

The Problem

Quick background. The system I have been building for about a year runs Claude Code (2.x as of September 2026) in a loop: a planner reads the repo and the backlog, breaks a goal into tasks, and hands each task to a fresh implementer sub-agent. The implementer edits, runs tests, and reports back. A separate verifier agent reviews the diff. Humans (me) only see the summary.

The planner and the implementers do not share a context window. That is deliberate. Fresh context keeps each implementer cheap and focused, and it lets me run four or five of them in parallel. But it also means every task handoff is a cold start. Whatever the planner forgets to write down simply does not exist for the implementer.

Early on, the handoff was a paragraph of natural language. Something like:

Add retry logic to the webhook sender so transient 5xx errors don't drop events. Keep it simple.

Reads fine, right? Here is what actually happened across a sample of 120 handoffs I logged in spring 2026:

Outcome	Count	%
✅ Done, verifier approved on first pass	68	57%
⚠️ Done, but scope crept or wrong file touched	37	31%
❌ Gave up or produced nothing useful	15	12%

The 31% row was the expensive one. Those tasks looked done. The implementer wrote a confident summary, tests passed, and only the verifier (or worse, me, the next morning) noticed that "retry logic" had been implemented as a brand new generic retry utility with its own config file, exponential backoff, jitter, a circuit breaker, and 400 lines of tests. For a webhook sender that already had a retry helper two directories over.

The war story that broke me

The specific incident that made me stop and redesign the handoff:

The planner emitted a task: "Fix the flaky date parsing in the export job." The repo had two export jobs. One was a legacy CSV exporter that nobody had touched in a year. The other was the active JSON exporter that had the actual flaky test. The implementer grepped for "export", found the legacy one first, "fixed" its date parsing by rewriting it to a different library, updated its tests, and reported success. The verifier approved because the diff was internally consistent. The actual flaky test kept flaking for three more days.

Nobody in that chain did anything wrong given what they knew. The planner knew which exporter it meant. It just never said so, because to the planner it was obvious. That is the core failure mode of cold-start handoffs: the sender's obvious is the receiver's unknown.

How I Solved It

I stopped treating the handoff as a message and started treating it as an interface. If the planner and implementer were two services, I would never let them talk in free text. I would give them a schema. So I did.

The task spec schema

Every task the planner emits must be a single fenced block that validates against this shape:

task_id: T-2026-0914-03
goal: >
  Make the JSON export job's date parsing deterministic so
  `test_export_dates_across_dst` stops flaking.
why: >
  The flaky test blocks CI ~2x/day; the root cause is naive
  datetime handling around DST transitions.
scope:
  allowed_paths:
    - src/export/json_exporter.py
    - tests/export/test_json_exporter.py
  forbidden_paths:
    - src/export/csv_exporter.py   # legacy, do NOT touch
    - src/shared/**                # shared helpers need a separate task
context:
  - "There is an existing tz helper at src/export/tz.py; use it, do not write a new one."
  - "The test currently fails ~30% of runs on the 2026-03-08 fixture."
acceptance:
  - cmd: "pytest tests/export/test_json_exporter.py -x --count=20"
    expect: "all 20 runs pass"
  - cmd: "git diff --stat"
    expect: "only the two allowed_paths appear"
forbidden_moves:
  - "Do not add new dependencies."
  - "Do not change the public signature of export_json()."
  - "Do not create a new utility module."
done_signal: >
  Reply with the exact acceptance command outputs, then the
  diff stat. If any acceptance check cannot pass, stop and
  report which one and why. Do not work around it.
budget:
  max_turns: 40
  max_minutes: 25

A few of these fields deserve explanation, because the obvious ones (goal, acceptance) are not where the value came from.

scope.forbidden_paths mattered more than allowed_paths. Allowed paths tell the implementer where to look. Forbidden paths tell it where not to fix things, which is exactly the information that was missing in the date parsing incident. I now make the planner write at least one forbidden path for every task, even if it has to think hard to find one. The act of choosing what to fence off forces the planner to surface the ambiguity it was carrying in its head.

context is for facts the implementer cannot discover cheaply. "There is already a tz helper" is a 10-second fact for the planner (it just read the repo) and a 15-minute discovery for a cold implementer, if it finds it at all. This field is where I put the things the planner "obviously knows."

forbidden_moves is for behaviors, not files. "Do not create a new utility module" kills the 400-line-retry-framework failure mode outright. My current top three forbidden moves, by how often they appear:

Do not create a new module/file when an existing one can be extended.
Do not add a dependency.
Do not "clean up" code adjacent to the task.

done_signal defines what the implementer's final message must contain. Before this, implementers reported success in prose. Now they must paste the acceptance command outputs verbatim. This one change made the verifier's job dramatically easier, because it could diff the claimed output against a re-run.

The echo-back step

The schema alone got me from 31% to about 18% scope failures. The second half of the fix was a mandatory echo-back: the implementer's first action, before reading a single file, is to restate the task in its own words in a fixed format.

## Task echo
- I will change: src/export/json_exporter.py, tests/export/test_json_exporter.py
- I will NOT change: src/export/csv_exporter.py, anything under src/shared/
- I am done when: 20 consecutive pytest runs pass AND diff stat shows only 2 files
- Things I must not do: add deps, change export_json() signature, create new modules
- Open questions: none

If the echo does not match the spec, a tiny check script rejects it and the implementer gets one retry. If Open questions is non-empty, the task is bounced back to the planner instead of proceeding. This felt like ceremony when I added it. It turned out to be the single highest-leverage step, because roughly half the remaining scope failures were the implementer misreading a correct spec, and the echo catches those for the cost of one short turn.

Here is the flow now:

flowchart LR
    P[Planner] -->|task spec YAML| V{Schema valid?}
    V -- no --> P
    V -- yes --> I[Implementer<br/>fresh context]
    I -->|task echo| E{Echo matches spec?}
    E -- no, 1 retry --> I
    E -- open questions --> P
    E -- yes --> W[Edit + run acceptance cmds]
    W -->|done_signal with raw outputs| R[Verifier]
    R -->|re-runs acceptance cmds| M{Match?}
    M -- yes --> Done[Merge]
    M -- no --> P

The validation script

The schema check is deliberately dumb. It is about 60 lines of Python and does not use an LLM. That matters: I want the gate to be deterministic and free.

REQUIRED = {"task_id", "goal", "scope", "acceptance", "forbidden_moves", "done_signal", "budget"}

def validate(spec: dict) -> list[str]:
    errors = []
    missing = REQUIRED - spec.keys()
    if missing:
        errors.append(f"missing fields: {sorted(missing)}")
    scope = spec.get("scope", {})
    if not scope.get("allowed_paths"):
        errors.append("scope.allowed_paths must be non-empty")
    if not scope.get("forbidden_paths"):
        errors.append("scope.forbidden_paths must be non-empty (yes, really)")
    for i, check in enumerate(spec.get("acceptance", [])):
        if "cmd" not in check or "expect" not in check:
            errors.append(f"acceptance[{i}] needs both cmd and expect")
    if len(spec.get("forbidden_moves", [])) < 1:
        errors.append("at least one forbidden_move required")
    return errors

Yes, the forbidden_paths must be non-empty rule is annoying for trivial tasks. I kept it anyway. A planner that cannot name one thing the implementer should not touch has not thought about the task hard enough.

Results

Same 120-task sample size, measured in August 2026 after the contract had been in place for six weeks:

Outcome	Before	After
✅ Approved first pass	57%	81%
⚠️ Scope creep / wrong target	31%	7%
❌ Gave up / nothing useful	12%	12%

Two things I want to be honest about. First, the "gave up" row did not move. The contract fixes misunderstanding, not capability. When a task is genuinely too hard for a single implementer, a better spec does not save it. Second, planner cost went up about 20% per task, because writing a spec takes more tokens than writing a paragraph. That is paid back many times over by the retries I no longer run, but it is not free.

Lessons Learned

1. Treat every cold-start handoff like an API boundary, because it is one. You would never let two microservices exchange free-text and hope. Two agents with separate context windows are exactly that situation. Give them a schema, validate it, and reject bad payloads before they cause work.

2. The sender's "obvious" is the receiver's "unknown." Design the schema to extract it. The most valuable fields (forbidden_paths, context, forbidden_moves) are all just structured ways of forcing the planner to write down what it was silently assuming. Free-form prompts let the planner skip that. A required field does not.

3. Negative space beats positive space. Telling an agent what to do is table stakes. Telling it what not to do is where the leverage is. An implementer with a clear goal and no fences will happily solve the goal in the most expansive way it can imagine. Fences make "the simplest thing that works" the path of least resistance.

4. Make the receiver echo before it acts. One short turn of restating the task catches misreads that would otherwise cost twenty turns of wrong work. It also gives you a clean place to surface open questions instead of letting the agent guess. This is the cheapest reliability upgrade I have ever added to the system.

5. Define the shape of "done," not just the definition. "Tests pass" is a definition. "Paste the raw output of these two commands" is a shape. Shapes can be checked mechanically by the next agent in the chain. Definitions require judgment, and judgment is where drift creeps in.

What's Next

The spec has become the backbone of more than just the planner-implementer handoff. The verifier now reads the same YAML and re-runs acceptance itself, so a task can only be marked done if two independent agents get the same outputs. I am working on two extensions:

Spec diffs for retries. When a task bounces back, the planner currently rewrites the whole spec. I want it to emit a diff against the previous spec so I can see what it learned from the failure, which should feed my self-improvement loop.
Budget-aware decomposition. The budget field is currently just a kill switch. I want the planner to use its own past budget overruns as a signal that a task should have been split in two.

I also want to try the echo-back pattern on the human side. If I have to write a task echo for my own tickets, I suspect I will discover I am just as bad at stating scope as my planner used to be.

Wrap-up

If you are running any kind of multi-agent setup with Claude Code, or even just handing tasks to a single fresh session, try the contract before you try a better prompt. Start with four fields: goal, forbidden_paths, acceptance with real commands, and a done_signal that demands raw output. Add the echo-back. Measure your retry rate before and after. I would genuinely like to hear whether your numbers look like mine.

If this was useful, follow me here on Dev.to 🚀. I write one post like this every week or so about building a fully autonomous implementation system, the stuff that broke, and what I changed. And if you have a handoff schema of your own, drop it in the comments. I am collecting them.

How I Migrated 90 Cypress Tests to Playwright With Claude Code in 4 Days

yureki_lab — Sat, 19 Sep 2026 14:32:12 +0000

TL;DR

I moved a 90-spec Cypress suite to Playwright in 4 working days using Claude Code. The trick wasn't "ask the AI to convert everything." It was: hand-translate one spec, extract the pattern into a written rulebook, then let the agent fan out across the other 89 while a screenshot-diff gate caught the silent behaviour drift. Here's the workflow, the two traps that cost me a day, and what I'd do differently.

The Problem

Our end-to-end suite was 90 Cypress specs, written over three years by six different people. It ran in about 38 minutes on CI, flaked roughly once every four runs, and had grown a 600-line commands.js full of custom helpers that nobody fully understood anymore.

We wanted Playwright for the usual reasons: real multi-tab support, first-class parallelism, and one runner for the three browsers we actually ship to. The problem was the migration itself. Every estimate I got from the team landed at "two sprints, maybe three." Nobody wanted to own it. It was pure toil with zero product upside until the very last spec crossed over.

That's exactly the kind of work I've been throwing at Claude Code (v2.1.x at the time, Node.js 22, Playwright 1.54). Mechanical, high-volume, pattern-heavy, and easy to verify. So I gave myself one week and a rule: no spec gets deleted from Cypress until its Playwright twin passes 10 times in a row.

How I Solved It

Step 1: Translate one spec by hand, badly

I picked a medium-complexity spec (the checkout flow: login, add to cart, apply coupon, pay with a test card) and translated it myself. No AI. About 90 minutes.

It was ugly, but that was the point. Doing it by hand surfaced every decision that a converter would otherwise make silently:

Cypress cy.get('[data-cy=x]') becomes page.getByTestId('x'), which meant changing the test-id attribute in playwright.config.ts.
Cypress's implicit retry-until-assert becomes an explicit await expect(locator).toHaveText().
Our cy.login() custom command hits an API and sets a cookie. In Playwright that became a storageState fixture created once per worker, not per test.
cy.intercept() stubs became page.route(), but the matcher semantics differ (glob vs. minimatch), and I got two wrong on the first try.

Each of those became a line in a file I called MIGRATION_RULES.md. By the end of spec one, it had 14 rules.

Step 2: Turn the rulebook into agent instructions

I put the rulebook into the project spec file that Claude Code reads on startup, along with the before/after of my hand-translated spec as a worked example. The key section looked like this:

## Cypress -> Playwright migration rules

1. Never translate a custom command inline. Look it up in
   cypress/support/commands.js, then map it to the fixture
   in tests/fixtures/*.ts. If no fixture exists, STOP and
   report which command is missing.
2. Every cy.get(...).should(...) chain becomes a single
   `await expect(locator).toX()`. Do not add manual waits.
3. cy.intercept(method, urlGlob) -> page.route(urlGlob).
   Convert Cypress glob to Playwright glob; verify with the
   table in MIGRATION_RULES.md section 3.
4. Keep the original spec's describe/it names verbatim so
   the test report diff is readable.
5. After writing a spec, run it 3x with `--repeat-each=3`.
   Only report success if all 3 pass.

Rule 1 turned out to be the most important line I wrote all week. More on that below.

Step 3: Fan out, in batches of five

I didn't run one giant "migrate all 89" session. I ran batches of five specs, each in a fresh session, with a prompt like:

Migrate these 5 Cypress specs to Playwright following the rules
in CLAUDE.md. Work on them one at a time. For each: write the
spec, run it 3x, then show me the pass/fail summary before
moving to the next. Do not touch any file outside tests/e2e/.

Why five? Three reasons:

Context stays sharp. After roughly 8 to 10 specs in one session, the agent started "remembering" helper names that didn't exist. Fresh sessions fixed that.
Failures stay cheap. If batch 7 went sideways, I lost 20 minutes, not a morning.
I could review at human speed. Five diffs is a coffee break. Eighty-nine is a weekend.

Here's the flow end to end:

flowchart LR
    A[Pick 5 Cypress specs] --> B[Agent translates spec N]
    B --> C{3x pass?}
    C -- no --> D[Agent reports failure + reason]
    D --> E[I fix rule or fixture]
    E --> B
    C -- yes --> F[Screenshot diff vs Cypress run]
    F -- drift --> E
    F -- clean --> G[Mark spec migrated]
    G --> B

Batches 1 through 4 went through in about a day and a half. Twenty specs, no drama. Then I hit the first trap.

Trap 1: The custom command that lied

Batch 5 included a spec that called cy.selectPlan('pro'). The agent, following rule 1, looked it up. The command did three things: clicked a plan card, waited for a modal, and silently dismissed a "confirm downgrade" dialog if it appeared.

That third behaviour was never documented. It existed because two years ago someone had a flaky test and patched the helper instead of the app. The Playwright fixture the agent wrote did the first two things, and the test passed. It passed because the test data never triggered the downgrade dialog.

I only caught it because rule 1 also said "report which command is missing," and the agent's report included a one-liner: "Note: the Cypress version also handles a confirm dialog conditionally; I did not port this since no test exercises it."

That single sentence saved a production bug. It turned out our real app did show that dialog for one plan transition, and the old Cypress helper had been hiding a broken flow for two years.

Lesson: when the agent says "I skipped this, here's why," read it. Every time.

Trap 2: Green tests that tested nothing

Around batch 9, the pass rate got suspiciously good. Every spec passed 3 of 3 on the first attempt. I got nervous and pulled up a diff.

The agent had discovered that Playwright's auto-waiting expect() is very forgiving, and had translated a Cypress assertion like this:

cy.get('[data-cy=total]').should('contain', '$49.00');

into this:

await expect(page.getByTestId('total')).toBeVisible();

Visible. Not "contains $49.00." The element was always visible. The test could never fail.

This wasn't the agent being lazy. It was me being vague. My rule 2 said "becomes a single expect" but never said "preserve the assertion's semantics exactly." Fixing that took one line in the rulebook and a re-run of 11 specs.

Step 4: The screenshot-diff gate

This is the part I'd keep even if I never migrate another test suite.

Before touching anything, I ran the full Cypress suite once with screenshots on at every it() boundary, and saved them to baseline/. For each migrated Playwright spec, I captured a screenshot at the same boundaries and diffed with pixelmatch:

import { test, expect } from '@playwright/test';
import { compareToBaseline } from '../fixtures/visual';

test('checkout: coupon applied', async ({ page }) => {
  // ... steps ...
  await compareToBaseline(page, 'checkout-coupon-applied', {
    threshold: 0.02,
  });
});

The diff caught four cases where the Playwright test passed, the assertions passed, but the screen looked different. Three were timing (Playwright was faster and captured mid-animation). One was real: a locator matched a different button with the same label, so the test was exercising the wrong path and getting lucky.

Lessons Learned

Do the first one by hand, always. The 90 minutes I spent translating spec one produced the 14 rules that made the other 89 possible. If you skip this, the agent makes every one of those decisions for you, silently and inconsistently.
"Stop and report" beats "figure it out." The single highest-value instruction was telling the agent to halt on unknown helpers instead of guessing. Every real bug I caught came through one of those reports.
Batch small, session fresh. Five specs per session was the sweet spot. Bigger batches got hallucinated helper names. Smaller ones wasted my review time on context-switching.
Passing tests are not evidence. Failing-when-they-should tests are. After trap 2, I added a step: for every migrated spec, deliberately break the app once and confirm the test goes red. The agent can do this too, and it takes 30 seconds per spec.
Visual diffs are the cheapest oracle you have. Assertions encode what someone remembered to check. Screenshots encode everything. For a migration, "does it look the same" catches a class of drift that no assertion will.

What's Next

The suite now runs in 11 minutes across 4 workers, down from 38, and I haven't seen a flake in three weeks. The commands.js file is gone. The 600 lines became about 180 lines of typed fixtures that a new hire can actually read.

Next up, I'm pointing the same "hand-translate one, extract rules, fan out" workflow at our 40-odd Jest snapshot tests, which have the same "green but meaningless" problem in a different costume. I'm also experimenting with having the agent write the deliberate break step itself, so the "does this test actually fail" check becomes part of the migration loop instead of something I remember to do.

If there's interest, I'll write up the fixture design in a follow-up. The storageState-per-worker pattern alone cut our login overhead by 80%.

Wrap-up

If you're staring at a test-suite migration that nobody wants to own, this is the workflow I'd hand you:

Translate one spec yourself. Write down every decision.
Turn those decisions into rules the agent can't misread.
Fan out in small batches. Make the agent report what it skipped.
Verify with something other than the tests themselves.

Got a migration war story of your own, or a rule that saved you a bad afternoon? Drop it in the comments. And if you want the follow-up on fixture design, hit follow so it lands in your feed. 🚀

How I Built a Remote Control Dashboard for My Autonomous Coding Agent

yureki_lab — Fri, 18 Sep 2026 14:31:57 +0000

TL;DR

I run a fully autonomous implementation system 24/7 on a Mac mini. For months, the only way to intervene was to SSH in and kill processes by hand. So I built a small remote control dashboard I can open on my phone to pause, steer, approve, and inspect the agent from anywhere. This post walks through the design (a file-based command queue, an approval gate, and a heartbeat), the code that matters, and five lessons from six months of running it.

The Problem

Here's the setup: an orchestrator module spawns parallel implementation agents built on Claude Code (v2.1.x at the time of writing), each working on a task from a queue. It plans, writes code, runs tests, commits, and moves on. It runs while I sleep. It runs while I'm at the gym. It runs while I'm on a train with no laptop.

That last part was the issue. 🚨

Three things kept happening:

The agent would go down a rabbit hole. A "rename this config key" task would turn into a 40-file refactor because the agent decided the old name was confusing everywhere. I'd find out 6 hours later.
It would hit a decision it shouldn't make alone. Delete a migration? Push to a shared branch? Rotate a credential? The agent was told to stop and wait for a human on those. But "wait for a human" meant "wait until I'm at my desk".
I had no idea what it was doing right now. Logs were on the box. Reading them meant a terminal, a VPN, and tail -f. Not a thing you do from a phone at dinner.

My first instinct was "just SSH from the phone". I tried it for two weeks. Typing kill -9 on a 6-inch screen at 2am while half-asleep is a great way to kill the wrong process. I did that. Twice.

What I actually needed was a remote control, not a remote shell. A handful of big, safe buttons and a clear status readout.

How I Solved It

Design constraints

I set three rules before writing any code:

The agent must not depend on the dashboard. If the dashboard is down, the agent keeps working. If the agent is down, the dashboard says so. No shared process, no shared database.
Every command must be idempotent and safe to replay. Mobile networks retry. Tapping "pause" twice must not do anything weird.
Read is cheap, write is gated. Anyone with the link can see status (it's behind auth anyway). Writing a command requires a second factor.

Architecture

The whole thing is three parts talking through the filesystem:

flowchart LR
    Phone[Phone browser] -->|HTTPS| Dash[Dashboard server\nFastAPI]
    Dash -->|writes JSON| Queue[(commands/ dir)]
    Dash -->|reads| Status[(status.json + logs)]
    Agent[Autonomous agent loop] -->|polls every 5s| Queue
    Agent -->|writes every 30s| Status

Yes, a directory of JSON files as a queue. No Redis, no Postgres, no message broker. I'll defend that choice in the lessons section.

The command queue

Each command is a single file. The filename is a UUID; the content is a tiny JSON document. The agent's main loop polls the directory between task steps:

# agent side: runs between every tool call / task step
import json, os, time
from pathlib import Path

COMMANDS = Path("~/agent/commands").expanduser()
PROCESSED = COMMANDS / "processed"

def drain_commands(state):
    for f in sorted(COMMANDS.glob("*.json")):
        try:
            cmd = json.loads(f.read_text())
        except json.JSONDecodeError:
            f.rename(PROCESSED / f"{f.name}.corrupt")
            continue

        kind = cmd.get("kind")
        if kind == "pause":
            state.paused = True
        elif kind == "resume":
            state.paused = False
        elif kind == "abort_task":
            state.abort_current = True
        elif kind == "steer":
            # appended to the next prompt as a user instruction
            state.pending_notes.append(cmd["text"])
        elif kind == "approve":
            state.approvals.add(cmd["request_id"])
        elif kind == "deny":
            state.denials.add(cmd["request_id"])

        f.rename(PROCESSED / f.name)

The steer command is the one I use most. It doesn't interrupt anything. It just says: "before your next step, read this note from me". Things like "Don't touch the billing module, I'm working on it locally" or "The flaky test is known, skip it and keep going". The orchestrator injects it into the next prompt as a high-priority user message.

The approval gate

This is the part that changed how I sleep. 😴

The agent has a list of actions it is not allowed to do without a human. Deleting files outside the repo, force-pushing, running migrations against anything but a local database, touching secrets. When it hits one, it doesn't fail and it doesn't guess. It writes an approval request and blocks on that request ID:

# agent side
def request_approval(action: str, detail: str, timeout_s: int = 6 * 3600) -> bool:
    req_id = uuid.uuid4().hex
    (REQUESTS / f"{req_id}.json").write_text(json.dumps({
        "id": req_id,
        "action": action,
        "detail": detail,
        "created": time.time(),
    }))
    notify_phone(f"Approval needed: {action}")   # push notification
    deadline = time.time() + timeout_s
    while time.time() < deadline:
        drain_commands(state)
        if req_id in state.approvals:
            return True
        if req_id in state.denials:
            return False
        time.sleep(5)
    return False   # timeout = deny, always

Timeout equals deny. Always. I went back and forth on this. An agent that treats silence as consent is an agent that will eventually do something irreversible at 3am because I was asleep. Silence means "not now", and the task gets parked, not dropped.

On the dashboard, an approval request renders as a card with the action, the detail (usually a diff or a command), and two big buttons. Green and red. That's it.

The status heartbeat

The agent writes a status.json every 30 seconds and after every task boundary:

{
  "ts": 1789740000,
  "state": "running",
  "current_task": "Add retry to webhook delivery",
  "step": 14,
  "tokens_today": 812000,
  "last_commit": "a1f3c9e",
  "pending_approvals": 1,
  "agents_active": 3
}

The dashboard reads it and does one thing I'd underrate if I hadn't lived without it: it shows how stale the heartbeat is. If ts is more than 90 seconds old, the header goes amber. More than 5 minutes, red. That single indicator caught two hung processes and one out-of-disk incident before anything else did.

The dashboard itself

Roughly 300 lines of FastAPI (Python 3.13) plus one HTML template with vanilla JavaScript. No framework. It fits on a phone because I designed it on a phone first:

Top: state badge, heartbeat age, current task.
Middle: pending approval cards (if any).
Bottom: four buttons. Pause, Resume, Abort current, Steer (opens a text box).
Last 50 log lines behind a collapsible section.

Writes go through a one-time code from an authenticator app. Reads are just behind the reverse proxy's basic auth. The server itself runs as a separate background service so it survives agent restarts, and it never imports anything from the agent codebase.

@app.post("/cmd/{kind}")
def post_command(kind: str, body: CommandIn, totp: str = Header(...)):
    if kind not in ALLOWED_KINDS:
        raise HTTPException(400)
    if not verify_totp(totp):
        raise HTTPException(403)
    payload = {"kind": kind, "created": time.time(), **body.model_dump()}
    tmp = COMMANDS / f".{uuid.uuid4().hex}.tmp"
    tmp.write_text(json.dumps(payload))
    tmp.rename(COMMANDS / f"{uuid.uuid4().hex}.json")   # atomic
    return {"ok": True}

Write to a temp file, then rename. The agent never sees a half-written command.

Lessons Learned

1. A directory of JSON files beats a queue you have to keep alive

I got roasted for this in a Discord. Fine. Here's the thing: the queue has zero dependencies, survives reboots, is inspectable with ls, and debuggable with cat. In six months it has never been the failing component. Every "real" queue I've run has needed babysitting at some point. For a single-machine, single-consumer, low-volume control channel, files win. 💡

2. "Steer" is worth more than "stop"

I built pause and abort first because they felt like the safety features. I use steer ten times more often. Most interventions aren't "stop everything", they're "you're missing context, here it is, keep going". Giving the agent a way to receive mid-task notes without losing its place turned a lot of would-be aborts into small corrections.

3. Silence must mean no

Any autonomous system with a human-in-the-loop gate will eventually run into the human being unavailable. Decide up front what happens. The answer that lets you sleep is "park the task and move on". The answer that ends your weekend is "assume yes after N minutes".

4. Heartbeat age is the most important pixel on the screen

Not the log. Not the task name. The number of seconds since the agent last said "I'm alive". Everything else can be wrong or stale; that number tells you whether to trust the rest of the screen.

5. Design the control surface for your dumbest future self

The person using this dashboard is me at 2am, on a phone, with one eye open. Big buttons. Confirmation on anything destructive. No text input required for the common path. The agent is sophisticated; the remote control should be boring. ⚠️

What's Next

Two things I'm working on:

Approval bundling. Right now each gated action is its own request. When the agent is migrating something, I get five approvals in a row. I want it to batch related requests into one card with one decision.
Read-only sharing. A teammate wants to watch the agent work on a shared repo without being able to steer it. That means separating the read and write auth properly instead of leaning on the reverse proxy.

Longer term, I want the dashboard to show why the agent is doing what it's doing, not just what. A short "current reasoning" field in the heartbeat, written by the agent in one sentence, would go a long way.

Wrap-up

If you're running any kind of long-lived AI coding agent and your intervention story is "SSH in and kill it", build the remote control. It's a weekend of work. The command queue is 40 lines, the approval gate is 30, the dashboard is an afternoon. You'll never go back to tail -f on a phone.

If this was useful, follow me here on Dev.to 🚀 — I'm writing up the rest of this autonomous system piece by piece: the orchestrator, the self-healing agent, and the observability layer. And if you've built something similar, tell me in the comments what your "steer" equivalent looks like. I want to steal your ideas.

How I Audited 1,400 npm Dependencies for License Risks With Claude Code

yureki_lab — Thu, 17 Sep 2026 14:32:28 +0000

TL;DR

A customer's procurement team asked us for a license inventory of our product, and I discovered our npm lockfile contained 1,400+ dependencies nobody had ever audited. I used Claude Code to classify every license, dig into the sketchy ones, trace how the risky packages got into the tree, and wire a CI gate so this never silently regresses. Total time: about two days instead of the two weeks I'd budgeted. 💡

The Problem

Here's an email you never want to receive: "Before we can renew, our legal team needs a complete inventory of third-party licenses in your product, including transitive dependencies."

Our package.json listed 62 direct dependencies. Innocent enough. Then I ran:

npm ls --all --parseable | wc -l
# 1417

1,417 packages. Every single one of them ships with a license, and every single one of those licenses is a tiny legal contract we had implicitly agreed to. Nobody on the team — including me — had ever read more than a handful of them.

The scary part isn't the well-behaved majority. Something like 90% of the npm ecosystem is MIT/ISC/Apache-2.0 and genuinely fine. The scary part is the tail:

Copyleft licenses (GPL, AGPL) hiding three levels deep in the tree
Packages whose license field says one thing while the bundled LICENSE file says another
Packages with "license": "SEE LICENSE IN LICENSE.txt" — which tells you exactly nothing

Reading 1,400 license files by hand is exactly the kind of soul-crushing, judgment-requiring-but-barely, high-volume work that I now refuse to do myself. So I opened Claude Code (v2.x, on Node.js 22.x) and made it a compliance intern for two days.

How I Solved It

Step 1: Get the raw inventory mechanically

First rule: don't make the AI do work a deterministic tool does better. The initial sweep came from license-checker-rss, which reads the metadata for the whole installed tree:

npx license-checker-rss --json --production > licenses.json
jq 'length' licenses.json
# 1417

Then I asked Claude Code to summarize the distribution before doing anything fancy:

Read licenses.json. Group packages by license identifier and give me
a count per license, sorted descending. Flag anything that is not
MIT, ISC, BSD-2-Clause, BSD-3-Clause, Apache-2.0, or 0BSD.

The distribution came back looking like most JS projects:

Bucket	Count
MIT / ISC / BSD / Apache-2.0	1,361
MPL-2.0, LGPL (weak copyleft)	19
GPL family flagged	4
`UNKNOWN`, `SEE LICENSE IN`, custom	33

So 1,361 packages needed zero further attention, and 56 packages needed actual eyes. That's the whole game: use cheap tools to shrink the haystack, use the agent on what's left.

Step 2: Let the agent read the weird ones

The 33 "unknown/custom" packages are where metadata-only scanners give up — and where an agent that can read files shines. Every one of those packages has something in node_modules: a LICENSE file, a license section in the README, a header comment in the source.

I gave Claude Code a classification rubric up front, in the project's instruction file, so it wouldn't improvise legal opinions:

## License classification rules
- Classify as exactly one of: PERMISSIVE / WEAK_COPYLEFT /
  STRONG_COPYLEFT / PROPRIETARY / CANNOT_DETERMINE
- Quote the exact sentence from the license text that justifies
  the classification. No quote, no classification.
- If the package.json license field and the LICENSE file disagree,
  report BOTH and classify by the LICENSE file.
- Never guess. CANNOT_DETERMINE is a valid and welcome answer.

Then the sweep itself:

For each package in unknowns.txt, read its files under node_modules/,
find the actual license text, and classify it per the rules in
CLAUDE.md. Output one JSON line per package: name, version,
classification, evidence quote, file path.

That last rule — quote the sentence or don't classify — is the single most important line. On my first attempt without it, the agent confidently labeled a package MIT because the README "looked standard." With the evidence-quote requirement, every classification came back verifiable, and I spot-checked them in minutes instead of re-reading everything.

Results from the 33 unknowns: 26 were permissive licenses with lazy metadata, 4 were custom-but-clearly-permissive texts, 2 were CANNOT_DETERMINE (dead packages with no license text anywhere — we replaced them), and 1 was a genuine surprise…

Step 3: The AGPL in the basement 😱

One of the four GPL-family flags turned out to be AGPL-3.0, sitting four levels deep under a charting library. If you're not familiar: AGPL is the license where even network use can trigger source-disclosure obligations. For a proprietary SaaS product, that's not a footnote — that's a drop-everything finding.

The immediate question was: how did this get here, and can it go away? This is where npm why plus an agent that can read changelogs saved me hours:

npm why aggressively-licensed-package

Ask: which of our direct dependencies ultimately pulls this in,
is it actually imported at runtime or only used at build time,
and does a newer version of the parent drop this dependency?

Claude Code traced the chain, checked our bundle output to confirm the package actually shipped to production (it did — build-time-only would have been a much easier conversation), and then found that the parent library had replaced this dependency two major versions ago, precisely because of the license. The fix was a scheduled upgrade we'd been putting off anyway. One npm install later, the AGPL was gone.

I want to be honest: an experienced engineer with npm why and patience would have found the same answer. The agent didn't do anything superhuman. It just did 45 minutes of archaeology in 4, and it read the parent library's changelog so I didn't have to.

Step 4: Make it impossible to regress

An audit that runs once is a snapshot, not a control. The lockfile changes every week; next month's npm install can quietly reintroduce the exact problem. So the last step was a CI gate — and this part I had Claude Code write end to end:

// scripts/check-licenses.mjs — runs in CI on every lockfile change
import { execSync } from "node:child_process";

const ALLOWED = new Set([
  "MIT", "ISC", "0BSD", "BSD-2-Clause", "BSD-3-Clause",
  "Apache-2.0", "CC0-1.0", "Unlicense",
]);
// Reviewed one-by-one; each entry links to the review note.
const EXCEPTIONS = new Map([
  ["some-mpl-package@4.2.1", "MPL-2.0 ok: unmodified, dynamically linked"],
]);

const raw = execSync("npx license-checker-rss --json --production");
const tree = JSON.parse(raw);

const violations = Object.entries(tree).filter(([pkg, info]) => {
  const license = String(info.licenses ?? "UNKNOWN");
  return !ALLOWED.has(license) && !EXCEPTIONS.has(pkg);
});

if (violations.length > 0) {
  console.error("License gate failed for:");
  for (const [pkg, info] of violations) {
    console.error(`  ${pkg} → ${info.licenses}`);
  }
  process.exit(1);
}
console.log(`License gate passed: ${Object.keys(tree).length} packages checked ✅`);

The design choice that matters: it's an allowlist, not a blocklist. A blocklist of "bad" licenses silently passes anything new or weird. An allowlist fails closed — a never-before-seen license identifier stops CI and forces a human (or an agent with a rubric) to look at it once, add it to ALLOWED or EXCEPTIONS with a written reason, and move on.

The whole flow now looks like this:

flowchart LR
    A[lockfile change] --> B[license-checker in CI]
    B --> C{all in allowlist?}
    C -- yes --> D[merge ✅]
    C -- no --> E[CI fails]
    E --> F[agent-assisted review]
    F --> G[allowlist or replace]
    G --> D

Lessons Learned

Shrink the haystack with dumb tools before you deploy the smart one. Metadata scanning removed 96% of the work for free. Pointing an LLM at all 1,417 packages would have been slower, pricier, and noisier. The agent earns its keep on the ambiguous residue, not the bulk.
Demand evidence quotes, not conclusions. "Classify this license" gets you plausible answers. "Classify it and quote the sentence that proves it" gets you checkable answers. This one prompt rule turned spot-checking from re-doing the work into skimming it.
CANNOT_DETERMINE is a feature. Explicitly telling the agent that "I don't know" is a welcome answer is the difference between an audit and a hallucination generator. The two packages it refused to classify were exactly the two that deserved human escalation.
The license field lies more often than you'd think. Out of 33 manually-reviewed packages, several had metadata that disagreed with the shipped license text. If your compliance story is built purely on package.json fields, it's built on the honor system.
An audit without a CI gate is theater. The one-time sweep satisfied procurement. The allowlist gate is what makes the answer stay true. Fail closed on anything unrecognized, and make every exception carry a written justification.

⚠️ Obligatory disclaimer: I'm an engineer, not a lawyer, and neither is Claude. The agent did the reading, sorting, and evidence-gathering; the actual risk calls on the flagged packages went through humans (and for the AGPL one, actual counsel). Use AI to prepare the decision, not to make it.

What's Next

Two things on my list: generating a proper SBOM (CycloneDX format) from the same pipeline so the next procurement request is a one-command answer, and running the identical playbook against our Python service — pip has its own flavor of license chaos, and I suspect its tail is even weirder than npm's.

Wrap-up

Dependency license auditing is the perfect AI-agent workload: massive volume, mostly mechanical, occasionally requiring real judgment — and an agent with file access plus a strict rubric handles the volume while surfacing exactly the cases that need your brain.

If you've never looked at your own lockfile's license tail, run npx license-checker-rss --summary today. It takes thirty seconds, and you might meet your own AGPL in the basement.

Have you found something scary in your dependency tree? Tell me your worst license surprise in the comments — and follow me here on Dev.to 🚀 for more write-ups on putting coding agents to work on the jobs nobody wants.

How I Built a Slack Bot That Answers Codebase Questions With Claude

yureki_lab — Wed, 16 Sep 2026 14:32:44 +0000

TL;DR

I built a Slack bot that answers "how does X work?" and "where does Y live?" questions about our private codebase, using Claude with tool-calling retrieval instead of embedding-based RAG. It took about two weekends, it answers most questions in under 30 seconds, and the biggest engineering problems had nothing to do with AI — they were about trust, staleness, and permissions. Here's the full build log, including the parts that went wrong. 🚀

The Problem

Every team has that one person who knows where everything is. On my projects, that person is me — and I was tired of being a human grep.

The questions were always the same shape:

"Where do we validate webhook signatures?"
"How does the retry logic work in the billing worker?"
"Is there already a helper for parsing these date formats?"

Each one costs the asker 20 minutes of digging or costs me a context switch. Multiply that by a few questions a day and you're losing real engineering time to archaeology, not building.

The obvious answer in 2026 is "point an LLM at the codebase." But the obvious implementation — chunk the repo, embed it, cosine-similarity your way to an answer — is where I started, and it's the first thing I threw away.

How I Solved It

The architecture that didn't work: embedding RAG

My first version was textbook RAG: split every file into 500-token chunks, embed them, stuff the top-k matches into a prompt.

It failed in a very specific way. Code questions are usually structural, not topical. When someone asks "how does retry logic work in the billing worker," the answer isn't in any single chunk. It's spread across a config file, a decorator definition, and the call site — three files that don't share much vocabulary. Embedding similarity finds you five chunks that all mention the word "retry" (including two from tests and one from a changelog) and none of the ones that matter.

I spent a week tuning chunk sizes and rerankers before admitting the approach was wrong for this job.

The architecture that worked: let the model drive the tools

The fix was to stop pre-deciding what context the model needs and instead give it the same tools I would use: search and read. Claude decides what to look for, reads what it finds, and follows the trail — the same loop a human does, just faster.

flowchart LR
    A[Slack mention] --> B[Bot server]
    B --> C{Claude agent loop}
    C -->|search_code| D[ripgrep over repo]
    C -->|read_file| E[File reader]
    D --> C
    E --> C
    C -->|final answer| F[Slack thread reply]

The whole thing is a Slack Bolt app (Python 3.13, slack-bolt 1.21) plus the Anthropic SDK. Two tools, that's it:

TOOLS = [
    {
        "name": "search_code",
        "description": "Search the codebase with a regex. Returns matching "
                       "lines with file paths and line numbers.",
        "input_schema": {
            "type": "object",
            "properties": {
                "pattern": {"type": "string"},
                "glob": {"type": "string", "description": "Optional file filter, e.g. '*.py'"},
            },
            "required": ["pattern"],
        },
    },
    {
        "name": "read_file",
        "description": "Read a file (or a line range) from the codebase.",
        "input_schema": {
            "type": "object",
            "properties": {
                "path": {"type": "string"},
                "start_line": {"type": "integer"},
                "end_line": {"type": "integer"},
            },
            "required": ["path"],
        },
    },
]

search_code is just ripgrep behind a subprocess call, with output capped so a careless .* doesn't blow up the context window:

def search_code(pattern: str, glob: str | None = None) -> str:
    cmd = ["rg", "--line-number", "--max-count", "8", "--no-heading", pattern]
    if glob:
        cmd += ["--glob", glob]
    out = subprocess.run(cmd, cwd=REPO_ROOT, capture_output=True, text=True, timeout=10)
    return out.stdout[:6000] or "No matches."

The agent loop is the standard tool-use pattern: call the model, execute any tool calls, feed results back, repeat until it produces a text answer. I run it on claude-sonnet-5 — fast enough that a 6-hop investigation still comes back inside Slack-attention-span, and cheap enough that I don't think about the bill.

A typical answer takes 3–6 tool calls. Watching the transcripts is genuinely fun: it greps for the obvious keyword, reads the hit, notices an import, follows the import, then answers with file paths and line numbers.

For the numbers people: median answer latency is around 20 seconds end-to-end, the slowest multi-hop investigations land near 45, and a month of real usage cost less than a single takeout lunch in API fees. Compare that to the 20 minutes of human digging each question used to burn and the math stops being interesting — it's just obviously worth it. 💡

The three problems that actually mattered

1. Hallucinated file paths destroy trust instantly. ⚠️

Early on, the bot answered a question with a confident reference to utils/validation.py — a file that did not exist. It had seen similar codebases in training and invented a plausible path. One wrong answer like that and your teammates stop trusting every answer.

The fix was mechanical, not prompt-based: before sending the reply, I extract every path-looking string from the answer and check it against the actual file tree. Any path that doesn't exist gets the whole answer flagged:

def verify_paths(answer: str, repo_files: set[str]) -> list[str]:
    candidates = re.findall(r"[\w./-]+\.(?:py|ts|tsx|go|sql|yaml|toml)\b", answer)
    return [p for p in candidates if p.strip("`") not in repo_files]

If verify_paths returns anything, the bot appends a visible warning and links the search query it ran, so the human can check. Since adding this, "confidently wrong path" incidents went from a few per week to zero reaching users.

2. Stale checkouts give stale answers.

The bot answers from a local clone. A clone that's three days old will cheerfully describe code that was refactored away on Tuesday. My first "fix" was pulling on a timer; the real fix was pulling on demand: every question triggers a git fetch + fast-forward before the agent loop starts. It adds ~2 seconds and eliminated the entire class of "that function doesn't exist anymore" answers. The answer also includes the commit SHA it read from, which turned out to be the single most trust-building detail in the whole project.

3. The bot must not see more than the asker can.

This is the one people skip. A codebase Q&A bot is effectively a read amplifier: anyone who can ask it questions can extract anything in its clone. Two hard rules came out of this:

The bot's clone excludes anything secret-shaped — .env files never land in git anyway (right? 😅), but I also deny-list config directories and run the same secret scanner we use in CI against every tool result before it enters the model context.
The bot only joins channels where everyone already has repo read access. No DMs. This one is organizational, not technical, and it's non-negotiable.

Lessons Learned

Tool-calling retrieval beats embedding RAG for code Q&A. Code questions are graph traversals, not similarity lookups. Give the model grep and read, and let it walk the graph. My answer quality jumped more from this one architectural change than from everything else combined.
Verify outputs mechanically, not with prompts. "Do not invent file paths" in the system prompt reduced hallucinations. Checking paths against the file tree eliminated them. Anything you can verify with code, verify with code.
Citations are a feature, not decoration. Answers that end with billing/worker.py:141 (at commit a3f9c21) get trusted and clicked. Answers without them get re-asked to a human. If your bot can't cite, it's a liability.
Freshness is a product requirement. Nobody forgives "that code was deleted last week." Sync on demand, stamp the SHA, and staleness stops being a conversation.
Scope the bot's eyes to the asker's eyes. Decide what the bot can read and who can ask it before the first deploy, not after the first awkward answer. Retrofitting access control onto a chatbot is miserable.

What's Next

Two things are on the list. First, cross-repo questions — "which services call the notification API?" needs the bot to search several clones and merge results, which mostly means smarter tool routing. Second, letting it answer with diagrams: it already understands structure well enough that generating a Mermaid diagram of a subsystem on request feels within reach.

I'm deliberately not giving it write access. An answer bot that occasionally opens "helpful" PRs is a different product with a very different blast radius — that's a post for another day.

Wrap-up

Total damage: ~600 lines of Python, two weekends, and one discarded RAG pipeline. The bot now handles the majority of "where is / how does" questions in our Slack, and the ones it can't handle come to me with context attached instead of from zero.

If you've been thinking about pointing Claude at your own codebase: skip the embedding pipeline, start with two tools and an agent loop, and spend your saved week on verification and access control. That's where the real product is. ✅

If this was useful, follow me here on Dev.to — I write weekly about building with AI coding agents, including the failures. And if you build your own version, I'd genuinely love to hear what broke first: drop it in the comments. 💬