Running 4 agents in parallel: the state files that keep them sane

#ai #programming #productivity #claude

One agent is a collaborator. Four agents are a concurrency problem.
I run four regularly now, and the difference between chaos and a
working fleet came down to state files: what gets written where,
who reads it, and when. This is the setup, with the numbers from
our own ops.

The lanes

Four agents, two kinds of lane:

research lanes (2): read-only investigation, no code changes
implementation lanes (2): code changes, one task each

The split matters more than the count. Research work (finding where
a bug lives, reading a library's source, mapping an API) burns
context fast and produces conclusions, not diffs. Mixing it with
implementation meant implementation agents inheriting 150k tokens of
exploration noise. Separate lanes fixed it: research concludes, a
human or orchestrator reads the conclusion, implementation gets a
clean brief.

The state file per lane

Each lane writes to exactly one file, append-mostly:

# state: lane-impl-1
task: add rate limiting to login route
branch: feat/rate-limit
done:
- middleware scaffold + tests
- 429 response shape per api-conventions.md
in-flight: redis-backed counter
blocked: none
next: wire counter into middleware, integration test

The discipline is boring and strict: update at task boundaries, not
mid-edit. The file is the handoff surface. If a lane dies, its
replacement reads the state file and continues in minutes instead
of re-deriving an hour of context.

Why not just share one plan

Tried it. Shared documents get write-contended: two agents editing
the same plan file mid-run corrupted entries twice in one week.
Per-lane files with a single writer each never corrupted once. The
orchestration layer (me, or a script) is the only reader that
crosses lanes.

The WIP cap

Two implementation lanes means at most two tasks in flight. When a
third task tempts me mid-run, the rule is: it queues, nothing
launches. The reason is merge surface. In our ops, two parallel
implementation branches merge cleanly almost every time; three
started producing conflict resolution sessions that cost more than
the parallelism saved. Your conflict threshold will differ with
codebase size, but there is one, and exceeding it is expensive.

Context budgets per lane

Rough numbers from our runs:

research lane:  100-150k tokens typical, conclusions in a report file
impl lane:       40-80k tokens typical, state file + tests as artifacts
reviewer pass:  10-20% of the task's tokens, fresh agent, diff only

The reviewer is the underrated one: a fresh agent with no prior
context reads the finished diff against the definition of done. No
shared enthusiasm for its own work, no inherited assumptions. It
catches the "tests pass but the requirement drifted" class of miss
better than any single-agent review.

Compaction and handoffs

Long lanes hit context limits, and what survives compaction is
files, not memory. So everything that matters gets written: state
file at boundaries, decisions with rejected alternatives, and a
brief file for any lane spawn. The pattern is the same as
single-agent discipline, just enforced more often because handoffs
happen more.

What still goes wrong

Two agents editing adjacent files in the same lane split (solved by
task slicing, not tooling). Research conclusions that age while the
impl lane waits (report files get a date, stale reports get
regenerated, not trusted). And the classic: I launch a third lane
"just this once" and pay the merge tax. The system works when the
caps hold.

DEV Community