DEV Community: Andréas Bodin

Measuring the real concurrency ceiling of an LLM agent runner

Andréas Bodin — Sat, 15 Aug 2026 14:24:21 +0000

I wanted to raise the concurrency limits on my local AI agent runner. The UI now supports multiple terminal panes running in flight, and my gut told me the runner process itself was becoming the bottleneck.

Before touching a single config setting, I wrote a benchmark to test that assumption: When N sessions run at once, what actually breaks — the model server, the hardware, or the scheduling policy?

It turns out my gut was entirely wrong.

The Benchmark Setup

I wrote a bench script that fires N concurrent chat sessions against the runner. Each turn hits a local model (hermes3 via Ollama) with a fixed prompt, running through the full pipeline: pre-turn intent classification, chat response, and post-reply memory extraction.

I tracked four specific metrics:

ttfa (Time to First Activity): POST request -> first internal event emitted. Measures the runner's event loop before touching Ollama.
first-Ollama: POST request -> first request hitting the model queue.
done: POST request -> user receives the answer.
wall (drained): Total time until the post-reply memory extraction tail completely finishes background work in Ollama.

The Results: A Hard Wall at N=1

N	Mean ttfa	Mean first-Ollama	Done (Min / Mean / Max)	Wall (Done)	Wall (Drained)	CPU Max	Min Free RAM
1	0.0s	0.9s	2.1s / 2.1s / 2.1s	2.1s	9.5s	23%	6,004 MB
2	0.0s	1.1s	3.5s / 4.2s / 5.0s	5.0s	12.9s	37%	5,957 MB
4	0.0s	1.5s	5.9s / 7.7s / 10.3s	10.3s	20.7s	63%	5,889 MB

At N=4, the wall-clock time scaled to 4.80× the single-session baseline. Because 4.8× exceeds 4×, concurrent requests actually performed worse than a pure, perfectly ordered serial queue.

The bottleneck breakdown was immediately clear:

It's not the runner: ttfa stayed at 0.0s across all runs. The runner's event loop processed and routed incoming POST requests in milliseconds without queuing.
It's not hardware: CPU peaked at 63% and free RAM never dropped below ~5.9 GB.
It's Ollama: Latency stacked inside Ollama's internal request queue for the single loaded model (hermes3 8.0B Q4_0).

Furthermore, wall (drained) proved that background memory extraction keeps Ollama saturated long after the user gets their answer — a hidden tax naive RPS benchmarks completely miss.

The Twist: Job Queues Fail by Policy, Not Load

Chat is only half the system. The real heavy lifting happens in coding jobs, which talk to the Claude API instead of Ollama.

When I checked why coding jobs weren't running concurrently, I didn't need a load test. I just needed to look at the scheduler code:

const MAX_CONCURRENT = Number(process.env.MAX_CONCURRENT_JOBS ?? 2)

// In the scheduler pump:
if (repoHasRunningJob(job.repo)) continue;   // One job per repo, unconditionally

Almost every ticket in my queue targets the same primary repository. Regardless of what MAX_CONCURRENT was set to, the system was hardcoded to run one job per repo at a time.

The rule exists for a valid reason: two AI agents branching off the same moving HEAD create messy merge conflicts. But serializing the entire repository by default was a blunt policy constraint, not a system capability limit.

Smart Parallelism: A Disjoint Path Rule

To fix job concurrency without causing merge chaos, I replaced the blind "one job per repo" check with a strict path-overlap predicate.

Two queued jobs (A and B) in the same repo can now run concurrently only if all three conditions are met:

Disjoint Named Paths: Both tickets explicitly declare target files/directories, and their path sets share no common files, ancestors, or subdirectories. (If a ticket names no path, it is treated as touching everything and stays serial.)
No Dependency Edges: Neither ticket references the other as a hard or soft dependency.
Historical Touched Files: Reopened tickets use their previous run's actual git diff file list rather than their written prose description.

If there's any ambiguity, the scheduler defaults to serial execution. A false positive (running serially when safe) costs a few minutes of queue time; a false negative (running concurrently and corrupting state) costs an hour of untangling merge conflicts by hand.

What I Actually Learned

Chat concurrency binds on the local model server — raising limits requires better GPU hardware or multi-model inference instances, not TypeScript optimizations.
Coding job concurrency binds on scheduling policies — fixable with smarter dependency tracking and file-path isolation.
The runner infrastructure was completely fine all along.

Measure first. The limit you think you're hitting is rarely the one actually holding you back.

I'm Andréas — full-stack dev, CTO at a B2B SaaS, building my own agent tooling. Portfolio: https://andreas-bodin.vercel.app

Can your verify gate actually fail?

Andréas Bodin — Sat, 15 Aug 2026 14:20:43 +0000

I let Claude Code commit directly to my repositories. I don't review the diffs. I didn't think I needed to, because I built a deterministic verify gate — a script that lints, typechecks, builds, and runs tests. If the gate goes green, the PR merges automatically.

I trusted that gate implicitly. Until I actually sat down and asked the one question that matters: If an agent pushes completely broken code right now, will this gate actually go red?

Not "is the script configured?" Not "does the file exist?"

Will it actually fail?

Turns out, for three out of my four repos, the answer was an emphatic no. The gate was returning GREEN without running a single line of code.

The line of code that lied to me

My gate script is repo-agnostic. It looks up the repository name in a policy JSON file (gates.<repoName>) and falls back to a default if it doesn't find one.

Here is the PowerShell logic driving the whole operation:

$steps = $policy.gates.$RepoName
if ($null -eq $steps) { $steps = $policy.gates.default }
if (-not $steps -or $steps.Count -eq 0) {
    Write-Log "GREEN (no gate steps configured for '$RepoName')"
    exit 0                       # <-- Unconditional green
}

And here is my config file:

"gates": {
  "app": [
    "npm run lint",
    "npm run test",
    "<design-gate script>"
  ],
  "default": []
}

Look at that exit code. If a repo doesn't have an explicit entry in the JSON file, it falls through to default: []. The script sees zero steps, prints GREEN, and exits with 0.

The runner sees exit code 0 and instantly merges the code. I built a system where having no tests configured is structurally identical to passing all tests.

The Audit: 4 Repos, 1 Real Gate, 0 Peace of Mind

When I audited all four repos the runner is allowed to touch, the reality was pretty grim:

app (The Runner itself): The only gated repo, but it's full of holes. npm run lint runs, but tsconfig explicitly excludes the runner/ directory — meaning 51 files of core executor code are never typechecked. Next.js build isn't in the gate at all, so client/server boundary breaks ship freely. Vitest runs 28 test files, but the glob pattern completely misses *.test.tsx files.
product (Next.js app): The most embarrassing row. It has full ESLint, tsc --noEmit, vitest (10 suites), and next build ready to go in package.json. None of them were wired into the gate config. It had 0% coverage purely because of lazy config debt.
notes (Markdown vault): Unconfigured. It doesn't need npm, but it does need a basic script to check for broken wikilinks or bad frontmatter. Instead, it was just auto-approving everything.
portfolio (Vite site): Unconfigured and the directory wasn't even a valid Git worktree. Jobs were dying before the gate even executed, but if they had reached it, they would have passed instantly.

How to fix a fake gate

The real bug here isn't missing npm scripts. The structural flaw is that an empty gate defaults to success.

To fix this properly:

Make empty gates loud. If $steps.Count -eq 0, the script must exit non-zero or return an explicit UNGATED status. A missing key should break the build, not bypass it.
Wire up the low-hanging fruit. The product repo was fixed with a single line in policy.json.
Add actual build checks. Typechecking doesn't catch bundler or framework errors. Adding npm run build to the gate costs execution time, but it's the only way to catch real deployment breakers.
Fix silent exclusion traps. Un-exclude critical core directories from tsconfig and fix the Vitest globs before someone writes a .tsx test that never actually gets executed.

If you run AI agents against your codebase without reading the diffs, go break a file on purpose and run your gate. If it doesn't yell at you, you don't have a safety gate — you just have a script that automatically approves bad code.

I'm Andréas — full-stack dev, CTO at a B2B SaaS, building my own agent tooling. Portfolio: https://andreas-bodin.vercel.app

Where the time actually goes in an AI coding-agent job

Andréas Bodin — Sat, 15 Aug 2026 14:16:18 +0000

I run an agent-orchestration platform: it takes a ticket, spins up a git worktree, lets a Claude Code agent build the feature, and runs a deterministic verify gate before merging. A typical job takes 20–30 minutes.

I used to blame slow model responses for the runtime, but stage-by-stage measurements proved that assumption wrong: infrastructure overhead costs 5–8 minutes of every job before the agent even finishes.

Here is where the time actually goes and how to optimize it.

The Baseline Overhead

On a standard dev machine (Windows 11, Bun, Next.js), a single job pays a heavy infrastructure tax:

Stage	Measured Cost	Impact / Notes
`git worktree add`	Seconds	Negligible overhead.
`bun install`	~91s (median)	Cold worktree setup, even with a warm cache.
Lint & Typecheck	~1–2 minutes	`eslint` + 2× `tsc` across the full repo with no shared cache.
`next build`	Minutes	The longest gate leg due to a cold `.next` directory every run.
Vitest	~26 seconds	Fast and acceptable.
Model Time	~15–22 minutes	Planning, execution, and review calls.

Fixed infrastructure costs account for 20% to 30% of total runtime. Eliminating this overhead reduces small job times from ~30 minutes to ~20 minutes, leaving the remaining time strictly bounded by model inference.

Ranked Infrastructure Optimizations

1. A Pre-Warmed Executor Slot Pool (The Highest-Value Win)

Instead of creating a fresh worktree and running bun install from scratch for every job, maintain a small pool of pre-warmed directories containing:

A long-lived Git worktree (re-pointed per job).
A pre-installed node_modules directory (invalidated only when the lockfile hash changes).
Persistent build caches (.next, tsconfig.tsbuildinfo, .eslintcache).

When a job starts, it claims a slot, checks out the target branch, and immediately executes. After completion, git clean resets the worktree while leaving cache directories intact. This eliminates the ~91-second install step on almost every job.

2. Persistent Next.js Build Caching

Next.js supports persistent filesystem build caching. Combining this with a warm slot pool carries the .next cache between jobs, turning the longest gate step from minutes into tens of seconds for standard diffs.

3. Incremental Linting and Typechecking

Adding --incremental to tsc and --cache to eslint within the persistent slot directory saves roughly 1 minute per job without changing verification outcomes.

4. Windows Filesystem Optimizations

Real-time Defender scanning severely throttles small-file I/O operations (like node_modules and .next writes). Adding directory exclusions or moving worktrees to a ReFS Dev Drive improves install and build I/O speeds by 2× to 5× with zero code changes.

5. Isolated Linkers (Proceed with Caution)

Switching to bun install --linker isolated provides pnpm-style symlinked stores, yielding faster warm installs. However, caution is required: symlinks spanning worktrees can cause catastrophic unintended deletions if cleanup commands traverse outside the worktree root.

Implementation Strategy

Environment Tweaks: Exclude worktrees from real-time antivirus scanning (immediate win).
Slot Pool Architecture: Build the persistent executor slot pool to retain installed dependencies and cache directories.
Incremental Caching: Enable tsc --incremental and eslint --cache.
Framework Upgrades: Enable persistent build caching in Next.js once the slot pool is stable.

By optimizing the infrastructure pipeline, fixed job overhead drops from 5–8 minutes to under 60 seconds. The remaining runtime represents actual model reasoning—the exact phase worth waiting for.

I'm Andréas — full-stack dev, CTO at a B2B SaaS, building my own agent tooling. Portfolio: https://andreas-bodin.vercel.app