DEV Community

João Eduardo F Bertacchi
João Eduardo F Bertacchi

Posted on Originally published at Medium AI-assisted

Feeling vs Data: My Multi-Agent Workflow measured

Two months of agent logs, about 1,100 agents and 45 features. What the data confirmed, what it corrected, and where the saved capacity went.


Where this started

A few weeks ago I wrote on LinkedIn (in Portuguese) that I used to think "10x productivity with AI" was a myth. Here is the short version for readers who missed it.

I have used AI for coding for years. I went from autocomplete to chat, then IDE plugins (Continue, Cline, Roo Code, Kilo Code), then TUIs (Claude Code, Codex, OpenCode). For a long time, I felt 2x was the ceiling. Agents ignored my conventions, duplicated code and drifted from project patterns, even with commit hooks, an AGENTS.md and skills.

Spec-driven development (SDD) changed that. Specs are bureaucratic, they cost time, and they drift from the code. But clear, verifiable goals let me give agents autonomy. I stopped babysitting them.

Autonomy brought new problems: bigger and messier PRs, plan limits, and the wrong model or reasoning level for a task. So I built a process:

  1. Planning
  2. Plan review
  3. Task breakdown and execution planning (dependencies, who does what)
  4. Task development
  5. A review-fix loop per task
  6. Open the PR
  7. Full review and fixes
  8. My review and fixes
  9. Merge to main
  10. Deploy to staging, then production

One orchestrator agent runs this process and dispatches other agents by task difficulty. My PRs got better. My reviews found fewer problems. But the orchestrator got slow as its context grew. A cache miss after lunch could burn my 5-hour plan limit in one message.

So I moved orchestration to Paseo. Paseo lets me run each lab's native CLI and define one agent profile per kind of work. I went back to Claude Opus as the orchestrator, and I set Claude Code's autocompaction at 400k tokens. I work on three plans: $20 Claude, $20 ChatGPT and $10 OpenCode Go.

I built JOBE Vox, a transcription service, with this workflow in one month. It has Stripe billing, three environments, CI/CD, observability, backups and models from three AI labs.

In the LinkedIn post I described what I felt. This post checks those feelings against the data.


The process in detail

These are the agent profiles I use today. The orchestrator picks one for each task.

Table: nine agent profiles. Orchestrator, Architect and Senior Frontend Developer run Claude Code with Opus; Senior Developer, Senior Code Reviewer, Plan Reviewer and Validator run Codex with GPT-Sol; Júnior Developer runs OpenCode with DeepSeek flash; Junior Code Reviewer runs OpenCode with GLM flash.

The workflow is driven by OpenSpec changes, one per feature. A change goes through these steps:

  1. Goals first. I agree goals and non-goals with the orchestrator before any planning.
  2. Plan, then plan review. A Plan Reviewer checks the proposal, design and tasks before I see them.
  3. Implementation in isolated worktrees. Each developer agent works on its own branch, and a reviewer agent checks every developer's work before it is integrated. The orchestrator does not fix code by hand. It sends findings back to a developer.
  4. One PR per feature. The PR gets CI, an automated Codex review and my review. Review fixes stay as separate fixup! commits until I approve, then the branch is squashed to a clean history.
  5. Validation, then release. Validation runs on dev, staging is a gate before production, and production deploys only from main.

The orchestrator also keeps a memory of rules. Most rules started as a correction from me: "never merge without a reviewer", "dispatch a fresh agent after one fix round", "reject findings that cannot happen in this system". The process is what is left after each mistake.

Steps 4 and 5 did not exist before Paseo. The orchestrator never touched GitHub then: no PRs, no CI runs, no comment reviews. It felt impossible with the context I had. Keep that in mind, because it explains most of what follows.


The data

I used five sources:

  • The OpenCode database: every session before Paseo, with tokens, models, compactions and subagents.
  • Claude Code transcripts: the orchestrator after Paseo, with per-request usage and compaction markers.
  • Paseo agent records: about 1,100 child agents, with their parent, provider, model and title.
  • Codex session logs: tokens for the Codex agents.
  • GitHub: PRs, reviews and bot comments.

I used OpenSpec changes as the proxy for features. There were 16 before Paseo and 29 after.

Method notes

  • The cutover is the first Paseo agent, on 2026-10-09.
  • Context tokens = input + cache read + cache write, summed over requests. They are not unique tokens and not billed cost.
  • Each orchestrator session is mapped to the feature it mentions most.
  • The model, the TUI and the orchestration tool all changed at the same time. The data cannot say which one caused an effect. I credit the switch as a whole.

Finding 1: latency stopped growing with context

This was my strongest feeling: orchestrator sessions got slow once they crossed a context threshold. The recorded turn latency shows the same pattern.

Line chart: median seconds per orchestrator turn by context size. Before Paseo it rises from 8.0 s below 100k to 62.4 s at 600k and above; after Paseo it stays between 4.5 and 6.1 s.

Before Paseo, the median turn got about 8x slower between a small context and 600k. At 400–500k, the slowest 10% of turns took at least about 3 minutes (p90 188 s). The rise remains when I control for output length: seconds per output token went from 0.047 to 0.35. After Paseo, the median stays at about 5 seconds (4.5–6.1 s) in every bucket.

One caveat: the two tools log time differently. For OpenCode I measure from message creation to completion, minus tool execution. For Claude Code I measure from the previous tool result to the last block of the response. Both approximate the time I waited for the model, but they are not identical measurements.

The data also sharpened one claim. I wrote that the orchestrator "frequently" went above 600k. Only one feature before Paseo did that. The real threshold was 400k:

  • 5 of 16 features (about 1 in 3) pushed the orchestrator above 400k. That is the limit I later chose for autocompaction.
  • Turns above 400k were only 5% of all turns, but they took 16% of the orchestrator's model time.
  • In the largest feature, turns above 400k took about half of the model time.

So my 600k figure was too high. The 400k threshold matches both the slowdown curve above and my autocompaction setting.


Finding 2: the orchestrator does half the work per feature

For the closest comparison, I split each post-Paseo feature at the moment its PR opened. I treat everything before that point as planning and implementation, the only work the old orchestrator ever did. PR creation is a proxy for the phase boundary, and features still differ in size.

Table: planning + implementation per feature, medians. Orchestrator context tokens 26.4M before, 12.7M after. Features with a compaction 25% (4 of 16) before, 18% (5 of 28 with a PR) after. My typed messages 30.5 before, 9 after.

The orchestrator now processes about half the tokens for the same phase. It also needs about a third of my messages. Five features have no recorded orchestrator work before their PR opened. Either the PR opened early, or the session mapping missed that work. Excluding them, the medians are 14.9M tokens and 12 messages, so the result holds.

This is the claim I made on LinkedIn, and it holds: the orchestrator's load dropped a lot.


Finding 3: I spent the savings

Total orchestrator tokens per feature did not go down. They went from 26.4M to 47.3M. The difference is the new phase:

  • 66% of the orchestrator's tokens now go to work after the PR opens: CI, the bot review, my review, fixups, the clean history, validation and the release.
  • 55% of my typed messages also land in that phase.
  • The orchestrator's share of all tokens, agents included, rose from 37% to 47%.
  • Compaction moved there too. In the post-Paseo features with a PR, 25 of 31 compactions happened after the PR opened.

This changes how I read my "1 in 8 features compacts" estimate. In planning and development, the phase where it matters most, 5 of 28 features (18%, about 1 in 6) compacted before the PR opened. That is close to my estimate, and down from 4 of 16 (25%, 1 in 4) before Paseo. This phase is the most sensitive to compaction, because losing context there can change design and code decisions. Most compactions now happen later, in the PR and release phase. Over the whole lifecycle, 13 of 29 features (45%) had at least one compaction.

The data shows a lower load in the old phases and a large new phase after the PR. It cannot show why. My own account: once the orchestrator had room in its context, I used that room to add the review and release steps I could not afford before.


Finding 4: almost half the agents are reviewers

Where does the extra capacity go? Mostly to review.

Table: agents by role. Before Paseo: 405 subagents, 266 development, 99 code review, 25 plan review, review share 31%. After Paseo: about 1,100 agents, about 340–400 development, about 405–415 code review, 75–95 plan review, review share 44–46%.

After Paseo, a typical feature used 7 developer agents, 8 code reviewers, 1 plan reviewer and 2 validators (medians). The roles come from agent titles, so the post-Paseo split is a range, not an exact count.

Reviews run on a model mix. DeepSeek flash and GLM flash, both on my $10 OpenCode Go plan, ran about 207 of the code reviews. GPT-Sol ran about 169, mostly for complex work.

Is the quality visible? One independent signal is the Codex bot that reviews every PR push. Before October 25 (UTC), it raised 0.64 findings per review pass (63 in 99). From October 25 onward, it raised 0.18 (12 in 67). Fewer bot findings does not prove fewer defects, and these data do not show why the rate fell. The samples are small, and the bot had outages. It does match my impression that problems now get caught before the bot looks.


Limits

  • One developer, one project, about six weeks after the switch. Claude Code transcripts start on October 14, so the first days after the switch are missing.
  • The model, the TUI and the orchestration tool changed together.
  • Features are not equal in size. Medians help, but they don't remove that.
  • Session-to-feature mapping is a heuristic. 28 post-Paseo sessions mention more than one feature.
  • Tokens are not cost. Most are cache reads on flat-rate plans.
  • I make no productivity-multiplier claim here, because the data cannot support one.

Takeaways

  1. If your orchestrator slows down with context, measure latency by context size. The curve tells you where the threshold is.
  2. Moving coordination out of the orchestrator's context can pay off. In my workflow, the switch coincided with about half the orchestrator load per phase.
  3. Expect to spend the savings. I added steps I had skipped before. Decide on purpose where the saved capacity goes. I chose quality.
  4. Make review a default step. Run it before every integration, and match the reviewer model to the task.

The product behind the data

Every feature in this post shipped in JOBE Vox. It is a transcription service I built with this workflow. You upload audio and get a clean, readable transcript, in the browser or through a public API. Every change in it went through the plan reviews, code reviews and release gates measured above.

If you need transcription, give it a try and tell me what you think. Your feedback goes into the same workflow.

Thanks for reading. If you run a multi-agent workflow and measure it differently, I'd like to hear how.

Top comments (0)