There are three honest answers to "the document does not fit": truncate, retrieve, or read all of it in pieces. Chain-of-Agents (Zhang et al., 2024) is the third.
Chunk the document. Give chunk 1 to a worker. Hand its communication unit — a short running summary, not the text — to the worker reading chunk 2. Repeat. A manager answers from the final unit alone.
CoA is worth knowing because of the questions retrieval cannot serve. "How many times does the narrator change their mind", "which clause contradicts clause 14" — the answer is distributed, and no top-k contains it.
Live, every number computed in your browser: https://dev48.infy.uk/prompt/day63-chain-of-agents.html
The cost, stated honestly
CoA reads more total tokens than stuffing the whole document in. Every worker re-reads the instructions and the incoming unit on top of its chunk. That is not a footnote, it is the trade.
What collapses is peak context — the largest single request — to roughly 10% of the stuffed one. Peak is the number that decides whether the job runs at all on a given model. Total tokens decide what it costs. They move in opposite directions and you should know both before choosing.
My verification asserts CoA is never cheaper in total. If that ever flips, the accounting has stopped counting the re-reads and the page is overselling.
The demo that could not fail, and why that was a bug
The page claims the communication unit is a lossy channel: a fixed budget means every hop can drop a detail.
I planted a fact and watched it survive 11 out of 11 positions no matter how tight the budget. The "needle DROPPED" state could never fire.
The reason was my haystack. It was random filler that never mentioned the question's keywords, so the planted fact was the only match in the entire document, ranked first at every hop, and always fit. Real documents discuss their own subject constantly — so I added distractor sentences that mention the keywords without answering, and made carried-forward content decay slightly per hop, because re-summarised details genuinely compete worse than text a worker can see in full.
Now: 2/11 survive at a tight budget, 7/11 at a loose one. The loss is real and you can watch which hop eats it.
Two smaller things worth stealing
Measure budgets on the assembled string, not a running sum. My token counter rounds up per call, so summing per-sentence estimates drifted from the real cost of the joined text once separators counted. Both the chunker and the unit crept over budget. Measure the candidate you are about to return.
Keep the backtracks. When serialising, the instinct is to tidy — drop dead ends, present the winning path. That destroys the method.
One caveat the page is strict about: it simulates the pipeline, never the model. Every number is arithmetic over real text. The paper's accuracy results are a separate claim and mixing them in would make both meaningless.
Top comments (0)