If you are building autonomous AI agents, you eventually hit a scaling wall. While developing Saturn AI, I noticed that pushing past five or six si...
For further actions, you may consider blocking this person and/or reporting abuse
"The issue was not the LLM API latency, it was OS process thrashing" is the key
insight, and JSON session-file locking has killed more agent systems than any
model behaviour. p-queue first is the right order. The other axis, for when
the work genuinely is parallel: one machine per agent, billed by the minute so
idle costs nothing. At five or six agents a queue is clearly better though.
At what concurrency did isolation start paying for itself?
Thanks Dhruv! Glad that insight resonated. You're completely right about JSON session-file locking—it becomes an absolute bottleneck the second multiple agents try to read/write state simultaneously.
To answer your question: since I’m managing the execution sandboxes using Docker containers on fly.io, the isolation really started paying for itself around 3-4 concurrent agents.
Below that threshold, a standard queue handled the load perfectly fine without the extra overhead of spinning up separate isolated environments. But once I pushed past that, the local I/O thrashing was too heavy, and true isolation became mandatory to prevent race conditions and keep the environment stable.
Have you experimented much with different state management databases when scaling up to one machine per agent?
Fly Machines are already Firecracker microVMs, each with its own kernel. So if you're running several Docker containers inside one machine, they're still sharing that machine's kernel , and the fix might just be one machine per agent rather than one container per agent. The boundary you want is already there.
On state: one-machine-per-agent often removes the need for a state database entirely. The JSON locking was a shared-filesystem problem, not a data-modelling one. What replaces it is persistence across disposable lifetimes , snapshot the disk, or take an external DB and its coordination point back on.
Yes, I am building Saturn with one machine per agent.
That's the right shape. The next thing that usually bites is boot cost , one machine per agent means one cold start per agent, and re-running installs on the way up is where the latency goes. Booting from a prepared snapshot instead of provisioning fresh tends to be the fix.
Did you go snapshot-restore, or does each agent build its own environment?