DEV Community

Cover image for Running AI agents in parallel: 6 failures and their fixes
Shyam Verma
Shyam Verma

Posted on AI-assisted

Running AI agents in parallel: 6 failures and their fixes

For the last month I've been building a Mac app with several AI coding agents working in parallel. Each agent has its own copy of the repo. I coordinate and make the calls.

The agents write working code most days. Here are 6 things that still slowed us down: what went wrong, how we noticed, the fix, and how long it took.

1. Too many agents building on one Mac

What went wrong: each agent ran its full build and tests before every push. On one day, 5 to 7 of these runs went at once on a 12-core Mac. The load average reached about 290.

How we noticed: tests failed that had nothing to do with the change. Every pull request got slower.

Fix: one machine-wide lock. Every heavy job waits its turn.

Time: 2 hours 26 minutes from the issue to the merged fix.

2. Timing tests failed under load

What went wrong:

  1. A timing test failed at a load average of 54 to 66. The branch never touched its code.
  2. A benchmark ran far slower at a load of 92 than at a load of 16.

How we noticed: red runs on unrelated branches.

Fix: tests that measure time moved to their own suite. Pull requests skip it. It runs once per release, on a quiet machine.

Time: about 4 days from the first logged failure to the merged fix. The fix itself was one pull request, merged in about 4 hours.

3. A hung test held the lock all night

What went wrong: the lock from the first fix created a new problem. One test run hung from 00:32 to 05:58 and held the lock the whole time. Every other agent waited.

How we noticed: late. Nothing alerted on a lock held for hours.

Fix:

  1. A watchdog stops a hung test within minutes and names it.
  2. An alert fires when the lock is held for over 60 minutes. It would have caught this hang about 4 hours earlier.

Time: merged the same day, about 13 and a half hours after the hang ended.

4. Local tests passed, production failed

What went wrong: our website has short share links like /r/reddit that redirect to /?s=reddit. I asked for one generic rule instead of one per channel. An agent wrote placeholder rules on Cloudflare Pages, like this one:

/r/:source  /?s=:source  302
Enter fullscreen mode Exit fullscreen mode

The local dev server (wrangler pages dev) filled :source in the query string. Production passed it through as plain text. Every link went to /?s=%3Asource.

How we noticed: on the live site. The change had passed the unit tests, a local end-to-end run and a review.

Fix: we rolled the site back and went back to one written-out rule per link. Every deploy now opens each share link live.

Time: 41 minutes from the bad merge to the merged fix.

5. Reading a status moved a real setting

What went wrong:

  1. A scratch build of the app used the real app id.
  2. It only read its login-item status, through SMAppService.mainApp.status.
  3. macOS then pointed my real login item at that scratch copy.
  4. The copy was deleted minutes later. My login item pointed at nothing.

How we noticed: an agent found the line in the macOS background-items log during a review.

Fix: every test copy gets its own app id. A build that would never register a login item never asks for its status.

Time: 1 hour 37 minutes from the issue to the merged fix.

6. The test VM borrowed a binary from the host

What went wrong: our end-to-end tests run in a throwaway macOS VM. The test script copied the host's node into it. That node came from Homebrew and needed a library that exists only on the host. In the VM it died with dyld: Library not loaded.

How we noticed: one check failed during a release run. The app itself was fine.

Fix: copy only a self-contained node into the VM.

Time: to be honest, the fix is still in review as I write this.

What I would recommend

  1. When agents share one machine, queue the heavy work.
  2. Put a timeout and an alert on every lock.
  3. Keep tests that measure time out of the per-change run.
  4. Check edge and OS behaviour live. A local dev server can differ.
  5. Give every test copy its own identity.

The app I'm building is Poxture App. It checks your posture on a Mac.

What has slowed you down when agents work in parallel, and how long did the fix take?

Top comments (0)