How do you test code that drives an AI coding agent?

I'm building a desktop app that starts the real agent CLIs (Claude Code, Codex, Gemini CLI) and talks to them over their protocols. That makes tests awkward. A test that starts a real agent spends someone's usage, needs a login and gets a different answer every run.

What I do for end-to-end tests: launch the built Electron app with Playwright, with fake claude, codex and gemini commands first on PATH. The app also gets a temporary user-data folder seeded with demo data, an in-memory keychain and a fixed clock. Each run starts from the same place and ends the same way.

That covers what I need most: the chat, an access request being answered, undoing a turn, the board. A test never starts a real agent or spends anyone's usage. Each fake has a .cmd twin for Windows, because a bare script on PATH doesn't run there.

The gap is that a fake only knows what I taught it. If a real CLI changes the events it sends, the fake keeps passing and the app breaks for users.

If you wrap or drive an agent CLI from your own code, how do you test it? Fakes on PATH, recorded sessions, a cheap real model on a schedule, something else? And how do you find out when the real CLI changes under you?

This came out of building Styx, an open-source app that runs these agents side by side.

Comments

[Auten]

Fakes on PATH for the main suite sounds right to me. For the drift gap, what helps is splitting "content" from "shape":

• Make the fakes replay recorded sessions instead of hand-written ones. Updating a fake then means re-recording, not re-teaching.
• Run a small canary on a schedule: each real CLI, cheapest model, one fixed tiny prompt, raw event stream saved. Diff only the shape (event types, field names, order of lifecycle events) against the fixture your fakes replay. Answers will differ every run; the shape should not.
• Pin the CLI versions your tests trust, and keep a second lane on "latest". A new CLI release then shows up as a red canary on your side before a user reports it.
• In the app, read --version at startup and flag versions you have not recorded yet, so a user report comes with that context.

The access-request and undo paths are where I'd expect shape changes to bite first, so those are worth recording per version.