DEV Community

Cover image for Your setup guide is an untested surface. Make the tool drive the agent instead
Gary Stupak
Gary Stupak

Posted on

Your setup guide is an untested surface. Make the tool drive the agent instead

My deploy wizard runs on the cheapest models on the market - Haiku in Claude Code, a free bundled model in OpenCode - and walks a stranger to a working, live-tested deployment in about an hour, two dashboard tours included: GitHub's and Stripe's. Not because those agents are clever. Because they don't have to be: the tool owns the sequence, the wording and the checks, and the agent only renders and relays.

Getting there meant deleting my prose setup guide. Here is why, and what replaced it.

Prose has no test suite

The setup path for my Cloudflare Worker used to be an 850-line markdown orchestrator. The agent read it, picked npm subcommands out of it, and hand-edited config files along the way. It worked, mostly - and "mostly" is the whole problem. Every edit to that file could break a live run, and nothing could catch it before a human walked the entire path again.

Dashboards make it worse. Payment providers rename tabs and buttons over time, and a guide that names the old one still reads as perfectly plausible - so the agent improvises something that sounds right. Improvised steps against a live payment dashboard are how an afternoon disappears.

A prose guide is a load-bearing component with no test suite. We would never accept that from code. We accept it from documentation because nobody is watching the moment it fails.

Every reliable installer already knows the answer

wrangler login does not hand you a markdown file and hope. Neither does gh auth login, or create-next-app. Every setup tool that people trust works one way: the tool drives and holds the state, and the human responds to the tool.

The moment AI agents arrived, we quietly lowered that bar. Hand the model an install README, call the result agentic, blame the model when it wanders. The fix is not a smarter agent. It is the old, boring inversion, applied to the new participant.

The contract: the driver decides, the agent renders

So the wizard became a program. The agent runs one command, npm run wizard:drive, which prints exactly one JSON record. The agent renders that record verbatim, collects the answer, and feeds it back. It never chooses the next step, never composes a shell command, and never diagnoses off-path.

A record is one of three things: a question - either a closed choice rendered as selectable options, or a named free-text field the human reads off a dashboard; a manual step the human confirms with the single word done; or a message to show. That vocabulary is the entire interface between the driver and whatever agent happens to be running it.

The current screen is a pure function of the state - cursor, answers, environment, flags. Which means every screen, every branch and every word a user reads is testable with no live agent and no side effects. The wording moved from prose into code, and code has tests.

"done" is verified, not trusted

A human says done; the driver checks. Wherever real state can be read - the GitHub org, the team, the token's capabilities, the deployed worker's URL - the driver reads it instead of believing the confirmation.

A failed check never advances the run. It emits that step's recovery block: the known failure modes, as data, routed back to the screen that owns the wrong input - a bad org goes to the org screen, a bad token to the token screen. The agent answers from the recovery instead of improvising a diagnosis. And the run ends with a live end-to-end test purchase, so "it works" is an observation, not a hope.

The agent never sees the secrets

The part I refuse to compromise on: you paste secret values into the env file yourself, and the deploy hands that file to wrangler, which reads it in its own process. The values never pass through the agent's context.

The permission rules that deny the agent's reads of the secrets files - by name - are committed in the repo, for every agent the wizard supports. And a test plants fake secret values, runs the wizard's file operations, and proves the refusal. You can read the enforcement before you run anything.

Why the cheapest model is enough

Most agentic-install demos run on a frontier model, and I think that has the causality backwards. If the setup needs a brilliant agent, the setup is underspecified - the brilliance is spent guessing what the author meant.

When the driver owns the sequence and the wording, there is nothing to be brilliant about. Rendering a record and relaying an answer is a job for the smallest model you can rent, which is exactly what my live runs use. Cheap models are also the discipline: they fail loudly on ambiguity instead of papering over it, so the driver gets better.

How to build one for your own repo

The pattern is small and does not depend on my product. What it takes:

  • A driver script with a pure currentRecord(state) and an advance(state, answer) that does the I/O through injectable dependencies, so tests pass fakes.
  • A record vocabulary - ask / do / say - with closed choices everywhere the answer set is known in advance.
  • Verify functions that run at the first point a credential for them exists, and recovery data attached to every step that can fail.
  • One npm script as the only entry point, so the agent never composes shell commands.
  • Committed permission configs for the agents you support, with the secrets files denied by name.
  • An AGENTS.md at the root, so any agent finds the door.

The working example is RepoAccess Core - an AGPL Cloudflare Worker that sells access to private GitHub repos, whose entire onboarding is this driver. The architecture behind the product itself is a separate write-up: How to sell access to a private GitHub repo without a SaaS in the middle.

If you maintain a repo that agents keep setting up wrong, steal the pattern. The guide you delete will not be missed.

Top comments (0)