Most automation backlogs are sorted by annoyance. The task you resent most goes to the top, and six hours later you have a script that saves ninety seconds a week and breaks the first time the input changes shape.
We run a scheduled publishing agent in production. It drafts articles, generates summaries, deploys, and syndicates to three platforms with nobody watching. Sorting that backlog by annoyance would have built the wrong three things first. Here is the sort key we ended up using, what each automation cost to build, and the failure modes that decide whether one survives past its second week.
Sort by decision entropy, not by annoyance
Every repeated task has three good endings: a script, an agent, or a checklist that stays a checklist. Picking wrong is the expensive part, and the wrong pick is usually "agent" for something that was always a script.
Script it when you can write the decision rule down. If the branches are enumerable, enumerate them. A script that runs in 40ms and cannot hallucinate beats a model call on every axis that matters: cost, latency, determinism, and your ability to debug it at 2am.
Hand it to an agent when the input varies in ways you can't enumerate ahead of time, but the output has a fixed shape you can check quickly. That second clause carries the whole argument. An agent whose output you have to read carefully to trust hasn't saved you anything — it moved the work from producing to reviewing, and reviewing is the slower of the two.
Leave it a checklist when verification costs more than the task, or when the action is hard to reverse and nobody is watching when it fires.
Three questions, in order:
- Can you write the rule as an
if? Then write theif. - Can you verify the answer faster than you can produce it? If yes, an agent is viable. If no, it isn't, regardless of how good the model is.
- What breaks if it fails unattended, and how long until you notice? This decides scheduled versus on-demand — not whether to automate at all.
| Task shape | Input entropy | Verify cost | Verdict |
|---|---|---|---|
| Format and post the same payload to N endpoints | Low | Trivial | Script |
| Summarize arbitrary prose into a fixed 4-bullet block | High | Seconds | Agent |
| Rank a feed of candidate topics | High | Seconds, and a human skims the queue anyway | Agent |
| Approve a refund, merge to main, rotate a key | Any | High or irreversible | Checklist |
What we shipped first, and what each one cost
1. Summary generation — agent, roughly three hours to build. Every article here renders a compact takeaways block above the body. It's one model call per article, keyed on a sha256 of the title, description, and stripped body, so unchanged articles skip on re-runs and the whole job is cheap to repeat. Input entropy is high: every article is different. Output shape is fixed: four bullets, each one either supported by the article or not, and a bad one is obvious within about five seconds of reading. That's the profile you want for your first agent.
One design choice did more for reliability than the prompt did: it never runs at build time. It runs as a separate command that writes a committed JSON file. Builds stay deterministic and offline, and every model-written sentence that ships passes through a diff someone can read before it goes out.
2. Syndication fan-out — script, roughly five hours. Three platforms plus a search-index ping, one dispatcher. Take the list of new URLs, format three payloads, respect three different rate limits — three seconds between posts on one network, seventy-five on another with backoff on 429, fifteen on the third. It feels fiddly, and fiddly is what tempts people toward an agent. Fiddly is not the same as ambiguous. Every branch here is enumerable, so it's a script, and it has never needed a model.
3. Topic discovery — agent, roughly four hours, and still the least reliable of the three. Pull candidates from a handful of public feeds, score them, write the survivors to a table. Roughly one in four candidates turns out worth writing. That hit rate would be unacceptable in a deploy step and is fine here, because the output is a queue a human skims rather than an action that fires.
Build the agent with the cheapest verification first, even if it saves less time than the one you actually hate. You are not just automating a task on the first build — you are learning what your prompts, your retries, and your error handling do under real input. Learn that on a job where a bad output costs five seconds of reading, not on the one that touches production.
The three failure modes that decide whether it survives
Building the automation is the short part. These are what kill it afterward.
Non-idempotent runs. A scheduled agent will get killed mid-run — deploy timeout, rate limit, closed laptop lid. If the second run repeats the first run's side effects, you get duplicate posts and duplicate rows, and you learn to stop re-running it, which means you've traded an automation for a manual recovery procedure. Our rule: every step checks whether its side effect already exists before performing it, and the pipeline is safe to re-run from the top at any point. Cheap to write on day one, genuinely painful to retrofit.
Silent success. This one cost us the most.
Our publish command built and deployed the site but did not trigger syndication — the two were separate commands, and only one of them was in the habit. Fifty-seven articles shipped and were never announced anywhere. Nothing errored. Exit code zero every time. We found it by reading a cross-post log for an unrelated reason. If a step can be skipped without failing, it will eventually be skipped every time, so make the composite command the only command and delete the partial one from your muscle memory.
Unverifiable output. If checking the agent's work requires the same context and attention as doing the work, you have built a second job. Either narrow the output until it's checkable — a fixed schema, a bounded list, a diff — or leave the task on the checklist. There is no third option where you trust it because the model is good.
A reasonable first month looks like this: one script for the enumerable fan-out you're currently doing by hand, one agent on a high-entropy task whose output you can check in seconds, and an honest list of the things you decided to leave as a checklist. That last list is the sign you sorted the backlog correctly. Teams that automate everything they can automate end up maintaining more surface than they eliminated.
Originally published at pickuma.com. Subscribe to the RSS or follow @pickuma.bsky.social for new reviews.
Top comments (0)