Most automation failures I worry about are not crashes. A crash is loud. The ones that cost money are quiet duplicates: the same email sent twice, the same order placed twice, the same post published twice because a request timed out and something helpfully tried again.
My setup is small. A scheduled script posts to X through the official API, up to five times a day. It runs while I sleep. So the question I designed it around was not "how do I make it post?" but "what happens when it is not sure whether it posted?"
The answer is a ledger with four possible states, and one rule: the ledger is written before the network call, not after.
The problem with "it returned 200"
When you call an API to do something (send, create, charge), there are three outcomes, not two:
It worked, and you got a response saying so.
It failed, and you got a response saying so.
You don't know. The connection dropped, the request timed out, the process was killed, the machine went to sleep.
Most code treats case 3 as case 2. It catches the exception and retries. That is exactly how duplicates happen: the first request did go through, the response just never came back.
The reverse mistake is also common. A 200 means the server accepted something. It does not always mean the thing you care about happened the way you think. That is why I say "treat a 200 as a claim, not a confirmation." Where it matters, read the object back.
The four rows
Every time my script decides to post, it appends rows to a plain JSONL file. One line per event:
attempt: written before the API call. It records the queue item ID, the date, the account, and a SHA-256 of the exact text.
sent: written after a successful response, with the returned post ID and URL.
failed: written when the API returns an error response, with the HTTP code and the first part of the body.
unknown: written when anything else goes wrong mid-send, such as a timeout or a dropped connection.
The last row for a given item is its final state. If the process dies between attempt and anything else, the file still says attempt, and that is treated the same as unknown.
The order matters more than the format. If you write the log entry after the call, a crash in between leaves no trace that you ever tried. Writing it first means the worst case is "we tried and don't know the result," which is recoverable by a human. The other worst case, "we tried, it worked, and we have no record," is how you get a second copy at 3 a.m.
The rule that makes it work: unknown blocks the queue
Before every run, the script reads the whole ledger. If any item's final state is attempt or unknown, it prints a STOP message and exits without posting anything.
It does not retry. It does not skip to the next item. It waits for me.
That sounds inconvenient, and it is, a little. In practice, it is the cheapest decision in the whole system. Checking whether one post went out takes me under a minute. Cleaning up a duplicate, on a platform that penalizes duplicate content, takes longer and looks worse.
So far the ledger has recorded every post as attempt followed by sent. It has never double-posted. It also cannot recover from an unknown on its own. Both of those are by design.
The other checks that run before the call
The ledger is the core, but it sits inside a short list of checks. They run in this order, and any one of them can stop the run:
Daily cap. At most five write attempts per day (Japan time). Attempts count, not just successes, so a failing run cannot burn through the cap by retrying.
Minimum gap. At least 50 minutes since the last attempt.
No unresolved attempt. The rule above.
Approved only. The script only posts items marked approved in the queue, whose date is today or earlier.
No surprise links. If the text contains a URL and the item was not explicitly allowed to have one, it stops. Link posts cost more, and a stray URL is also a common sign that the wrong text got into the queue.
Length. It computes X's weighted length itself and stops above 280, instead of letting the API reject it.
No identical text. If a post with the same SHA-256 was already sent, it stops.
Identity. Before the first write of the day, it asks the API which account the token belongs to and compares it to one hard-coded handle. I run more than one account on the same machine. A mismatch stops everything. The result is cached for the day, so this costs one read per day, not one per post.
Only after all of that does it write the attempt row and make the call.
There is also a --dry-run flag that runs every check, prints the next post, and makes no network calls and no file changes. My weekly job uses it to confirm the queue is valid before I walk away.
What this looks like for emails, orders and payments
I use this for social posts because that is what I run. The same shape applies anywhere a retry could create a second real-world effect. If you send emails from a job, place orders through an API, or move money:
Give every action an ID before you act. Generate it when you decide to do the thing, not when you do it. If the provider supports idempotency keys, send that ID as the key, so a retried request is recognized as the same request.
Write "attempt" before the call. Same reason as above.
Separate "failed" from "unknown." An error response is information. A timeout is the absence of information. Only the first is safe to retry automatically, and even then only if the provider tells you nothing happened.
Make unknown stop the line. Not forever. Until a person, or a read-back check, resolves it.
Read the object back when it matters. If a payment API returns success, look up the payment. If an email API returns accepted, check the provider's log. A 200 is a claim.
A checklist you can run against your own job
If you have one scheduled job that writes to the outside world, go through this in ten minutes:
☐ Is there a record of each write attempt that exists before the network call?
☐ Does the record distinguish failed from unknown?
☐ Does an unknown result stop further writes until resolved?
☐ Do retries reuse the same ID, or generate a new one? (New ID = possible duplicate.)
☐ Is there a cap on attempts, not just on successes?
☐ Does the job check which account it is about to act as?
☐ Is there a dry-run mode that makes no changes at all?
☐ When the job stops itself, does a person find out?
If any answer is "no," that is the next thing to fix. None of these require a framework. Mine is a single Python file and a text file that only ever gets appended to.
Why this matters more with agents
Everything above is old advice from payments and distributed systems. It matters more now because AI agents make a lot of API calls, and their default response to an error is to try again, slightly differently. An agent that is told "the request timed out" will often just resend. If the tool it is calling has side effects, the ledger and the unknown rule need to live in the tool, not in the prompt. You cannot instruct your way out of a duplicate charge.
If you want the broader list I run before letting any agent or bot act unattended, it is a free one-page checklist: Agent Ops Pre-Flight Checklist.
I write here weekly about running AI agents and automations in production: what broke, what I check now, and what it cost. Not financial, legal or security advice. Adapt everything to your own systems.
Originally published on NeuralAlpha. I write weekly about running AI agents in production.
Top comments (0)