My Cron Job Double-Fired and Placed the Same Order Twice — Here's My Postmortem

I keep a Mac Mini in a closet running a handful of small scheduled Python jobs. One of them scans market data on a schedule and, when its rules fire, places tiny orders through an exchange API. Positions are around $12 each. It ran for months without drama.

Then one morning the exchange said I held 16 contracts, and my local state file said 8.

Not a rounding error. Not a stale cache. The same order, placed twice, ten seconds apart — by two instances of the same script that were both alive at the same time, each keeping its own private idea of reality.

What follows is the postmortem I wrote for myself, cleaned up. The punchline: the scheduler was fine. The API was fine. Both root causes were me.

The morning I noticed two of everything

Context first, because the boring design matters later:

• A Python script runs on a schedule (and occasionally on demand)
• It fetches about a thousand tickers, applies some rules, sometimes opens or closes a small position
• Every decision lands in a JSON state file; all my reports read from that file

That morning the fee log showed two order fees for the same position, timestamps ten seconds apart. The exchange held 16 contracts. My state file tracked 8. The state file wasn't lying about what it knew — it had only ever been told about one of the two orders.

I eventually found two distinct ways this had happened. Both were my fault, in different ways.

Incident 1: I restarted a job that was never dead

The first one is embarrassing in the most ordinary way possible.

The scan isn't instant — a thousand tickers plus math takes a bit. One evening I triggered a manual run, and the terminal just sat there. No output. No prompt back. "Hung," I thought — and ran it again.

Here's what actually happens when a chatty Python script runs with stdout connected to a pipe or a background handle instead of a real terminal: output is block-buffered. The job was alive and working the whole time. Its output was sitting in a buffer, waiting to be flushed. I read the silence as "dead," and my restart created a second live instance.

Those two order fees, ten seconds apart? One order from the run I thought had died, one from the resurrection.

The lesson that stings: silence is not a signal. "It's not printing" and "it's not running" are unrelated statements. Before you rerun anything: check the process table, check the log file's modification time, check whether it's burning CPU. Poll the thing. Re-running a silent-but-alive job is how you manufacture concurrency you never designed for.

Incident 2: The scheduler and I raced each other

The second one took an evening of forensics.

A few days later: same symptom, but one position was triplicated. The order timestamps lined up with the top of the hour — and my scheduler fires at :02. In my shell history, right there: a manual run of the same script at :02. I'd been poking around and wanted a fresh scan "right now," and it simply never occurred to me that the clock was about to do the same thing.

Two entry points, same script, same minute, zero coordination. A scheduler is just a clock — it doesn't know what you're doing in a terminal, and I wasn't thinking about the clock.

The rule I added: manual runs never land on a scheduled minute. If the job runs at :02, I run manual checks at :15. It feels comically simple. It has prevented at least two repeats that I know of.

Why this wasn't a disaster (this time)

I want to be honest here, because the honest version is more useful than the heroic one. Three things saved me, and none of them was foresight:

1. My exits are close-all. The exit logic closes the entire position on the exchange — not "the size I think I have." A duplicated position resolves itself at the next exit. The bug was self-healing by accident.
2. The money was small. $12 doubling to $24 is a shrug. The same code with position sizes that scale with an account would not be a shrug.
3. The exchange is the source of truth. My reports were wrong and my history file had gaps, but the actual money was always consistent with reality. The drift lived in my shadow copy — the state file — not in the world.

That third point rewired how I think about every state file I keep. They're caches. The API on the other side is the truth. Once that lands, "reconcile before acting" stops being an enterprise buzzphrase and becomes just reading the real number before making a decision.

The fixes

A lockfile — the boring kind

The core fix is about fifteen lines at the top of main():

import fcntl, os, sys

LOCK = os.path.join(os.path.dirname(os.path.abspath(__file__)), ".job.lock")
lock_fd = open(LOCK, "w")
try:
    fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
    print(f"[lock] another instance holds {LOCK}; exiting", flush=True)
    sys.exit(0)
lock_fd.write(str(os.getpid()))
lock_fd.flush()

A second instance starts, tries to grab the lock, fails, exits with a message. On Linux and macOS that's fcntl; on Windows it's msvcrt.locking — same idea, different syscall. And if you'd rather not touch Python for it, the shell version is one line:

flock -n /tmp/job.lock python3 job.py || echo "already running"

Announce yourself in the logs

Every run now logs start and end with its PID:

import logging

logging.basicConfig(
    filename="job.log",
    format="%(asctime)s pid=%(process)d %(levelname)s %(message)s",
)
logging.info("scan start")

When the lockfile blocks a second instance, the log says so in plain words. And looking back at the incident logs now, the duplicate jumps out in ten seconds — two different pid= numbers in the same minute — instead of needing an evening of timestamp archaeology.

Flush like you mean it

flush=True on prints — or just run with python3 -u. This one directly kills Incident 1: the terminal would have shown live output, and "no output" would have actually meant something.

Reconcile before acting

Before the job opens anything, it now pulls actual positions from the exchange API and treats the local state file as a hint, not as truth. Anything the file claims that the API doesn't confirm gets flagged for review instead of acted on.

The part I'd been getting wrong for years

The reframe that made all of this click:

I used to think of idempotency as an API-design topic — something server people worry about. But the moment your cron job calls an API that mutates state, you're running a distributed system. Multiple processes, lying transports (buffered stdout counts as a lying transport — it lied straight to my eyes), shared mutable state on the other side of a network. All the classic distributed-systems failure modes are now your failure modes. Just at hobby scale.

A lockfile doesn't make a distributed system safe. What it does is collapse the most common failure — scheduler plus human, or human plus impatient human — into a no-op. That's most of the practical risk, for fifteen lines of code.

Wrapping up

If you take away two things:

1. A silent job is not a dead job. Poll, don't rerun.
2. If a scheduled job of yours talks to anything that mutates state, add the lockfile today — not after your own double-fire. Mine cost an evening of forensics and $12 of unplanned exposure. The same bug with real sizing would have cost a weekend and my confidence in the whole setup.

I'm curious how others handle this, especially anyone running scheduled jobs against mutating APIs on a homelab box. Lockfiles, per-action idempotency keys, a real job queue instead of cron? Drop what's actually held up for you in the comments — I'm always stealing ideas for my checklist.