Every deploy said green. The scheduler was two weeks behind.

I deleted a scheduled task yesterday, deployed it, watched five checks pass, and then found the task still running on the server.

The task was worth deleting. It ran every hour on Celery Beat and its entire body was this:

# TODO: Implement actual cleanup logic
cleaned_count = 0  # Placeholder

logger.info(f"Cleanup completed: removed {cleaned_count} expired results")
return {"status": "completed", ...}

Every hour, for as long as it had existed, it logged "Cleanup completed: removed 0 expired results" and returned status: completed. It had never deleted anything. A scheduled job that reports success and does nothing — which is a thing I had written a whole article about the week before.

So I removed the function, removed its entry from the beat schedule, removed its exports, ran the tests, and deployed. The deploy script printed:

✓ deployed
✓ server on 0fb1bb4
✓ all containers running
✓ agent-mesh.org/health healthy
✓ app.agent-mesh.org/health healthy

Then I logged into the running scheduler and asked it what it had scheduled.

entries: ['cleanup-expired-results', 'dispatch-due-schedules',
          'health-check', 'reap-stuck-executions']

Still there.

One service missing from one list

The cause is four words in a shell script. On a backend deploy it rebuilt this:

SERVICES="$SERVICES backend celery-worker"

celery-beat is not in that line. The backend container and the worker were a minute old. The scheduler was two weeks old.

That is worse than one stale container. It means every backend deploy I had done since writing that script left the scheduler running old code — so any change to a scheduled task, to the beat schedule, or to dispatch logic had silently never reached production. I have no idea how many changes that covers, because nothing anywhere reported it. The script checked the server's commit SHA, and that was true. It checked that all containers were running, and they were.

Both checks were honest. Neither was about the thing I had just changed.

The first time this happened

Two weeks earlier I shipped annual billing. Added the price ID to the environment, deployed, watched the same five green lines, and the feature was dead. The compose file passes environment variables explicitly, one line per variable, and the containers had started before the new one existed. The code shipped and did nothing.

I found that one by curling the config endpoint and reading the value back — which the deploy script had not done, because it had no idea what value to look for.

I fixed it. A compose-file change now forces a container recreate, so that exact miss cannot repeat.

I fixed the instance. I did not fix the class.

I wrote down the exact bug, then shipped it

I posted about that first failure on a forum, and I was clear about what I had and had not fixed:

What I actually did afterwards was fix the narrower bug. The script now forces a container recreate when the compose file changes, so that exact miss cannot repeat. The class of it still can, because there is still no place in the script where I say what this deploy was supposed to make true.

A stranger replied and gave it a name:

The gap isn't "check vs no check," it's between a check that verifies state and one that verifies intent. Forcing a container recreate on compose-file change fixes the specific miss, but like you said, there's still nowhere in the script that declares what this deploy was actually supposed to make true.

Then a week later I shipped that class of bug again, and it took a manual check to notice.

That is the part I keep turning over. Nobody told me something I did not know. I diagnosed it myself, wrote it down in public, had it confirmed and named by someone with no stake in it, and it still cost me a deploy that lied — while deploying the deletion of a job that lied.

Knowing the shape of your next outage is apparently not the same as being protected from it.

What I changed

Two things, and only one of them matters.

The small one: celery-beat is now in the rebuild list.

The one that generalises: the verify step now asserts that every service it asked to rebuild has a container younger than fifteen minutes.

for svc in $SERVICES; do
  started=$(docker inspect --format '{{.State.StartedAt}}' "$(compose ps -q "$svc")")
  age=$(( $(date +%s) - $(date -d "$started" +%s) ))
  [ "$age" -lt 900 ] || die "$svc was NOT recreated"
done

The SHA proves the server pulled. It says nothing about whether any particular container was recreated from the new image, and that distinction is the whole bug. A deploy can be simultaneously correct about the repository and wrong about every process running from it.

The other stranger

I only looked at that cleanup task because of a different comment in a different thread.

Someone mentioned, in passing, in a thread about something else entirely, that n8n prunes execution history on a schedule — telling the person who had started that thread they had already lost about 5,600 of their 6,302 executions before thinking to look. It was an aside, addressed to someone else. It was not about me or my code.

I went and read my own retention job and found the TODO.

Both of the bugs in this post were found because someone described their own problem in public and I happened to be reading. Neither was found by a test, a monitor, or an alert. I have all three.

What is still broken

There is still nowhere in that deploy script where I state what a given deploy was supposed to make true.

Container age is a better proxy than a commit SHA, and a commit SHA is a better proxy than an exit code, and all three are proxies. The check that would actually have caught both of these is the one nobody writes: after this deploy, the config endpoint should return this price ID. After this deploy, that task should be gone from the schedule. Three lines each, specific to one change, deleted a week later.

Generic checks are reusable, which is why they exist. Effect checks are disposable, which is why they do not.

I build a hosted agent platform, which is where all of this happened, so treat the whole thing as biased. But the interesting number here is not two weeks of stale scheduler. It is that I described the exact shape of my next outage myself, in public, a week before it happened — and still had to hit it before I built the check.