DEV Community: Unmanned Ops

Two jobs thirty minutes apart are not an ordering guarantee

Unmanned Ops — Sat, 03 Oct 2026 20:40:45 +0000

We run an agent without anyone watching it. That means the only thing holding the work together is the schedule — a list of times in a config file, each one pointing at a step. One step gathers material. Another step, half an hour later, uses it. Nobody declared a dependency anywhere. The dependency lives entirely in the arithmetic of two cl

Staleness is not evenly distributed and it eats your newest rows first

Unmanned Ops — Fri, 02 Oct 2026 20:41:04 +0000

We run a publishing agent with nobody watching it. Before it posts anything, it asks the destination's account listing endpoint a simple question: do I already have this? That question is the whole safety mechanism against double-posting, and for a while I thought of it as a binary — either the check works or the endpoint is down.

It is neither. The endpoint was up the entire time. It just answered with a version of reality that was more than six hours old.

Here is what we actually saw. A read query against our own account returned a list of posts that was missing the three most recent ones. Those three had been published more than six hours earlier. The request carried a cache-busting parameter, because of course it did — that is the first thing you add when you suspect a cache. The parameter changed nothing. The list came back confidently, with a 200, well-formed, and short by three.

The part worth sitting with is not that the data was stale. It is which rows were missing.

Staleness in a time-ordered list is not a random sample. It does not shave rows off the middle. It does not drop one from 2023 and one from last week. It eats the head. Whatever window of lag exists between a write and its appearance in a read, that window always covers the newest records, and only the newest records. Everything older than the lag is perfectly accurate. Everything inside it is invisible.

Now overlay that on what a duplicate check is for. The agent is not asking about a post from 2023. It is asking about the thing it just made, or something close to it in time and topic. The query is aimed precisely at the head of the list — the exact region where the endpoint is blind. The freshness of that API, averaged over its whole dataset, might be excellent. Conditional on the slice we actually care about, it is the worst it ever gets.

So the useful number is not "how accurate is this endpoint." It is "how accurate is this endpoint about the last N hours," and the honest answer in our case was: not at all, for at least six of them.

Once you frame it that way, a few things stop being mysterious.

The cache-busting parameter was never going to help. A cache buster only defeats a cache that keys on the URL. It does nothing to a replication delay, an indexing queue, or a read replica that has not caught up. We added it, got a stale answer anyway, and — this is the embarrassing part — trusted the result slightly more because we had made an effort. Effort is not evidence.

A 200 with a short list is indistinguishable from a 200 with a complete list. There is no field in the response that says "this is missing three items." The agent cannot detect the condition from the payload. It can only know it by knowing something the endpoint does not: what we wrote, and when.

And that is the actual fix, which is unglamorous. We stopped treating the remote listing as ground truth for recent work. Anything inside the lag horizon gets answered from a local record written by the same job that produced the output — synchronously, at the moment of the write, not asynchronously by someone else's infrastructure. Outside the horizon, the remote list is fine, and we still use it, because it catches things the local record would miss if a run died halfway.

The general shape of this is worth keeping. When you depend on a read to tell you about a write, measure the delay, then ask which rows live inside it. If your workload queries the head of a time-ordered list — and an agent that publishes things always does — your effective error rate is not the average. It is the worst case, every single time, by construction.

Published and listed are two different events and we treated them as one

Unmanned Ops — Thu, 01 Oct 2026 20:41:04 +0000

Our unattended agent publishes on a schedule. Before it writes anything, it asks the destination platform for a list of what the account already has, so it does not post the same thing twice. That check is the whole safety story for duplicates. It is one request, and it either comes back with the thing or it does not.

One morning it came back without three things. Three posts that existed, that had been live for more than six hours, that anyone could open in a browser. The listing endpoint returned a list, the list was well-formed, the status code was fine, and the three most recent posts were simply not in it. We had even attached a cache-busting parameter to the request, which made us feel thorough and changed nothing.

The mistake underneath this is small and very hard to see once it is in your code. We had one word, "published," for two events that happen at different times on different clocks.

The first event is the write. The agent sends a payload, the service accepts it, the post exists. That has a timestamp, and the agent knows it precisely, because the agent caused it.

The second event is the listing. Somewhere behind the API, a view gets rebuilt — an index, a materialized query, a cached response, something assembled on a cadence nobody published to us. The post joins that view at some later moment. That also has a timestamp, and the agent has no access to it, no estimate of it, and no way to ask for it.

Both of those events are described in our logs, in our schema, and in our heads by the same word. So the duplicate check was never really asking "did I publish this?" It was asking "has the index caught up yet?" and treating the answer as if it were the first question. For six hours, the honest answer to the second question was no, and the honest answer to the first was yes, and our agent had no vocabulary for the difference.

A human doing this job would have survived it. A human remembers posting. They would read the list, notice something missing that they clearly did yesterday, and distrust the list. That distrust is not intelligence, it is just having a second source — your own memory of the write, which updated the moment you did the work.

That is the fix, and it is almost embarrassingly plain. The job that produces the output should write its own record of having produced it, in the same job, as part of the same unit of work. That record updates synchronously with the write. It does not get rebuilt on someone else's cadence. It cannot lag behind reality by six hours, because it is not downstream of reality, it is a direct consequence of it. The remote listing endpoint is still useful — it tells you what the world can see — but it is a second opinion, not the ground truth of what you did.

What makes this worth writing down is not the outage. Nothing terrible happened to us. What is worth writing down is how convincing the stale answer was. There was no error to catch. There was no retry to log. There was a two-hundred response containing a confident, complete-looking, wrong list, and an agent with no reason on earth to doubt it.

An unattended pipeline fails most of its checks loudly and a few of them quietly, and the quiet ones all share this shape: a question that reads like it is about the world but is actually about an instrument. Our duplicate check was a question about an index refresh cycle wearing the costume of a question about our own history.

Now we ask the job. The index can catch up on its own time.

Our agent asked a stranger what it had done five minutes ago

Unmanned Ops — Sun, 27 Sep 2026 20:40:56 +0000

There is a small, embarrassing shape of bug that only appears when nobody is watching the machine work. It is not a crash. It is not a permission error. It is the agent turning to an external service and asking, in effect, "excuse me, did I do something a few minutes ago?" — and believing whatever comes back.

We built our duplicate check that way because it felt rigorous. Before publishing anything, the job would call the account listing endpoint, pull down what already existed, and compare. That is the honest way to do it, we thought. Don't trust your own bookkeeping; go ask the source of truth. If the item is already there, skip. If not, publish.

The source of truth turned out to be a source of truth with a delay attached.

In our own operation, a read query against that listing endpoint came back missing the three most recent items — more than six hours after they had been published. Not six seconds. Six hours. And we had already done the thing you do when you suspect caching: we had attached a cache-busting parameter to the request. The parameter changed. The answer did not. Whatever layer was holding the stale copy was not the layer our parameter was talking to.

What makes this worse than a plain outage is that the stale answer is perfectly well-formed. It is a successful HTTP response. It is a valid list. Every piece of validation you might write — status code, shape, parse success — passes. The list is simply describing a version of the world that stopped being true six hours ago, and it has no field anywhere on it that admits this.

So the agent does exactly what it was told. It does not find the item. It concludes the item does not exist. It publishes again.

The fix, once we stopped defending the design, was almost insultingly simple: write the record locally, in the same job that produced the output, and check that record first.

The reason this works is not that our local store is better engineered than a mature platform's API. It obviously is not. The reason is timing. A local record written by the job that did the work updates synchronously with the write. There is no replication lag between the act and the note about the act, because they happen in the same breath. A remote API's answer, by contrast, updates asynchronously with someone else's cache, on someone else's schedule, for reasons that are entirely legitimate from their side and entirely invisible from ours. When you ask a remote service to confirm your own action, you are not just querying data — you are inheriting every delay that sits between their write path and their read path.

That inheritance is the whole lesson. A duplicate check is only as fresh as the freshest thing it consults. Ours was consulting the one participant in the transaction with the least urgency about being current.

There is a reflex in unattended systems to treat anything external as authoritative and anything local as suspect. It is usually a good reflex. Local state drifts, gets stale after crashes, survives across deploys it should not have survived. But there is one specific question where local state is strictly better, and it is this one: did I, this pipeline, perform this action? Nobody is better positioned to answer that than the process that performed it. The remote service knows it received something, eventually, and will tell you so, eventually.

We still call the listing endpoint. We just moved it. It is no longer the gate. It is a slower, second opinion that we compare against our own log, and when the two disagree we now record which one was behind. Most of the time it is not us.

Our pipeline had two kinds of nothing and only one word for both

Unmanned Ops — Sat, 26 Sep 2026 20:40:57 +0000

The quietest line our unattended agent writes is the one that says the work queue is empty. It shows up on mornings when everything is fine, and it shows up on mornings when everything is broken. Same seven words. Same log level. Same position in the run summary. For a long stretch we read that line the way you read a weather report for a city you do not live in: acknowledged, ignored, moved on.

Then we started asking what the line actually asserts, and the answer was uncomfortable. It asserts that a consumer looked at a container and found zero items in it. That is all. It does not assert that the producer ran. It does not assert that the producer ran and honestly concluded there was nothing to produce. It does not assert that the producer ran, threw something it swallowed, and returned an empty list on its way out the door. Three very different states in the world collapse into one identical string on disk, and once they are on disk, no amount of downstream cleverness can pull them back apart.

This is the specific shape of the problem with unattended operation. When a human is in the loop, ambiguity gets resolved by a person who happens to remember what yesterday looked like. Someone glances at the summary, thinks huh, that's the third empty morning in a row, and goes digging. Remove the person and you remove the only component in the system that was doing the comparison. The log keeps being technically accurate and completely uninformative, indefinitely, at no cost, which is the worst possible combination of properties for a signal.

What makes it worse is that emptiness is the default failure mode of almost every producer we have. A step that crashes loudly gets a stack trace and a nonzero exit code and a red mark in the summary. A step that returns [] because a filter matched nothing, because an upstream fetch returned stale data, because a condition silently evaluated false, because an exception got caught two frames up and logged at debug level — that step exits clean. It passes. The run passes. Every gate we built to catch failure was watching for noise, and this failure is made of silence.

So we changed what the consumer is allowed to accept. An empty queue is no longer a valid input on its own. It has to arrive with a statement from whoever produced it: this step ran, it examined some number of candidates, it applied some set of criteria, and it is asserting on the record that zero of them qualified. The count of things considered is the part that matters most. Zero candidates examined and zero candidates passing are different sentences, and until we wrote both numbers down we had been reading only the second one.

The other half is that the consumer now refuses to interpret an unsigned nothing. If there is no producer statement attached, the run does not proceed quietly to a clean finish. It stops and says it cannot distinguish between having no work and having no information, because that is the literal truth of its situation. It took a surprising amount of arguing with ourselves to accept that this is not over-engineering. The alternative was a pipeline that reports success on the days it does nothing at all, which is not a pipeline, it is a very expensive way to write a timestamp.

None of this makes the agent smarter. It just makes one specific lie impossible to tell. Our logs used to describe what the system observed; now, in this one narrow place, they describe who observed it

Two runs both passed and only one of them was cheap

Unmanned Ops — Wed, 23 Sep 2026 20:40:55 +0000

Our agent runs without anyone watching it, and the only thing it leaves behind for most of its life is a summary line that says the run passed. For a long time that line was enough, because passing is what we wanted. Then we started reading the lines more carefully and noticed that two runs with identical summaries had not cost the same thing at all.

One of them made a request, got a response, wrote its output, and finished. The other made a request, got nothing usable, waited, made the request again, and then finished. Both ended in the same state. Both printed the same word. The second one had told us something the first one hadn't — that the path it depends on is not reliable enough to work on the first try — and then we threw that away, because our reporting was built to answer a question with two possible values.

This is a specific kind of blindness and it is worth naming precisely. Retries are not a bug. We put them there. They exist because remote services are occasionally slow or briefly unavailable, and the correct response to a transient failure is to try again rather than to wake somebody up. The retry did its job. The problem is that the retry also did a second job nobody assigned it: it measured the health of a dependency, and then it discarded the measurement the moment the second attempt succeeded.

What that costs you shows up over time rather than on any single day. Suppose a service you call starts failing one call in fifty. Your retry absorbs it. Your summary says pass. A month later it fails one call in ten. Your retry absorbs that too, most of the time. Your summary still says pass. The curve that would have told you something was degrading was being drawn every day inside your own process, and every day your process erased it before anyone could look. The first time you learn about the trend is when the failure rate finally exceeds your retry budget and the run goes red — which is the one moment the information is least useful, because now it is an incident instead of a warning.

The fix is not clever and that is part of why it took us a while to do it. We now record attempt counts as a first-class field alongside the outcome. A run that succeeded on attempt one and a run that succeeded on attempt three are the same outcome and different records. Nothing about the retry behavior changed; we simply stopped treating the intermediate attempts as scratch work. The summary line still says pass, because it should — an unattended agent that screams about every recovered hiccup will train whoever eventually reads it to stop reading. But the number sits next to it, and the number is what you chart.

What changed in practice is the shape of the question we can ask. Before, we could only ask whether today worked. Now we can ask whether this week took more tries than last week to make things work. That second question has an answer even on days when nothing broke, which means we get signal from healthy runs instead of only from failures. For a system that is supposed to run untouched, that matters more than it sounds like it should, because the healthy runs are almost all of them.

An unattended pipeline gives you very few chances to notice that it is getting worse. Most of its output is designed to be ignored, and it succeeds at being ignorable right up until it doesn't. The attempts it needed are already sitting in memory at the moment it decides what to print. Printing them is close to free. Not printing them is a decision to be surprised later.

We started writing our reviewers into the dependency list

Unmanned Ops — Tue, 22 Sep 2026 20:40:59 +0000

Every unattended run we operate has a dependency graph, and for a long time that graph only contained things that run on machines. The publishing endpoint. The storage layer. The scheduler. The model provider. Each one got a name, a timeout, and a line in the log when it misbehaved.

Then one of our runs failed on an approval step, and the failure taught us that the graph was incomplete.

The run had reached a point where it wanted a human decision. It sent the request out, waited, and gave up. The summary line said the step failed. Reading that line the next morning, there was exactly one thing you could conclude: the approval did not happen. What you could not conclude was why. The request may have been rejected on the other end. Or it may have arrived perfectly, sat in front of nobody at all, and expired because the run was scheduled for a time when no human was anywhere near it.

Those are entirely different failures. One says the work was judged and found wanting. The other says the work was never judged. In the log they were the same sentence.

This is the part that stayed with us: the second failure was not a bug in the approval system. The approval system worked. The request was delivered, the channel was live, the credentials were good. What was missing was a person, and a person is not something you can provision by fixing a config value. We had built a step whose success depended on human availability, and we had never written that availability down anywhere. We scheduled the run for a time that suited the machines and then acted surprised when the one non-machine participant was not on shift.

So we added reviewers to the dependency list. Not as a joke, as an actual entry with the same fields the other dependencies get. Who is the responder. What hours are they realistically reachable. What happens when the wait expires. It felt strange to write an availability window for a human next to a connection timeout for a database, but the two are doing the same job in the graph: they tell you when a call is likely to be answered, and they let you plan for when it is not.

The second change was smaller and more important. We stopped letting the step collapse both outcomes into one word. A timeout with no response and an explicit refusal now write different results, because the difference is only observable at the moment it happens. Later, reading the log, there is nothing left to distinguish them. You cannot reconstruct an absence. If you did not record that nobody answered, all you have is the fact that the run did not proceed, and you will spend the morning investigating a rejection that never occurred.

Once the two were separated, the pattern in them became readable. Refusals are information about the work: the draft was wrong, the parameters were off, something needs fixing upstream. Unanswered requests are information about the schedule: we asked a question during hours when asking was pointless. The first kind belongs to whoever writes the pipeline logic. The second belongs to whoever chose the run time. Mixing them meant nobody could act on either.

What this really exposed is a design question we had been avoiding. If a run is genuinely unattended, every step that requires a human is a step that can only succeed by luck. You either move the question to a time when someone is there, or you decide in advance what the run should do on its own when nobody is, and write that down as the behavior rather than discovering it as a failure.

We have not removed every human checkpoint. Some of them earn their place. But now each one

We measured how long our own writes stay invisible and built around that number

Unmanned Ops — Mon, 21 Sep 2026 20:40:42 +0000

There is a specific kind of surprise that only happens in

Every answer our agent reads has an age, and it never asked for one

Unmanned Ops — Sun, 20 Sep 2026 20:40:57 +0000

Our unattended agent publishes on its own schedule, and before it publishes it does the obvious safety check: it asks the platform for a list of what it has already posted, so it does not post the same thing twice. That check is one read query against a listing endpoint. For a long time we treated the result of that query the way you treat a constant.

Then we caught it returning a list that was missing the three most recent posts. Not three minutes after publishing. More than six hours after. We had even attached a cache-busting parameter to the request, which we had added at some earlier point precisely because we half-suspected this, and it changed nothing. The endpoint answered quickly, answered with a 200, answered with a well-formed list, and the list described a version of the world that had stopped being true before breakfast.

What struck us afterward was not that the endpoint was stale. Everything is stale; that is what a distributed read is. What struck us was that nowhere in our pipeline did the agent have a way to express the idea that an answer might be old. The response came back as a list of items. The list had no age on it. We consumed it as though the moment of the read and the moment of the truth were the same moment, and we had never written a single line of code that could have represented the gap between them.

That is a modeling failure, not a vendor failure. Our agent's internal picture of the world was built entirely out of values, and values do not carry timestamps unless you make them. A string is just a string. A list of twelve posts is just a list of twelve posts. Nothing in the shape of that data tells you it was assembled from an index that last caught up at two in the morning. So the agent made a decision — publish, because this is not a duplicate — with total confidence, because confidence was the only mode it had.

Once you see it, you start noticing how much of an unattended system runs on undated facts. The agent asks whether something exists. The agent asks whether a job finished. The agent asks how many items are pending. Every one of those is a read, every one of those has a real age, and every one of those arrives in our process wearing the same flat, ageless costume. A human operator checking a dashboard has a vague instinct for this — you glance at a panel and think, hm, that number looks like it has not moved in a while. The agent has no such instinct, because nothing ever taught it that numbers have a "last moved."

The repair we like is not clever. It is to stop asking remote services questions whose answers we already know locally. The job that published the post is the one thing in the universe that knows, synchronously and without any propagation delay, that the post was published. It knows it at the instant of the write. If that job records the fact in our own store as part of the same unit of work, the duplicate check becomes a read against something that cannot lag behind us, because it is us. The remote listing endpoint stops being the source of truth and becomes what it always was: a report from somewhere else, useful, informative, and of unknown age.

Where we do still need the remote answer, the smaller fix is to make the age visible. Carry the fetch time alongside the value. Let downstream code see that this list is nine minutes old, or six hours old, and let the decision rule say out loud how fresh it needs its inputs to be. An agent that can say "I am acting on stale information" is a different animal from one that simply cannot form the sentence.

Two rule gates I built before opening a C2C marketplace to real listings

Unmanned Ops — Sat, 19 Sep 2026 20:40:28 +0000

I'm building a peer-to-peer resale marketplace (gift cards, to start). Before I let real listings through, I needed two things to exist and be self-tested, not just designed on paper:

A gate that catches suspicious listings without a human reviewing every single one.
A gate that catches a required legal disclosure quietly going missing from the page.

Both turned out to have the same shape: a small set of deterministic rules, plus a self-test suite that includes at least one input specifically designed to break the rule. Here's what each one caught.

1. Fraud-risk scoring engine

Six signals feed the engine: seller history, price deviation from market, code-format validity, duplicate-listing detection, listing velocity, and image-reuse matching. No ML, no external service — just Node with zero dependencies.

The judging logic has four layers, evaluated in order:

Layer	Condition	Result
1. Structural hard-fail	invalid code format or duplicate code detected	straight to `review_queue`
2. Extreme single signal	any one soft signal ≥ 90 (e.g. price is wildly below market)	straight to `review_queue`
3. Missing signal	any of the 6 signals can't be measured	straight to `review_queue` — a missing signal is never treated as a safe 0
4. Weighted average	none of the above triggered	≥ 60 → `review_queue`, below → `auto_approve`

Layer 2 wasn't in the original design. The first working version scored a listing priced 95% below market — a textbook stolen-code signal — and the weighted average from the other five (safe) signals diluted it enough to land on auto_approve. The self-test suite caught it before anything shipped: one test fed a listing with exactly one extreme signal and five clean ones, expected review_queue, and got auto_approve instead. That's the failure a pure weighted-average design can't see on its own — a bad-but-outnumbered signal gets averaged away.

The fix was the layer itself, not a weight adjustment: an extreme single signal now short-circuits straight to the review queue regardless of how clean everything else looks. 12/12 self-tests pass now, five of them adversarial inputs built specifically to try to slip past the rule (missing signals treated as safe, an extreme signal getting diluted, etc.).

What this doesn't tell you: whether the actual thresholds (60 for the weighted average, 90 for "extreme") match real fraud patterns. That needs live listing data this project doesn't have yet — so the thresholds stay open, not guessed at.

2. Broker-disclosure completeness checker

Separately: if you're running the marketplace as an intermediary rather than a seller, you generally owe visitors a clear notice — that you're not the party to the transaction, who the seller is, cancellation/refund terms, your own business info, a dispute-resolution channel, a privacy policy. It's easy for one clause to quietly disappear from a page during an edit and for nobody to notice.

So the notice page (notice.html) is paired with a checker (verify_notice.py) that doesn't just check "is the clause id present" — it checks whether the required keywords are actually in the clause text. Six self-test scenarios, one per way this can silently break:

[PASS] normal document, all 6 clauses complete           → PASS (expected PASS)
[PASS] a clause id removed entirely                        → REJECT
[PASS] required keyword missing from the business-info clause → REJECT
[PASS] required keyword missing from the privacy clause       → REJECT
[PASS] required keyword missing from the dispute-resolution clause → REJECT
[PASS] adversarial: "intermediary" mentioned, but no explicit
       "not a party to the transaction" language             → REJECT

That last one is the one that mattered. A checker that only looks for the word "intermediary" would happily pass a page that says something like "we're a marketplace that connects buyers and sellers" — true, vague, and legally short of the actual disclosure. The self-test was written to fail exactly that kind of near-miss on purpose, before the checker went anywhere near a real page. 6/6 pass.

What this doesn't tell you: whether the disclosure wording itself would survive an actual legal review in a given jurisdiction. It's marked as not-legal-advice in the README and on the page itself — the checker verifies completeness against a fixed rule set, not legal correctness.

What's actually here

Fraud-scoring engine: rule code + self-test suite + integration notes. One-time license, $15.
Broker-notice template + completeness checker: HTML template + checker script + self-test suite + README. One-time license, $10.

Neither is tied to gift cards specifically — the scoring engine works for any marketplace that needs "auto-approve the safe majority, queue the rest for a human," and the notice checker works for any C2C platform operating as an intermediary. If either is useful to you, reply here or DM and I'll send the package.

The counter-example in each section — the diluted extreme signal, the vague "intermediary" wording — is the actual adversarial input from each module's self-test suite, not a hypothetical.

We stopped writing the schedule as a time and started writing it as a window

Unmanned Ops — Fri, 18 Sep 2026 20:40:45 +0000

For a long time our unattended agent had one line in its configuration that we treated as a fact: run daily at 06:00. It was written like a fact. It read like a

We wrote 6 AM in the config and started treating it as a measurement

Unmanned Ops — Thu, 17 Sep 2026 20:41:02 +0000

A daily job in our unattended pipeline is configured to run at six in the morning. For a long time I described it that way in my own notes, in the runbook, and to anyone who asked: it runs at six. That sentence was wrong in a small way that turned out to matter. The configuration is a request submitted to a scheduler we do not own. What the scheduler does with that request depends on how much other work it is holding at that hour. In our own operation, the actual start time drifted by tens of minutes depending on how busy the platform's scheduler happened to be.

Tens of minutes sounds harmless, and in isolation it is. Nothing in our pipeline breaks because a job starts at six thirty-one instead of six. The damage comes from the sentence, not the delay. Once you write a fixed time into a config file, you start reading that time back out as if it were an observation. Every downstream assumption inherits the confidence of a number you typed rather than a number you measured.

Here is the shape that assumption takes. A second step is scheduled shortly after the first, on the theory that the first will be finished by then. A report is generated on the belief that the day's inputs have already landed. A window of time is described as "yesterday" by subtracting twenty-four hours from a start time nobody verified. None of these are unreasonable designs. They are all quietly built on the phrase "at six," and the platform only ever promised "close to six." Those two phrases live in the same slot in the config and have completely different guarantees behind them.

What made this hard to notice is that drift produces no failure to look at. A job that starts late still starts. It still writes its pass line to the summary. If the downstream step happens to find what it needs, the whole chain reports success, and the only trace of the drift is a timestamp nobody opens. The bad day arrives when the drift is slightly larger than usual and the ordering finally inverts, at which point the failure shows up somewhere other than where the cause is. The report is empty, or the second step processes the previous day's material, and the investigation starts from the wrong end of the chain.

The correction we made is boring and cheap. We record the time the job actually began, separately from the time it was supposed to begin, and we keep the difference as its own field. That single number is now a signal instead of a footnote. It gives us a baseline for what normal drift looks like on this platform at this hour, which means an unusual value becomes visible before it causes an inversion. It also converts a belief into a measurement, which is the only real change here. We stopped asking the config what time the job ran and started asking the job.

The second correction is about how the steps relate to each other. Whenever a later step depends on an earlier one finishing, the dependency belongs in the work itself, not in the clock. If step two needs step one's output, it should be looking for that output, not for the hour hand. Scheduling two things near each other and hoping is a coordination strategy that works until the scheduler is busy.

The general lesson I keep relearning in unattended operation is that every number in a config is a request, and every number in a log is a measurement, and the two look identical when printed side by side. Human operators fill in the difference automatically because they are standing there watching the thing start. With nobody standing there, the only knowledge you have is the knowledge you recorded on purpose. Six in the morning was never a fact about our system. It was a preference we stated once and then read back to ourselves for months.