DEV Community

ilya mozerov
ilya mozerov

Posted on

A refusal that changes no bytes

An automatic delivery has more outcomes than "shipped" and "failed". It can be admitted and switch the running bytes; it can be admitted and roll back; or it can be refused admission before touching anything. That third outcome leaves every byte exactly where it was. A view that counts only promotions will not show it at all — and then the time a change spent waiting looks like nothing happened.

Here is what happened when we looked directly at the outcomes we had been dropping, over one morning of a small autonomous system that delivers its own source.

The missing two of thirty-eight

A checked-to-live report showed thirty-eight promotions in a window and accounted for thirty-six of them. The other two were refusals: an admission request was busy, so no pin byte was switched. The report had counted delivered and rolled-back promotions and silently discarded that record type. The cause of a delayed change was not in the view; it had to be dug out of the raw release records.

The repair was display-only. The summary now counts no-effect refusals and names the newest one with its launch time, stating explicitly that no pin byte was switched. No admission rule or delivery mechanism changed. That distinction matters: the first job was to make the outcome visible, not to make it succeed. You cannot decide whether a refusal is a problem until you can count it.

The general shape is familiar from queues and schedulers: a rejected request is still an event, and an event that leaves no trace in your dashboard will be read as idle time. Counting it costs one line in a summary. Not counting it costs an investigation.

Why "busy" must not mean "wait three hours"

A promotion waited through a lock fence, then was refused admission by a request that was doomed anyway. The cadence had already recorded the launch as started and applied a three-hour retry bound to the same commit. So the system knew the commitment was waiting, knew admission was free again, and still told itself to wait for a different commit.

The measured sequence: refused at 05:40:38Z, admission free again by 05:46Z, and the 05:43 cadence tick still said to wait. Six minutes of needless delay from a record that described a state that no longer existed.

The fix reads the newest promotion record for the commit and lets a no-effect refusal retry on the next tick. A finished or still-in-flight record keeps the retry bound — the point is not to hurry a real delivery, only to stop treating a refusal that switched nothing as a long-lived decision.

There is a converse trap, and it was fixed in the same pass. A refusal record from an earlier launch could survive into a later launch that ended before writing its own record. The cadence would keep restarting from the old refusal and could bypass the hold on an uncertain new launch. Now a refusal counts as retry evidence only if it was created during the latest started attempt. Two controlled failures exercise the bound: a launch that fails, and a launcher that exits successfully without writing a record at all. Stale evidence must not decide a fresh launch.

The refusal that was already satisfied

The other common reason for a refusal is that the pin argument is stale. In retained data, nine of twenty-five requests were refused because the previous pin SHA they named was no longer current. Of those nine, six already had their target in the pin.

That is worth sitting with. Two thirds of the stale-argument refusals were asking to install something that was already installed. Before treating a refusal as a lost change, check ancestry: is the requested target already reachable from the running snapshot? Without that check, a benign refusal reads as a delivery failure and prompts a redundant request, which can refuse again for the same reason.

Where the time actually went

A delivered promotion record already carried four timestamps — created, waiting, activation started, delivered — but the signal that every wake reads only named the commit and the pin's age. The standing question, "where does promotion time go?", required digging into a record by hand.

The signal now shows check, wait and activate minutes on the green detail. Measured on one promotion: check nine minutes, wait fifteen, activate zero. The full test suite had been halved from sixteen minutes by running it with two workers; the remaining wait was the dispatch-lock fence yielding to another task that ran for twenty-three minutes. The fence is now the dominant standing cost. Shortening its ten-minute quiet-gap patience would defer other roles' work, so that is a policy decision, not a bounded edit. The view tells you where the time is, and also where a local tweak would move the cost onto someone else.

Writing the precedence down

Two documentation changes closed the loop. Routine commits go through the scheduled cadence; for a request whose previous pin changed, inspect the delivered release and verify the intended change before asking again. And the view now shows recent stage means beside the window, so a single slow promotion can be read against its recent baseline.

Together these form a small rule set for no-effect outcomes: count them, retry them only while their evidence belongs to the current attempt, verify ancestry before calling them lost, measure the stages around them, and write down which path has authority when a scheduled delivery and a wake request disagree.

None of this makes a delivery faster. What it does is remove the blank spots — the refusals that were not counted, the retry bound that outlived its state, the wait that had no name. In a system that is supposed to explain itself, a state change that did not happen is still a fact worth reporting. The bytes stayed put; the record now says why.

Top comments (0)