The failure that does not announce itself
An automation that stops is a small problem. It stops, somebody notices within the hour, the cause is obvious because the last thing that changed is the thing that broke.
The expensive failure is the one that keeps running. A field is renamed upstream, the code that reads it gets nothing, and nothing is a legal value in most processes - an empty note, an unset flag, a missing second line of an address. Nothing raises. The workflow produces records that look exactly like last week's records, and it does that for as long as it takes somebody to compare two numbers by hand.
What actually changes
Four shapes account for most of it.
A field is renamed or moved, usually as part of a tidy-up nobody thought was external. A payload gains a level of nesting when a vendor adds a wrapper for pagination or metadata. An enum gains a value - a new order status, a new document type - and the branch that handles the known values silently drops the unknown one. An API version sunsets, and the fallback turns out to be an older shape rather than an error.
None of these are outages. Every one of them is a Tuesday afternoon in somebody else's release notes.
Three defences, all cheap
One record, end to end, every morning. A synthetic item that goes the whole way and is checked at the far end against a known answer. It exercises the joins between systems, and the joins are where drift lives. Four separately healthy services can still be handing each other something that changed.
Assert on shape, not on status. A two hundred with a parseable body is not evidence that the body means what it meant last month. Check that the fields you read are present and typed as expected, and fail loudly when they are not. This is ten lines and it converts a silent wrong answer into a visible stop.
Alert on distribution, not on exceptions. If one in twenty items took the manual path last month and one in six takes it today, that is the signal - and no exception was raised to produce it. This only works if the ordinary numbers were written down first, which is the argument a measured baseline makes for itself.
Why this is a scoping question, not a maintenance one
Every integration is a dependency on somebody else's release schedule. That does not make it a bad idea; the rental delivery in where the four hours go reads availability from a system we do not control, and it still pays for itself. It makes the dependency a thing to price.
The practical version: when a workflow is scoped, list what it reads from outside itself, and for each one say what happens if the shape changes. Most answers will be "it stops, and that is fine". The ones where the answer is "it keeps going and we would not know" are the ones that need a canary before they ship, not after the first bad month.
The part nobody wants to name
A workflow that no person is responsible for is a workflow being checked by whoever happens to look, which in practice means after a customer complains. The name goes in the handover document with everything else, and it belongs to somebody who has opinions about the design rather than to whoever was free that week. The argument for treating that as a stage rather than an afternoon is in what a process audit must actually produce, and the reason it cannot be added later is that the person who forgot what was confusing cannot write it down.
Top comments (0)