The service started cleanly.
It ran for five hours.
Then it fell over
on the one path
that sends a receipt.
A key in the configuration
had an underscore
where the code expected a dash.
Nothing caught that at boot,
because nothing read it at boot.
It was read at the moment of use,
inside a function
that only runs
after a customer has actually paid.
That is the shape of the bug.
Your configuration is not a document.
It is a pile of strings
that becomes a program
one lookup at a time,
in whatever order
the day happens to take.
The parts you exercise constantly
are proved on every deploy.
The parts you touch once a week
are not proved at all
until the week turns.
Now the second half of it.
The lookup usually has a fallback.
Somebody wrote a sensible default
so the service would not crash,
and now a missing value
does not fail at all.
It quietly takes the safe option,
which is the wrong option,
and the only evidence
is behaviour nobody ordered.
A timeout that is not the one you set.
Messages going to the wrong queue.
A feature switched off
in the one environment
where it mattered.
Do it the other way round.
Read the whole configuration once,
at startup,
before you accept a single request.
Turn it into one object
with real types.
A number that is a number.
An address that has been parsed.
A setting that can only hold
the values you allow.
If anything is missing or nonsense,
refuse to start,
and print which key it was
and what you expected to find there.
A process that will not boot
is a five minute problem,
discovered by the person deploying it,
who is already watching.
The same mistake at half past two
is an incident,
with a customer in it,
and an hour of reading code
to learn that one line in one file
has the wrong punctuation.
Find it in the first second,
loudly,
while somebody is still looking.
– Serguey Asael Shinder
Top comments (0)