DEV Community: Dmytro Polhorodnyk

What "Fully Automated" Actually Costs

Dmytro Polhorodnyk — Mon, 14 Sep 2026 16:02:02 +0000

I run an automated system on a remote machine with nobody watching it. When I
started, I described it to people as passive.

Over two days it needed five separate interventions. None of them were exotic, and
that's the point of this post. The gap between "automated" and "passive" is made of
very ordinary things.

Here's what actually broke, and the patterns I use now.

The five failures, named

Two copies of the same process running side by side for ten hours. Both were
writing. Neither complained. The watchdog that was supposed to prevent exactly this
compared strings after formatting instead of before, so the duplicate didn't look
like a duplicate to it.

A lost login that nobody noticed. The service kept running, kept polling, kept
reporting healthy. It just wasn't authenticated any more.

A component quietly detached from its host. Still installed, still configured,
no longer attached to the thing it was supposed to be attached to. Nothing logged it.

A dead event that never fired. A file was written for a consumer that had
stopped reading it months earlier. Correct, current, and read by nobody.

A config that silently reverted to defaults after a restart. The system came
back up with different behaviour than it had before, and the only sign was that the
numbers looked slightly different.

I'd expected the dramatic failures: crashes, disk full, network down. All five
were quiet ones. Not one produced an error message a human would see.

Pattern 1: restart is a state change, not a no-op

Two of those five happened only because something restarted.

We tend to think of a restart as returning to a known good state. In practice it's a
transition with its own failure modes. Parameters reset to defaults. Identifiers
regenerate. Connections re-establish in a different order than they did the first
time.

I hit a third version of this elsewhere: a startup sequence read a value from a
remote service before the connection to that service was authorized. The read
returned a clean, plausible zero. The system then used that zero as a measurement and
configured itself three hours away from correct.

The journal shows it by the second: component loaded at :29, value read at :36,
connection authorized at :39. The measurement happened three seconds before the thing
it measured existed.

A measurement taken before its input exists isn't a measurement. It's a default
value wearing a measurement's clothes, and if your priority rules trust measurements
over configuration, it will win every time.

Anything that must survive a restart has to be designed to survive it, and anything
read at startup needs to answer "is the thing I'm reading from actually available yet".

Pattern 2: scheduled jobs fail in complete silence

I moved a directory. The wrapper script coped fine, because it resolved its own
location at runtime. The registered scheduled tasks did not: they held absolute
paths recorded at registration time.

They failed. Nothing on the machine said so.

This is worth sitting with, because it's the quietest failure mode a system has.
A crashed process leaves a trace. A failing request returns a status. A scheduled
job that stops running produces nothing at all — and "nothing" is exactly what a
working system that had nothing to do also produces.

What I do now:

every scheduled job writes a heartbeat on every run, including runs where it decided there was nothing to do
absence of heartbeat for longer than the interval is itself an alert
anything path-dependent resolves its own location rather than trusting a value stored elsewhere

The general rule: if the only evidence of health is the absence of errors, you
have no evidence of health.

Pattern 3: an error is not "nothing new"

My message loop checked whether a response contained items. If it didn't, the loop
went back to sleep.

Token revoked returns an error, not items. A second instance stealing the connection
returns an error. Rate limiting returns an error. All three looked identical to
"nothing new happened", and the process stayed alive, the watchdog stayed green, and
the system was completely deaf.

Now: status gets checked before contents, an unexpected status is loud, and no
network call goes out without a client-side timeout. "Waiting forever" is another way
to look alive while doing nothing.

Pattern 4: confirm delivery before you mark it done

My notification pipeline deleted each event from disk before confirming it had been
sent. Send fails, event is already gone. No retry, no record, nothing to recover.

I'd built at-most-once delivery by accident, in the one place where at-least-once was
the entire point. A duplicate notification is annoying. A missing one is invisible,
and invisible is worse.

The order is the whole fix: confirm, then mark done. Never the reverse.

Pattern 5: updates on a machine you can't babysit

This is the part that took longest to get right, so I'll describe the shape I landed on.

Pull, not push. The machine checks a manifest on a schedule and decides for
itself. Nothing is pushed at it from outside.

Verify before applying. The manifest carries a hash of every artifact. Hash
mismatch means refuse and alert, not "try anyway".

Refuse during work. If the system is mid-operation, the update waits for the next
window. An update that interrupts work is worse than an update that's a day late.

Back up, swap, then self-verify. After applying, the system proves it's running
the new version by reading it back from the running process, not from the file on
disk. If verification fails, it rolls back automatically and alerts.

A canary ring of one. New versions go to a single machine first, and that machine
is always mine. Whoever writes the update should be the first person it breaks.

The honest summary

Automation moves work, it doesn't remove it. The work changes shape: instead of
doing the thing, you maintain the thing that does the thing, and you build the
instruments that tell you whether it's still doing it.

That second part is most of the cost, and it's the part nobody estimates.

The question I ask before automating anything now isn't "can this run without me". It's
"which failure am I prepared to discover late" — because that's the one I'll get.

Plausible Is What Gets Through

Dmytro Polhorodnyk — Mon, 14 Sep 2026 14:49:16 +0000

I ran the same logic against two market data vendors for the same period. Twelve
percent of the resulting signals disagreed.

Neither feed was broken. Neither validator complained. Every value was a number, in
range, correctly typed, and wrong.

That gap taught me the thing I now check for first: validators check shape, and
shape is not meaning. "Is this a number" and "is this the number" are different
questions, and almost every validation library I've used only answers the first.

Here are four cases from my own system, all of which passed every check I had.

1. Two sources, same question, different answers

I only ran that comparison because I was switching providers. I expected them to
agree to within a rounding error, and I was doing it as a formality.

Twelve percent of signals differed.

The cause wasn't corruption. Each vendor timestamped and rounded slightly
differently, and my logic sat on top of those boundaries. Small differences at the
edges, compounded through a pipeline, produced materially different outputs.

Nothing in the data looked wrong. You could stare at either feed all day and see
nothing to report. The disagreement was only visible when I diffed the outputs
rather than inspecting the inputs.

That's now a rule: when you swap a data source, don't validate the new feed. Run
both through your full pipeline and compare what comes out the other end. The
inputs will always look fine. They're supposed to.

2. The thousands separator that became a decimal point

The string 2,650 parsed as 2.65 in my system, and the result stayed a perfectly
valid number.

Different sources write thousands differently. Mine assumed one convention. On a
source that used the other, prices quietly shrank by three orders of magnitude and
stayed numeric, stayed positive, stayed parseable.

Every downstream check passed, because every downstream check asked the same
question: is this a number? It was. Beautifully.

This is the shape of the problem in one line. A type check cannot tell you that
2.65 should have been 2650, because 2.65 is a completely reasonable price for
something.

3. Position in a string is not meaning

A number in my logs read 414 when the real value was 4414. The parser had eaten
a digit, and nobody noticed for weeks.

The input format numbered its items: TP 1: 4414. My pattern matched the index and
the value together. On a source that omitted the index, the first digit of the
value became the index.

4414 became 414. Which is plausible. It's in range for other instruments, it
doesn't look like garbage, and no sanity check I had would flag it.

That's the thing about silent corruption: it never produces obviously broken
values. It produces slightly wrong ones that survive every check you thought to
write, because you wrote those checks imagining garbage, not plausibility.

The fix was to stop treating position in a string as meaning, and to parse the label
explicitly rather than relying on where things sat.

4. If your clock starts when you look, you will never measure a delay

My freshness check measured age from the moment my code noticed a message, not from
when the message was sent.

Which means after any outage, the backlog was fresh by construction. The service
comes back up, reads a queue of messages hours old, stamps every one with "now", and
hands me yesterday's decisions with today's timestamp.

I found it when a stale instruction showed up with live action buttons attached.

The general form of this bug is worth stating plainly: any measurement taken from
your own observation time will always look current. Everything is fresh, including
the things that aren't. Age has to come from the source, or it isn't age.

The related case that isn't about data at all

While auditing this, I found the same class of error in a completely different place.

My own analysis tool marked 87 ideas as "refuted" when nobody had actually judged
them. I'd built a two-pass system: one pass generates findings, a second sends
skeptics to challenge each. The script counted a finding as refuted when it had zero
supporting votes.

The verifiers never ran. They crashed on startup and returned nothing.

Zero votes and zero verifiers look identical if you only count votes. So 87 ideas
went into the graveyard with a confident label, and the report printed a clean number
with no warning anywhere.

Absence of evidence rendered silently as evidence of absence, inside a tool I built
specifically to be skeptical. I now keep "not checked" as a third bucket, visible in
the output, separate from "checked and rejected".

What I actually do now

Four habits, all cheap:

Diff outputs, not inputs. When you change a source, a library, or a version, run
the full pipeline both ways and compare the end result. Input validation will pass
on both sides. That's what it's for.

Assert ranges that mean something, not just types. Not "is this a number" but "is
this number within the band this field can physically occupy". Most silent
corruption lands outside a tight band and inside a loose one.

Parse labels, not positions. Anything that relies on where a value sits in a
string will eventually meet a source that formats it differently, and it will fail
quietly rather than loudly.

Keep "unknown" as a separate state. Not zero, not false, not missing. If your
code can't distinguish "checked and found nothing" from "never checked", it will
report the second as the first at the worst possible moment.

None of this is complicated. It's just that the bugs worth fearing don't announce
themselves. They arrive looking exactly like the values you expected, which is
precisely why they get through.

The Test That Could Not Fail

Dmytro Polhorodnyk — Mon, 14 Sep 2026 13:45:39 +0000

I shipped a safety feature that passed every test I had and could never have fired
in production. Not "rarely fired". Could not, structurally, under any market
condition, ever.

It took me two days to find out, and the only reason I found out at all is that I
stopped trusting green checkmarks.

Here's what happened, and the four habits I picked up from it. I build an automated
trading system, so my bugs settle in money rather than in tickets. That tends to
sharpen the lessons.

The feature

The system takes one position per day. I wanted a daily loss limit: if the account
drops by more than a set percentage today, stop trading until tomorrow.

Simple enough. Parse the threshold, compare it against the current drawdown, block
the entry if we're past it, log an alert.

I wrote it. Compilation clean. The test suite went green. The regression harness
confirmed the results matched the previous version byte for byte. I shipped it.

The part where nothing happened

Two days later I ran what I call a dose test. The idea is simple: set the threshold
so absurdly tight that the feature must fire. If it doesn't, something's wrong
with the feature, not with the market.

I set the daily loss limit to a value the account crosses within minutes of opening.

The results came back byte for byte identical to the run without the feature.

Not "mostly similar". Identical. The feature had done nothing at all, and it had
been doing nothing in production for two days.

Why every test was green

The check sat in the right place logically and in the wrong place temporally.

It ran before the entry block: "are we past the daily loss? then don't enter."
That reads correctly. But the system takes one position per day. By the time the
check ran, today's position had already been opened. The guard was standing in
front of a door nobody was going to walk through again until tomorrow.

Every test I'd written asked the same question in different clothes: given a
drawdown value, does the function return the correct decision? And it did. Every
time. Correctly.

Not one test asked the other question: does this function get called at a moment
when its answer can still change anything?

Those two things fail independently, and I'd only been testing one of them.

Habit 1: a test that can't go red isn't a test

This is the one that changed how I work.

A test proves nothing unless you've seen it fail. Until then you haven't tested
your code, you've tested the happy path of your test.

So now every meaningful check gets a dose: I deliberately break the thing under
test and require the suite to go red. Set the threshold absurdly tight and the
feature must trigger. Feed the parser a malformed input and it must refuse. Point
the monitor at a dead process and it must alert.

If the suite stays green while the system is visibly broken, the test was
decoration.

This has caught more of my mistakes than any coverage report ever did. Coverage
tells you a line was executed. It doesn't tell you that executing it proved
anything.

Habit 2: count, don't claim

While I was auditing tests, I found this in my own harness: