A crash is honest. It interrupts you, prints a stack trace, and tells you where to look. You fix it because it insists.
This article is about the other kind. Code that runs. Code that returns. Code that satisfies every assertion you wrote, and hands the person using your software nothing at all.
I have shipped that at least twenty-four times across two projects: a Windows system monitor that is on the Microsoft Store, and a strategy game in development. Both solo, both public, so the record is a commit history I cannot edit.
Here is the whole thing in two snippets.
This test passes:
def test_grants_corpus():
company = Company()
node = ResearchNode("curated_corpora")
company.try_acquire_data_source(node.grant) # <- the test calls it
assert "curated_corpora" in company.data_sources
And this is the production path, in a different file, written weeks apart:
def on_research_complete(self, node):
self.company.research_points -= node.cost
self.company.cash -= node.price
self.ui.toast(f"{node.name} complete")
# nothing here ever calls try_acquire_data_source
The function works. It has always worked. The test proves it works. Nobody in the product ever calls it.
A test proves that a function, when called, does what you expected. It does not prove that anything calls it. Almost every class below is a variation on that sentence.
SF-01. The function nobody calls
Real case, 15 August 2026. A research node in the game printed exactly what it granted: a curated corpus, a mixture-of-experts architecture. The player paid research points, cash, and four months of in-game calendar. They received neither, in any campaign, ever. A campaign ran locked to its starting data source and architecture from day one to the last day.
526 tests passed on it.
Every test called TryAcquireDataSource and TryAdoptArchitecture directly. Nothing in the UI layer ever did.
The sweep
#!/usr/bin/env bash
# caller_audit.sh - domain methods with no caller in the interface layer
grep -rhoE "public [A-Za-z<>]+ (Try[A-Z]\w+|Set[A-Z]\w+|Add[A-Z]\w+)" Simulation/ \
| awk '{print $NF}' | sort -u \
| while read -r m; do
n=$(grep -rl "\b$m\b" UI/ 2>/dev/null | wc -l)
[ "$n" -eq 0 ] && echo "UNREACHABLE $m"
done
Output on my repository:
UNREACHABLE TryAcquireDataSource
UNREACHABLE TryAdoptArchitecture
UNREACHABLE CancelTraining
UNREACHABLE CancelArchitectureProgramme
...
46 public mutators, 21 with no caller, 5 meant to be player actions
The fix, which is a test shape, not a patch
// BAD: proves the function works. Proves nothing about reachability.
company.TryAcquireDataSource(node.Grant);
Assert.Contains("curated_corpora", company.DataSources);
// GOOD: banned from calling the granting function at all.
// Finish the node the way the day loop finishes it, then read
// what the company owns afterwards.
var before = company.DataSources.Count;
company.CompleteResearchNode(node); // the real completion path
Assert.AreEqual(before + 1, company.DataSources.Count);
Assert.Contains("curated_corpora", company.DataSources);
Same class, other projects. Thermal baselines using Welford's algorithm ran and were fully tested for months; the chat assistant never imported the engine. A method set_turbo() existed on the optimizer daemon with zero callers, so the entire coupling it was written for was dead from the day it was written. Sixteen assistant intents matched user phrasings at confidence 1.00 with no handler behind them.
SF-02. The key that never existed
Real case, 9 July 2026. Every row in the events log rendered as Unknown event, for months. The database held the type, the metric, the value, the baseline, and a written description. The UI read evt["message"]. That key did not exist in any layer of the codebase, ever.
# ui/events_page.py
label = evt.get("message", "Unknown event") # key never written, anywhere
# hck_stats_engine/events.py - what the row actually contains
{"ts": ..., "type": "spike", "metric": "cpu",
"value": 87.0, "baseline": 32.0,
"description": "CPU spike: 87% (usual 32%, +55)"}
The most informative label my product ever showed a user was the word "Unknown", with the real answer one dictionary key away.
The sweep
# key_diff.py - what the UI reads vs what storage produces
import re, sqlite3, pathlib
db = sqlite3.connect("data/logs/hck_stats.db")
cols = {r[1] for r in db.execute("PRAGMA table_info(events)")}
used = set()
for f in pathlib.Path("ui").rglob("*.py"):
used |= set(re.findall(r'evt\[[\'"](\w+)[\'"]\]', f.read_text(encoding="utf-8")))
used |= set(re.findall(r'evt\.get\([\'"](\w+)[\'"]', f.read_text(encoding="utf-8")))
print("read but never stored:", sorted(used - cols))
Why the tests were green: the fixtures were hand-written, and a hand-written fixture contains the keys you believe exist. That belief is the thing under test. Test against row shapes pulled from the real store.
Variant worth stating. A learning engine reported it was working and taught nothing. It learned one metric and the query filtered WHERE cpu_temp > 0. Reading CPU temperature on Windows needs a sensor service most people do not run, so the column was always zero and the filter excluded every row. Honestly: a correctness rule I added the week before, never learn from an estimated temperature, is what zeroed the column. A good decision created a silent outage.
SF-03. The import that never worked
Real case, 16 July 2026. The hardware scanner had never worked, on any machine, since it was written.
def _scan_wmi(self):
try:
import wmi # never installed, never bundled
c = wmi.WMI()
...
except Exception:
pass # every machine, every time, forever
The first successful hardware identity write in the project's history happened the day I rewired it to the scanner that already worked.
The sweep
# import_audit.py - modules imported but never declared
import ast, pathlib, sys
declared = {l.split("==")[0].strip().lower()
for l in open("requirements.txt") if l.strip()}
std = set(sys.stdlib_module_names)
for f in pathlib.Path(".").rglob("*.py"):
tree = ast.parse(f.read_text(encoding="utf-8"))
for n in ast.walk(tree):
if isinstance(n, ast.Import):
for a in n.names:
root = a.name.split(".")[0]
if root.lower() not in std | declared:
print(f"{f}:{n.lineno} undeclared: {root}")
Variant, and this one is mine to own. I published three times that psutil.sensors_temperatures() returns an empty dict on Windows. It does not. The attribute does not exist:
>>> import psutil
>>> hasattr(psutil, "sensors_temperatures")
False # Windows, psutil 7.2.1
Every call had been an AttributeError swallowed by a try block. I only checked because I was writing about repeating claims from memory.
For optional platform APIs, record which branch you took:
if hasattr(psutil, "sensors_temperatures"):
temps, src = psutil.sensors_temperatures(), "sensor"
else:
temps, src = _estimate_from_load(), "est"
# src travels with the value. An absent sensor must be visibly
# different from a normal reading, in every consumer.
SF-04. The success message that is a lie
Real case, 18 July 2026. The fan curve editor had an Apply button. It flashed "applied successfully" and persisted nothing. Every restart discarded the user's curve, silently, for two releases.
def _on_apply(self):
self.curve = self._draft_curve # in memory only
self._flash("Applied successfully") # a string, not evidence
Nobody reported it. The user cannot tell the difference between "it saved" and "it said it saved." They set the curve, saw the confirmation, restarted a week later, found defaults, and assumed they had done something wrong.
The sweep
grep -rniE "(applied|saved|success|complete)" ui/ --include=*.py -l \
| xargs -r grep -LiE "(open\(|json\.dump|\.write\(|commit\(|save\()" \
| sed 's/^/CLAIMS SUCCESS, NEVER WRITES: /'
The test shape
# BAD
assert page._on_apply() is True
# GOOD - the product is the side effect, so read it back
page._on_apply()
with open(SETTINGS) as f:
assert json.load(f)["curve"] == page._draft_curve
Same class. ShowWindow(SW_HIDE) succeeded and the console stayed open, because under Windows Terminal GetConsoleWindow() returns a hidden proxy window. A Store shortcut pointed at an Application Id absent from the published manifest, so it opened nothing. Nobody reports a broken shortcut. They stop using it.
SF-05. The silent catch
Real case, 24 June 2026.
try:
ctx = build_llm_context(lang, window) # `window` belongs to another function
...
except:
pass
NameError, on every call, for four months. No crash, no log, no warning. The entire fallback path was dead from day one. I found it because some answers felt slightly worse than they should, which is not a debugging method.
A bare except also swallows SystemExit and KeyboardInterrupt, so it is a shutdown hazard on top of everything else. A repository-wide count found 53 of them.
The sweep
grep -rn "except:" --include=*.py . | sed 's/^/BARE EXCEPT /'
grep -rn -A2 "except Exception" --include=*.py . \
| grep -B1 -E "^\s*(pass|continue)\s*$" \
| sed 's/^/SWALLOWED /'
The rule that fixed it
Swallow reads, never writes.
# fine: a default is a reasonable answer to "I do not know"
try:
temp = read_sensor()
except SensorUnavailable:
temp = None
# never: the user believes this happened
try:
save_settings(cfg)
except Exception as e:
log.error("settings not saved: %s", e)
raise
The one that hurt most. A refactor split a 6,533-line module into seven and dropped a module-level singleton. The caller wrapped its import in a broad except, so HAS_AI_LAYER quietly became False and the product's entire AI layer switched itself off. Everything ran. Tests were green. A guard test written the same day, for a blind spot I had not predicted, is the only reason I know.
SF-06. The environment that hides the bug
Real case, 17 June 2026.
# insights.py
def summarise(self, label: Optional[str] = None) -> str: # Optional never imported
...
On my machine this works perfectly. I develop on Python 3.14, where PEP 649 defers annotation evaluation, so the missing import never fires. On 3.9 through 3.13, the versions I officially support, it raises on import and the whole module silently turns itself off.
My own runtime was hiding the bug from me.
for v in 3.9 3.10 3.11 3.12 3.13 3.14; do
echo "=== python $v ==="
py -$v -m unittest discover tests -q 2>&1 | tail -3
done
The Store case, 4 July 2026. Microsoft approved the app into C:\Program Files\WindowsApps, which is read only. The app wrote its database, preferences and learning baselines next to its own executable, so every write failed silently on every Store install. Certification does not check this. Users do not report it, because nothing appears to go wrong. The app is simply permanently amnesiac.
def _app_dir():
exe = os.path.dirname(sys.executable)
# A name check is a guess. A write probe is a fact.
try:
probe = os.path.join(exe, ".w")
with open(probe, "w") as f:
f.write("1")
os.remove(probe)
return exe
except OSError:
return os.path.join(os.environ["LOCALAPPDATA"], "PC_Workman_HCK")
Do not detect an environment by name. Probe it.
Same class, different engine: Object.Destroy is a no-op outside play mode, so a texture cache that correctly released memory at runtime released nothing in the editor.
if (Application.isPlaying) Object.Destroy(tex);
else Object.DestroyImmediate(tex);
SF-07. Two sources of one truth
Real case, 19 August 2026. The model creator quoted the player eleven days for a training run. The run finished in one. Both numbers came out of my own code.
// projection, shown to the player before they commit money
days = petaflopDays / (throughput * PrecisionMultiplier(precision));
// daily tick, what actually advances the run
progress += baseRate * founderSkill * teamUtilisation;
// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the projection
// has never heard of either of these,
// and never multiplies by precision
Two formulas producing one quantity, for weeks, with nothing comparing them.
The sweep
grep -rhoE "^\s*def (_?[a-z_]+)\(" --include=*.py . \
| sed -E 's/.*def (_?[a-z_]+)\(/\1/' | sort | uniq -c \
| awk '$1 > 1 {print "DUPLICATE DEF x"$1" "$2}'
That found _base_dir in six copies and _is_admin in five. Version strings lived in eight files and were already wrong in three: the main window titled itself 1.8.1 while the single-instance check searched for a 1.8.2 title, so launching the app twice stopped focusing the running copy, for two releases, with no test on it.
If the same quantity is computed in two places, that is not duplication. It is a future disagreement with a date on it. The fix is never to keep the copies in sync. It is one function both sides call, so they cannot drift, plus a ratchet test:
def test_version_has_exactly_one_source(self):
hits = [p for p in Path(".").rglob("*.py")
if re.search(r'["\']\d+\.\d+\.\d+["\']', p.read_text())
and p.name != "app_version.py"]
self.assertEqual([], hits, f"hardcoded version literal in {hits}")
SF-08. The rule that only exists in the interface
A parameter ceiling was enforced inside TryStartTraining. But Project(), the function the creator screen calls on every slider movement, checked neither the ceiling nor the precision gates. The creator happily priced a run the START button then refused, with no warning attached to anything the player had touched.
// the test that matters: it never opens a screen
var blueprint = new Blueprint { Parameters = ceiling * 10 };
var result = company.TryStartTraining(blueprint);
Assert.IsFalse(result.Accepted,
"parameter ceiling is enforced in the UI only");
Order of operations is the same class in different clothes:
var blueprint = CurrentBlueprint(); // reads the raw slider
var projection = Project(blueprint);
RefreshParameterCeiling(); // clamps the slider, too late
The handle snapped back and every number on screen kept the value the player had dragged to. Validation that runs after the read is not validation.
A limit that lives in a screen is a suggestion with a nice font. It expires the moment a second entry point to that action exists: another screen, a loaded save, an edited config, an API call.
Running all of it as one pre-release check
None of these is worth much as something you remember to do. Vigilance does not scale past a few thousand lines and it fails exactly when you are tired, which is when this class ships.
#!/usr/bin/env bash
# preflight.sh - before tagging. Exit code is advisory, not a gate.
set -u
fail=0
echo "== SF-01 unreachable domain methods =="; ./tools/caller_audit.sh
echo "== SF-02 keys read but never stored =="; python tools/key_diff.py
echo "== SF-03 undeclared imports =="; python tools/import_audit.py || fail=1
echo "== SF-04 success with no write =="
grep -rniE "(applied|saved|success)" ui/ --include=*.py -l \
| xargs -r grep -LiE "(open\(|json\.dump|\.write\(|commit\()" || true
echo "== SF-05 bare excepts =="
grep -rn "except:" --include=*.py . && fail=1
echo "== SF-06 runtime matrix =="
for v in 3.9 3.10 3.11 3.12 3.13 3.14; do
py -$v -m unittest discover tests -q >/dev/null 2>&1 || { echo "FAILS ON $v"; fail=1; }
done
echo "== SF-07 duplicate definitions =="; ./tools/duplicate_defs.sh
exit $fail
Read the comment on line 2. This does not block a release. It prints a list a human reads, because half of what it finds is legitimate: internal plumbing with no user-facing caller, a helper that genuinely should exist twice, a success message on an operation that really is read-only.
A check that cries wolf gets disabled within a week, and then it protects nothing. Only a narrow set are hard failures: no bare excepts, no dead imports, no hardcoded version literals, no second copy of a single-source helper. Those are ratchets. Everything else is a report.
Where AI assistance actually changes this
I build with an assistant, openly, and have for over a year. The honest account is narrower than either side of the current argument.
It did not lower the quality of individual functions. Most of the cases above are ordinary mistakes with ordinary causes, and several predate any assistant.
What changed is where the standard has to be enforced. An assistant is extremely good at producing a function that satisfies a description. It has no view of whether a path exists from a human hand to that function, because that path lives in another file, in another layer, usually in another conversation. Writing got faster. Verifying that anything reaches the result did not. Every class above lives in the gap between those two speeds.
The incident I think about most. This project keeps a rule against generated-sounding prose and a test that fires every assistant response and fails on any em dash. On 17 July an automated cleanup removed 312 em dashes across 42 files. It also rewrote the em dash literal inside the test guarding against them, converting that test into "does this answer contain a hyphen".
# before: eaten by the sweep it was guarding against
self.assertNotIn("—", response)
# after: pure ASCII, no future text pass can match it
EM_DASH = chr(0x2014)
self.assertNotIn(EM_DASH, response)
Then verified with a negative control: inject an em dash into a real response, watch the test fail. If you have never watched your guard fail on purpose, you do not know that it can.
The tool did exactly what I asked, quickly and correctly on its own terms, and disabled the only thing standing between me and the problem it was hired to solve.
Why one person misses all eight
These failures are invisible from the inside. Nothing crashes, nothing logs, nothing looks wrong on the screen you happen to be looking at. Each one needs somebody to ask a question nobody thought to ask, and alone, the set of questions asked is exactly the set you already know to ask.
A reviewer supplies that for free, not because they are smarter but because they arrive without your assumptions. The reviewer reading your test file has never been told that calling the function directly is fine here. They just see a test that does not resemble what a user does, and they say so.
Without one you need a mechanical substitute. A grep does not get tired, does not assume, and does not politely skip the file it read last week. It is a worse reviewer than a person in every respect except the one that matters: it is not you, and it does not share your blind spots.
The second substitute is real users, and it is uncomfortable how much better they are. Three testers spent five hours each on a release I considered finished and found six stability issues, seventeen intents matching at full confidence and returning nothing, and a function calling the system snapshot nineteen times per message. None of that was in my suite. All of it was in their first evening.
Limits, stated out loud
One person, two projects. I have no data on whether this generalises to a team, and a team has review, which removes some of these entirely and creates others I have never met.
I also cannot cleanly separate "failures caused by AI assistance" from "failures I would have shipped anyway". Anyone claiming they can, in either direction, is selling something. What I can offer is what the log shows, with dates, in public repositories.
The suite went from 21 tests in June to 331 in August, and the growth was not the point. The point is that every silent failure that shipped once now has a ratchet test that fails the build if the class returns. That does not make the software correct. It makes one specific category of wrongness unable to ship twice.
Who wrote this
Marcin Firmuga, 22, Radom, Poland. Fourteen months on PC Workman, a Windows system monitor with a fully local assistant: 331 tests, 110 intents in Polish and English, 521 process definitions, a 330-entry offline hardware compatibility library. MIT licensed, Sigstore signed, CodeQL on every commit, on the Microsoft Store since July. Alongside it Scaling Laws, a strategy game about running a frontier AI lab, with 697 tests across 65 files and 35 save migrations.
Also 25 diagnostic guides in English and Polish, free and without a signup, and three build-in-public posts a week for twenty one weeks with no buffer and no queue of drafts.
None of it started at a desk. It started in a Netherlands warehouse where I was an order picker on a forklift, then driving in Poland, then welding plastics in a repair shop, and now a taxi at ten to twelve hours a day. One system underneath all of it, unchanged for fourteen months: work by day, coding by night.
Every figure above was read out of the source on the day this was published, not from memory. That habit exists because I have been wrong from memory before, in public, three times about the same API. It is written up in the catalogue as its own entry.
I am looking for my first job in software. Python, desktop applications, local AI without cloud dependencies, Windows internals, release engineering, technical writing in two languages. Open to remote, hybrid, or Krakow, Warsaw and Wroclaw.
- The full catalogue, eight classes with a sweep for each: pcworkman.dev/guides/silent-failure-catalogue
- Code: github.com/HuckleR2003
- Everything in one place: linktr.ee/marcin_firmuga
I write a public log every week, including the weeks that go badly.
Top comments (0)