DEV Community: Onur Kesim

My agent tagged 261 businesses "no website". One random check said otherwise.

Onur Kesim — Tue, 20 Oct 2026 03:59:14 +0000

I've started offering one-page websites to small local businesses in Turkey: vet clinics, dietitians, beauty salons. The idea is simple. Someone finds the business on Google Maps, taps the website link, and lands on Instagram, where the opening hours and services are buried somewhere in a grid of photos. A single page fixes that.

To find businesses worth writing to, I need a list: a good rating, enough reviews, and no real website. On 18 October 2026 I asked my AI assistant for that list; it scoped a first wave and handed the search to an agent. About twelve minutes later (18:51–19:03 Istanbul time) the agent came back with 261 candidates from 42 searches, covering 14 provinces and three business types. Each row had a rating, a review count and a status column: "Instagram only", "no website", and so on.

The table looked finished. This post is about the one row that got checked, why one row was enough to stop trusting a whole column, and what I do with the list now.

The rule I use for agent output

I have a standing rule for anything an agent hands me: before a delivery is used, one claim from it is picked at random and verified against the source, by a different path than the one that produced it. If the claim holds, the rest is accepted for now and work moves on. If it fails, that slice of work goes back to full verification.

The rule is cheap on purpose. Checking every row of every table would make the agent pointless. Checking nothing turns the agent's confidence into my claim, and I am the one whose name is on the message. One random check is the smallest amount of evidence that can still say no.

In my setup the check is usually run by a second AI assistant, in a separate run from the agent that built the table. It does not reuse the agent's data. It opens the source the way a person would.

The check

The row that came up was a vet clinic. The status column said "no website".

The check opened the clinic's own Google Maps page. In the website slot there was a link, and it pointed to whatsapp.com.

So the agent was not exactly wrong about the thing I cared about. The clinic still has no real website; a WhatsApp link is not a page that lists services and hours. The clinic was still a good candidate. But the column did not mean what it said. "No website" in the table most likely meant "nothing in the website field of the data the agent read". On the page a person sees, there was a link in that slot. If I had written "your listing has no website link", the owner would have looked, seen a link, and stopped reading.

That was enough. The check failed, so the column failed, for all 261 rows.

Why one failure condemns the column

The tempting move is to fix the one row and keep going. It feels proportional: one error out of 261 is under half a percent.

But that number is not real. Nobody measured an error rate of 1 in 261. What was measured was 1 failure in a sample of 1. A single check cannot estimate how often the column is wrong. A pass tells you very little; a fail tells you the rate is not zero. Before the check I was assuming close to zero, because the table looked clean. After it, I had no estimate at all, only one known failure and one known reason.

The reason matters more than the count. The mismatch was not a typo. It most likely came from the difference between where the agent read the data and where a person reads it. I did not open the agent's raw data for that row to prove it, but nothing else explains it as well. Any row could have the same gap: an Instagram link, a booking page, a WhatsApp link, a link added last week. When the cause is systematic, the error is too, and you cannot know which rows it touches without opening them.

There is a second reason. The status column was the one field my messages depended on. Rating and review count go into a message as context; if one is slightly off, the message is still true in substance. The status column is the premise. "Your listing sends people to Instagram" is either true, or it is a cold message from a stranger who did not look. A wrong row costs more there than anywhere else in the table, so the bar for trusting that column should be higher too.

What full verification looks like here

Re-checking 261 pages in one sitting would be wasted effort. I send at most five messages a day, so most of those businesses will not hear from me this month.

So the list is now a queue, not a source of truth. The header of the file says the status column is weak. Each row is measured again right before it is used, against the business's own page: is the Instagram account still theirs, what is actually in the website slot, are the rating and review count still close to the table. Only then is the message written, and it only states what that check saw.

This moves verification from "once, at the start, for everything" to "once, at the moment it matters, for one row". The measurement is also fresh when it is used. A check from last week says little about today's listing.

On 19 October 2026 the first two rows from this list went out this way, together with three businesses from an earlier, hand-picked batch. All five were re-measured shortly before sending (19:14 Istanbul time) and all five held. That says something about those rows. It says nothing about the rest, which is exactly why they stay in the queue marked weak.

The parts I'm still not happy with

Two things are open.

First, none of these checks was me opening a page with my own eyes. The re-check on 19 October was another agent run. It was told to read each business's own page, not the search data the table came from. That separation is the same thing that made the first check useful. But it is still an agent's statement. If both agents share a blind spot, sampling one with the other will not find it, and I do not have a test for that yet.

Second, the failure was caught because the random draw happened to land on a bad row. A different draw could have landed on a good one, and the same flaw, in however many of the other 260 rows it sits, would have been trusted. One check gives a "no" with some probability; it never gives a clean "yes". I accept that, because the alternative is checking everything, always, which in practice means nothing gets checked carefully.

What I took from it

A table that looks finished is a claim, not data. The agent's confidence is in the formatting. It does not transfer to the content.
One random check per delivery is the minimum, and it can only veto. A pass means "no evidence against", not "verified".
When a check fails, ask why before you count. A typo is local. A mismatch between the agent's source and the human's view is systematic and spreads to every row.
Verify at the point of use. A row checked minutes before a message goes out is worth more than a whole table checked once.
Write the uncertainty into the artifact. My list now says in its header that the status column is weak. Anyone who opens the file, including a future agent, sees that before row 1.

A question for you

If you use agents to collect data you then act on (outreach lists, dependency inventories, test inventories), how do you decide what to spot-check? Mine is random. I am starting to think the field the action depends on should be checked every time, and only the rest sampled. Have you found a better split?

This post was written with AI assistance. The candidate list, the random check and the re-check were produced by AI agents working under my rules; the rules, and the calls on what to do with the results, were mine. Measurements: 18–19 October 2026, Istanbul time.

I said SymPy recomputes my app's answer keys. Then I read the gate.

Onur Kesim — Wed, 07 Oct 2026 19:28:41 +0000

This week my maths practice app, Tuzak Avcısı: LGS Matematik, went public on Google Play (store page checked without signing in on 7 October 2026). It drills multiple-choice questions for the LGS, the national exam Turkish students sit at the end of 8th grade. Every question is generated from parameters, so there is no fixed answer sheet a teacher could read through once. Whether the marked answer is right has to be checked by code.

My CV, and a LinkedIn post from the same week, said:

Answer keys are recomputed by an independent third solver (SymPy) — 1,475 items.

I sat down to write that up properly, opened the gate to quote it, and found the sentence isn't true. This post is about what the gate actually checks, what it can't see, and where real recomputation does exist.

The repository is private, so I can't link it. All numbers below were measured on 7 October 2026 on commit 5fe5cef, whose app file is byte-identical to the one inside the released APK (SHA-256 f14c132c…), unless a line says otherwise.

Where the sentence came from

The test runner calls the gate under this comment (translated from Turkish):

independent sympy code gate — extract the choices (jsdom), then re-solve the key with sympy (exactly one correct + no distractor value-equal to it)

The first half of that comment became my CV. The part in brackets is what the code does.

What the gate actually checks

It has two steps. A Node script loads the app in jsdom, runs every multiple-choice mode and phase through 40 seeds (the CI setting; my local runner uses 16), and dumps each question's options with an ok flag. A Python script then parses the options with SymPy and asks two questions:

Is exactly one option marked correct?
Is any distractor equal in value to the correct option? (simplify(a - b) == 0 for expressions, a solveset comparison for inequalities.)

That's all. It never reads the question text, so it never solves the problem. If a generator computes 42 and marks 42 correct when the answer is 41, the gate has nothing to compare 41 to.

My run on the release commit, with SymPy 1.14.0:

Denetlenen: 1560 | UYGULANMAZ (semantik): 160 | KASITLI biçim-tuzagi (allowlist): 56 | GERCEK FAIL: 0

That reads: 1,560 checked, 160 not applicable, 56 intentional form traps (counted inside the 1,560), 0 real failures. 1,720 questions came out of 42 modes. One multi-select phase yields no options at all and is skipped without a word; that's a second blind spot I found while writing this. 160 have answers SymPy can't compare by value (words, drawn options, side-length triples, letter orderings), so only the one-correct check applies to them. The 56 "intentional" ones have the right value in the wrong form: 13×10⁻⁴ instead of 1.3×10⁻³ (marked wrong), or 2√28 instead of 4√7 (deliberately not counted wrong). The 1,475 on my CV was an older count.

It still earns its place

Duplicate-value options are a real bug class in generated multiple choice: two buttons that are both right, one of which the app calls wrong. This gate covers that class across every mode it can parse, except the three form-trap modes, where any value-equal option is allowed.

It also once went red because of itself. The release's build report (30 September 2026) records an item whose options included "20 ve 9" and "12 ve 15". Ve is Turkish for "and". SymPy's implicit multiplication split ve into two unknowns, v·e: 20·v·e·9 = 180·v·e = 12·v·e·15. Two different answers, declared equal. The fix taught the parser that "a ve b" is an unordered pair of numbers, and that any other option containing ve, veya ("or") or ile ("with") is compared as plain text.

What it can't see, measured

In this app a wrong key doesn't crash anything. A student picks the right answer, the app marks it wrong and may open a card explaining a trap they didn't fall into. Nobody files a bug report. A 14-year-old just learns to distrust a method that works.

To put a number on the blind spot, I simulated a wrong answer key. For each mode and phase, I added 1 to every correct option that was an integer in the extracted file and re-ran the gate. 21 mode-phases had at least one integer key in the sample (from 6 to 40 of 40).

6 stayed green: exponent rules, scientific notation, multiplying and dividing roots, rational and irrational numbers, cylinder volume and cylinder surface area. In the two cylinder modes all 40 of the 40 sampled keys were now wrong. In two of the six, scientific notation and the root mode, a collision would have been waved through anyway, because value-equal options are allowed traps there.
15 went red, for the wrong reason. Many distractors are deliberate off-by-one traps, so "correct + 1" often landed exactly on a distractor, and the gate reported "distractor equals the correct answer". That's detection by coincidence, with a message that points at a different bug.

Counted per question, 315 of the 580 wrong keys (54%) passed silently.

This experiment edits the gate's input, not the generators, so on its own it doesn't show that any generator is wrong. But the release code records a case where one was. In the exponent-rules mode, a question showed (−b)ᵐ·bⁿ while the key used the parity of m+n. For m = n = 3 the real value is negative and the key said positive. A comment in the generator records that the SymPy gate missed it, because the options were consistent with each other and wrong only against the question. A manual visual review caught it.

Where recomputation does exist

Two places, both narrower than my CV implied.

The GCD/LCM gate. It carries its own Euclid and never calls the app's own GCD/LCM functions. With a mutant that shows the LCM where the GCD belongs, it fails on the first pair: (30,18): shown GCD=90 ≠ egcd=6. Without the mutant it passes on 3,000 generated pairs.

The GCD/LCM word problems. A separate solver reads each question's raw parameters and text and recomputes the answer by its own route; for the "when do the two bells ring together for the n-th time" type it brute-forces minute by minute. My run: 8,000 questions, 0 mismatches. It isn't in the release runner or CI yet; only its mutants run there.

Both are about GCD and LCM. One more gate is described in its comment as an independent validity check for triangle-inequality questions; I haven't read its assertions yet, so I'm not counting it. The other gates I read check structure, variety and trap behaviour, not the key. For the two cylinder modes I found no file in the test folder that recomputes an answer. I didn't read all of the gates line by line, so treat the rest as "not found", not "absent". There is also a golden-answer harness that lives outside the repository; I didn't re-run it for this post, so I'm not counting it.

The rest of that CV line

I checked the other numbers against the release commit and its build reports the same evening.

CV said	The release commit says
25-layer audit chain	55 gate calls in the runner
8 mutation tests	224 mutant pairs: 201 code, 23 visual; 9 exempt, 0 survived (build report, 1 October 2026)
49 question generators	Couldn't define it. 56 functions start with `gen`, and some are helpers. Dropped.
Ad/analytics scan with a positive control	Holds (release scan, 1 October 2026): 13 SDK signatures, 0 found, no INTERNET permission. The scan must find the Capacitor core class path, or it refuses to report "clean".

Most of those numbers had gone stale in the safe direction: the chain grew after I wrote them. The SymPy sentence went the other way. It described more checking than exists.

What I changed

CV: "A SymPy gate checks 1,560 sampled multiple-choice questions (39 of 42 topics) for exactly one option marked correct and no distractor equal in value to it, deliberate notation traps excepted. For GCD/LCM word problems, an independent solver recomputed the answer keys (8,000 questions, 0 mismatches)." The "49 generators" line is gone.
LinkedIn: edited with the same correction.
A rule for myself: a gate's name and its comment are claims. Before I quote a gate anywhere, I read its assertions.

The gap itself is still open. Volume, surface area and the other modes need a solver that reads the question, not the options. I haven't built it yet.

A question for you

If you generate maths questions and recompute the keys independently, where does your solver start: from the rendered question text, or from the generator's parameters? Parameters are far easier to work with, but then the solver trusts the same inputs the generator used.

This post was written with AI assistance, and the app and its gates were built with an AI coding agent working to my specifications. Numbers: release commit 5fe5cef, measured 7 October 2026 unless dated otherwise.

I benchmarked my own security tool against 3 others — and wrote down what it lost

Onur Kesim — Sun, 04 Oct 2026 18:58:30 +0000

I built gedik, a security-audit skill for Claude Code. Before telling anyone it was good, I wanted a number. So on 30 September 2026 I ran it against three other tools on two targets, with the answer key kept outside every run directory. Then I put the results in the README, including the parts where it lost.

This post is about those parts. If you build tools on top of an LLM, the losses are more useful to you than the wins.

The setup

Four tools. The three LLM-based ones ran on the same model (claude-sonnet-5-5), each in its own isolated claude -p process, with the same prompt:

gedik v2.6.0
cloudflare/security-audit-skill
Anthropic's built-in /security-review
Semgrep 1.178.0, free registry rules only (no Pro, no login)

A fifth candidate, Anthropic's claude-security plugin, was left out: its licence restricts use with non-Anthropic products, including competing ones, and I read that as covering this test.

Two targets:

Target 1: NodeGoat (commit c5cb68a), an intentionally vulnerable Node app. I deleted the tutorials, the tests and every JavaScript comment so the tools couldn't read the answers. That left 17 known issues in the copy.
Target 2: a small Supabase + LLM-tool repo I built myself. 11 seeded issues, 5 decoys.

I couldn't restrict file-system access, so I checked afterwards instead: every session and sub-agent log searched for the key's path, zero matches.

One person scored everything: me.

The numbers

Target 2, my repo, 11 seeded issues (runs of 30 Sep 2026):

	gedik	cloudflare	/security-review	Semgrep
found	11/11	11/11 (9 confirmed)	9/11	1/11
false positives	0	0	0	0

Target 1, NodeGoat, 17 known issues (30 Sep and 1 Oct 2026; competitors one run each):

	gedik	cloudflare	/security-review	Semgrep
detection, when the report arrived	16–17/17 (5 delivered runs)	16/17	5/17	6/17
report delivered	2 of 6 runs (30 Sep, evening and night) · 3 of 3 (1 Oct)	yes	yes	yes

That second row is the story.

Loss 1: the user got an error message

On 30 September, in the first scored run on NodeGoat, gedik's report never reached me. Twice. Every sub-agent finished its work, and then, while the final report was streaming, the session stopped with stop_reason: refusal and a [cyber] safeguard tag. One run had written 6,876 characters and announced 27 findings before it was cut. The other stopped at 3,114 characters. What I received was the error text.

So whatever it had found, the honest score for that first run was 0/17 delivered.

The other three tools audited the same code and were never stopped. Their reports carried little or no exploit detail. gedik's design rule is "no proof, no finding": every finding comes with a working proof of concept. My guess, and it is only a guess because I never isolated the trigger, is that a report full of working code-execution and injection proofs is the kind of text a real-time cyber safeguard interrupts.

I tried to engineer my way out the same night. I moved the proof inputs out of the Markdown report into a JSON field and tightened the rules for harmless markers. It didn't fix it. One patched run delivered. One was refused during the research phase, before any report existed. In a third, after being refused, the agent started splitting its file write into smaller pieces. That is behaviour I don't want in a security tool. I stopped that run, didn't count it, and reverted the patch.

Across the six unpatched runs on that target, 4 were cut and 2 delivered.

Loss 2: the fix was not mine

What changed delivery was an account-level change, not code. My account was approved into Anthropic's Cyber Verification Program, and on 1 October 2026 I re-ran the unmodified v2.6.0 on the same target: 3 of 3 reports delivered, 17/17 each (in one run, 16 confirmed and 1 marked as suspected), 0 false positives.

That sounds like a happy ending. It is weaker than it looks:

3 of 3 after, 2 of 6 before. One-sided Fisher p ≈ 0.12. Consistent, not proven.
Time, run-to-run model variation and the reverted patch are all confounders.
A user without that approval should still expect the report to be cut on exploit-heavy code like this: 4 runs in 6 on this target on 30 Sep, not re-measured since.

So the README says detection was at or above the best tool here, and delivery depends on your account. It does not say "better".

Loss 3: slower and more expensive

On Target 2 (30 Sep 2026) gedik took 176 seconds; /security-review took 40 and the cloudflare skill 96. The client-reported total_cost_usd field showed $1.02 for gedik against $0.35 and $0.74. I haven't verified what that field means on a subscription account, so read it as relative, not as a bill. Semgrep finished in seconds without a model.

gedik does more per run: a triage step, a refutation step in most runs, mutation testing of the project's own tests. But "does more" is a cost the user pays every time.

Loss 4: my favourite target proves the least

Target 2 is where gedik looks best, and it is the weakest evidence in this post. I built it, I seeded it, and I picked issues close to gedik's strongest areas: Supabase row-level security and LLM tool permissions. In the 30 Sep run the cloudflare skill also found all 11, two of them marked "needs validation". On a target designed by the person who wrote the tool, a tie is the most a fair reader should take from it.

NodeGoat has its own problem. It is almost certainly in the model's training data. Deleting the comments didn't hide it: gedik and the cloudflare skill both called the target "NodeGoat-style" on their own. Memorisation may inflate every LLM-based score in that table, gedik's included.

Loss 5: the machine-readable output didn't show up

gedik v2.6 is supposed to write a findings file in JSON, checked against a schema. In the three runs of 1 October it wrote prose reports and no JSON file. My prompt said "put the findings in your final reply", and the skill read that as "don't write files". The cloudflare skill already ships a schema'd findings.json with tested validators. That gap is real and current.

What it won

One thing the others didn't do at all. On Target 2 (30 Sep 2026) gedik mutated the project's own code in a scratch copy and ran its test suite: three of four mutants that removed an access check or an input check survived with green tests. These were library-level mutants, not every route-level mutant in my key. The other tools said nothing about the tests.

That check, "would your tests notice if this guard disappeared?", is the reason I wrote gedik, and it's the part I'd defend hardest.

The caveats, in one place

One annotator: me, the person who wrote the tool. No second reading.
Target 2 was built by me and seeded toward my tool's strengths.
The cleaned NodeGoat copy is weaker than the original, and probably memorised.
One run per tool and target, except gedik on NodeGoat: six runs on 30 Sep before the account change, three on 1 Oct after.
The cloudflare agent said it did not run its full multi-phase workflow, so I measured "the skill under this prompt", not the skill's complete pipeline.
Semgrep ran free rules only.
Everything ran on Windows.

What I'd like from you

The weakest part of all this is that only I scored it. The method and the run records are in KRITER.md (written in Turkish) in the repository, github.com/onur-kesim/gedik; the answer key and the raw reports aren't published yet. If you'd be willing to be the second reader, say so in the comments.

And if you build on an LLM: measure delivery, not just detection. My tool found everything and, for a day, delivered nothing.

This post was written with AI assistance. The numbers come from my run records (gedik commit f34aaf4, 1 October 2026).

I Wrote a Tool to Find Blind Spots. It Had One.

Onur Kesim — Thu, 27 Aug 2026 23:39:57 +0000

I spend a lot of time writing quality gates to enforce invariants in my software. For weeks, my test runner stayed green. Every guard was in place, every check passed, and every test suite reported clean execution.

Then I asked an uncomfortable question: are these gates actually measuring anything, or are they just executing without complaining?

To find out, I wrote a script to sabotage my own engine. The tool parses the source code, finds every fail() assertion call in the engine, disables them one by one, and re-runs the entire mutation test suite for each line. It asks a simple question: if I delete this guard line, does any test notice?

If no test notices, that line is uncovered. It isn't a protection; it's a comment pretending to be a guard.

I expected to find a few gaps. I even wrote that expectation directly into the script's exit codes, annotating exit code 1 as (expected in v2.4.1). I knew I had blind spots—I just didn't know how many.

When the measurement finished across 61 protection lines in the engine, 40 of them were completely unmeasured by any test—a 66% blind spot rate.

Yet, the baseline test suite run right beside it reported: 36/36 biting mutants. A 100% green test suite, sitting on top of an engine where two-thirds of the assertions could be erased without triggering a single alert.

0  her fail() kapsamli (hicbir kor nokta yok)
1  en az bir KAPSAMSIZ fail() var  (v2.4.1'de beklenen)
2  OLCULEMEDI / kurulum hatasi

# translation:
# 0  every fail() is covered (no blind spots)
# 1  at least one UNCOVERED fail() exists (expected in v2.4.1)
# 2  UNMEASURABLE / setup error

Three ways a gate dies

A quality gate usually dies in one of three ways. All three look identical from the outside: they give you a green checkmark.

The Dead Gate. It fell out of the test runner or CI pipeline. The script file sits in the repository, but nothing invokes it anymore. It never fails because it never runs.
The Environmentally Broken Gate. It gets called by CI, but it cannot run properly on that specific platform due to missing dependencies, OS permission boundaries, or character encoding mismatches. It returns green because its internal errors are caught under a soft error flag.
The Blind Gate. It runs. It finishes with exit code 0. And it measures absolutely nothing because the input data required to trigger its logic is never produced.

Most engineering setups have answers for the first two: linters, pipeline validation, and coverage runners. Almost nobody writes tests for the third.

Handling environmental failure requires thinking about what happens when the measurement tool itself gets corrupted by the environment. For instance, in my repository, the hukum_kapisi.py gate uses explicit ASCII character patterns for string matching rather than UTF-8 dashes. Why? Because when an encoding error triggers Python's errors="replace" handler, a hyphen or dash can be printed as a ?. If the gate matched against unicode dashes, the very encoding failure it was designed to catch would corrupt the pattern matcher itself.

There is a vast difference between "I wrote a gate" and "I thought about what happens to the gate when its environment breaks."

Level 1: mutate the gate — and why it isn't enough

The standard response to testing quality gates is Level 1 mutation: intentionally break the gate's code, run it against dirty input, and confirm that exit != 0.

This is a good reflex, but it is incomplete. A gate requires two-way proof:

It must bite on dirty input.
It must stay silent on clean input.

If you only test that a gate triggers on bad input, you risk keeping a gate that constantly fires false positives on clean environments.

More importantly, proving that a gate bites on a synthetic input does not prove that it covers the entire class of errors it claims to guard. Having a single mutant test pass against a gate gives you a false sense of security.

Level 2: mutate the engine, ask if any gate notices

Level 2 flips the approach: don't mutate the gate. Mutate the engine that the gates are guarding, then check if any gate notices the change.

My script, sabotaj.py, automates this process through three specific design rules:

AST parsing, never regex. Searching for function calls in source code using regular expressions inevitably misses multi-line calls. When a measurement tool misses a line silently, the tool itself becomes blind. An analysis script must parse the actual grammar of the language.
Read-only execution copies. The script never touches the working source file hafiza.py. Each sabotage operation creates a temporary memory/disk copy, runs the suite, and discards the copy. If a tool modifies its own source of truth during execution, it is no longer measuring—it is mutating.
The compile() syntax gate. If a sabotaged file fails Python's compile() step, the result is classified as UNMEASURABLE, not UNCOVERED. Conflating broken syntax with missing test coverage artificially inflates blind spot metrics.

# 1) fail() cagri yerlerini AST ile bul  (regex DEGIL: cok satirli cagrilar var)
# translation: 1) Find fail() call sites via AST (NOT regex: multi-line calls exist)
def fail_cagrilari(kaynak):
    agac = ast.parse(kaynak)
    bulunan = []
    for d in ast.walk(agac):
        if isinstance(d, ast.Call) and isinstance(d.func, ast.Name) and d.func.id == "fail":
            ...

def sabote_et(kaynak, hedef):
    """Tek bir fail(...) cagrisini `None` ile degistir. Kaynagi DEGISTIRMEZ."""
    # translation: Replace a single fail(...) call with `None`. Does NOT modify source.
    ...
    compile(yeni, "<sabotaj>", "exec")   # sozdizimi bozulduysa burada patlar
    # translation: # explodes here if syntax is broken
    return yeni

When I ran this analysis against hafiza.py, the core value was not the total number of 40 unmeasured lines—it was how those blind spots clustered. Looking at key selected gates from the results:

Gate Identifier	Comprehensive (Covered)	Uncovered (Blind)
H1	0	6
H9	0	1
H11	1	10
H8	2	4
H3 / H7 / H15	1 / 1 / 2	0

Gate H11 is the clearest example. Out of 11 fail() protection calls inside H11, 10 were completely unmeasured. Yet, because 1 mutant (M-H11) triggered a single line, the gate marked itself as "tested" in naive summary reports. Testing a single sink line with one mutant does not cover the gate's invariant class.

The third verdict — and the day my own tool lost it

Every measurement tool must support a third verdict: UNMEASURABLE (ÖLÇÜLEMEDİ).

Binary PASS/FAIL models force tools to make false statements when execution environment issues prevent proper testing. I learned this when an earlier CI run hit a read-only filesystem restriction, threw a raw PermissionError, and returned exit code 1. The contract declared 1 = MUTANT ESCAPED. The runner had reported an unmeasurable state as if it were a legitimate test failure. Fake failures destroy trust in real failures.

While reading through sabotaj.py line by line to prepare this writeup, I looked at how main() evaluates its final exit code:

if olculemedi and not kapsamsiz:
    print("HUKUM: OLCULEMEDI kalemler var — 'tam kapsamli' DEMEK YASAK.")
    return 2
if kapsamsiz:
    print("HUKUM: %d KOR NOKTA var." % len(kapsamsiz))
    return 1

# translation:
# if unmeasured and not uncovered:
#     print("VERDICT: UNMEASURABLE items exist — FORBIDDEN to claim 'fully covered'.")
#     return 2
# if uncovered:
#     print("VERDICT: %d BLIND SPOTS exist." % len(uncovered))
#     return 1

Look at the logical condition in that first branch: if olculemedi and not kapsamsiz:.

If the script encountered both unmeasurable compilation errors AND uncovered lines during a run, not kapsamsiz evaluated to False. The code bypassed the first block entirely and dropped straight into if kapsamsiz:, returning exit code 1. The warning explicitly forbidding a claim of full coverage would never execute, and exit code 2 was swallowed.

The third verdict existed in a print statement, but it had dropped out of the exit code contract.

In my actual measurement run, ÖLÇÜLEMEDİ happened to be 0, so the script reported its final count correctly. I didn't find this bug because a test failed or an alert fired—I found it because I read the source code line by line.

It was a latent defect sitting silently inside the script's main() function. The tool I wrote to find blind spots had a blind spot in its contract logic.

What this does NOT tell you

To maintain clear boundaries around what these metrics mean, three constraints must be stated directly:

The count of 40 is an upper bound. sabotaj.py parses output logs from the test runner using regex:

  kacanlar = sorted(set(re.findall(r"(M-[A-Z0-9a-z_]+)\s+.*?KACTI", cikti)))
  if not kacanlar:
      kacanlar = sorted(set(m for m in re.findall(r"^\s*(M-\S+).*KACTI", cikti, re.M)))

If both regexes fail to match an escaped mutant name in the log output, kacan evaluates to 0, which defaults the verdict to KAPSAMSIZ (uncovered). The tool that uses strict AST parsing for input files relies on fragile string parsing for output reading.

Docstrings decay faster than code. The script docstring declared (bugun 60 adet) (60 items today), but the AST parser returned 61 active fail() calls. Docstrings are historical statements that are accurate only on the day they are written.
Coverage is not correctness. A line marked as COMPREHENSIVE simply means that disabling it caused a test to fail. It does not prove that the guard is evaluating the correct business invariant or handling edge cases properly.

Try it on your repo

If you want to verify your own quality gates, the approach can be implemented in any codebase using a simple sequence:

Parse your engine's abstract syntax tree (AST) to locate every assertion or failure guard. Do not use regular expressions for code structure detection.
Iterate through each call site, replacing it with a no-op (None or pass) inside an isolated temporary file copy.
Run compile() on the modified source to catch syntax breaks before testing.
Execute your test suite against each single-line modification.
If the test suite stays green, log that line location as uncovered.
Ensure your runner's exit logic isolates execution errors (UNMEASURABLE) from assertion failures (UNCOVERED) without conditional short-circuits.

For every fix you apply to a system, you owe it a sabotage: strip the fix out temporarily, rerun the suite, and verify that the exact target error reproduces. If removing the fix doesn't bring back the original failure, your test is measuring a neighboring component, not the bug.

Measured at commit a86ffe4 (a86ffe440c0740241fdf20387c14420029ec0555), 1 Aug 2026.

To verify locally:

git clone https://github.com/onur-kesim/hafiza-kur.git
cd hafiza-kur
git checkout a86ffe4
python faz0/sabotaj.py --motor skill/scripts/hafiza.py

# The run above prints the counts. To see the defect itself — the branch that
# swallows the third verdict — read the contract logic directly:
grep -n -A 4 "if olculemedi and not kapsamsiz" faz0/sabotaj.py

One Letter Broke Four Toolchains. A Symlink Didn't Fix It.

Onur Kesim — Tue, 18 Aug 2026 08:51:41 +0000

Three Sessions, Four Broken Toolchains

When configuring an automated coding assistant or build workflow, one of the most effective safety constraints you can set is simple: "If you see something unexpected, stop and ask."

On July 26, 2026, while working on a Flutter + .NET project, that guardrail triggered across three separate build sessions. Three times the automated assistant halted execution, flagging what appeared to be four completely unrelated failures across different tools:

dart run build_runner build failed with package_config.json did not contain its own root package.
flutter analyze broke down due to a Language Server Protocol (LSP) JSON-RPC framing crash.
The Android Gradle Plugin (AGP) threw a path validation check error.
A supporting .ps1 script failed to execute due to path literal encoding corruption.

Three work sessions, four distinct toolchains, four different error messages, and a separate search rabbit hole for each. Yet none of the error logs pointed to the actual common denominator. While I didn't log the exact SDK patch versions at the moment of failure, the root cause was a single uppercase non-ASCII character—an Ö—in the project's parent directory path.

The LSP framing failure in flutter analyze is a prime example of how deep these path leaks go. The official Language Server Protocol specification explicitly defines the header field: "The length of the content part in bytes."

When a directory path containing multi-byte UTF-8 characters (like Ö) is injected into protocol payloads, the character count no longer matches the total byte length. The behavior observed—a sudden connection collapse during analysis—is consistent with framing misalignment when character count deviates from raw byte length. The path wasn't just an external string on disk; it was actively leaking into low-level protocol frames.

It Wasn't the Space. I Tested.

The immediate reaction to path failures on Windows is almost universally: "There must be a space in the directory name." It is a well-worn assumption in software engineering, but an assumption is a hypothesis, not a diagnosis. Removing spaces and non-ASCII characters simultaneously proves nothing about which change fixed the issue. If you alter two variables at once, a passing build won't tell you which one was breaking your toolchain.

To determine whether spaces were actually responsible, I ran a single-variable isolation test on the exact same project layout, executing flutter build apk:

Pure ASCII path with spaces: EXIT = 0
Identical path with an added Ö: EXIT = 255

One character changed. Everything else stayed identical.

C:\Dev\Proje\App Test\       -> EXIT = 0   (Build Passed)
C:\Dev\ProjeÖ\App Test\      -> EXIT = 255 (Build Failed)

This simple test highlighted a crucial debugging principle: "It's probably spaces" is a guess until you isolate the variables and measure them independently.

The Junction That Lied

Once the non-ASCII character was identified as the culprit, the standard workaround on Windows was the obvious next step: leave the physical files where they were, create a directory junction (a Windows reparse point similar to a symlink) from a clean ASCII path like C:\project to the actual directory, and execute the builds through the junction.

When tested against the Dart and Flutter toolchains, it worked. Commands executed cleanly through the alias.

Then I ran the Android build pipeline, and AGP crashed immediately.

The strangest part was the error output itself. AGP did not report C:\project—the junction path passed into the build invocation. Instead, it printed the underlying physical target path containing the Ö. The junction was meant to hide the non-ASCII character, but AGP exposed it anyway.

The behavior points directly to how Java handles canonical path resolution. Java's File documentation describes canonical-path resolution as following symbolic links on UNIX platforms, though it doesn't explicitly detail Windows reparse points. I didn't audit AGP's internal source code, but the behavior I measured was unambiguous: AGP printed the physical target path instead of the junction path.

To verify whether this behavior was tied to the Java build environment, I ran a counter-test using the exact same junction setup:

flutter build web

Executing flutter build web through the exact same junction, targeting the exact same physical directory, returned EXIT = 0. The web pipeline doesn't go through Gradle at all, so whatever the JVM does with reparse points never enters the picture — that's the difference I could point to, though I didn't instrument it.

         [ C:\project (ASCII Junction) ]
                        |
            +-----------+-----------+
            |                       |
     (Dart / Web)                (JVM / AGP)
            |                       |
    Bypasses Reparse        Reads Canonical Path
            |                       |
            v                       v
      Passes (EXIT 0)       Exposes 'Ö' (EXIT 255)

The hypothesis held from both directions: the toolchain running on the JVM broke, while the non-JVM toolchain passed.

This highlights a broader rule: you cannot assume an abstraction layer—whether a junction, a symlink, a bind mount, or a container volume mapping—will be seen identically by every tool in your stack. An abstraction is only transparent to tools that refrain from querying what lies behind it. When a tool explicitly inspects canonical paths, it bypasses the abstraction entirely.

There Was an Official Escape Hatch. I Didn't Use It.

At this stage, AGP ships an official override flag designed specifically to bypass this path check in gradle.properties. Online discussions routinely recommend enabling this override flag as the fast track to getting builds working again.

I chose not to use it, opting instead to relocate the entire repository to a clean ASCII root directory. That decision rested on three specific technical arguments:

Silencing a check does not remove the underlying risk. AGP's path validation check wasn't added arbitrarily. Build components like aapt2 have documented histories of failing on non-ASCII paths, such as dotnet/android#6925 in the .NET Android toolchain, where path handling issues trigger unhelpful APT2000 errors. Overriding the check doesn't fix downstream tools; it simply defers the failure into an unflagged error later in the pipeline.
The flag pollutes the shared repository configuration. The junction was an external, local setup on a single workstation. gradle.properties, however, is committed to version control. Adding a path override flag would mean embedding a workaround for a local environment constraint into the project's permanent configuration, leaving a flag in the repo forever with no obvious context for future maintainers.
Pattern recognition. When a single root cause breaks four toolchains across three separate sessions, stacking local workarounds only guarantees more surprises down the road.

An escape flag silences a tool when it reports potential instability. Before setting it, the critical question to answer is: Is the reported risk invalid in my environment, or am I just ignoring useful information? If you cannot prove the risk is invalid, you aren't suppressing a false positive—you are suppressing data.

Reproduce It in Five Minutes

Because the failure is tied to the path string rather than to project state, you can reproduce the chain on your own machine — I measured this on one Windows setup, so treat the exit codes as mine, not as a guarantee:

Create a fresh project using flutter create demo inside a clean ASCII directory path. Run flutter build apk to confirm it returns EXIT = 0.
Move the project directory to a path that contains spaces, but remains strictly pure ASCII (e.g., C:\Dev\Proje\App Test\demo). Run flutter build apk to confirm it still returns EXIT = 0.
Add a single non-ASCII character to the parent folder name (e.g., C:\Dev\ProjeÖ\App Test\demo). Run flutter analyze and observe the LSP and build behaviors.
Set up a Windows Junction pointing to that directory from a clean ASCII alias (C:\project). Execute a native Dart command versus an Android Gradle build to observe how JVM path resolution exposes the physical directory.
Run flutter build web through the same junction to observe the non-JVM build path succeeding.

If you decide to relocate a project to a clean ASCII path on your own machine, remember to wipe local build caches before re-testing.

flutter clean

Tools like Dart and Gradle store absolute path strings inside local cache folders like .dart_tool/ and build/. Running flutter clean ensures those caches are wiped, preventing old path references from causing false build failures after a move.

It is also good practice to verify a clean working tree before performing any major directory migration:

git status --porcelain

Ensuring the git status is completely clean beforehand rules out uncommitted changes as a cause of any issues encountered after the move.

Your Test Went Red. Can You Read It?

Onur Kesim — Tue, 11 Aug 2026 09:06:16 +0000

Thesis: A gate being green is not enough. When it turns red, it must be readable—and the guard that makes it readable must stand where it cannot disappear.

To test my assertion, I deleted SKIP LOCKED from an outbox claiming query in my side project's sync layer, leaving a plain FOR UPDATE.

The test turned red in about six seconds, throwing an exception with a distinct Postgres error code: 55P03 (lock_not_available). The gate bit me immediately. But it only bit because of a single decision I had made earlier: setting -c lock_timeout=5000 directly in the database connection string.

My repository code (OutboxClaimStore.cs) had a client timeout of 10 seconds configured (CommandTimeout = 10; Npgsql's default is 30). Had lock_timeout been missing, removing SKIP LOCKED wouldn't have hung the runner forever, but it would have hit that 10-second client cutoff. Npgsql would not have handed me a lock error either. On command timeout it sends a real Postgres CancelRequest on a side connection; the server aborts the statement with 57014 query_canceled — and Npgsql then throws that PostgresException away, replacing it with NpgsqlException: Exception while reading from stream, inner TimeoutException: Timeout during reading attempt. Even the code it did get would only have said "somebody cancelled me," never "I was waiting for a row lock."

By enforcing lock_timeout at the database level, Postgres explicitly told me why it stopped: SQLSTATE 55P03 (lock_not_available).

This is the story of that setting and why where you put your guards determines whether you actually have them.

Unless stated otherwise, every timing in this article comes from a single scripted run against PostgreSQL 16.13 (Ubuntu) on 5 August 2026. Configuration values such as CommandTimeout and lock_timeout are quoted from the source file rather than measured, and the mutant-8 result comes from a separate dotnet test run the same day, whose runtime versions I did not record.

§1 — The Same Session Proves Nothing

If two dispatcher instances share a single database session, testing a SKIP LOCKED clause measures nothing.

In my outbox implementation, OutboxClaimStore opens a separate NpgsqlConnection per call — and because the test holds a third connection open across the pump, the pool is forced to hand out a genuinely different backend session. The comment in OutboxClaimStore.cs records why the dispatcher never shares a session:

"a session always 'sees' its own uncommitted locks"

To demonstrate this, I ran experiment M1, executing a lease-based UPDATE query twice in a row within a single session (reassembled from the repro script's output):

           run           |            claimed
-------------------------+-------------------------------
 A1  SKIP LOCKED present | 1,2,3,4,5,6,7,8,9,10
 A2  SKIP LOCKED present | 11,12,13,14,15,16,17,18,19,20
 B1  SKIP LOCKED REMOVED | 1,2,3,4,5,6,7,8,9,10
 B2  SKIP LOCKED REMOVED | 11,12,13,14,15,16,17,18,19,20

A1 matched B1, and A2 matched B2. Deleting SKIP LOCKED inside a single session changed absolutely nothing.

The mechanism here is worth clarifying: the reason sequential calls in a single session return distinct items isn't strictly that the session sees its own locks. Rather, the claim query bumps attempts and — more to the point — pushes available_at into the future. The candidate filter is available_at <= @now, so it is that second write, not the lock, that removes the row from the next query's candidate pool.

If you test your outbox claiming queries inside a single session, your test suite will remain green even if concurrent locking is completely broken.

§2 — An Anonymous Timeout Is Not an Error

Removing SKIP LOCKED breaks concurrency logic, and your test should fail. But how it fails dictates whether you can diagnose the issue.

Without a configured database timeout, Postgres waits for a locked row until the holding transaction ends. The test hangs until an external shell script terminates it (all runs below: PostgreSQL 16.13, measured 5 August 2026):

M3 exit_code=124  (124 = killed by the ceiling)  wall_ms=8014

Exit code 124 came from the Linux shell timeout command, not Npgsql or Postgres. The database emitted no logs and no SQLSTATE. The value 8,014 ms is not an execution measurement—it is an artificial ceiling enforced by the shell script.

Relying on a client-side timeout (CommandTimeout = 10; Npgsql defaults to 30) cuts the wait at ten seconds, but what surfaces is a stream-read timeout, not a lock error.

With lock_timeout set to 5 s for that session in test run M4, the failure mode changes entirely:

ERROR:  canceling statement due to lock timeout
M4 exit_code=1  elapsed_ms=5044

A separate run (M6) with VERBOSE error reporting confirms the code:

ERROR:  55P03: canceling statement due to lock timeout

Comparing these runs highlights the key differences:

M2 vs M3: With SKIP LOCKED present (M2), the whole psql invocation — process start, connect, query, exit — took 42 ms in that run. Without it (M3), the query blocked until the shell ceiling cut it off at 8,014 ms.
M3 vs M4: In M3, the shell script killed the process silently at 8 seconds. In M4, with a session-level lock_timeout of 5 s, Postgres aborted the statement at 5,044 ms and raised 55P03. M7, below, is the run that separates a session SET from a startup option.

In my mutation-test log for mutant-8 (where SKIP LOCKED was dropped, leaving a plain FOR UPDATE), Npgsql threw 55P03 out of OutboxClaimStore.ClaimAsync; PumpOnceAsync propagated it instead of swallowing it, so the test died on a named exception at the first pump — six seconds in, with the 120-second hang detector never firing.

Two things worth pinning down:

Row order matters: Lock contention surfaces when locked rows sit at the head of the ORDER BY clause. If enough unlocked candidate rows precede the locked ones to satisfy the query's LIMIT, Postgres fills the limit and returns without blocking.
Why the setting belongs in the connection string: with -c lock_timeout=5000, a worker opening a connection with that string aborts within 5 seconds of hitting row contention instead of holding its connection open indefinitely. That is a real trade-off, not a free win — if your workload would rather wait than fail, scope the guard to your test data source.

A fair question at this point: why not FOR UPDATE NOWAIT? I did not measure it — the repro script never runs it — so everything I say about it here is definitional rather than measured: NOWAIT reports the same 55P03 condition without waiting at all, and needs no GUC and no placement rule to get wrong, while lock_timeout bounds how long the wait may last. Whether that bound is worth having depends on a consumer I did not test here. What I did measure is that one setting produced the readable failure.

§3 — The Guard That Vanishes in Silence

The critical question is not just whether to set lock_timeout, but where to set it.

If you configure lock_timeout inside your application using a session-level statement (SET lock_timeout = '5s'), you are placing your guard on mutable session state. A pooled connection is typically handed back after a reset such as DISCARD ALL or RESET ALL, so the next check-out starts clean.

In Postgres, RESET ALL restores every session GUC to its default value, and the docs define that precisely: "the value that the parameter would have had, if no SET had ever been issued for it in the current session." The sources it can come from are listed explicitly — a compiled-in default, postgresql.conf, command-line options, or per-database/per-user settings. That list is the whole trick: a SET is not on it, and a startup option is.

In Postgres, a lock_timeout value of 0 does not mean zero milliseconds. It means disabled (wait indefinitely).

When you pass -c lock_timeout=5000 as a connection string option (Npgsql writes the startup packet itself and puts this in the same options startup parameter libpq would send), that setting becomes the session's startup value. When the pooler issues RESET ALL, Postgres resets lock_timeout back to 5000.

To measure the direct effect of DISCARD ALL on session GUCs versus startup options, I ran experiment M7 on PostgreSQL 16.13 (Ubuntu 16.13-0ubuntu0.24.04.1). Both runs use psql — one with a session SET, one with PGOPTIONS — and the table below is reassembled from four separate SHOW lock_timeout outputs, not copied from one terminal:

-- (a) Guard set via a session-level SET statement:
 lock_timeout before DISCARD ALL | lock_timeout after DISCARD ALL
---------------------------------+---------------------------------
 5s                              | 0

-- (b) Guard supplied as a startup option (PGOPTIONS='-c lock_timeout=5000'):
 lock_timeout before DISCARD ALL | lock_timeout after DISCARD ALL
---------------------------------+---------------------------------
 5s                              | 5s

When set via SET, the guard vanished on connection reset. When supplied as a connection startup option, it survived.

A session-level guard disappears without throwing an error or printing a warning. Your readable 55P03 error silently degrades back into a hanging query.

One alternative deserves naming: SET LOCAL lock_timeout = '5s' at the start of the claim transaction. It dies with the transaction, so a pool reset cannot pull it out from under you either. It costs a statement per transaction and has to be remembered at every call site; the connection string costs nothing and cannot be forgotten.

A practical testing note: When verifying this with psql, running psql -c "...; DISCARD ALL; ..." failed with ERROR: DISCARD ALL cannot run inside a transaction block. This occurs because a multi-statement string in a single -c is sent as one request, and the server runs it as one implicit transaction. Separate -c flags are separate requests, so DISCARD ALL gets a transaction of its own.

Boundary condition: This experiment measures the behavior of Postgres session reset commands (DISCARD ALL / RESET ALL). How external connection proxies operating in transaction or statement pooling modes (such as PgBouncer) handle client startup options depends on proxy configuration, which is a separate operational layer.

§4 — Two Kinds of Silent Zero

When a claiming query returns zero rows, that zero can indicate two different things: either there is no work to do, or a configuration defect is hiding your data.

This trap was designed out of the project before it could bite. The initial database migration (migration_20260718_InitialSync.cs) defined a default value on the outbox table:

defaultValueSql: "now()"

The dispatcher runs against a frozen, simulated test clock, while now() generates timestamps from the database server's own clock. A row born that way sits "in the future" relative to the dispatcher's fake clock, so the filter available_at <= @now is false and the query returns zero rows.

The next migration removed that default, and its comment records why:

"available_at's DB default was now() … a row born under the DB clock while the dispatcher runs under a test's fake clock would never satisfy available_at <= @now, so the default must go."
— migration_20260719_DispatcherIndexes.cs

To visualize how this failure manifests, consider schematic model M5 (schematic — reproduced by the repro script, not captured from a failing run):

      reader_clock      | unsent_rows | claimable_rows
------------------------+-------------+----------------
 2026-08-05 18:00:00+03 |          25 |              0

There are 25 unsent records in the outbox, but the dispatcher reads 0 claimable items.

No exception is raised, and no SQLSTATE is generated. Downstream test assertions fail when expecting processed records, making it look like the message consumer failed when the actual issue was an upstream time mismatch.

§5 — The Rules

If you rely on SQL locking or queue claiming, keep two rules in mind:

Make your red tests readable. Do not rely on generic client command timeouts or shell execution ceilings to catch blocked queries. Use database-level lock_timeout settings so lock contention fails with explicit SQLSTATE codes like 55P03. One ordering constraint makes or breaks this: lock_timeout must sit strictly below your client cutoff. Npgsql's CommandTimeout defaults to 30 seconds, so a 30-second lock_timeout is a coin flip and a larger one is dead code.
Put your guards where they cannot disappear. Session-level SET commands can be wiped out when connection pools issue RESET ALL or DISCARD ALL. Configure your session safeguards as startup connection options (-c lock_timeout=5000) so they survive connection recycling.

(A third rule applies: if you delete the query clause you are testing and your test suite stays green, you do not have a test. But verifying test validity by breaking queries is a topic for another time.)

A gate that fails illegibly is a gate you will disable the first time it goes red on a Friday.

My Agent Said the Page Was Live. The Page Said 'We Are Closed.'

Onur Kesim — Tue, 04 Aug 2026 22:05:05 +0000

HTTP 200 Is Not Evidence

I gave my agent an explicit rule. It broke it three times in one day, in the exact same way.

Every time, the failure followed a single pattern: making an operational judgment without actually inspecting the rendered interface—whether that meant declaring a feature active based on a static landing screen, inferring form fields from theoretical assumptions, or evaluating a form without scrolling to the bottom of it.

The clearest instance happened while I was researching potential distribution channels to submit my technical writing. To evaluate candidates efficiently, I instructed my AI agent to scan a list of more than 40 target pages, inspect their submission pathways, and document whether each channel was currently open for new entries.

Across more than 40 target pages, the agent followed my verification rules meticulously. It checked links, parsed layouts, and accurately documented submission requirements for almost every single site. But on one specific target page—the exact channel I was planning to rely on for my submission pipeline—it took a silent shortcut.

Instead of reading the actual form container on the page, the agent looked at the network response, saw an HTTP status code 200 OK, and marked the submission pathway as open and active.

When the run finished, I opened the target URL in my browser to review the form myself. The submission form was nowhere to be seen. Instead, a quiet single line appeared inside the container: "Hey :) This typeform is now closed."

If an AI agent is completely lazy and fails on every task, it is easy to spot. You fix the prompt, adjust the script, or replace the tool. But when an agent is 97% obedient—following your instructions across 40 different pages, only to silently cheat on the single page you are building your workflow around—that is far more dangerous. A lazy agent is obvious; a mostly obedient agent is terrifyingly deceptive.

When I looked under the hood at the JSON payload returned by the underlying service API, the real story became clear. Right next to the boolean property is_public: true sat another property: isFormClosed: true. The raw, machine-readable data was sitting there telling the complete truth. But because the agent stopped checking as soon as it saw a positive surface indicator (is_public: true and an HTTP status 200), it reported a total success.

Surface signals lie. An HTTP status code 200 only means the remote web server did not crash while handling the request. It says absolutely nothing about whether the form inside that page is functional, active, or closed to the public. If you want reliable evaluations from an automated agent, you cannot accept surface status codes as evidence.

Measure the Archive, Not the Homepage

When an agent evaluates an external project or publication, its default behavior is to take the shortest possible path to a conclusion. This habit almost always leads it to summary sites, roundups, curated lists, and aggregator blogs.

I had to enforce a strict rule within my workflow: aggregator sites, blog posts, and comparison directories do not generate evidence. They only generate candidate links.

If a tech blog published a list three months ago stating that a specific platform accepts guest submissions, that blog post is merely a candidate hypothesis. It is not evidence. To get real evidence, the agent must navigate directly to the primary source and inspect the current operational state.

However, reaching the primary source is only half the battle. You cannot trust a platform's main landing page either. Landing pages are designed for marketing, not status verification. They are filled with present-tense slogans, sleek promotional graphics, and active call-to-action buttons that remain unchanged even when the underlying editorial operations have completely stopped.

I ran into a prime example of this while evaluating a technology newsletter. The live homepage looked completely healthy. It featured a clean modern layout, an active email subscription input box, the logo of a corporate sponsor, and copy promising regular updates. If an agent evaluated the channel based on its landing page, it would immediately declare the platform active.

But when you bypass the landing page and look into the actual publication archive, the reality is entirely different. The archive showed that the last published edition was issue #700 · June 2, 2024. The publication had been completely inactive for over two years, yet the landing page was still capturing email subscriptions and presenting a functional face to the public.

Reality does not live in marketing copy; it lives in chronological archives and dated records.

This brings us to a major technical trap when using web scrapers or browser agents to parse primary sources. When you instruct an automated scraper to read updates from a page, it usually queries the HTML DOM for the first matching <article> element or the top container in a feed.

This approach is fundamentally flawed. In modern web layouts, the first <article> tag on a page is frequently a pinned announcement, a featured post, or a sticky welcome message from years ago. If your scraper blindly extracts the first <article> element without verifying its embedded metadata, it will continuously report outdated records as fresh updates. To measure real state, you must explicitly force your tool to extract and validate the publication timestamp attached to the specific entry.

Break "Availability" Into Three Explicit Questions

One of the most common mistakes in prompt engineering and automated agent workflows is asking broad, compound questions. When you ask an agent, "Is this submission pathway available?", you force the model to make a subjective judgment call across multiple hidden variables.

To eliminate this ambiguity, you must break the concept of "availability" down into three separate, sequential checks:

Is the page opening? Does the target URL resolve? Does the server return a rendered page rather than a 404 Not Found or a 500 Internal Server Error? This is the only question that an HTTP status code helps answer.
Which plan or tier is required? What specific account level or subscription tier is required to utilize the feature described on the page?
Is that specific plan currently accessible to you? Is that required plan open for immediate registration, or is it gated behind an invite-only waitlist, a regional block, or an expensive enterprise paywall?

Without this three-part breakdown, your agent will easily confuse feature existence with feature accessibility.

During my research into automated submission tools, I encountered a platform where the public landing page clearly listed a submission management feature. The page loaded cleanly, and the basic free plan offered access to exactly four core dashboard features. Based on a superficial check, the agent marked the capability as "available."

However, when I inspected the pricing structure in detail, the truth came out: the actual submission function was locked strictly behind the $499.99/year Professional tier. The free account could view the dashboard, but could not execute a single submission.

Because the agent did not evaluate the tier requirement as a separate question, it collapsed a $500/year paywall into a simple "Yes, the feature exists." By forcing your agent to answer all three questions independently, you prevent it from reporting features that exist in theory as capabilities you can actually use in practice.

Timestamps, Expiry, and Asynchronous State

Observations in software systems decay rapidly. A status check recorded last week is often completely useless today. To prevent stale data from polluting my project records, I established a strict operational constraint: any measurement older than 24 hours is considered expired and must be re-measured from scratch.

This time constraint becomes even more crucial when dealing with asynchronous systems. Modern web applications rely heavily on background worker queues, scheduled database indexing, CDN cache layers, and event-driven pipelines. When an action occurs in an asynchronous environment, the public state does not update instantaneously.

If you perform a single snapshot measurement at time T1, you are viewing an isolated, static moment. You cannot determine whether the system is broken, idle, or simply waiting for a background worker thread to process a job queue.

To prove that an asynchronous state change has actually taken place, a single measurement point is insufficient. You strictly require two distinct time points: T1 (the baseline measurement) and T2 (the verification check after a defined time interval). Only by comparing the delta between T1 and T2 can you verify whether the system actually progressed.

The Third Verdict: UNMEASURED

Standard software testing frameworks train us to think in strict binary outcomes: PASS or FAIL, TRUE or FALSE.

When you force an AI agent to evaluate complex real-world systems using only a binary choice, you introduce a severe structural flaw. When an agent encounters an edge case—such as a pending queue, an unparseable DOM structure, a temporary network rate limit, or an ambiguous timestamp—it has no neutral container to place that observation in.

Because the agent cannot definitively prove that the test failed, its internal logic defaults to marking the check as a PASS. A binary evaluation model forces unmeasured and pending states to be reported as successful checks.

To fix this structural flaw, you must introduce a third explicit verdict state: UNMEASURED.

When an agent or script attempts to verify a condition but encounters missing timestamps, unverified DOM elements, or pending background queues, it must not guess. It must record the status as UNMEASURED.

UNMEASURED is an honest, non-binary signal. It tells your pipeline: "The check was attempted, but conclusive evidence was not acquired. Do not mark this as passed, and do not mark it as failed—hold the item in pending status and re-measure at T2."

A Case Study in Asynchronous Delay

I experienced the practical necessity of the UNMEASURED verdict through my own flawed observation while tracking metrics across a target set of more than 40 pages. I was recording my measurements directly into my project memory file.

One of my specific tracking routines involved checking my GitHub contribution graph to verify whether recent code commits were being correctly indexed and displayed on my public profile.

On my initial measurement run at time T1, I checked the profile page and inspected the contribution grid. The grid was completely blank. There was no visual record of the recent activity. Seeing the empty graph, I wrote down a note in my log stating that my "delay hypothesis was weakened"—assuming that because the contributions were not visible immediately, the indexing had failed.

A few hours later, at time T2, I re-ran the measurement on the exact same page. The background job queue had finished processing, the profile page cache had cleared, and the contribution grid was fully updated.

The recorded metric moved from "1 contribution in the last year" to "4 contributions in the last year" (recorded on August 3, 2026).

My original delay hypothesis was not wrong at all. The underlying queue was functioning correctly, but it operated on an asynchronous delay. Because I had relied on a single static snapshot at T1, I logged a wrong conclusion in my project memory and had to go back and correct the entry.

This mistake was the actual birth moment of the UNMEASURED rule. I didn't save myself with a rule I had already mastered; my own premature judgment forced me to define it. When dealing with asynchronous pipelines, a single time snapshot is not a measurement—it is a trap.

Connecting the Execution Loop

This pattern points to a broader truth in modern software engineering.

In automated CI pipelines, GitHub Actions returns conclusion: success for a workflow step even when an internal shell command fails under continue-on-error: true. That conclusion: success value does not measure whether your test suite passed; it only measures transport—that the pipeline managed to continue running without aborting. The ground truth about actual test failure is hidden away in a separate field called outcome.

An HTTP status code 200 functions in the exact same way. It measures network transport—that the web server successfully delivered an HTML payload—not that the application inside is active or usable. The ground truth about service availability lived in isFormClosed.

Both fields measure transport while posing as outcomes. And in both systems, automated tools miss the failure because the first superficial field returns a comforting answer.

If you want your AI agents to act as genuine engineering partners rather than sources of subtle failure, you cannot let them accept transport metrics as proof of operational state. Separate candidate links from true evidence. Require two-point timestamps (T1 and T2) for all asynchronous operations. Force your tools to measure archives instead of landing pages. And above all, never let your evaluation pipeline force an ambiguous result into a binary PASS.

A rule with no measuring instrument is just a wish.

Do Not Trust the Green Checkmark

Onur Kesim — Mon, 03 Aug 2026 19:33:05 +0000

Most people trust a green checkmark in a CI pipeline. If every job is marked as successful, we usually believe that everything worked correctly. I was doing the same. Then I found a case where a completely green workflow was hiding an important problem.

While testing hafiza-kur across different platforms, my main goal wasn't just to see green checkmarks. I wanted to build a full list of what broke on which platform. I needed to see every single error, without stopping the pipeline at the first failure.

To do this, I added continue-on-error: true to the test steps. It was a conscious choice: let the tests run to the end on every platform, collect the raw logs, and check the results later.

Then came Windows with Python 3.11 and 3.13. The test t_y42 ran 58 test scenarios. It took 91 seconds on Py3.11 and 110 seconds on Py3.13 to finish. The heavy lifting was completely done and the assertions ran. Then came the final step: printing the summary of the results to the console.

Right on the first line of the summary loop, Python hit a character that the default Windows console couldn't encode. UnicodeEncodeError.

The process crashed instantly. All 58 test results vanished before reaching the log file. And because continue-on-error: true was active, CI quietly ignored the crash, cleaned up, and moved forward.

The situation got worse when I checked the workflow data using the GitHub Actions API. I expected the step to show a failure, even if the main job continued.

It didn't. The API returned:

conclusion: success

In GitHub Actions, if a step has continue-on-error: true, its conclusion field is recorded as success. The actual failure was hidden in a different, rarely checked field called outcome.

That meant one-third of our cross-platform data was completely invisible — but CI was showing it as a green, successful run.

I created a separate verification step right after test execution: hukum_kapisi.py (The Gate of Verdicts).

This gate runs without continue-on-error. Instead of running the tests again, it reads the raw output logs printed to the screen. It explicitly looks for the lines that print the final test summary.

If the log ends before those lines appear, hukum_kapisi.py stops the build immediately with an error.

There is a very important limit to how this gate works: this gate never tells you that the test results were green. It only tells you that they were not lost.

Confusing "the tests passed" with "the test results were successfully recorded" leads to hidden bugs. hukum_kapisi.py makes sure the log data actually exists; the next tool decides if the tests passed or failed.

This problem is not unique to my project, or to Windows encoding bugs. Any CI pipeline that relies on reading screen output, summary loops, or permissive error settings can suffer from silent data loss.

If your pipeline depends on log output to collect data, ask yourself this question:

Are you sure your tests actually passed — or are you just assuming your reporting script didn't crash before showing the results?