DEV Community: Onur Kesim

One Letter Broke Four Toolchains. A Symlink Didn't Fix It.

Onur Kesim — Tue, 18 Aug 2026 08:51:41 +0000

Three Sessions, Four Broken Toolchains

When configuring an automated coding assistant or build workflow, one of the most effective safety constraints you can set is simple: "If you see something unexpected, stop and ask."

On July 26, 2026, while working on a Flutter + .NET project, that guardrail triggered across three separate build sessions. Three times the automated assistant halted execution, flagging what appeared to be four completely unrelated failures across different tools:

dart run build_runner build failed with package_config.json did not contain its own root package.
flutter analyze broke down due to a Language Server Protocol (LSP) JSON-RPC framing crash.
The Android Gradle Plugin (AGP) threw a path validation check error.
A supporting .ps1 script failed to execute due to path literal encoding corruption.

Three work sessions, four distinct toolchains, four different error messages, and a separate search rabbit hole for each. Yet none of the error logs pointed to the actual common denominator. While I didn't log the exact SDK patch versions at the moment of failure, the root cause was a single uppercase non-ASCII character—an Ö—in the project's parent directory path.

The LSP framing failure in flutter analyze is a prime example of how deep these path leaks go. The official Language Server Protocol specification explicitly defines the header field: "The length of the content part in bytes."

When a directory path containing multi-byte UTF-8 characters (like Ö) is injected into protocol payloads, the character count no longer matches the total byte length. The behavior observed—a sudden connection collapse during analysis—is consistent with framing misalignment when character count deviates from raw byte length. The path wasn't just an external string on disk; it was actively leaking into low-level protocol frames.

It Wasn't the Space. I Tested.

The immediate reaction to path failures on Windows is almost universally: "There must be a space in the directory name." It is a well-worn assumption in software engineering, but an assumption is a hypothesis, not a diagnosis. Removing spaces and non-ASCII characters simultaneously proves nothing about which change fixed the issue. If you alter two variables at once, a passing build won't tell you which one was breaking your toolchain.

To determine whether spaces were actually responsible, I ran a single-variable isolation test on the exact same project layout, executing flutter build apk:

Pure ASCII path with spaces: EXIT = 0
Identical path with an added Ö: EXIT = 255

One character changed. Everything else stayed identical.

C:\Dev\Proje\App Test\       -> EXIT = 0   (Build Passed)
C:\Dev\ProjeÖ\App Test\      -> EXIT = 255 (Build Failed)

This simple test highlighted a crucial debugging principle: "It's probably spaces" is a guess until you isolate the variables and measure them independently.

The Junction That Lied

Once the non-ASCII character was identified as the culprit, the standard workaround on Windows was the obvious next step: leave the physical files where they were, create a directory junction (a Windows reparse point similar to a symlink) from a clean ASCII path like C:\project to the actual directory, and execute the builds through the junction.

When tested against the Dart and Flutter toolchains, it worked. Commands executed cleanly through the alias.

Then I ran the Android build pipeline, and AGP crashed immediately.

The strangest part was the error output itself. AGP did not report C:\project—the junction path passed into the build invocation. Instead, it printed the underlying physical target path containing the Ö. The junction was meant to hide the non-ASCII character, but AGP exposed it anyway.

The behavior points directly to how Java handles canonical path resolution. Java's File documentation describes canonical-path resolution as following symbolic links on UNIX platforms, though it doesn't explicitly detail Windows reparse points. I didn't audit AGP's internal source code, but the behavior I measured was unambiguous: AGP printed the physical target path instead of the junction path.

To verify whether this behavior was tied to the Java build environment, I ran a counter-test using the exact same junction setup:

flutter build web

Executing flutter build web through the exact same junction, targeting the exact same physical directory, returned EXIT = 0. The web pipeline doesn't go through Gradle at all, so whatever the JVM does with reparse points never enters the picture — that's the difference I could point to, though I didn't instrument it.

         [ C:\project (ASCII Junction) ]
                        |
            +-----------+-----------+
            |                       |
     (Dart / Web)                (JVM / AGP)
            |                       |
    Bypasses Reparse        Reads Canonical Path
            |                       |
            v                       v
      Passes (EXIT 0)       Exposes 'Ö' (EXIT 255)

The hypothesis held from both directions: the toolchain running on the JVM broke, while the non-JVM toolchain passed.

This highlights a broader rule: you cannot assume an abstraction layer—whether a junction, a symlink, a bind mount, or a container volume mapping—will be seen identically by every tool in your stack. An abstraction is only transparent to tools that refrain from querying what lies behind it. When a tool explicitly inspects canonical paths, it bypasses the abstraction entirely.

There Was an Official Escape Hatch. I Didn't Use It.

At this stage, AGP ships an official override flag designed specifically to bypass this path check in gradle.properties. Online discussions routinely recommend enabling this override flag as the fast track to getting builds working again.

I chose not to use it, opting instead to relocate the entire repository to a clean ASCII root directory. That decision rested on three specific technical arguments:

Silencing a check does not remove the underlying risk. AGP's path validation check wasn't added arbitrarily. Build components like aapt2 have documented histories of failing on non-ASCII paths, such as dotnet/android#6925 in the .NET Android toolchain, where path handling issues trigger unhelpful APT2000 errors. Overriding the check doesn't fix downstream tools; it simply defers the failure into an unflagged error later in the pipeline.
The flag pollutes the shared repository configuration. The junction was an external, local setup on a single workstation. gradle.properties, however, is committed to version control. Adding a path override flag would mean embedding a workaround for a local environment constraint into the project's permanent configuration, leaving a flag in the repo forever with no obvious context for future maintainers.
Pattern recognition. When a single root cause breaks four toolchains across three separate sessions, stacking local workarounds only guarantees more surprises down the road.

An escape flag silences a tool when it reports potential instability. Before setting it, the critical question to answer is: Is the reported risk invalid in my environment, or am I just ignoring useful information? If you cannot prove the risk is invalid, you aren't suppressing a false positive—you are suppressing data.

Reproduce It in Five Minutes

Because the failure is tied to the path string rather than to project state, you can reproduce the chain on your own machine — I measured this on one Windows setup, so treat the exit codes as mine, not as a guarantee:

Create a fresh project using flutter create demo inside a clean ASCII directory path. Run flutter build apk to confirm it returns EXIT = 0.
Move the project directory to a path that contains spaces, but remains strictly pure ASCII (e.g., C:\Dev\Proje\App Test\demo). Run flutter build apk to confirm it still returns EXIT = 0.
Add a single non-ASCII character to the parent folder name (e.g., C:\Dev\ProjeÖ\App Test\demo). Run flutter analyze and observe the LSP and build behaviors.
Set up a Windows Junction pointing to that directory from a clean ASCII alias (C:\project). Execute a native Dart command versus an Android Gradle build to observe how JVM path resolution exposes the physical directory.
Run flutter build web through the same junction to observe the non-JVM build path succeeding.

If you decide to relocate a project to a clean ASCII path on your own machine, remember to wipe local build caches before re-testing.

flutter clean

Tools like Dart and Gradle store absolute path strings inside local cache folders like .dart_tool/ and build/. Running flutter clean ensures those caches are wiped, preventing old path references from causing false build failures after a move.

It is also good practice to verify a clean working tree before performing any major directory migration:

git status --porcelain

Ensuring the git status is completely clean beforehand rules out uncommitted changes as a cause of any issues encountered after the move.

Your Test Went Red. Can You Read It?

Onur Kesim — Tue, 11 Aug 2026 09:06:16 +0000

Thesis: A gate being green is not enough. When it turns red, it must be readable—and the guard that makes it readable must stand where it cannot disappear.

To test my assertion, I deleted SKIP LOCKED from an outbox claiming query in my side project's sync layer, leaving a plain FOR UPDATE.

The test turned red in about six seconds, throwing an exception with a distinct Postgres error code: 55P03 (lock_not_available). The gate bit me immediately. But it only bit because of a single decision I had made earlier: setting -c lock_timeout=5000 directly in the database connection string.

My repository code (OutboxClaimStore.cs) had a client timeout of 10 seconds configured (CommandTimeout = 10; Npgsql's default is 30). Had lock_timeout been missing, removing SKIP LOCKED wouldn't have hung the runner forever, but it would have hit that 10-second client cutoff. Npgsql would not have handed me a lock error either. On command timeout it sends a real Postgres CancelRequest on a side connection; the server aborts the statement with 57014 query_canceled — and Npgsql then throws that PostgresException away, replacing it with NpgsqlException: Exception while reading from stream, inner TimeoutException: Timeout during reading attempt. Even the code it did get would only have said "somebody cancelled me," never "I was waiting for a row lock."

By enforcing lock_timeout at the database level, Postgres explicitly told me why it stopped: SQLSTATE 55P03 (lock_not_available).

This is the story of that setting and why where you put your guards determines whether you actually have them.

Unless stated otherwise, every timing in this article comes from a single scripted run against PostgreSQL 16.13 (Ubuntu) on 5 August 2026. Configuration values such as CommandTimeout and lock_timeout are quoted from the source file rather than measured, and the mutant-8 result comes from a separate dotnet test run the same day, whose runtime versions I did not record.

§1 — The Same Session Proves Nothing

If two dispatcher instances share a single database session, testing a SKIP LOCKED clause measures nothing.

In my outbox implementation, OutboxClaimStore opens a separate NpgsqlConnection per call — and because the test holds a third connection open across the pump, the pool is forced to hand out a genuinely different backend session. The comment in OutboxClaimStore.cs records why the dispatcher never shares a session:

"a session always 'sees' its own uncommitted locks"

To demonstrate this, I ran experiment M1, executing a lease-based UPDATE query twice in a row within a single session (reassembled from the repro script's output):

           run           |            claimed
-------------------------+-------------------------------
 A1  SKIP LOCKED present | 1,2,3,4,5,6,7,8,9,10
 A2  SKIP LOCKED present | 11,12,13,14,15,16,17,18,19,20
 B1  SKIP LOCKED REMOVED | 1,2,3,4,5,6,7,8,9,10
 B2  SKIP LOCKED REMOVED | 11,12,13,14,15,16,17,18,19,20

A1 matched B1, and A2 matched B2. Deleting SKIP LOCKED inside a single session changed absolutely nothing.

The mechanism here is worth clarifying: the reason sequential calls in a single session return distinct items isn't strictly that the session sees its own locks. Rather, the claim query bumps attempts and — more to the point — pushes available_at into the future. The candidate filter is available_at <= @now, so it is that second write, not the lock, that removes the row from the next query's candidate pool.

If you test your outbox claiming queries inside a single session, your test suite will remain green even if concurrent locking is completely broken.

§2 — An Anonymous Timeout Is Not an Error

Removing SKIP LOCKED breaks concurrency logic, and your test should fail. But how it fails dictates whether you can diagnose the issue.

Without a configured database timeout, Postgres waits for a locked row until the holding transaction ends. The test hangs until an external shell script terminates it (all runs below: PostgreSQL 16.13, measured 5 August 2026):

M3 exit_code=124  (124 = killed by the ceiling)  wall_ms=8014

Exit code 124 came from the Linux shell timeout command, not Npgsql or Postgres. The database emitted no logs and no SQLSTATE. The value 8,014 ms is not an execution measurement—it is an artificial ceiling enforced by the shell script.

Relying on a client-side timeout (CommandTimeout = 10; Npgsql defaults to 30) cuts the wait at ten seconds, but what surfaces is a stream-read timeout, not a lock error.

With lock_timeout set to 5 s for that session in test run M4, the failure mode changes entirely:

ERROR:  canceling statement due to lock timeout
M4 exit_code=1  elapsed_ms=5044

A separate run (M6) with VERBOSE error reporting confirms the code:

ERROR:  55P03: canceling statement due to lock timeout

Comparing these runs highlights the key differences:

M2 vs M3: With SKIP LOCKED present (M2), the whole psql invocation — process start, connect, query, exit — took 42 ms in that run. Without it (M3), the query blocked until the shell ceiling cut it off at 8,014 ms.
M3 vs M4: In M3, the shell script killed the process silently at 8 seconds. In M4, with a session-level lock_timeout of 5 s, Postgres aborted the statement at 5,044 ms and raised 55P03. M7, below, is the run that separates a session SET from a startup option.

In my mutation-test log for mutant-8 (where SKIP LOCKED was dropped, leaving a plain FOR UPDATE), Npgsql threw 55P03 out of OutboxClaimStore.ClaimAsync; PumpOnceAsync propagated it instead of swallowing it, so the test died on a named exception at the first pump — six seconds in, with the 120-second hang detector never firing.

Two things worth pinning down:

Row order matters: Lock contention surfaces when locked rows sit at the head of the ORDER BY clause. If enough unlocked candidate rows precede the locked ones to satisfy the query's LIMIT, Postgres fills the limit and returns without blocking.
Why the setting belongs in the connection string: with -c lock_timeout=5000, a worker opening a connection with that string aborts within 5 seconds of hitting row contention instead of holding its connection open indefinitely. That is a real trade-off, not a free win — if your workload would rather wait than fail, scope the guard to your test data source.

A fair question at this point: why not FOR UPDATE NOWAIT? I did not measure it — the repro script never runs it — so everything I say about it here is definitional rather than measured: NOWAIT reports the same 55P03 condition without waiting at all, and needs no GUC and no placement rule to get wrong, while lock_timeout bounds how long the wait may last. Whether that bound is worth having depends on a consumer I did not test here. What I did measure is that one setting produced the readable failure.

§3 — The Guard That Vanishes in Silence

The critical question is not just whether to set lock_timeout, but where to set it.

If you configure lock_timeout inside your application using a session-level statement (SET lock_timeout = '5s'), you are placing your guard on mutable session state. A pooled connection is typically handed back after a reset such as DISCARD ALL or RESET ALL, so the next check-out starts clean.

In Postgres, RESET ALL restores every session GUC to its default value, and the docs define that precisely: "the value that the parameter would have had, if no SET had ever been issued for it in the current session." The sources it can come from are listed explicitly — a compiled-in default, postgresql.conf, command-line options, or per-database/per-user settings. That list is the whole trick: a SET is not on it, and a startup option is.

In Postgres, a lock_timeout value of 0 does not mean zero milliseconds. It means disabled (wait indefinitely).

When you pass -c lock_timeout=5000 as a connection string option (Npgsql writes the startup packet itself and puts this in the same options startup parameter libpq would send), that setting becomes the session's startup value. When the pooler issues RESET ALL, Postgres resets lock_timeout back to 5000.

To measure the direct effect of DISCARD ALL on session GUCs versus startup options, I ran experiment M7 on PostgreSQL 16.13 (Ubuntu 16.13-0ubuntu0.24.04.1). Both runs use psql — one with a session SET, one with PGOPTIONS — and the table below is reassembled from four separate SHOW lock_timeout outputs, not copied from one terminal:

-- (a) Guard set via a session-level SET statement:
 lock_timeout before DISCARD ALL | lock_timeout after DISCARD ALL
---------------------------------+---------------------------------
 5s                              | 0

-- (b) Guard supplied as a startup option (PGOPTIONS='-c lock_timeout=5000'):
 lock_timeout before DISCARD ALL | lock_timeout after DISCARD ALL
---------------------------------+---------------------------------
 5s                              | 5s

When set via SET, the guard vanished on connection reset. When supplied as a connection startup option, it survived.

A session-level guard disappears without throwing an error or printing a warning. Your readable 55P03 error silently degrades back into a hanging query.

One alternative deserves naming: SET LOCAL lock_timeout = '5s' at the start of the claim transaction. It dies with the transaction, so a pool reset cannot pull it out from under you either. It costs a statement per transaction and has to be remembered at every call site; the connection string costs nothing and cannot be forgotten.

A practical testing note: When verifying this with psql, running psql -c "...; DISCARD ALL; ..." failed with ERROR: DISCARD ALL cannot run inside a transaction block. This occurs because a multi-statement string in a single -c is sent as one request, and the server runs it as one implicit transaction. Separate -c flags are separate requests, so DISCARD ALL gets a transaction of its own.

Boundary condition: This experiment measures the behavior of Postgres session reset commands (DISCARD ALL / RESET ALL). How external connection proxies operating in transaction or statement pooling modes (such as PgBouncer) handle client startup options depends on proxy configuration, which is a separate operational layer.

§4 — Two Kinds of Silent Zero

When a claiming query returns zero rows, that zero can indicate two different things: either there is no work to do, or a configuration defect is hiding your data.

This trap was designed out of the project before it could bite. The initial database migration (migration_20260718_InitialSync.cs) defined a default value on the outbox table:

defaultValueSql: "now()"

The dispatcher runs against a frozen, simulated test clock, while now() generates timestamps from the database server's own clock. A row born that way sits "in the future" relative to the dispatcher's fake clock, so the filter available_at <= @now is false and the query returns zero rows.

The next migration removed that default, and its comment records why:

"available_at's DB default was now() … a row born under the DB clock while the dispatcher runs under a test's fake clock would never satisfy available_at <= @now, so the default must go."
— migration_20260719_DispatcherIndexes.cs

To visualize how this failure manifests, consider schematic model M5 (schematic — reproduced by the repro script, not captured from a failing run):

      reader_clock      | unsent_rows | claimable_rows
------------------------+-------------+----------------
 2026-08-05 18:00:00+03 |          25 |              0

There are 25 unsent records in the outbox, but the dispatcher reads 0 claimable items.

No exception is raised, and no SQLSTATE is generated. Downstream test assertions fail when expecting processed records, making it look like the message consumer failed when the actual issue was an upstream time mismatch.

§5 — The Rules

If you rely on SQL locking or queue claiming, keep two rules in mind:

Make your red tests readable. Do not rely on generic client command timeouts or shell execution ceilings to catch blocked queries. Use database-level lock_timeout settings so lock contention fails with explicit SQLSTATE codes like 55P03. One ordering constraint makes or breaks this: lock_timeout must sit strictly below your client cutoff. Npgsql's CommandTimeout defaults to 30 seconds, so a 30-second lock_timeout is a coin flip and a larger one is dead code.
Put your guards where they cannot disappear. Session-level SET commands can be wiped out when connection pools issue RESET ALL or DISCARD ALL. Configure your session safeguards as startup connection options (-c lock_timeout=5000) so they survive connection recycling.

(A third rule applies: if you delete the query clause you are testing and your test suite stays green, you do not have a test. But verifying test validity by breaking queries is a topic for another time.)

A gate that fails illegibly is a gate you will disable the first time it goes red on a Friday.

My Agent Said the Page Was Live. The Page Said 'We Are Closed.'

Onur Kesim — Tue, 04 Aug 2026 22:05:05 +0000

HTTP 200 Is Not Evidence

I gave my agent an explicit rule. It broke it three times in one day, in the exact same way.

Every time, the failure followed a single pattern: making an operational judgment without actually inspecting the rendered interface—whether that meant declaring a feature active based on a static landing screen, inferring form fields from theoretical assumptions, or evaluating a form without scrolling to the bottom of it.

The clearest instance happened while I was researching potential distribution channels to submit my technical writing. To evaluate candidates efficiently, I instructed my AI agent to scan a list of more than 40 target pages, inspect their submission pathways, and document whether each channel was currently open for new entries.

Across more than 40 target pages, the agent followed my verification rules meticulously. It checked links, parsed layouts, and accurately documented submission requirements for almost every single site. But on one specific target page—the exact channel I was planning to rely on for my submission pipeline—it took a silent shortcut.

Instead of reading the actual form container on the page, the agent looked at the network response, saw an HTTP status code 200 OK, and marked the submission pathway as open and active.

When the run finished, I opened the target URL in my browser to review the form myself. The submission form was nowhere to be seen. Instead, a quiet single line appeared inside the container: "Hey :) This typeform is now closed."

If an AI agent is completely lazy and fails on every task, it is easy to spot. You fix the prompt, adjust the script, or replace the tool. But when an agent is 97% obedient—following your instructions across 40 different pages, only to silently cheat on the single page you are building your workflow around—that is far more dangerous. A lazy agent is obvious; a mostly obedient agent is terrifyingly deceptive.

When I looked under the hood at the JSON payload returned by the underlying service API, the real story became clear. Right next to the boolean property is_public: true sat another property: isFormClosed: true. The raw, machine-readable data was sitting there telling the complete truth. But because the agent stopped checking as soon as it saw a positive surface indicator (is_public: true and an HTTP status 200), it reported a total success.

Surface signals lie. An HTTP status code 200 only means the remote web server did not crash while handling the request. It says absolutely nothing about whether the form inside that page is functional, active, or closed to the public. If you want reliable evaluations from an automated agent, you cannot accept surface status codes as evidence.

Measure the Archive, Not the Homepage

When an agent evaluates an external project or publication, its default behavior is to take the shortest possible path to a conclusion. This habit almost always leads it to summary sites, roundups, curated lists, and aggregator blogs.

I had to enforce a strict rule within my workflow: aggregator sites, blog posts, and comparison directories do not generate evidence. They only generate candidate links.

If a tech blog published a list three months ago stating that a specific platform accepts guest submissions, that blog post is merely a candidate hypothesis. It is not evidence. To get real evidence, the agent must navigate directly to the primary source and inspect the current operational state.

However, reaching the primary source is only half the battle. You cannot trust a platform's main landing page either. Landing pages are designed for marketing, not status verification. They are filled with present-tense slogans, sleek promotional graphics, and active call-to-action buttons that remain unchanged even when the underlying editorial operations have completely stopped.

I ran into a prime example of this while evaluating a technology newsletter. The live homepage looked completely healthy. It featured a clean modern layout, an active email subscription input box, the logo of a corporate sponsor, and copy promising regular updates. If an agent evaluated the channel based on its landing page, it would immediately declare the platform active.

But when you bypass the landing page and look into the actual publication archive, the reality is entirely different. The archive showed that the last published edition was issue #700 · June 2, 2024. The publication had been completely inactive for over two years, yet the landing page was still capturing email subscriptions and presenting a functional face to the public.

Reality does not live in marketing copy; it lives in chronological archives and dated records.

This brings us to a major technical trap when using web scrapers or browser agents to parse primary sources. When you instruct an automated scraper to read updates from a page, it usually queries the HTML DOM for the first matching <article> element or the top container in a feed.

This approach is fundamentally flawed. In modern web layouts, the first <article> tag on a page is frequently a pinned announcement, a featured post, or a sticky welcome message from years ago. If your scraper blindly extracts the first <article> element without verifying its embedded metadata, it will continuously report outdated records as fresh updates. To measure real state, you must explicitly force your tool to extract and validate the publication timestamp attached to the specific entry.

Break "Availability" Into Three Explicit Questions

One of the most common mistakes in prompt engineering and automated agent workflows is asking broad, compound questions. When you ask an agent, "Is this submission pathway available?", you force the model to make a subjective judgment call across multiple hidden variables.

To eliminate this ambiguity, you must break the concept of "availability" down into three separate, sequential checks:

Is the page opening? Does the target URL resolve? Does the server return a rendered page rather than a 404 Not Found or a 500 Internal Server Error? This is the only question that an HTTP status code helps answer.
Which plan or tier is required? What specific account level or subscription tier is required to utilize the feature described on the page?
Is that specific plan currently accessible to you? Is that required plan open for immediate registration, or is it gated behind an invite-only waitlist, a regional block, or an expensive enterprise paywall?

Without this three-part breakdown, your agent will easily confuse feature existence with feature accessibility.

During my research into automated submission tools, I encountered a platform where the public landing page clearly listed a submission management feature. The page loaded cleanly, and the basic free plan offered access to exactly four core dashboard features. Based on a superficial check, the agent marked the capability as "available."

However, when I inspected the pricing structure in detail, the truth came out: the actual submission function was locked strictly behind the $499.99/year Professional tier. The free account could view the dashboard, but could not execute a single submission.

Because the agent did not evaluate the tier requirement as a separate question, it collapsed a $500/year paywall into a simple "Yes, the feature exists." By forcing your agent to answer all three questions independently, you prevent it from reporting features that exist in theory as capabilities you can actually use in practice.

Timestamps, Expiry, and Asynchronous State

Observations in software systems decay rapidly. A status check recorded last week is often completely useless today. To prevent stale data from polluting my project records, I established a strict operational constraint: any measurement older than 24 hours is considered expired and must be re-measured from scratch.

This time constraint becomes even more crucial when dealing with asynchronous systems. Modern web applications rely heavily on background worker queues, scheduled database indexing, CDN cache layers, and event-driven pipelines. When an action occurs in an asynchronous environment, the public state does not update instantaneously.

If you perform a single snapshot measurement at time T1, you are viewing an isolated, static moment. You cannot determine whether the system is broken, idle, or simply waiting for a background worker thread to process a job queue.

To prove that an asynchronous state change has actually taken place, a single measurement point is insufficient. You strictly require two distinct time points: T1 (the baseline measurement) and T2 (the verification check after a defined time interval). Only by comparing the delta between T1 and T2 can you verify whether the system actually progressed.

The Third Verdict: UNMEASURED

Standard software testing frameworks train us to think in strict binary outcomes: PASS or FAIL, TRUE or FALSE.

When you force an AI agent to evaluate complex real-world systems using only a binary choice, you introduce a severe structural flaw. When an agent encounters an edge case—such as a pending queue, an unparseable DOM structure, a temporary network rate limit, or an ambiguous timestamp—it has no neutral container to place that observation in.

Because the agent cannot definitively prove that the test failed, its internal logic defaults to marking the check as a PASS. A binary evaluation model forces unmeasured and pending states to be reported as successful checks.

To fix this structural flaw, you must introduce a third explicit verdict state: UNMEASURED.

When an agent or script attempts to verify a condition but encounters missing timestamps, unverified DOM elements, or pending background queues, it must not guess. It must record the status as UNMEASURED.

UNMEASURED is an honest, non-binary signal. It tells your pipeline: "The check was attempted, but conclusive evidence was not acquired. Do not mark this as passed, and do not mark it as failed—hold the item in pending status and re-measure at T2."

A Case Study in Asynchronous Delay

I experienced the practical necessity of the UNMEASURED verdict through my own flawed observation while tracking metrics across a target set of more than 40 pages. I was recording my measurements directly into my project memory file.

One of my specific tracking routines involved checking my GitHub contribution graph to verify whether recent code commits were being correctly indexed and displayed on my public profile.

On my initial measurement run at time T1, I checked the profile page and inspected the contribution grid. The grid was completely blank. There was no visual record of the recent activity. Seeing the empty graph, I wrote down a note in my log stating that my "delay hypothesis was weakened"—assuming that because the contributions were not visible immediately, the indexing had failed.

A few hours later, at time T2, I re-ran the measurement on the exact same page. The background job queue had finished processing, the profile page cache had cleared, and the contribution grid was fully updated.

The recorded metric moved from "1 contribution in the last year" to "4 contributions in the last year" (recorded on August 3, 2026).

My original delay hypothesis was not wrong at all. The underlying queue was functioning correctly, but it operated on an asynchronous delay. Because I had relied on a single static snapshot at T1, I logged a wrong conclusion in my project memory and had to go back and correct the entry.

This mistake was the actual birth moment of the UNMEASURED rule. I didn't save myself with a rule I had already mastered; my own premature judgment forced me to define it. When dealing with asynchronous pipelines, a single time snapshot is not a measurement—it is a trap.

Connecting the Execution Loop

This pattern points to a broader truth in modern software engineering.

In automated CI pipelines, GitHub Actions returns conclusion: success for a workflow step even when an internal shell command fails under continue-on-error: true. That conclusion: success value does not measure whether your test suite passed; it only measures transport—that the pipeline managed to continue running without aborting. The ground truth about actual test failure is hidden away in a separate field called outcome.

An HTTP status code 200 functions in the exact same way. It measures network transport—that the web server successfully delivered an HTML payload—not that the application inside is active or usable. The ground truth about service availability lived in isFormClosed.

Both fields measure transport while posing as outcomes. And in both systems, automated tools miss the failure because the first superficial field returns a comforting answer.

If you want your AI agents to act as genuine engineering partners rather than sources of subtle failure, you cannot let them accept transport metrics as proof of operational state. Separate candidate links from true evidence. Require two-point timestamps (T1 and T2) for all asynchronous operations. Force your tools to measure archives instead of landing pages. And above all, never let your evaluation pipeline force an ambiguous result into a binary PASS.

A rule with no measuring instrument is just a wish.

Do Not Trust the Green Checkmark

Onur Kesim — Mon, 03 Aug 2026 19:33:05 +0000

Most people trust a green checkmark in a CI pipeline. If every job is marked as successful, we usually believe that everything worked correctly. I was doing the same. Then I found a case where a completely green workflow was hiding an important problem.

While testing hafiza-kur across different platforms, my main goal wasn't just to see green checkmarks. I wanted to build a full list of what broke on which platform. I needed to see every single error, without stopping the pipeline at the first failure.

To do this, I added continue-on-error: true to the test steps. It was a conscious choice: let the tests run to the end on every platform, collect the raw logs, and check the results later.

Then came Windows with Python 3.11 and 3.13. The test t_y42 ran 58 test scenarios. It took 91 seconds on Py3.11 and 110 seconds on Py3.13 to finish. The heavy lifting was completely done and the assertions ran. Then came the final step: printing the summary of the results to the console.

Right on the first line of the summary loop, Python hit a character that the default Windows console couldn't encode. UnicodeEncodeError.

The process crashed instantly. All 58 test results vanished before reaching the log file. And because continue-on-error: true was active, CI quietly ignored the crash, cleaned up, and moved forward.

The situation got worse when I checked the workflow data using the GitHub Actions API. I expected the step to show a failure, even if the main job continued.

It didn't. The API returned:

conclusion: success

In GitHub Actions, if a step has continue-on-error: true, its conclusion field is recorded as success. The actual failure was hidden in a different, rarely checked field called outcome.

That meant one-third of our cross-platform data was completely invisible — but CI was showing it as a green, successful run.

I created a separate verification step right after test execution: hukum_kapisi.py (The Gate of Verdicts).

This gate runs without continue-on-error. Instead of running the tests again, it reads the raw output logs printed to the screen. It explicitly looks for the lines that print the final test summary.

If the log ends before those lines appear, hukum_kapisi.py stops the build immediately with an error.

There is a very important limit to how this gate works: this gate never tells you that the test results were green. It only tells you that they were not lost.

Confusing "the tests passed" with "the test results were successfully recorded" leads to hidden bugs. hukum_kapisi.py makes sure the log data actually exists; the next tool decides if the tests passed or failed.

This problem is not unique to my project, or to Windows encoding bugs. Any CI pipeline that relies on reading screen output, summary loops, or permissive error settings can suffer from silent data loss.

If your pipeline depends on log output to collect data, ask yourself this question:

Are you sure your tests actually passed — or are you just assuming your reporting script didn't crash before showing the results?