DEV Community: Guangyi Zhao

SSH, systemd, kubectl: the one-page cheatsheets I keep open during incidents

Guangyi Zhao — Sun, 25 Oct 2026 22:24:48 +0000

I work on data center infrastructure. When something breaks at 2 a.m., nobody re-reads a 400-page manual — you reach for the handful of commands you half-remember and hope you get the flags right.

So I started writing one-page cheatsheets for the commands I keep looking up, and publishing them as open source. The first set covered everyday tools (git, docker, curl, jq, regex). This one is the infrastructure set: SSH, systemd, and kubectl — the three I actually keep open during incidents.

Every command below is copied verbatim from the cheatsheets, and every flag in them was checked against the real tools (OpenSSH 9.6 man pages, systemd 255, the official kubectl reference).

SSH: tunnels and jump hosts, not just login

Most people use SSH to log in. The parts that save me are the forwarding trio and ProxyJump.

Bring a remote port to your machine:

ssh -L 8080:localhost:80 deploy@example.com
# http://localhost:8080 on YOUR machine now hits port 80 on the server
ssh -L 5433:db.internal:5432 deploy@example.com
# ...or reach a third machine (db.internal) through the server you're logged into

Bake a bastion hop into ~/.ssh/config once, and every tool inherits it:

Host bastion
    HostName bastion.example.com
    User ec2-user
    IdentityFile ~/.ssh/id_ed25519

Host internal
    HostName 192.0.2.10
    User deploy
    ProxyJump bastion

ssh internal    # the hop happens automatically —
                # scp / sftp / rsync to "internal" get it for free too

And the smallest gotcha in the whole set: scp's port flag is capital -P; ssh uses lowercase -p. That mix-up is the #1 reason "scp works but ssh doesn't" (and vice versa).

Full sheet (28 entries: keys, config, forwarding, hardening): https://github.com/Ezra-Zhao/ssh-cheatsheet

systemd: read the journal like you mean it

The service commands everyone knows are start/stop/status. The ones that matter in an incident:

sudo systemctl restart nginx          # full stop + start; drops connections
sudo systemctl reload nginx           # re-read config WITHOUT restarting, if the unit supports it
sudo systemctl reload-or-restart nginx # reload if supported, otherwise restart — the safe default

After a crash or a hang, the previous boot's journal is where the answer lives:

journalctl -b         # this boot only
journalctl -b -1      # the PREVIOUS boot — where you look after a crash or a hang
journalctl --list-boots   # list recorded boots with IDs and time ranges, then pick with -b <id>

Filter by priority instead of scrolling:

journalctl -p err          # errors and worse (this boot)
journalctl -p warning -b -1  # warnings and worse, previous boot

And see the unit the way systemd actually sees it — base file plus every drop-in merged:

systemctl cat nginx   # base file + every drop-in merged, each piece labelled with its file path
# this is the ground truth — what you edited is not always what is running

Full sheet (30 entries: services, journal, unit files, timers): https://github.com/Ezra-Zhao/systemd-cheatsheet

kubectl: triage in the right order

My triage order is: describe first, events second, previous-container logs third.

kubectl describe pod web-6d9f7c5b4-x2abc   # why pending/crashing? read the Events at the bottom
kubectl describe node node-1               # capacity, allocatable, taints, conditions

kubectl get events --sort-by=.lastTimestamp        # OOMKilled, FailedScheduling, pull errors live here
kubectl get events -n prod --sort-by=.lastTimestamp

kubectl logs -p web-6d9f7c5b4-x2abc   # CrashLoopBackOff? this is where the real error is

(-p reads the previous, crashed container — the current one just shows you the restart loop.)

When a release goes wrong, the rollback is one command:

kubectl rollout history deployment/web        # list revisions
kubectl rollout undo deployment/web           # back to the previous revision
kubectl rollout undo deployment/web --to-revision=2

Full sheet (30 entries: status, deploy, logs, inside the container): https://github.com/Ezra-Zhao/kubectl-cheatsheet

The whole set

All fifteen one-page cheatsheets (git, docker, ffmpeg, curl, jq, regex, sql, ssh, kubectl, vim, tmux, systemd, and more) are MIT-licensed, each with a Chinese / Spanish / Portuguese README:

Index site: https://ezra-zhao.github.io/cheatsheet-pages/
First article in this series: https://dev.to/ezrazhao/9-one-page-cheatsheets-for-commands-you-keep-looking-up-ffmpeg-git-docker-curl-jq-regex-225d

If you find a wrong command or an outdated flag, a PR or a comment is the fastest way to fix it — the sheets only stay useful if they stay accurate.

Elder-Sentinel: Pose-Based Action Recognition for Elder-Care Safety Monitoring

Guangyi Zhao — Wed, 21 Oct 2026 14:11:55 +0000

Pose-based action recognition for elder-care safety monitoring.

A real-time pipeline that watches body movement — not identity — and flags safety-relevant events: help-seeking gestures, wandering into off-limits areas, prolonged inactivity that may signal a problem, and falls. Built for the realities of care homes, where caregivers can't be in every room at once, and where a fall detected minutes earlier can save a life.

Project status: scaffold v0.1 (honest edition).
The end-to-end pipeline runs today on procedurally generated, simulated skeleton sequences. What is real: pose data model, rule-based action detectors (hand-raise counting, loitering, fall alarm), event reporting.
What is still TODO (clearly marked in code): the field-validated action thresholds and the MediaPipe live-camera backend. This repo contains no face recognition and no identity matching — pose only.

Background

In a care home, caregivers can't watch every room at once. This system defines measurable action standards — e.g. "hand raised above shoulder N times in M seconds" as a call-for-help signal, "resident stationary near a stairwell or exit for T seconds", "sudden vertical drop consistent with a fall" — and raises alerts the moment a live pose stream matches one. The same architecture applies anywhere safety depends on body movement without identifying the person: elder-care fall detection, hospital ward monitoring, assisted-living wandering alerts.

What it does

Module	Input	Output
Pose estimation	camera frame (or simulated skeleton)	`SkeletonFrame` joint coordinates
Hand-raise counter	wrist/shoulder trajectories	count + timestamps per side
Loitering detector	body centroid trajectory	`loiter_alert` after dwell threshold
Fall detector	hip-height trajectory	`fall_alarm` (critical) on sudden drop
Event reporter	all detector outputs	console summary + JSON report

Architecture

                +------------------+
                |  Pose source     |
                |  MediaPipe cam   |  (TODO: live backend)
                |  Simulated seq   |  <-- demo runs on this
                +--------+---------+
                         | SkeletonFrame stream
          +--------------+--------------+
          |              |              |
   HandRaiseCounter LoiteringDetector FallDetector
   (config thresholds — TODO: field values)
          |              |              |
          +--------------+--------------+
                         v
                  EventReporter
                  console + report.json

Quickstart

cd Elder-Sentinel
pip install -r requirements.txt
python examples/demo.py        # simulated end-to-end demo
python -m pytest tests/ -q     # unit tests

The demo runs three simulated scenarios — hand raises, loitering, a fall — and writes examples/demo_report.json.

Configuration

All action thresholds live in config/action_thresholds.yaml. The shipped values are placeholders, not standards:

hand_raise:
  raise_margin: 0.05   # TODO(ezra): field-validated value

TODO(ezra): replace every threshold with values validated from your real deployments (raise margins, dwell times, drop ratios, zone geometry). The detectors are deliberately dumb rules — your experience is what makes them trustworthy. Do not present placeholder values as operational standards.

Live camera backend (TODO)

pose/mediapipe_backend.py is a stub. To go live:

pip install -r requirements-optional.txt (MediaPipe + OpenCV)
Implement MediaPipeBackend.estimate() mapping the 33 MediaPipe pose landmarks onto this repo's joint set, emitting SkeletonFrame(simulated=False).

Ethics & lawful use

Pose only. This repository contains no face recognition, no identity matching, and no person re-identification. It answers "what is the body doing", never "who is this".
Authorized safety use only. Intended solely for lawful, authorized safety deployments (e.g. licensed elder-care facilities, with family/guardian consent). Do not use for covert surveillance, and do not deploy without proper legal authorization and oversight.
Simulated data. Every demo frame is procedurally generated and labeled simulated: true. Nothing here was recorded from real people.

Roadmap

[ ] Field-validated thresholds (Ezra's deployment experience)
[ ] MediaPipe live backend + zone polygons on floor plans
[ ] Multi-person tracking (ID by track, not by face)
[ ] Alert routing (SMS/dashboard webhook)
[ ] Recurring-pattern analytics (e.g. agitation escalation over hours)

What a Second Model Actually Catches: Seven Articles, Forty Review Comments, Measured

Guangyi Zhao — Tue, 20 Oct 2026 01:52:32 +0000

Author: Guangyi Zhao

1. Problem and background

This article series runs under a standing rule: every draft gets reviewed by a different model before it can ship. The reviewer is told to hunt for factual errors, logical leaps, and overclaims; every comment must be resolved, one by one, before the piece is final. After seven days and seven articles, a fair question is whether the ritual earns its keep. What does the second model catch that a careful author re-read would not? And how does it fail when it fails?

No new apparatus is needed to answer that. The review records themselves are the dataset: every comment, every disposition, and every failed API call is on disk in the project's review/ directory. This piece does two things with that record. First, it codes all forty comments from the seven articles and counts what kinds of problems a cross-model review actually produces. Second, it runs a planted-error probe: a short passage with six defects buried in it, sent to the reviewer with the key withheld, to measure recall against a known denominator — something the historical record alone can never give you.

One clarification up front, because the title invites the wrong reading: of the forty comments, 29 came from the cross-model reviewer and 11 from the human three-role fallback used on the two nights the model never answered. The two sets are counted separately throughout. The headline result: the model reviewed five of seven articles and produced 29 comments, 27 of which changed the text. The other two were a false alarm of its own and a style note I kept anyway. What it catches is concentrated — not typos, but numbers that don't reconcile, causal claims that outrun their evidence, and conclusions stated without boundaries. The catch is availability: getting those five reviews took 13 API calls, of which 6 died with HTTP 503 and 2 with 429 quota exhaustion. It is a good reviewer who calls in sick a lot.

2. Method

Dataset A is the complete review log of the seven articles published October 13–18, 2026: seven review records, plus the raw reviewer receipts where they exist (JSON receipts for three successful reviews — Sentinel, Vuln-Scanner, Bloom; the Wiper receipt file was lost to a capture bug and its comments and usage figures survive in its record; the REA review survives as record text; two failure receipts document the nights the model never responded). The reviewing configuration was Gemini throughout — gemini-flash-latest as primary, with one article falling back to gemini-3.8-flash after two 503s. I coded each comment on two axes: category (numbers/definitions, causality/concepts, overclaim/generalization, scope qualification, method disclosure, structure/cross-references, style) and disposition (changed the text, verified as correct on re-check, rejected as reviewer error, already covered elsewhere, kept by author). Counting was done by a script, not by hand; the script and its JSON output ship with this article's data files.

Dataset B is the probe: a roughly 330-word excerpt about adding a cache to a diagnostics gateway, with six planted defects — a breakdown whose parts sum to 45 ms against a stated 41 ms total, a causal attribution asserted without control conditions, a boundless "no side effects, at any scale" conclusion, a stale cross-reference, a conclusion "supported" by an unverifiable internal newsletter metric that measures something else, and a success rate of 100% in the same breath as two timed-out failures. Only the passage went to the reviewer; the answer key did not. One design flaw disclosed in advance: the cross-reference defect carries an editorial note in the text admitting the numbering is stale, which half-cues that item. Discount it accordingly.

Also disclosed: the historical coding counts only the reviewer's comments. The single largest catch of the week — a draft that claimed "we tried this, it doesn't work" about an experiment that had never been run — was caught by the author's own pre-submission check, not by the reviewer. It is discussed in Section 4 but not credited to the reviewer.

3. Measurements

3.1 Availability: 13 calls, 5 reviews

The call log, article by article: REA failed once with 503, then succeeded on retry. Web-Vuln-Scanner failed twice with 503 and succeeded only after switching to the fallback model. AI-Node-Sentinel succeeded on the first call but a day late — drafted on the 14th, reviewed and published on the 15th. Wiper and Bloom-Counter succeeded first try. The remaining two never got through: Go-Diagnoser hit a 429 quota wall, and Emotion-Guard burned four calls in one evening (three 503s, one 429) before I gave up and ran the human fallback. Totals: 5 of 13 calls succeeded (38%); of the 8 failures, 6 were 503 and 2 were 429. Five of seven articles received a cross-model review; only four received it the same day. This is what a free-tier quota buys you: a 429 that says "come back in about 22 hours" and a 503 that means the provider is busy are both failures that retrying for a few minutes will not fix.

3.2 The probe: six planted, six caught, one unplanted

All six planted defects were identified explicitly. The arithmetic error came back with the sum worked out. The success-rate contradiction was quoted back in adjacent fragments. The causal claim was faulted for ignoring second-order effects — downstream relief, network and GC behavior — and for asserting complete attribution without controls. The overclaim was quoted word by word. The newsletter evidence was flagged, though filed under causality rather than sourcing: ticket volume moves with business cycles and rosters, so it cannot certify "stability." The cross-reference was caught too, subject to the half-cue disclosed in Section 2.

More interesting than the perfect score is the seventh finding, which I did not plant. The probe states an 87% cache hit rate and also books the full 19 ms origin fetch into the average per-request breakdown. The reviewer objected: if only 13% of requests reach the origin, that line should enter the average weighted (≈2.5 ms), not at full value — the metric definition contradicts the hit rate. On re-checking, the reviewer is right; the breakdown described a full request path while claiming to be an average. Planting defects, I had accidentally planted a real one. Final probe score: 6 of 6 on the key, plus 1 valid finding outside it.

3.3 What the 29 comments look like

By category: causality/concepts 6, overclaim/generalization 6, numbers/definitions 5, method disclosure 5, structure/cross-references 3, scope qualification 3, style 1. The top three families are 17 of 29 (59%).

Representative catches. Numbers: in the Bloom-Counter piece I had summarized an overexposure condition as essentially unaffected; the reviewer pointed out that its 15 false positives (precision 0.933) were bleached leaf fragments masquerading as buds — brightening manufactures targets while darkening loses them, two different failure modes my summary had flattened into one. In the Web-Vuln-Scanner piece, a target's 15 findings were itemized as 14 in my text; the reviewer counted. Causality: in Bloom again, my explanation of an over-count contradicted my own limitations section — adhesion blamed in one place, a "conservative estimator" claimed in another; the reviewer laid the two passages side by side. Overclaim: "not a bottleneck at any realistic scale" in the Sentinel piece was flagged as evidence-free extrapolation and cut down to the measured scale. The most instructive comment was method-related, in the Wiper piece: the reviewer said the rounding convention behind the quoted prices was undocumented, so a reader recomputing from the printed numbers would not balance. I re-ran all 8 records under both pricing branches — 16 combinations — and every total balanced to the cent. That comment's product was not a corrected number but a paragraph documenting the convention and the reconciliation.

Dispositions: 27 of 29 changed the article (including the Wiper comment, resolved by documenting the re-check). One was rejected: the reviewer called the Sentinel draft's date, 2026-10-14, a "future date typo." The draft was written on the 14th and reviewed on the 15th; the review prompt carried no current date, and the model's knowledge ends before 2026, so a true date read as future. Section 4 returns to this. One was kept: a style objection to a turn of phrase in the REA piece, noted for the record; I preserved the first-person register.

3.4 The fallback group: what human three-role review produces

The two fallback articles generated 11 comments: numbers 5, method 2, scope 2, overclaim 1, style 1. Their disposition profile is the mirror image of the model's: only 5 changed the text, while 5 were "verified correct on re-check" and 1 was already covered by the limitations section. The Emotion-Guard reviewer (also me, wearing a different hat) recomputed every probability in the draft — the teacher-spike figure, 1 − 0.95⁶ ≈ 26.5%, checks against the text's "more than a quarter" — and reconciled every number against the raw data file. That is audit confirmation, not fault-finding, and the reason is structural: in the fallback, reviewer and author share one brain. It is good at re-deriving its own arithmetic and poor at stepping outside its own framing. The Go-Diagnoser fallback did catch an unsupported attribution and one "tests pass" claim that had never actually been run — both are evidence-state problems, neither required an outside frame.

3.5 Cost

The direct cost is small. The four successful reviews with complete usage records: Sentinel 5,095 prompt tokens, Vuln-Scanner 5,365, Bloom 5,149, Wiper 24,372 (that receipt folds the model's internal thinking tokens into its totals, so it is not strictly comparable to the other three), with responses of 1,820, 1,764, 1,974, and 1,605 tokens — all under two thousand. The REA review's usage was not preserved, the one gap in the ledger. The probe cost 443 prompt and 1,009 response tokens. The real cost of the pipeline is not tokens but latency and uncertainty: retries against 503s, a model swap, a review that landed a day late, and two evenings of human fallback labor.

4. Analysis

Four judgments survive the data.

First, the reviewer's highest-value skill is reconciliation, not judgment. Roughly ten of the 29 comments turn on numbers, definitions, or count completeness, and nearly all were right. Reconciliation needs no theory of the whole article — it needs the stamina to collect every instance of the same quantity scattered across sections and compare them. That is exactly where a single author fatigues: the same figure appears in an abstract, a body paragraph, and a table, gets edited in one place, and drifts in the other two.

Second, it is more sensitive to unbounded language than its author is. Overclaim plus scope comments form the largest bloc at 9. The reviewer has no sunk cost in a pretty sentence. The probe reproduced the pattern in miniature: "no side effects" was quoted back and condemned in the same reflexive way "at any realistic scale" had been a few days earlier.

Third, its false alarms are context-shaped, not confabulation-shaped. The week's single rejection traces to a missing date in the prompt plus a knowledge cutoff that predates the draft's year. The defenses are procedural, not rhetorical: resolve every comment individually rather than accepting them wholesale, record rejections, and hand the reviewer the context it cannot guess. For this article's own review I injected the current date into the prompt; the failure mode did not recur — one observation, not a cure.

Fourth, the fallback path is part of the main path. Two of seven articles ended up there, so "what happens when the model doesn't answer" is not an edge case; it is a weekly event. The fallback produces a different substance (verification over reframing), and the ledger says so in plain language, so readers can discount the review strength of those two articles correctly. The one unforgivable version of this pipeline is a vague methods line that lets a fallback review masquerade as a cross-model one.

And the question from Section 1 — can author self-check substitute? Within this workflow — one author, unaided re-reads, no checklist scripts — the evidence says the two catch different species of error. The week's worst defect (a never-run experiment reported as run) was invisible to the reviewer: on the page it read perfectly. The reconciliation errors the reviewer caught had survived two of my own read-throughs. Bolting extraction scripts and checklists onto self-check would move that boundary; this dataset doesn't measure that configuration.

5. Limitations and open problems

First, the sample is seven articles, one reviewer family (one primary model, one fallback variant), one author, and one coder — me — with no second coder to check the category boundaries; the script guarantees arithmetic, not coding. Second, the probe is a single short passage with a defect density no real draft has, and one half-cued item; 6-of-6 says these defect types don't slip past it in a short text, and says little about recall in a 5,000-word article where defects hide deeper. Third, the fallback and model groups reviewed different drafts, so the contrast in Section 3.4 mixes reviewer effects with manuscript effects; it is not a controlled comparison. Fourth, the availability figures are bound to a free-tier quota and one particular week of provider load; 5-of-13 will move with a paid tier or a different week, and every conclusion here is scoped to this configuration. Fifth, nothing here measures the endpoint that ultimately matters — whether reviewed articles earn more readers or more trust. Comments changed texts; the rest of that chain has no data in this study.

6. References

Review records for the seven articles: review/2026-10-13-rea-gemini-review.md, review/2026-10-14-sentinel-review.md, review/2026-10-15-vuln-scanner-review.md, review/2026-10-16-wiper-review.md, review/2026-10-17-bloom-review.md, review/2026-10-18-emotion-guard-review.md; raw Gemini receipts in the same directory (*-gemini-raw.json).
This round's data: data/2026-10-19-cross-review-bench.json (script: data/2026-10-19-cross-review-bench.py, re-runnable); probe text, answer key, and reviewer output: data/2026-10-19-probe.txt, review/2026-10-19-probe-gemini-raw.json.
The pipeline's standing rule: the project's writing standard, quality-gate section (cross-model review and honest fallback labeling).

Author: Guangyi Zhao. The pipeline, records, and experiments described here are the author's own clean-room work; the experimental material consists of the series' own manuscripts and a synthetic probe passage. No client information is involved.

Letting an AI Agent Drive Ghidra: A Hands-On Test of REA’s 170 MCP Tools, One Recovered Password, and the Limits

Guangyi Zhao — Mon, 19 Oct 2026 05:05:58 +0000

Author: Guangyi Zhao
Date: 2026-10-13
Subject: morluto/rea (REA: Reverse Engineer Anything), npm package rea-agents 6.4.0, MIT license
Environment: Ubuntu 24.04 (Linux x64), Node 24.20; target is an ELF built from my own C source

1. Problem and background

Reverse-engineering an unfamiliar binary the traditional way means sitting in front of Ghidra, paging through assembly, following cross-references, guessing what each function does, and writing the conclusions down by hand. The slowness is not even the worst part. The worst part is how easily conclusions drift away from evidence: three days later I reread my own notes and cannot reconstruct what I actually saw when I wrote a given claim.

REA proposes a different division of labor. It does not implement decompilation itself. Instead, it standardizes the operating surface of reverse-engineering tools as MCP tools, so agents such as Claude Code, Codex, Cursor, or Gemini CLI can drive them directly: the agent asks the question, opens the binary, lists functions, pulls pseudocode, checks references, runs the target and diffs its behavior, then comes back with evidence attached. The human's job shifts from paging through assembly to asking the right questions and auditing the evidence chain.

Is that division real, or does it only demo well? I spent a day running it end to end, from install to password recovery, and I did not dodge the failures. Every number below comes from that day's logs and Evidence JSON files, and I re-checked each one against the raw files before finalizing this article. Anything I could not verify is either deleted or explicitly labeled as inference.

One ground rule up front: the only program analyzed here is one I wrote myself. The legal boundary is covered in Section 5, and it is not filler.

2. Method and architecture

REA is easiest to understand in four layers.

The first is the agent access layer. rea mcp starts an MCP server over stdio. I performed a real JSON-RPC handshake: initialize succeeded and tools/list returned 170 tools, with names like open_binary, analyze_function, batch_decompile, and inspect_binary_layout. npx rea-agents setup registers the server with the agent client you choose and installs an investigation-workflow skill that teaches the agent what to check first, what to check next, and how to follow up when the evidence is thin.

The second is the workflow layer, shared by the CLI and MCP. The CLI exposes a hundred-plus commands — analyze, function, decompile, xrefs, search for native analysis; capture-process for behavior capture; compare-process-captures for diffing two runs. The only difference is invocation: the CLI is a fresh process per call with results saved to disk, while MCP is a persistent session.

The third is the analysis-engine layer, and the most commonly misunderstood. Deep native analysis — decompiled pseudocode, function dossiers — is not implemented by REA. It bridges to one of three existing engines: Ghidra, through a Java bridge script (ReaGhidraBridge) running headless and talking over a local unix socket, requiring Ghidra 12.1.x; Hopper, through a Python bridge; and IDA, through IDA's own MCP registration, bring-your-own-license and bring-your-own-instance. REA is MIT-licensed, but that license grants you no commercial engine. IDA and Hopper still cost what they cost. Ghidra is free, which is why it is the route tested here.

A separate set of capabilities needs none of those engines: offline ELF layout parsing (through a Python environment you supply, pinned to pwntools 4.15.0, pyelftools 0.33, and Unicorn 2.1.2; read-only parsing of sections, segments, symbols, relocations, and mitigations, never executing the target), inspection of metadata embedded in Go binaries, purely static parsing of JavaScript and .NET files, and process behavior capture that actually runs the target under a PTY.

The fourth layer is the evidence layer, and in my judgment it is the real design center. Every call returns a structured Evidence record: an evidence_id, the target file's SHA-256, the provider and its version, the raw and normalized results, a confidence label such as observed or inferred, and an explicit list of limitations — what was not checked and what is inferred rather than seen. REA deliberately refuses to hand you a verdict. That is a different temperament from one-click report generators, and it is what makes automated follow-up questions defensible: every step can be traced back to specific evidence and open items.

Two further design notes. According to REA's own documentation, analysis runs entirely on your machine and target files are not uploaded to REA's servers. I did not verify that with packet capture, so treat it as a vendor claim, not a measurement — and note that what your agent does with tool results afterward is governed by your model provider's data policy, which is a separate question. The claimed target coverage is broad: beyond native binaries it lists Go, .NET, JavaScript/Electron, Android APKs, firmware, WASM, EVM bytecode, SQLite, pcap files, and crash dumps. This article tests exactly one of those lanes, native ELF, and draws no conclusions about the rest.

The target, checkme, is 30 lines of my own C: it takes a key on the command line, and check_key() first compares the length, then XORs the input byte-by-byte against a built-in constant and accumulates; only a zero sum passes. I built a symbolled and a stripped version. The correct key is 13 bytes. The question for the day: starting from "we do not know the answer," how far does this toolchain get, and at which step does a human — or an agent — still have to actually think?

3. Measurements

3.1 Readiness check: half the tool works out of the box

Installing rea-agents globally (version 6.4.0) took about 1 minute 51 seconds. rea doctor was refreshingly honest: Node and the host were healthy, but all three deep engines — Hopper, Ghidra, IDA — were missing, and the workflow skill was not installed. Out of the box you get only the built-in capabilities; deep decompilation requires bringing an engine. Doctor spelled out what was missing and how to fix each item, which already puts it ahead of tools that hand you a stack trace.

One piece of friction runs through everything: every rea command cold-starts in roughly 20 to 30 seconds, rea --help included. I did not profile where that time goes; my guess is Node loading the full tool catalog at startup, but that is a guess and I am labeling it as one. If you explore interactively at the rhythm you would use in Ghidra's GUI, this will wear on you. The tool rewards deciding your questions first and fetching evidence in batches.

3.2 Offline ELF layout: the manual jigsaw, done in one pass

I created a Python venv with pwntools — the versions landed exactly on REA's pins — pointed REA_PWNTOOLS_PYTHON at it, and ran rea inspect-binary-layout ./checkme --json. About 22 seconds later I had roughly 45 KB of Evidence JSON. I reopened that file while finalizing this article; the numbers are:

31 sections, 13 segments, 48 symbols, 11 relocations, with the file anchored by SHA-256;
mitigations matching the build reality item by item: PIE on, NX on, non-executable stack, Full RELRO, and no stack canary — correct, because the source has no local arrays. The whole set is labeled inferred rather than observed, which is the honest label;
a symbol table that names main, check_key, and strlen outright, so you know where to start;
12 limitations attached, including that link-time addresses are not runtime addresses and that relocation inventory completeness is unknown. Nothing swept under the rug.

This replaces the manual checksec-plus-readelf jigsaw with one structured pass. Repeating it on the stripped binary left only 9 dynamic symbols — the libc imports — with check_key and main gone. That is expected, and it draws the boundary of this lane precisely: layout only, no function recovery.

3.3 Behavior diffing: locate the divergence before reading code

Next, black-box capture. I ran rea capture-process once with a wrong key and once with the right one (about 30 seconds each, PTY startup included) and diffed them with rea compare-process-captures. The result names a first_divergence: terminal event 0, ACCESS DENIED on the left, ACCESS GRANTED on the right, with the exit dimension also marked changed. I checked the raw comparison JSON; its confidence is labeled derived — a conclusion computed from two captures rather than a single observation — which is the correct label.

On a toy target this looks trivial. The value is the ordering habit: confirm behaviorally which input decides success before you open the static analysis. Get the order right and you skip a lot of wandering later.

3.4 Ghidra deep analysis: it worked — the second time

First, the setup bill, because it is real: this machine had no Java. Temurin JDK 21 (198 MB, about 2 minutes) plus the Ghidra 12.1 release (542 MB, about 10 minutes), roughly 874 MB unpacked, plus GHIDRA_INSTALL_DIR and JAVA_HOME to set. Call it about 740 MB of downloads before the deep lane opens. Any honest evaluation has to count that.

Then rea analyze ./checkme --provider ghidra --json:

Attempt one: about 3 minutes 26 seconds, failed with error code cleanup_incomplete. Ghidra's own log showed import and post-analysis succeeding, but REA's teardown refused to confirm cleanup because the process tree contained orphaned or PID-reused processes, and failed the whole run. My inclination is that its process-ownership checking clashed with this containerized environment, but I did not root-cause it — container PID namespaces and process reaping were never individually ruled out — so this is recorded as an unlocalized failure, not an explained one. No workaround was applied and it is not reported as a success.
Attempt two: about 2 minutes 50 seconds, success — with a caveat. The overview's normalized result is only about 2.7 KB (the saved JSON file is about 4.4 KB): segment layout plus counts. It counted 30 procedures and 80 strings and gave pseudocode for none of them. Deep answers require naming the function you want.

So I named it: rea function ./checkme check_key --provider ghidra --json took about 2 minutes 42 seconds and returned a roughly 52 KB dossier with three things that matter. First, decompiled pseudocode matching the source logic line for line, with placeholder names like param_1 and DAT_00102004: compare strlen against 0xd — 13 — then XOR the input byte-by-byte against the constant at DAT_00102004 and accumulate; pass only if the sum is zero. Second, the full assembly listing: a 76-byte function body at 0x101189–0x1011d4, which I verified against the raw JSON. Third, the call graph edges: caller main at 0x1011d5, callee the strlen PLT thunk, with incoming reference addresses.

That closed the loop. Within this target's value range — password and constant are plain ASCII, so every XOR result lands between 0 and 127 — a zero sum is equivalent to byte-for-byte equality, so the 13 bytes at that address are the key; I read them out by file offset and confirmed. One precision worth adding: the pseudocode accumulates the XOR result truncated to a char, so with high-bit bytes, signed interpretation could in theory let several mismatches cancel out to zero. As a teaching target this check is fine; as a real password check it would be an exploitable weakness, and "sum equals zero" should never be mistaken for a cryptographic equality test. From opening an unfamiliar binary to recovering both the checking logic and the key, no human read any assembly. For completeness: with no engine configured, the same analyze command fails politely — a structured provider_unavailable error with remediation guidance instead of a crash.

3.5 The MCP channel, and the full timing table

rea mcp runs over stdio; the JSON-RPC initialize handshake succeeded, the server identifies as rea 6.4.0, and tools/list returned 170 tools. Scope discipline matters here: 170 is the catalog size. The tools I actually exercised and verified are a small subset — binary opening, layout inspection, behavior capture and comparison, Ghidra analysis, and the function dossier. The other 160-plus tools are unverified by this article, and a directory count must not be passed off as capability verification. Within the subset I used, CLI and MCP are two front doors to the same workflows.

Full run log, all from that day:

Shallow repo clone (56 MB): success, about 28 s
npm i -g rea-agents: success, about 1 min 51 s
rea doctor: success, three engines reported missing, about 30 s
Offline ELF layout, symbolled and stripped: both success, about 22 s each
Process capture twice plus comparison: success, first behavioral divergence pinpointed, about 30 s each
Analyze with no engine configured: failed cleanly with provider_unavailable, as expected, about 23 s
JDK 21 plus Ghidra 12.1 download and unpack: success, about 740 MB, about 14 min
Ghidra analyze, attempt one: failed, cleanup_incomplete (engine analysis succeeded, teardown verification failed), about 3 min 26 s
Ghidra analyze, attempt two: success, segment-level overview only, about 2 min 50 s
Ghidra function dossier for check_key: success, logic fully recovered, about 2 min 42 s
MCP initialize plus tools/list: success, 170 tools, about 23 s

4. Analysis and trade-offs

Where the division of labor genuinely holds: it splits the Ghidra learning curve into two stages. First you understand what a function does by questioning an agent that fetches evidence; then you learn the tool itself against the assembly and pseudocode you already understand. For anyone studying malware analysis or binary fundamentals, that order beats staring at an empty Ghidra window. Behavior diffing is the second habit worth stealing: behavior first, statics second is the same discipline as comparing normal and anomalous samples in dynamic analysis.

Four selection judgments for anyone evaluating this stack:

First, Ghidra versus IDA. REA does not change either engine's economics. Ghidra is free and headless-friendly; the price is the setup bill above and minute-scale calls. IDA's recovery quality and ecosystem are their own argument, but the license is yours to buy — REA's MIT license does not extend to it. My recommended order: teams that already hold IDA licenses should use that integration; everyone else should get the workflow running on Ghidra first.

Second, CLI versus MCP. For one-off scripts and auditable reruns, use the CLI; evidence on disk is naturally traceable. For multi-step investigations where an agent keeps asking follow-ups, use MCP. A persistent session should amortize repeated cold starts in theory — but I did not measure per-call latency inside a session, and the handshake alone took about 23 seconds, the same order as a single CLI call. So "MCP is faster" is an architectural expectation in this article, not a measured result; measure session latency yourself before choosing on speed. Also, a persistent session means evidence management depends on session semantics; do not mix the two modes' outputs in one folder and expect clean accounting.

Third, is the evidence contract worth its verbosity? Every result carrying limitations and a confidence label looks like overhead until you are the reviewer: you no longer have to guess which sentence was observed and which was inferred. In report-writing, delivery, or graded casework, that verbosity is net profit. For a one-off glance at a binary, it is pure cost. Know which mode you are in.

Fourth, and most important: every step that requires judgment was left to the caller, by design. Understanding the pseudocode's semantics — why a zero XOR sum means byte equality here, and when it would not — was the caller's work. Deciding to read the bytes at DAT_00102004 was the caller's work. Translating length 0xd into 13 characters was the caller's work. REA supplies evidence and open items; the conclusion belongs to whoever called it, and an agent that misreads pseudocode will produce a wrong conclusion that looks thoroughly evidenced. "Agent plus REA" does not mean unmanned reverse engineering. It means automated evidence collection with judgment still on the human side — and anyone who quietly drops the second half of that sentence will eventually be bitten by it.

5. Limitations and open questions

Measured limitations:

Cold starts are slow: 20 to 30 seconds per CLI command, and Ghidra-backed calls are measured in minutes. Function-level answers must be requested one function at a time, so cost grows roughly linearly with function count. I did not test full deep analysis of a large binary and will not extrapolate to one.
The analyze overview is shallow by design — layout and counts only. If you expect one command to hand you function-level conclusions, you will be disappointed; the design pushes you to arrive with a question and fetch named evidence.
Against stripped, packed, or obfuscated targets, the layout lane degrades to dynamic symbols, and the deep lane inherits whatever recovery ability the engine itself has. REA adds no magic. My target was an unobfuscated program I wrote, so this article has no standing to quote recovery rates on real-world samples.
The process-ownership check failed once on containerized Linux with cleanup_incomplete and passed on retry. Unattended batch runs need retries and failure logging budgeted in; do not assume a 100 percent first-pass success rate.
Only the native ELF lane was tested. APK, firmware, WASM, EVM, and the rest of the coverage list went unexercised, and unexercised means unjudged.
The ceiling on conclusion quality is the agent and the human reviewing it. Evidence chains can skip steps and pseudocode can be misread, and such errors arrive wearing Evidence IDs, which makes them look more credible, not less. That is the most dangerous failure mode in the whole setup.

Legal and compliance boundaries, which are not optional:

Analyze only software you own or are explicitly authorized to analyze: your own programs, CTF and lab targets, samples you purchased under terms that allow it, client deliverables with written authorization.
Unauthorized reverse engineering of commercial software can trip three wires at once: anti-reverse-engineering clauses in the EULA; DMCA Section 1201 anti-circumvention rules, where bypassing a technical protection measure can be unlawful in itself and legitimate purposes have only bounded exceptions; and unauthorized-access statutes. REA's own DISCLAIMER takes the same line: lawful research, education, interoperability, and software preservation.
Commercial engine licenses are yours to obtain legitimately, and trial-license restrictions on automation and sharing outputs apply in full.
Practical rule: learn on crackmes, CTF challenges, and programs you wrote — exactly what this article does. At work, with a client binary, confirm the authorization paperwork is in hand before you touch anything.

Open questions I am leaving for follow-up measurements rather than pre-spending conclusions: recovery-quality differences on the same target with the IDA provider, deep-lane behavior on a packed sample, and the context cost of accumulated evidence in long MCP sessions.

6. References and sources

REA repository and documentation: morluto/rea (MIT), npm package rea-agents 6.4.0; architecture descriptions checked against its README, docs, and the ReaGhidraBridge sources under bridge/ghidra.
Primary test materials: target source checkme.c (30 lines, my own), and the Evidence JSON files — layout_checkme.json (31 sections, 13 segments, 48 symbols, 11 relocations; provider pwntools 4.15.0 / pyelftools 0.33 / Unicorn 2.1.2), layout_stripped.json (9 dynamic symbols), capture_compare.json (first_divergence at terminal event 0), ghidra_func_checkkey.json (check_key dossier, Ghidra 12.1, 76-byte body at 0x101189–0x1011d4). Each was re-opened and re-checked before this article was finalized.
Engine versions: Ghidra 12.1 PUBLIC, Temurin JDK 21, Node 24.20, Ubuntu 24.04.
Legal reference: DMCA anti-circumvention provisions, 17 U.S.C. § 1201. Consult a lawyer for specific cases; this article is not legal advice.

Disclosure: All code, target programs, and experiments in this article are the author's own clean-room work. The experiments were run only against the author's own program. No client information, client code, or unauthorized software is involved in any way.
Author: Guangyi Zhao

“Can You Make It Cheaper?” — and the Agent Booked the Order: A State-Machine Field Test of Wiper-Service-Agent

Guangyi Zhao — Mon, 19 Oct 2026 05:03:42 +0000

Author: Guangyi Zhao
Date: 2026-10-16
Subject: Wiper-Service-Agent (my own open-source project, github.com/Ezra-Zhao/Wiper-Service-Agent), scaffold v0.2, commit 9675291, Python 3.12.3
Environment: Linux x86_64 VM, agent exercised in-process. No WhatsApp connection, no real messages sent, no real customer data. The fitment catalog, price table, and NLU are the repository's own simulated implementations, and the article says so wherever it matters.

1. Problem and background

The hard part of a WhatsApp service agent for a small shop is not wiring up a model. It is walking one conversation down a path where money changes hands: ask for the vehicle, look up the size, quote a price, answer questions, record an order intent. Misread the customer once and the cost is real — hear a haggle as an order and the shop holds an order nobody placed; hear a hesitant "I don't know" as a rejection and a nearly closed sale is shut down on the spot.

Wiper-Service-Agent is the agent I built for a wiper-blade shop. The conversation is not left to a language model. An explicit state machine runs it — GREETING → COLLECT_VEHICLE → QUOTE → DONE, four states — and the understanding layer defaults to a deterministic rule-based parser (MockLLM, marked SIMULATED in the repo). The real LLM parser (RealLLMProvider) is only ever asked to parse a message into an intent and vehicle slots; it never decides prices, sizes, or whether a deal closed. That division of labor is written into the README. Status, honestly: scaffold v0.2, a simulated catalog of 8 fitment records across 5 makes, a demo price table. This is not ready to take real orders.

So this article does not measure whether the demo runs — the bundled demo and all 56 tests pass (I re-ran them: 56 passed; the README still says 44, which is stale). It measures how the machine behaves off the demo path: 27 scripted conversations, 62 messages, 13 intent-collision probes, a full recomputation of quotes for all 8 catalog records, one persistence round-trip, and one two-customer isolation check. The headline first: the main flow is steadier than I expected, the quote math is exact to the cent, and all six FAQ topics stay inside the quoting state. But the intent layer's substring matching produces three misreadings that would cost a real shop money — the worst being the title of this article. Under the default rule-based parser (MockLLM), a customer in the QUOTE state who asks "可以便宜点吗" ("can you make it cheaper?") is taken straight to DONE with a confirmed order intent, at the full, undiscounted price. Scope, stated once and binding for the whole article: every intent-layer finding here is about that default parser; behavior under the real LLM parser is a separate question, revisited in §5.

2. Method and architecture

The core package, tools, and WhatsApp boundary total 1,659 lines, including a 258-line webhook server. Conversation state lives in a Conversation object: current state, three vehicle slots (make / model / year), the fitment result, the quote, a boolean haggled flag (has the one-time haggle discount been used), the outcome (confirmed / declined), detected language and confidence, and full history.

The transition rules, state by state — everything later follows from these:

GREETING: a message carrying all three vehicle slots goes straight to a fitment lookup — found means QUOTE; a miss (a year outside the catalog range, say) falls back to COLLECT_VEHICLE with a "not found" reply. A partial vehicle moves to COLLECT_VEHICLE and asks for what is missing. Anything else gets the greeting text and the state stays GREETING. This state has no branch for decline, FAQ, or price questions. None.

COLLECT_VEHICLE: only two things are handled — a decline (DONE, marked declined) and completing the slots (lookup). Confirm, price, haggle, and FAQ intents are all unhandled here and get the same "what's still missing" reply.

QUOTE: the fullest state. Haggle (once only, an extra 5%), FAQ (answered without leaving QUOTE), price question (re-present the quote), confirm (DONE / confirmed), decline (DONE / declined), a message carrying vehicle info (re-run the lookup — treated as a vehicle change), and anything else re-presents the quote with a nudge.

DONE: intercepted in handle_message. Any new message after DONE discards the conversation and starts a fresh one — vehicle, quote, and haggle flag all reset. I verified the reset by round-trip.

Intent parsing (MockLLM.parse) is where this article lives. The precedence order in code is: decline > confirm > haggle > FAQ > price > greeting > provide-vehicle > unknown. Keywords match by substring (w in low); only three short tokens — sí, да, نه — get word-boundary matching. The confirm list contains everyday words: "买" (buy), "好的" (okay), "可以" (can/sure), "ok", "yes". The decline list contains a bare "no". Vehicle extraction only recognizes catalog vehicles: a (make, model) pair from the simulated catalog must both appear as substrings; the year is the first 4-digit number matching (19|20)\d\d.

The two tools: wiper_db.lookup searches the 8 simulated fitments by make, model, and year range. pricing.build_quote prices blades in three length tiers (up to 20" $12.99, 21–24" $14.99, 25"+ $16.99), takes 10% off the subtotal (the code comment calls it a bundle discount), and a further 5% after a successful haggle.

Method: repository code unmodified, conversations driven in-process under different "phone numbers" (arbitrary string IDs), logging every message, resulting state, quote total, haggle flag, and full order intent. Quotes for all 8 fitments were recomputed independently and compared. Collisions were first classified by calling parse directly, then the three worst were replayed inside real conversations to observe the consequences. The bench script and raw JSON are archived with this article and can be re-run.

3. Measurements

3.1 The main flow: split vehicle entry, a full Spanish run, exact quote math

The standard path (hello → Toyota Camry → 2018 → how do I install it → okay, I'll take it) walks GREETING → COLLECT_VEHICLE → QUOTE → QUOTE → DONE and produces a confirmed intent with vehicle, sizes (26/20"), and a $26.98 total. All three slots in one message jump straight to QUOTE. A full Spanish run (Hola → Corolla 2021 → ¿Cuánto cuesta? → ¿Incluye instalación? → Vale, sí) answers in Spanish at every step, locks the language to es on the first message, and records detected_language: es on the final intent.

All 8 quote recomputations matched. Camry, worked in full: $16.99 + $12.99 = $29.98 subtotal, $26.98 after 10% off, $25.48 after the haggle (85% of subtotal). The RAV4, the only record with a rear blade: $42.97 subtotal, $38.67 quoted, $36.52 haggled. One bookkeeping discrepancy worth recording: the comment describes the 10% as a driver+passenger bundle discount, but the discount base is the whole subtotal including the rear blade — the RAV4's $4.30 discount is computed on all three blades. The error favors the customer; the books should still say what the code does. A rounding note, so nobody's calculator disagrees: the Quote's discount field is rounded to cents on its own (the Camry's exact 10% is 2.998, stored as 3.0), while the total is computed from the unrounded discount and then rounded. I reconciled all 16 combinations (8 records, with and without haggle): discount plus total reproduces the subtotal exactly in every case.

3.2 Haggle once, and only once — until the vehicle changes

Altima 2019 quotes at $26.98. "太贵了" (too expensive) drops it to $25.48 and sets the flag. "再便宜点" and "还能便宜吗" both get "that's already my lowest," and the total does not move. As a one-shot allowance enforced by a boolean and a recomputation, there is no path around it. That part holds.

What the tests never covered is haggling across a vehicle change. Haggle the Altima down to $25.48, then name a Civic 2020: the lookup rebuilds the quote without the haggle, the new total is back to $26.98 — and the haggled flag stays set. Haggle again and the agent replies "this is already the lowest price" about a price that was never discounted. The customer is told something factually false. The cause is granularity: the flag lives on the conversation, not on the quote. A vehicle change neither carries the discount over nor clears the flag; of the two coherent choices, the code makes neither.

3.3 FAQ only lives inside QUOTE

In QUOTE, all six FAQ topics (install, warranty, lifespan, payment, pickup, shipping) answer correctly, stay in QUOTE, leave the quote untouched, and close with an ordering prompt. That matches the documented design — FAQ must not break the sales flow — and it held in every probe.

Ask before the quote and the question disappears. At GREETING, "怎么安装" (how do I install it?) gets only the greeting; the question is neither answered nor logged (no FAQ note is recorded). At COLLECT, the same question gets a repeat of "which year is it?", the question vanishes, and after completing the vehicle the customer has to ask again. Price and haggle attempts at COLLECT meet the same fate. There is no holding area in this machine for "remember this, answer it when we get there." Out-of-flow input is not refused; it is silently dropped, and the reply gives the customer no sign their question was never heard — it just reads as a non sequitur.

Declines are state-dependent too. "算了" (forget it) at COLLECT and at QUOTE correctly ends as declined. "不要了" (I don't want it) as the very first message is not handled at all — GREETING has no decline branch — and earns a cheerful greeting. Minor on its own; included because it shows GREETING's intent coverage is not partial, it is zero.

3.4 Intent collisions: 13 probes, three failure families

Feeding everyday phrasings to parse, 7 of 13 came back with a classification visibly at odds with what the speaker meant, in three families:

Confirm words are too broad and outrank everything else they co-occur with. "可以便宜点吗" (can you make it cheaper?) classifies as CONFIRM — "可以" is in the confirm list and confirm is checked before haggle. "好的，多少钱" (okay, how much?) is CONFIRM, not a price question. "我想买，先问问保修" (I do want to buy, but first let me ask about the warranty) is CONFIRM — "买" fires confirm even though the FAQ topic was correctly detected as warranty and never gets used. "yes but too expensive" is CONFIRM for the same structural reason.

The decline list contains a bare substring. "no" is matched as a substring, so "I don't know" and "another question" both classify as DECLINE — both contain "no". The codebase already has a word-boundary mechanism for short tokens (CONFIRM_TOKENS / DECLINE_TOKENS); "no" simply was not put in it.

Decline outranks haggle. "不要太贵了" (don't make it too expensive) and "不买了太贵" classify as DECLINE. Whether the first of those is really a refusal is debatable — it reads more like a haggle opening — but the machine offers no clarifying move. Between "kill the conversation" and "treat as haggle," it picks kill.

The three worst, replayed inside real conversations:

In QUOTE, "可以便宜点吗" → straight to DONE, a confirmed order intent at the full $26.98. The customer asked for a discount; the system recorded a sale.
In QUOTE, "我想买，先问问保修" → straight to DONE, confirmed, warranty question never answered. The buying intent was genuine, but the flow closed before the customer's actual question, leaving the human follow-up holding a "confirmed" record and an unanswered conversation.
In QUOTE, "I don't know" → straight to DONE, declined, logged as "declined at quote." The customer was undecided; the system recorded a rejection. The same sentence at COLLECT ends the same way.

Controls, so the word lists are not condemned wholesale: "能便宜点吗" (the same haggle without "可以") classifies correctly as HAGGLE; "先看看保修再说" goes to FAQ; "好的" and "OK" in QUOTE are legitimately CONFIRM. What breaks is the compound sentence — and real customers rarely speak in single keywords.

3.5 Changing vehicles: an off-catalog car is quietly merged into the old one

In QUOTE, "Honda Civic 2020" correctly re-quotes a new vehicle (same $26.98 total, but sizes change from 26/20 to 26/18, so the change is visible). "BMW X5 2020" produces the reply "Found it! Toyota Camry 2020." BMW and X5 are not in the simulated catalog, so vehicle extraction fails entirely and only the year, 2020, is captured; slot merging folds that year into the old vehicle, the lookup succeeds for a Camry 2020, and a fresh Camry quote is presented with total confidence. The customer thinks they asked about a BMW. The agent quoted them a Camry. At no point does anything say "I can't find that car."

The same off-catalog car at the start of a conversation behaves almost honestly: extraction fails and the agent asks which vehicle — phrased as "still missing the vehicle," which will confuse a customer who just named one, but at least nothing is fabricated. Two different failures should not be conflated here: that was an extraction failure (make and model never parsed, slots incomplete, so all it can do is ask). A lookup failure is different — "Toyota Camry 2005" as a first message parses completely, misses the catalog, and gets an explicit "not found" with a fallback to COLLECT, which I also tested. In QUOTE, the pre-existing slots mask the failure and turn it into a silent substitution. The only difference is whether old slot values exist to paper over the gap.

A year-only change has the mirror-image problem. "2025" merges into Camry, falls outside the catalog range (2018–2024), and this time the lookup properly misses: the state falls back to COLLECT and asks the customer to double-check. The handling is right; the cleanup is not — the old $26.98 quote stays attached to the conversation. When the customer then gives up ("算了"), the declined order intent carries that stale quote, with the vehicle field reading Camry 2025: a car with no fitment, wearing a 2018 quote, filed as if the customer had rejected a real offer.

3.6 Persistence and isolation: two passing grades

Mid-conversation — quoted and haggled — I restored the conversation into a brand-new agent instance from the same JSON file store. State QUOTE, vehicle, haggled total $25.48, the flag, and all 4 history entries came back exact, and the restored conversation answered a warranty question while staying in QUOTE. The to_dict / from_dict round-trip is lossless. Two interleaved customers (one haggling, one completing a vehicle) never leaked slots, quotes, or flags into each other; keying by phone number holds.

Gibberish in QUOTE ("asdf qwer") re-presents the quote with a nudge and changes nothing. Not smart, but it neither crashes nor closes anything — and in context it is the intent layer's most trustworthy behavior. Only UNKNOWN gets this treatment. Every failure above came from a match that was almost right.

4. Analysis and trade-offs

4.1 What the architecture genuinely gets right: the parser never holds the pen on a deal

Read against the design, the result worth keeping is the division of authority. The state machine owns the flow, the tools own the numbers, and the parsing layer — rules today, an LLM tomorrow — only emits intents and slots. Exact quote math, a hard one-time haggle cap, FAQ containment, and lossless persistence all follow from decisions being kept in deterministic code. Swapping in RealLLMProvider will improve parsing, but its intents still pass through the same transition table, and it still cannot mint a price or close a deal. For a small-shop agent that is the right border: language may be probabilistic; the ledger may not.

4.2 The defects are not in the machine; they are in its entry contract

None of the three measured problem families is a wrongly drawn transition. The table's flaw is that it trusts whatever the entrance hands it: whatever intent parse returns gets executed, whatever the slot merger produces gets looked up. Every business-harming action in the tests — the wrongly closed order, the wrongly quoted car, the fabricated intent record — happened at a point where input was over-trusted:

CONFIRM and DECLINE are irreversible (DONE, plus a recorded intent), yet they are triggered by the weakest evidence in the system: a single substring. Irreversible actions should demand stronger evidence than reversible ones (re-presenting a quote, answering an FAQ). The current precedence inverts that — the irreversible intents are checked first.
Slot merging overwrites field by field and cannot distinguish "the customer added the year" from "the customer changed cars and the new car failed extraction." The second case should route to clarification, not be healed with stale values.
haggled and quote have mismatched lifetimes. A quote can be replaced while its flag persists; a failed lookup retreats while its quote persists. Derived state has no rule that says "a new quote invalidates everything derived from the old one."

4.3 Fix order: gate the irreversible actions first, upgrade the NLU later

In cost-benefit order. First, move short words like "no" out of the substring lists into the existing word-boundary sets, and tighten confirm words like "可以" and "买" with co-occurrence conditions (confirm only when no haggle or price wording is present). That is an afternoon of word-list and ordering work and it dismantles most of §3.4. Second, reorder precedence so reversible intents win: FAQ, price, and haggle are evaluated before confirm and decline, and confirm only stands when the sentence carries no other request. "Wants to buy, but first the warranty" then gets its answer and stays in QUOTE instead of being closed early. Third, give CONFIRM a read-back step: before DONE, echo vehicle and total — "shall I book this?" — and record the intent only on an explicit yes. One extra CONFIRMING state holds it, and it mirrors what a human clerk does anyway. Fourth, add a merge rule: when a message carries a year or vehicle-like text but extraction comes back empty and inconsistent with the current vehicle, do not merge — clarify; and when a lookup miss retreats to COLLECT, clear the quote and the haggle flag with it. Fifth, and only then, swap MockLLM for RealLLMProvider, with this article's 13 probes plus real conversation logs as the regression set. A better brain fixes part of the compound-sentence problem; it fixes none of the merging or lifecycle problems, because those belong to the machine itself.

The judgment underneath the ordering: in a demo, the thing most easily overlooked is not answer quality but closing correctness. A wrong FAQ answer gets asked again. A wrongly recorded confirmed intent gets acted on — the owner preps stock, reaches out, and finds nobody there.

5. Limitations and open questions

Stated plainly. First, everything ran on MockLLM; the collision patterns are specific to a rule-based parser. A real LLM endpoint would very likely handle the compound sentences better, but I did not re-run the 13 probes against RealLLMProvider, so the post-swap error rate is unmeasured. Second, the catalog has 8 simulated records at demo prices. In a real catalog of hundreds of records, extraction failures get rarer but not harmless — the silent-merge mechanism is size-independent, and the claim here is about the mechanism. Third, the scripts were written backwards from the transition table, so they cover the paths I could think of; real customers' speech-to-text errors, code-switching, and two cars in one sentence were only spot-checked, not systematically tested. Fourth, the WhatsApp boundary layer (webhook signature verification, Cloud API delivery, usage counters) is out of scope — this round exercised the in-process agent API only; the repo's own tests, all 56 passing, cover parts of that layer, but no end-to-end message test was run. Fifth, beyond Spanish, the Russian and Persian flows were verified only at greeting and language-detection level, following the repo's own test coverage; the templates have not had native-speaker review, which the README also states.

Three open questions. Adding a CONFIRMING read-back state will cost some completion rate — how much is only knowable with real traffic. How fine-grained the merge-clarification rule should be (year conflicts only, or any "vehicle-like text, empty extraction") needs real conversation logs to calibrate. And two cars in one order — a customer buying for themselves and a family member — does not fit the one-conversation-one-vehicle data model at all; whether that is a notes-field workaround or a multi-intent model change is undecided.

6. References

Wiper-Service-Agent source (version tested, commit 9675291): github.com/Ezra-Zhao/Wiper-Service-Agent. Key files: wiper_agent/agent.py (state machine, slot merging, language upgrade), wiper_agent/llm.py (MockLLM precedence and word lists, the RealLLMProvider parsing contract), wiper_agent/conversation.py and store.py (state and persistence), wiper_agent/schemas.py (the four states and data models), tools/wiper_db.py (8 simulated fitments, marked SIMULATED in the file header), tools/pricing.py (three price tiers and discounts), tools/faq.py (six FAQ topics, five-language templates).
Bench script and raw data for this article: data/2026-10-16-wiper-bench.py and data/2026-10-16-wiper-bench.json (27 conversation traces, 13 intent-collision probes, 8 quote recomputations, persistence round-trip, two-customer isolation), archived with the article; the numbers can be reproduced by running the script against a clone of the repository.
Repository test suite: tests/run_tests.py — 56 passed, 0 failed in this run (the README Quickstart says 44, which is out of date).

Author: Guangyi Zhao. The code, experiments, and text in this article are clean-room work of my own: the subject is an open-source scaffold I maintain, every conversation was script-generated, no WhatsApp account was connected, and no real customer's chats, phone numbers, or orders were used. No client information, client systems, or third-party non-public data are involved. All vehicles, prices, and language templates are the repository's simulated data and are not presented as real product information; the scaffold is not presented as a production system in service.

Before You Wire an LLM Into GPU Fault Triage: AI-Node-Sentinel's Integration Points and Six Measured Failure Modes

Guangyi Zhao — Mon, 19 Oct 2026 05:01:17 +0000

Author: Guangyi Zhao
Date: 2026-10-14
Subject: AI-Node-Sentinel (open-source project by the author, github.com/Ezra-Zhao/AI-Node-Sentinel), scaffold v0.1, Python 3.12.3
Environment: Linux x86_64 VM, repository code run unmodified; LLM experiments use local fake providers implementing the project's own protocol — no real model API was called

1. Problem and background

In a GPU cluster, the first signal of failing hardware is usually one line in the kernel log: an Xid. XID is NVIDIA's numbering scheme for GPU hardware and firmware errors — a GPU falling off the PCIe bus, an uncorrectable double-bit ECC error, an NVLink fault, a GSP firmware timeout each has its own code. The on-call ritual is manual: pull dmesg from a thousand GPUs, look up what each code means, then decide — drain the node, reset the GPU, reboot, or start an RMA. Decide slowly and the training job keeps retrying on broken hardware, burning GPU-hours that cost real money.

The obvious impulse is to hand this to an LLM agent. Let the model read telemetry, look up error codes, and recommend actions — isn't that exactly what agents are for?

My own open-source project, AI-Node-Sentinel, was scaffolded around that idea: telemetry comes in, XID events are extracted, a lookup table enriches them, triage rules are applied, and a structured report comes out, with a LangChain-based slot reserved for LLM reasoning. Let me state the honest status up front, because everything after this paragraph depends on it: the project is scaffold v0.1, and its README says so. The triage decision tree is a deliberate stub — every event gets NEEDS_HUMAN_REVIEW with confidence 0.0. The LangChain provider is also a stub that raises NotImplementedError on construction. All telemetry is synthetic, produced by a bundled simulator. Only three stages actually run: ingestion, lookup, and report rendering.

So this is not a write-up of a production diagnostic system. It is the opposite: I put the scaffold on the dissecting table and ran six fault-injection experiments to measure where this pipeline would break once an LLM is actually wired in. The results generalize to anyone attaching an LLM to an operations system, because none of the six breakpoints is a model-capability problem. They are wiring problems: an output contract that doesn't match, evidence fed too thin, one malformed record poisoning the whole batch, a prompt path coupled to the working directory, stateless single-event myopia, and corroborating signals discarded by the pipeline itself. A stronger model fixes none of these.

2. Method and architecture

The codebase is small: eight core Python files, roughly 470 lines including tests and the demo. Four parts. agent/schemas.py defines the data types: TelemetryEvent (a normalized event with source, kind, and payload), XidFinding (an XID event enriched by lookup), and TriageVerdict (action, confidence, rationale). tools/xid_lookup.py is a static table mapping codes to name, category, and severity, seeded only from NVIDIA's public XID documentation and Google Cloud's GPU troubleshooting guide; a miss returns None — it never guesses. tools/triage_rules.py holds the decision tree, currently the stub. agent/orchestrator.py is the TriageAgent: ingest, extract XID events, triage each one, render the report.

The LLM integration point lives in agent/llm.py: a minimal protocol, LLMProvider, requiring one method, reason(system_prompt, evidence) -> str. The orchestrator first computes the deterministic verdict; if a provider is present, it builds an evidence string, calls the provider, and appends the returned text to the verdict's rationale with an [LLM] prefix. The draft system prompt in prompts/triage_system.md instructs the model to cite only the evidence given, to say explicitly when a code is unknown, to return exactly one action with a confidence between 0 and 1, to force NEEDS_HUMAN_REVIEW whenever confidence is below 0.7, and to answer in JSON.

Method: five experiment groups against the unmodified repository, with the scripts and raw JSON archived alongside this article. (1) Baseline scale: the bundled simulator generates telemetry windows for 4, 16, and 64 nodes; I record event composition, verdict distribution, and pipeline latency. (2) Lookup coverage: table size plus ten out-of-table probe codes. (3) Malformed-data injection: events with a missing xid key, a non-numeric xid, and xid=None, mixed into otherwise good batches. (4) Fake-LLM experiments: a local provider that returns exactly the JSON the prompt demands, to observe what actually changes in the verdict; then a provider that always raises, to observe fault propagation. (5) Failure-mode probes: repeat faults on one GPU, the fate of corroborating events, and prompt loading from a different working directory.

3. Measurements

3.1 Baseline: the deterministic part is too fast to matter, and the verdicts are the stub's

The simulator emits one thermal and one ECC baseline event per GPU, then injects one or two XID events on about 35% of GPUs:

Scale	Events	XID events	Verdicts	Unknown verdicts	Severity (critical/high/unknown)	Pipeline time
4 nodes × 8 GPUs (seed 42)	76	12	12	1	10 / 1 / 1	~0.1 ms
16 nodes × 8 GPUs (seed 42)	313	57	57	5	39 / 13 / 5	~0.2 ms
64 nodes × 8 GPUs (seed 42)	1,285	261	261	34	157 / 70 / 34	~0.7 ms
64 nodes × 8 GPUs (seed 7)	1,295	271	271	35	177 / 59 / 35	~0.7 ms
64 nodes × 8 GPUs (seed 123)	1,282	258	258	28	167 / 63 / 28	~0.7 ms

Three observations. First, every verdict is NEEDS_HUMAN_REVIEW at confidence 0.0. That is stub behavior, not measured intelligence, and I record it as the control group for what follows. Second, roughly 1,300 events process in under a millisecond — about 1.77 million events per second on the raw single-run perf_counter timing (the table rounds to 0.1 ms; sub-millisecond figures are order-of-magnitude only, not a precise benchmark) — because lookup is a dictionary read. At this article's scale the deterministic part is free; streaming ingestion and window state will need their own measurement later, but the bulk of future latency and cost will come from the LLM step. Third, do not misread the 11–13% unknown share: the simulator's injection pool has eight codes, one of which (999) is a deliberately synthetic unknown, so the unknown rate is an artifact of pool design — one eighth by construction — not a field measurement of unknown codes in a real fleet.

The repository's six bundled tests all pass when invoked directly (determinism, known-code lookup, unknown codes not guessed, stub routing to human review, report shape). This machine has no pytest installed, so I ran the test functions directly rather than through the runner; noted for accuracy.

3.2 Lookup coverage: seven codes, everything else is "unknown"

The table contains exactly seven codes: 13, 31, 48, 74, 79, 119, 120. Ten out-of-table probe codes (43, 45, 56, 63, 64, 68, 92, 94, 109, 154) all return None and are treated exactly like 999: severity unknown, known=False, routed to human review. The routing behavior is right — refusing to guess is this pipeline's most important safety property. The coverage is another matter: any XID outside those seven earns the operator exactly one piece of information, "an unknown error occurred, go look." The table was hand-seeded from public documents; extending it is unglamorous work, and section 4.3 argues it is still not the first thing to fix.

3.3 Malformed data: one bad event poisons the batch

The orchestrator extracts the code with int(event.payload["xid"]) — no validation, no per-event isolation. Mixing malformed events into good batches:

Injection	Outcome
One well-formed event	Completes, 1 verdict
Good + missing xid key + good	Entire run raises KeyError; completed verdicts are lost with it
Good + xid as a non-numeric string	Entire run raises ValueError
Good + xid=None	Entire run raises TypeError

run() processes events in a list comprehension, so the first exception discards every verdict in the window. Translate to production: a dmesg parser will eventually emit a field-incomplete event, and when it does, the failure mode is not "one event undiagnosed" but "the whole window's verdicts vanish, as a process-level exception instead of a line in the report." Per-event isolation with a bad-record counter is table stakes for this kind of pipeline, and it is absent here.

3.4 The fake-LLM experiment: the output contract is broken, and the model doesn't get a vote

This is the experiment the article exists for. I implemented a fake provider that obeys the system prompt perfectly, returning JSON that recommends DRAIN_NODE at confidence 0.95 for XID 79 (GPU fallen off the bus). Wired into the orchestrator, the verdict's actual state:

Field	Fake provider output	Final verdict
action	DRAIN_NODE	NEEDS_HUMAN_REVIEW (unchanged)
confidence	0.95	0.0 (unchanged)
rationale	The full JSON	Appended verbatim as text, prefixed [LLM]

The code makes the reason plain: the protocol returns a string, the orchestrator only concatenates it, nothing ever parses the JSON, and nothing overwrites the deterministic verdict. The prompt's output format and confidence gate are a gentlemen's agreement with the model that the code side never honors. Even if the real LangChain provider were implemented tomorrow and the model followed the format perfectly every time, the triage decision would remain 100% determined by the stub; the LLM's total influence would be a paragraph pasted into the rationale.

The second half of the experiment is evidence thickness. The exact evidence string the orchestrator assembles is: node=node-00 gpu=0 xid=79 name=GPU has fallen off the bus severity=critical ts=. That is all. The lookup entry has category and notes fields — including remediation hints and sourcing notes — and neither enters the evidence. The same GPU's thermal and ECC readings cannot enter it either, because the evidence is built inside the per-event triage function, whose signature has no window context at all. The prompt's first rule orders the model to cite only the evidence provided. The pipeline starves the evidence down to a restatement of the error name, then forbids the model from reaching beyond it. If the model gives an informative answer anyway, it can only do so by breaking the rule. That contradiction is a wiring decision, not a model defect.

Third: fault propagation. Replace the provider with one that always raises RuntimeError, and triage_event lets the exception straight through — the whole run dies, the same death as section 3.3. A provider is an external dependency; timeouts, rate limits, and network jitter are its normal weather. An optional reasoning step should degrade to the deterministic verdict, never take the batch down with it. There is no degradation path here.

3.5 The prompt path is coupled to the working directory

The orchestrator loads the prompt with a relative path, open("prompts/triage_system.md"). From the repository root it works. From /tmp, the identical code raises FileNotFoundError immediately. (The LangChain provider raising NotImplementedError on construction is an honest stub, not a defect.) The relative path is the kind of bug that bites exactly once, in production: services started by systemd, containers, or schedulers rarely have the repository root as their working directory, while a developer's terminal always does — which is why it never reproduces at a desk.

3.6 No memory, and corroboration blindness: two failure modes with one root

Repeat faults do not escalate. Three XID 79 events on the same GPU within three hours produce three verdicts that have never heard of each other: same action, same confidence, same rationale. Causality, stated precisely: the identical verdicts are directly caused by the decision tree being a stub — any architecture would produce them today. The architectural problem sits one layer down. The stub's own comments sketch the intended real rule — XID 79 twice in 24 hours on one GPU should mean drain plus reboot — but the orchestrator triages events one at a time, statelessly, so even once that rule is written, the pipeline has nowhere to put it. This is not only a missing rule; it is missing state.

Corroborating signals are filtered out by the pipeline itself. I built a three-event window by hand: one GPU reports an ECC double-bit error count of 17, then a temperature of 104 °C, then XID 13. The prompt's second rule says app-level faults like XID 13/31 should not trigger a drain recommendation without corroborating evidence — which implies corroboration should change the judgment when present. The measured result: three events in, one verdict out. The two non-XID events are dropped at the extraction stage; they produce no verdicts and are never attached to the XID 13 verdict as context. At baseline scale this is 1,024 of 1,285 events in the 64-node window. The pipeline demands corroboration-aware reasoning in its prompt while deleting the corroboration from its data path.

4. Analysis and trade-offs

4.1 Deterministic first, LLM second — and the order is not negotiable

The scaffold's best judgment call is its layering: lookup and rules in a deterministic core, the LLM as a downstream reasoning supplement, and — until the decision tree exists — everything routed to humans with an honest confidence of 0. The reasoning is simple. Drain, reset, reboot, and RMA are real operations on production hardware. Anything a table or a rule can decide should never be improvised by a probabilistic model at 3 a.m.; the LLM's value is synthesis and explanation, not authority.

Section 3.4 yields a counterintuitive corollary: the broken output contract is currently an accidental safety net. Because the model's JSON is never parsed into the verdict, a hallucinating model today can pollute a rationale but cannot drain a node. But it is an accident, not a design. The correct fix is not to preserve the break; it is to add parsing plus validation — actions must land inside the site's runbook vocabulary, confidence below 0.7 must be rewritten to human review in code (the prompt already states this rule; move it from prose into the program), and parse failure must degrade to the deterministic verdict alone. Whoever wires up the parsing without the validation has personally removed the safety net.

4.2 The failure-mode list, in fix order

None of the six failure modes needs a better model or a better prompt. All six are ordinary engineering, and I would fix them in this order:

Per-event isolation. Wrap each event's triage in exception handling; count and surface bad records in the report. Batch survival comes before everything else.
Honor the output contract. JSON parsing, vocabulary validation, and the confidence gate in code, per section 4.1.
Evidence assembly. Include the lookup's category and notes, and pass the triage function a per-GPU window summary — recent same-code counts, thermal and ECC extremes — so the prompt's "cite only the evidence" rule becomes executable.
A state layer. Repeat counting and escalation need window-level aggregation, which means restructuring from stateless per-event triage to aggregate-then-triage. It is an architectural change, so it goes after the contract and evidence fixes.
Resource paths and graceful degradation. Load the prompt as a package resource or absolute path; wrap provider calls with timeout and exception handling that fall back silently — but visibly in the report — to the deterministic verdict.
Corroboration plumbing. Stop discarding non-XID events at extraction; attach them to the same GPU's XID verdicts as context. It shares a root with item 3 but is listed separately: item 3 changes the evidence template, this one changes the data path.

Extending the lookup table is deliberately not on this list. It is data-dictionary work, not a failure mode; seven codes is thin coverage and the table should grow — after these six, because without them a 70-code table still dies batch-wide on the first malformed event.

4.3 A judgment: trustworthiness in systems like this comes from restraint, not from the model

The design choices I would defend most strongly in this scaffold are the things it refuses to do. Unknown codes are not assigned invented meanings. An unfinished decision tree issues no actions. Simulated data is labeled SIMULATED from the generator to the report header. The stub raises NotImplementedError instead of returning a fabricated rationale. Together these refusals make every output auditable: for any line in the report you can tell whether it came from the table, a rule, or a model. That is precisely what gets lost first when an LLM is bolted onto an operations system — once fluent model prose washes over the deterministic core's true state, the on-call engineer can no longer tell what was looked up from what was made up, and the report stops deserving trust. All six failure modes point at the same repair: make the model's scope, evidence, and failure behavior explicit in code, not implied by prompt wording.

5. Limitations and open questions

The limitations are material. First, all telemetry is synthetic, generated by the bundled simulator; its distributions are set by injection probabilities and the injection pool, so every proportion in this article validates pipeline behavior only and says nothing about fault distributions in a real fleet. Second, the LLM experiments use a local fake provider and verify wiring contracts and fault propagation; no real model was called, so this article makes no claim about any model's reasoning quality. How often a real model, fed evidence this thin, would reach beyond it and fabricate is a separate measurement that requires wiring one up. Third, the scale ceiling here is 64 nodes and about 1,300 events on a single VM; concurrent ingestion, persistence, and alerting channels are untested. Fourth, lookup coverage is reported as table size plus out-of-table probes; I did not diff the table against NVIDIA's complete published catalog, so no coverage denominator is claimed.

Three open questions remain. The aggregation grain for the state layer — per GPU, per node, or per NVLink domain — implies different escalation rules and false-positive costs, and calibrating it needs real fault sequences. The 0.7 confidence gate in the draft prompt was picked, not calibrated; once a real model is wired in, it should be set by replaying historical tickets. And once corroborating evidence enters triage, the baseline thermal and ECC stream is roughly four times the XID volume — what evidence assembly costs in tokens and context length has to be measured when the real model arrives.

6. References

AI-Node-Sentinel source (version measured here): github.com/Ezra-Zhao/AI-Node-Sentinel — key files: agent/orchestrator.py (orchestration and evidence assembly), agent/llm.py (LLMProvider protocol and LangChain stub), agent/schemas.py (data types), tools/xid_lookup.py (lookup table), tools/triage_rules.py (decision-tree stub), prompts/triage_system.md (draft system prompt), telemetry/simulator.py (simulator).
Experiment scripts and raw data for this article: data/2026-10-14-sentinel-bench.py and data/2026-10-14-sentinel-bench.json, archived with this article; every number can be re-checked by re-running the script.
NVIDIA, Xid Errors documentation: docs.nvidia.com/deploy/xid-errors — the authoritative source for the lookup metadata, also cited in the header of tools/xid_lookup.py.
Google Cloud, Troubleshooting GPU issues: cloud.google.com/compute/docs/troubleshooting/troubleshooting-gpus — cross-check source for the lookup table, as above.

Author: Guangyi Zhao. The code, experiments, and text in this article are clean-room work by the author: the subject is an open-source scaffold project maintained by the author, all telemetry is synthetic data from that project's simulator, and all experiments ran in the author's own local environment. No customer information, customer systems, or third-party non-public data are involved. Every number comes from the measurements listed at the top of this article; the simulated nature of the data and the limits of the conclusions are stated in the text and in section 5, and no scaffold state is presented as a production system.

The Total Was Right and the Structure Was Wrong: Stress-Testing a Flower-Counting Pipeline Against Light and Occlusion

Guangyi Zhao — Mon, 19 Oct 2026 04:58:38 +0000

Author: Guangyi Zhao

1. The problem

A crop-counting pipeline does not deliver one number. For yield estimates and labor scheduling, a grower needs two: how many flowers have opened, and how many buds are still closed. The first drives this week's harvest, the second next week's. The real deliverable is the split, not the sum.

My open-source project Bloom-Counter does exactly this: photos in, HSV color segmentation plus contour analysis out, with per-image bloom and bud counts aggregated by region and date. Its README carries a handsome scorecard: 6 simulated photos, 357 flowers, 99.7% detection recall. This article starts from an uncomfortable claim: that 99.7% proves nothing. Two reasons, both sitting in the code for anyone to check.

First, the simulator and the detector were tuned to each other. A comment on line 23 of simulator/generate.py says it plainly: the flower colors in BGR are "tuned to sit inside the detector's HSV petal ranges." The generator paints flowers to the detector's taste; the detector then counts them. A perfect score is structural.

Second, the simulator's _place function uses rejection sampling to guarantee that no two flowers ever overlap, and the demo's scoring in examples/demo.py is (detected blooms + buds) / (true blooms + buds) — totals only, no per-object matching, no per-class breakdown.

A real greenhouse inverts both assumptions: light swings across the day, leaves occlude as a matter of course, and flowers grow in clusters. So I took the unmodified detector and systematically broke the simulator's two hidden assumptions, measuring how it fails first, how badly, and whether two cheap remedies actually help.

2. Method

Detector: the stock ColorSegmentationDetector (detector/color_segmentation.py), parameters untouched: min_area=120 px², bloom_area_min=900 px² (a connected component at or above 900 px² counts as a bloom, below it as a bud), morphology kernel 7. It thresholds pink, white, and yellow petal ranges in HSV, applies closing and opening, then counts components.

Scenes: four 1280×960 images from the repo's own simulator (seeds 101/202/303/404), 210 objects in total: 134 blooms, 76 buds, each with a ground-truth center and radius.

Scoring: not the totals ratio. Greedy center-distance matching — a detection counts as a hit only if its center lands within max(38 px, 1.1 × radius) of a ground-truth object of the same label — yielding precision, recall, and F1, with bloom recall and bud recall reported separately, micro-averaged over the four scenes. Sanity check first: on the untouched scenes this stricter scoring still returns precision = recall = F1 = 1.000 with zero false positives. The baseline is perfect because the pair is co-tuned, which is the point.

Two stress groups:

Lighting: parametric whole-image transforms standing in for time-of-day and weather: gamma 0.6 / 1.6 / 2.2, exposure −50 (α=0.78, β=−50) and overexposure +60 (α=1.25, β=+60), a left-to-right shadow gradient (0.35→1.0), a half-frame shadow (left 45% at ×0.40), a warm color cast (B×0.72, R×1.18), and low light (gamma 1.9 plus Gaussian noise σ=12).
Occlusion: leaf-colored ellipses painted over 60% of objects at their true centers, with radius 0.5 / 0.8 / 1.1 × the object's radius; and extra blooms drawn beside existing ones at offsets of 2.2 / 1.6 / 1.2 / 0.8 / 0.45 × radius (+10 blooms per scene, true blooms rising to 174) to simulate clustering.

Two remedies: gray-world white balance followed by CLAHE on the V channel as preprocessing; and an area-based split estimator that, for bloom components much larger than the median single-bloom box area (5,889.5 px² in this batch), adds counts proportional to area.

Raw data and scripts: data/2026-10-17-bloom-bench.json and data/2026-10-17-bloom-bench.py, reproducible with one command in the repo's virtualenv.

3. Results

3.1 Lighting: demotion before disappearance

Condition	Recall	Bloom recall	Bud recall	Blooms found / 134	Buds found / 76	Total error
Clean	1.000	1.000	1.000	134	76	0.0%
Gamma 0.6 (brighter)	1.000	1.000	1.000	134	76	0.0%
Gamma 1.6 (darker)	1.000	1.000	1.000	134	76	0.0%
Low light, gamma 1.9 + noise	1.000	1.000	1.000	134	76	0.0%
Gamma 2.2 (much darker)	0.576	0.336	1.000	45	179	+6.7%
Exposure −50	0.810	0.702	1.000	94	116	0.0%
Overexposure +60	1.000	1.000	1.000	135	90	+7.1%
Shadow gradient 0.35→1.0	0.895	0.836	1.000	112	98	0.0%
Half-frame shadow ×0.40	0.905	0.851	1.000	114	96	0.0%
Warm cast	0.638	1.000	0.000	135	25	−23.8%

Look at exposure −50 and the two shadow conditions: total error 0.0%. Under the repo demo's scoring, all three are perfect runs. Yet a quarter to a sixth of the blooms are gone, and twenty-odd phantom buds appeared. Splitting the first half-shadow scene makes it vivid: of 20 objects inside the shadow, 70% were recovered; of 28 in the light, all of them.

The mechanism is easy to trace. Dimming eats the petal mask from the edges first: V falls below the pink range's floor, the mask shrinks, and the component's area shrinks with it. Once area crosses below the 900 px² line, the object does not vanish — it is reclassified as a bud. This classifier demotes before it deletes: at gamma 2.2 only 45 blooms remain while "buds" swell to 179, and the total is off by just +6.7%.

The warm cast fails in the opposite direction. Blooms survive intact (white and yellow ranges back them up), but bud recall goes to exactly zero: buds exist in only one color range, pink, and the cast pushes their hue out of it. All 76 buds disappear; the 25 detected "buds" are fragment false positives.

Overexposure fails differently again: recall holds at 1.000 but precision drops to 0.933, with 15 false positives — washed-out leaf fragments impersonating buds. Getting brighter does not lose objects; it manufactures them.

The good news is real, too: gamma 0.6 brightening, darkening up to gamma 1.6, and low light with noise leave the detector untouched. Its robust zone is wider than I expected. Between the intact 1.9 run and the collapsed 2.2 run the failure threshold sits somewhere in that interval — but sampling is sparse, and the 1.9 run also carried noise rather than being a pure gamma control, so I cannot pin the cliff down.

3.2 Preprocessing: fixes global shifts, breaks clipped and noisy images

Gray-world white balance plus CLAHE on V splits cleanly by failure type (false positives below are totals across the four scenes):

Condition	Recall before	Recall after	F1 after	Cost
Gamma 2.2	0.576	1.000	0.981	8 FP
Warm cast	0.638	1.000	0.961	17 FP
Shadow gradient	0.895	1.000	0.963	16 FP
Half-frame shadow	0.905	1.000	0.959	18 FP
Gamma 1.6 (was perfect)	1.000	1.000	0.970	13 FP
Overexposure +60	1.000	0.319	0.404	Recall collapse
Low light + noise (was perfect)	1.000	0.991	0.726	155 FP

For global brightness and hue shifts the pipeline recovers almost everything. It is not free: on images that were already fine it invents 13–18 phantom buds. On overexposed images the highlights are clipped — the information is gone — and recall falls from perfect to 0.319. On the noisy low-light image it produces 155 false buds. One honest caveat: white balance and CLAHE were tested as one pipelined package, never ablated separately, so I cannot say which step does the damage in the bad cases. The operational conclusion stands either way: gate the preprocessing. Measure the frame first (mean luminance, cast statistics); turn it on only when a global shift is actually present.

3.3 Occlusion: small targets die first, and fragments impersonate buds

Cover radius	Covered recall	Uncovered recall	Bloom recall	Bud recall	FP	Total error
0.5 × radius	0.690	1.000	1.000	0.382	4	−20.5%
0.8 × radius	0.207	1.000	0.530	0.290	287	+81.0%
1.1 × radius	0.014	1.000	0.321	0.290	98	−22.4%

At half-radius cover, every uncovered object is still found, but 31% of covered ones are gone — and buds take the hit. A bud's radius is only 11–17 px; cover its center and the surviving fragments fall below min_area=120 and are discarded as noise. Blooms, being larger, keep a ring of petals and survive the same treatment.

At 0.8 × radius comes the strangest number in this study: total error +81%. Occlusion does not only delete; it creates. A partly covered bloom breaks into fragments, each landing in the 120–900 px² band, each booked as a bud: 297 buds detected against 76 real. A heavily leafed photo will undercount blooms and overcount buds at the same time — and the overcount is the bigger error.

One parameter ablation: dropping min_area from 120 to 40.

Cover	min_area	Recall	Precision	Bud recall	FP
0.5 ×	120	0.776	0.976	0.382	4
0.5 ×	40	0.981	0.808	0.947	49
0.8 ×	120	0.443	0.245	0.290	287
0.8 ×	40	0.510	0.226	0.474	366

Under light cover, lowering the floor genuinely rescues buds — at ten times the false positives, flipping total error from −20.5% to +21.4%. Under heavy cover it buys almost nothing and adds fragments. It is a precision-for-recall trade that only pays in the light-occlusion regime, not a fix (script: data/2026-10-17-bloom-minarea-ablation.py, results merged into the same raw data file).

3.4 Overlap: clustering always undercounts; area-splitting works in one band

Offset	Blooms found / 174	Raw error	After split correction
2.2 × radius	139	−35	+27
1.6 × radius	117	−57	+20
1.2 × radius	120	−54	+10
0.8 × radius	125	−49	−4
0.45 × radius	132	−42	−11

Once two blooms touch, morphological closing welds them into one component, and every offset undercounts by 42–57 blooms. Area-splitting genuinely helps in the 0.8–1.2 × band (−49 → −4, −54 → +10). At wider offsets it overcorrects (+27 at 2.2 ×). The overcorrection is mostly the estimator's own doing: it rescales every bloom box by area, and single-bloom radius naturally varies from 30 to 49 px, so a large lone bloom gets rounded up into "one and a half." The wider the offset, the less true merging there is, and the more this systematic inflation dominates. A second confound: at 2.2 × offset in these dense scenes, the added bloom often lands on a third flower, so the raw −35 at that end mixes three-way merges and is not a clean pairwise reading.

Combined stress (gamma 1.6 plus 0.8 × leaf cover) leaves recall at 0.429; preprocessing lifts it only to 0.500. Preprocessing repairs global shifts. It cannot repair something physically hidden, and that boundary deserves to be stated plainly.

4. What I take from this

Four things a practitioner can lift directly:

Acceptance metrics for a counting pipeline must be per-class, per-object matched precision/recall, plus an explicit error on the bloom-to-bud ratio. Totals alone certified three conditions in my runs as 0.0% error while a sixth to a quarter of the blooms were already wrong. I will change the repo demo's scoring and annotate the README's 99.7% with its methodology.
Under the two most common degradations — dimming and moderate occlusion — an area-threshold classifier's dominant failure is class migration, not misses. Watching the class mix drift is an earlier alarm than watching the total. The boundary matters: under extreme underexposure and 1.1 × cover, objects simply vanish (covered recall 0.014) and there is no migration to watch.
Preprocessing (white balance, CLAHE) should be conditional, triggered by measured global shift. On clipped or noisy input it is a net harm.
Small-class targets need their own protection strategy under occlusion. Lowering min_area only pays under light cover. The candidate beyond a single frame is cross-day tracking — yesterday's bud, hidden by a leaf today, restored from the time series. That idea is not validated by anything here (every experiment is a static single frame; wind sway and viewpoint shift are untested). It is a direction the failure modes point to, and it moves the roadmap's video / multi-frame de-duplication up in priority.

Why not jump straight to a trained detector? The BloomDetector interface is already there; swapping the backend touches nothing else. But without real annotated photos there is nothing to train on. What this study does is map the baseline's failure boundaries under light and occlusion, so that any future backend has to pass the same destructive test suite, with the numbers side by side.

5. Limitations and open problems

Everything is still synthetic: program-drawn flowers, flat-painted leaf occluders, global parametric lighting. Real greenhouse light is directional and spectral; real leaves have texture and translucency. Treat these numbers as a lower bound on failure direction and magnitude, not as field accuracy.
Lighting transforms are whole-image; local specular reflection and lens flare are not modeled.
The white-balance/CLAHE pair was never ablated separately; their individual contributions and harms are entangled.
At 2.2 × offset the overlap group is confounded by third-flower merges: the raw error there mixes them in, and the split correction overshoots (+27). Neither end of that row is a clean reading.
Four scenes, 210 objects, no confidence intervals computed. The claims I kept are the ones consistent across conditions; exact point values (such as where the gamma cliff sits) need denser sampling.
The split estimator's unit area is the median from this same batch; it must be recalibrated per cultivar and camera distance, and I have not tested that transfer.
Real-photo validation is still missing. The roadmap's trained detector backend and a real annotated set have not been started. That remains the largest open problem on this line.

6. References

Bloom-Counter repository (MIT; all code used here): https://github.com/Ezra-Zhao/Bloom-Counter . Detector: detector/color_segmentation.py; the co-tuning comment is on line 23 of simulator/generate.py, non-overlap placement in its _place function; totals-only scoring in examples/demo.py.
This study's scripts and raw data: the companion data/2026-10-17-bloom-bench.py, data/2026-10-17-bloom-minarea-ablation.py, and data/2026-10-17-bloom-bench.json (4 scenes, seeds 101/202/303/404, fixed and re-runnable), archived with the article in the tech-deep-dives repository.
OpenCV documentation: HSV color space and inRange, morphological operations, createCLAHE: https://docs.opencv.org/4.x/ .

Author: Guangyi Zhao. All code and experiments described here are the author's own clean-room work: the detector, the simulator, and the stress-test scripts were written by the author, and the experiments ran on locally generated synthetic data. No client data, client imagery, or identifying information is involved.

How Many of a Scanner's Findings Are Real? Web-Vuln-Scanner's Detection Pipeline and False-Positive Control, Measured

Guangyi Zhao — Mon, 19 Oct 2026 04:56:09 +0000

Author: Guangyi Zhao
Date: 2026-10-15
Subject: Web-Vuln-Scanner (open-source project by the author, github.com/Ezra-Zhao/Web-Vuln-Scanner), scaffold v0.1, Python 3.12.3
Environment: Linux x86_64 VM; every target is the author's own application on 127.0.0.1 (the bundled demo app plus five target apps written for these experiments). No third-party system was scanned.

1. Problem and background

A vulnerability scanner's report has a question that rarely gets asked to its face: how many of these findings are real?

The cost of a false positive is not "a second look." One fake high-severity SQL injection drags the on-call engineer through a full incident loop — confirm, reproduce, read the code — until it turns out the page merely mentioned database errors in its prose. Enough false positives and nobody reads the report carefully anymore; the real findings get swept away with the noise. In a detection pipeline, false-positive control is not polish. It is what decides whether the report deserves trust.

My own open-source project, Web-Vuln-Scanner, is a small ethics-first scanner: it refuses to run without an explicit target and an explicit authorization confirmation. It currently implements four checks — reflected XSS, error-based SQL injection, security response headers, and sensitive-path probing. The honest status, stated up front: scaffold v0.1, and the README says so. Stored XSS, blind SQLi, CSRF, and authenticated scanning are not implemented; they sit in the source as TODOs, not claimed features.

This is not a usage tutorial. I ran the pipeline against six targets: the bundled intentionally vulnerable demo app; a hardened app written the careful way; a single-page-app shell that answers HTTP 200 for every unknown path; an article site whose prose legitimately talks about database errors; an app whose only form is POST and whose backend really is injectable; and an API app that reflects input verbatim into a JSON response. The six targets measure four things: whether real vulnerabilities are caught, whether clean targets stay clean, what shape the false positives take, and where the pipeline's field of view ends. The headline result: each check's logic is restrained in its own way, but the two checks that did produce false positives — sensitive-path probing and error-based SQLi — are missing the same thing: a baseline. Without a "first look at what the normal response looks like" step, a soft-404 site makes the sensitive-path check go 14 for 14 wrong, and one ordinary sentence about a "database error" buys you a high-severity injection alert.

2. Method and architecture

The codebase totals 846 lines including the demo app and tests. The pipeline is four serial stages: scope.py validates authorization; crawler.py crawls the same host collecting pages, forms, and parameterized URLs; the four checks run independently against the crawl results; report.py emits a console and JSON report. Checks share no state, and the CLI wraps each check so one check's exception cannot kill the scan.

Each check's decision logic, stated precisely, because every conclusion below follows from it:

Reflected XSS (scanner/checks/xss.py): generates an 8-hex-character random token, builds a payload of the form "><svg/onload=alert('TOKEN')>, and injects it into every URL parameter and form field. A high-severity finding requires both the token and the complete payload to appear verbatim in the response body. Escaped reflection (quotes or angle brackets encoded) cannot match, because the full payload string no longer appears.

Error-based SQLi (scanner/checks/sqli.py): four payloads only (single quote, double quote, two tautologies). The response body is matched against six database error-signature groups — SQLite, MySQL/MariaDB, PostgreSQL, MSSQL, Oracle, and a generic group — 21 regular expressions in total, including generic words like syntax error and database error. A match is a high-severity finding, at most one per parameter. The check never sends a baseline request and never diffs the response against the un-injected one.

Security headers (scanner/checks/headers.py): examines exactly one response — the base page. Missing Content-Security-Policy, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, or Permissions-Policy headers are reported individually; HSTS is checked on HTTPS targets; a Server header earns an info-level version-disclosure note.

Sensitive files (scanner/checks/sensitive_files.py): a fixed table of 14 probe paths (.git/HEAD, .env, admin, backup.zip, db.sqlite, and so on), requested one by one without following redirects. HTTP 200 with a non-empty body is a finding, at a per-path preset severity from info to high. The response content is never examined — the source's own TODOs admit soft-404 detection and content sniffing are unimplemented.

The crawler's field of view bounds the checks': same-host links only, at most 20 pages by default, and only pages served as text/html are recorded — a non-HTML response is skipped entirely, neither stored nor parsed for links.

Method: the repository ran unmodified. The demo target is the bundled demo/vulnerable_app.py; the other five targets were written for this article and are archived with the experiment script. Every target went through the full pipeline (crawl plus four checks) with per-check request counts and timings recorded; the demo scan was repeated three times for stability; the authorization guard got five dedicated probes; and the JSON endpoint got one manual payload-reflection probe. Scripts and raw JSON are archived with this article so every number can be re-run.

3. Measurements

3.1 Demo app: all four planted problem classes caught, 31 requests per run

The demo app plants four problem classes: reflected XSS on /search, string-concatenated SQL with verbose error echo on /user, no security headers at all, and exposed /admin and /.git/HEAD. The scan reported 13 findings:

Check	Requests	Findings	What
Crawl	4	4 pages, 1 form, 2 parameterized URLs	—
reflected-xss	3	3	parameter q, parameter id, form field q
error-based-sqli	9	1	parameter id (SQLite error disclosed)
security-headers	1	6	all five headers missing, plus Server version disclosure
sensitive-files	14	3	/robots.txt, /.git/HEAD, /admin

Thirty-one requests total; per-check time 0.04–0.19 s; a full local run finishes in under a second. Three repeated runs agreed exactly: 13 findings each time, same severity distribution (high 5, medium 2, low 3, info 3). The pipeline is deterministic; there is no random component in its verdicts.

One hit deserves unpacking. XSS also fired on /user?id= — SQLi's home turf. The XSS payload was concatenated into the SQL statement, the database errored, and the error page echoed the exception — payload included, unescaped — inside a pre element. So the payload was reflected, through the error page. That is a true positive, but the reflection path is "via verbose error disclosure," not "the app renders the parameter into a normal page." A reader who only sees the title "Reflected XSS via parameter 'id'" will go looking in the wrong place: the check supplies the evidence string, but not the context in which the reflection happened. That is a reporting debt.

The bundled test suite — 11 tests covering the four checks against the demo app plus the scope guard — passes under pytest in 1.61 s.

3.2 Hardened app: zero vulnerability findings, one unavoidable info

The second target was written the careful way: html.escape on output, parameterized queries with type validation, all five security headers set, real 404s for unknown paths, no sensitive paths. Result: XSS, SQLi, and sensitive-files all returned 0. The single finding was the security-headers info — the Server header disclosing the Werkzeug version.

That info is worth a sentence of its own. It is not the application code's fault: the development server adds the Server header at the response layer, and my hardened app did not attempt to suppress it. Against any target running on a framework dev server with default settings, this info will very likely appear regardless of how disciplined the application code is. The lesson for report readers: judge info-level items by which layer produced them, and do not charge the framework's default behavior to the application.

The positive conclusion matters just as much: on a "written properly" baseline, the four checks produced zero vulnerability-class findings. False positives are not this pipeline's default state. They appear on specific site shapes — the next three sections.

3.3 Soft 404: 14 of 14 probes wrong, a report that is 93% noise

The third target mimics the common SPA/fallback shape: every unknown path returns 200 with the app shell. It sets all five security headers and escapes its search output — an app with nothing planted. The scan reported 15 findings, 14 of them from sensitive-files: the entire 14-path probe table hit, including .env (high), db.sqlite (high), database.sql (high), .git/HEAD (high), backup.zip, phpinfo.php. The 15th finding was the Server version-disclosure info, same as section 3.2. All 14 sensitive-path hits are false positives — a 100% false-positive rate for that check — and 14 of the report's 15 findings, 93%, are noise.

The cause is written into the decision rule: status 200 plus a non-empty body, content never examined. On a site where every path is a 200, the probe table's length is the false-positive count. The source's TODO already names the fix — fetch one random nonexistent path as a soft-404 sample and compare similarity — but until that lands, this check's output on SPA-shaped sites should be stamped untrusted as a block, not interleaved with real findings sorted by high severity. The practical damage is exactly there: the four high-severity findings at the top of the report are all fake, and the reader's trust in any real finding further down is spent along with them.

3.4 No baseline comparison: one sentence of prose buys a high-severity alert

The fourth target is an article site whose page legitimately discusses database errors — the prose contains the phrases "database error" and "SQL syntax" — and whose topic parameter feeds no SQL query at all. There is no injection. The scan reported one high-severity finding: "Possible SQL injection via parameter 'topic' (Generic SQL error disclosed)."

The mechanism is easy to dissect. The check appends its four payloads to the parameter one at a time and matches every response against the error signatures. The article's prose contains signature words regardless of any payload, so the first payload already matches; the verdict has nothing to do with whether injection occurred. The missing step is the baseline: never request the page without a payload and check whether the signature words were already there. Baseline comparison is the textbook move for injection detection — this project's own blind-SQLi TODO sketches exactly that, comparing true-condition and false-condition responses — but the error-based path never got one. And because the generic signature group uses everyday words like "database error," the tripwire is low: any site that writes about database operations can set it off.

This and section 3.3 are the same disease in two organs: the verdict asks only "is the thing I am looking for in this response," never "was it already there before I attacked."

3.5 POST forms: a real injection in plain sight, skipped by design

The fifth target has a single POST form whose backend query is string-concatenated with verbose errors — a real, exploitable SQL injection. The SQLi check reported 0. By design it tests GET forms and URL parameters only; POST forms are skipped, with a TODO saying so. The miss is a scope boundary, not an implementation bug — but nothing in the report mentions that this form was never tested. The reader sees a clean zero.

The same target produced a telling contrast. The XSS check does test POST forms, and it reported one high. Its HTML payload happened to contain quotes that broke the backend SQL; the error page echoed the payload unescaped, and the XSS check duly confirmed a reflected XSS — the same echo path as section 3.1. What it recorded is an XSS, not a SQLi. But that response objectively carried a full database error message, and no part of the pipeline ever matched it: the SQLi check had skipped the form, and the XSS check does not look at error signatures. The evidence appeared in the traffic and no check was responsible for claiming it. That is the price of hermetic check separation: every check answers for its own conclusions, and nobody answers for coverage completeness. A "not covered" section in the report — which forms, which parameters, which checks never ran on them — costs little and turns this silent miss into a stated boundary.

3.6 JSON endpoints: outside the crawler's field of view

The sixth target's home page links to a JSON endpoint, /api/search?q=, which places q verbatim into its JSON response. I verified with a separate manual probe: the endpoint returns 200 with Content-Type application/json, and both the token and the complete payload appear verbatim in the body — had this endpoint been probed, the XSS check's match condition would have been satisfied. In the real scan it was never probed. The crawler fetched the link, saw a non-HTML content type, and skipped it: the endpoint entered neither the page list nor the parameterized-URL list, so neither injection check ever saw it as input. The target's six findings were all headers-class; injection findings were 0.

The characterization needs restraint. Verbatim reflection inside an application/json response is not the same as XSS in an HTML context; modern browsers will not execute it as a page. Calling this a "missed XSS" would overstate it. The real conclusion is about the field of view: this pipeline can only see the HTML-page portion of an application. API endpoints — JSON backends, mobile APIs — are entirely outside it, and the report never tells you how large the visible portion was. For a scaffold that calls itself a web scanner, that belongs in the report, not just the README.

3.7 The authorization guard: five probes

The last experiment measures not detection quality but whether the pipeline will run at all — five probes against scope.py:

Probe	Result
Target without an http(s) scheme	Refused (ScopeError: explicit URL required)
Authorization not confirmed	Refused (ScopeError)
Confirmed, local target	Allowed
Allowlist not containing the target host	Refused
Allowlist containing the target host	Allowed

All five matched the design: no default target, no confirmation no scan, and an allowlist, when given, actually narrows scope. The guard is enforced in code rather than advised in documentation — not a given among learning-grade scanners, and worth recording.

4. Analysis and trade-offs

4.1 The four checks' false-positive control, graded individually

Ranked by decision method, the four checks are not equally armored. XSS carries the hardest evidence: a random token plus the complete payload as a dual condition excludes escaped reflection and static content by construction, and it produced a clean zero on the hardened target. Its only weakness is context expression — reflection via an error page is reported identically to reflection in normal rendering (sections 3.1 and 3.5). Sensitive-path probing has the weakest decision rule — status code only, zero content verification — and it fails wholesale on the most common modern deployment shape, the SPA. SQLi sits between: its payloads and signatures are not unreasonable, but the missing baseline makes it systematically wrong on sites that talk about databases, and the more generic the signature words, the worse it gets. Headers is the most deterministic check — a header is present or it is not — with one caveat: the Server info originates at the server layer, not in application code, and should not be graded alongside headers the application itself failed to set.

4.2 Fix order: baselines before breadth

In return-on-effort order. First, baselines and soft-404 sampling: the sensitive-path check should first fetch one random path as a baseline and downgrade or drop probe responses that closely resemble it; the SQLi check should send a baseline request before injecting, and skip any parameter whose normal response already contains the signature words. This one item treats sections 3.3 and 3.4 together — it is the hinge of false-positive control. Second, content sniffing: a .git/HEAD hit should begin with "ref:", an .env hit should contain key-value lines; verify content after a probe hits before assigning high severity. The source TODO already has this idea; it ranks after baselines because the baseline helps all 14 probes at once, while sniffing rules are written per path. Third, a not-covered section in the report: POST forms, non-HTML endpoints, pages beyond the crawl cap — listed explicitly, so misses become stated boundaries instead of silence. Fourth, and only then, breadth: POST SQLi, stored XSS, and blind SQLi are all on the TODO list, but breadth widens the promise; widen it before the baselines land and you manufacture false positives at a larger scale.

The judgment underneath the ordering: a scanner's credibility comes first from "what it reports is mostly real," and only second from "it reports a lot." This pipeline currently leans the other way — coverage grows, at least in its TODO plans, while the baselines that make verdicts trustworthy are still missing.

4.3 One judgment: false-positive control means giving every verdict a refutation procedure

Looking across the four checks, the reliable verdict (XSS) has a shared structure: its evidence is exclusive. A random token cannot pre-exist in the page; the complete payload appearing verbatim excludes escaping. The unreliable two lack exactly that. A 200 status is not exclusive evidence — a soft 404 is also a 200. A signature word in the prose is not exclusive evidence — it was there with no injection at all. Doing false-positive control on a detection pipeline means, in engineering terms, attaching a refutation procedure to every verdict class: first prove "without my attack, it is not there," then report "after my attack, it appeared." Baseline requests, soft-404 samples, and random tokens are three implementations of the same idea. That is why baselines rank ahead of every breadth item in section 4.2 — the gap is not a missing patch on one check, it is a missing step in the verdict methodology.

5. Limitations and open questions

The limitations, stated plainly. First, all six targets are minimal local apps, each isolating one site shape. Real sites mix shapes — an SPA with a genuinely exposed file, an article site with a real injection — and this article does not measure how false and true positives interact in mixed shapes. Second, the hardened app was written by the author against the checks' own list, a "studying to the test" bias: it proves the checks stay quiet on an ideal baseline, not that they stay equally quiet on mediocre real-world code. Third, every experiment ran over loopback; the timing figures are local orders of magnitude only (section 3.1's 0.04–0.19 s is one demo run's per-check times; across all six targets, per-check times spanned 0.003–0.22 s). Under real network latency, the 14 serial sensitive-path requests and the per-parameter serial payloads will stretch totals considerably, and concurrency and timeout behavior went unmeasured — the per-request timeout is a uniform 10 s in the source, and worst-case waits were not exercised. Fourth, the XSS-via-error-page path is classified here as a true positive because the payload does enter an HTML response unescaped; its exploitability in a real browser depends on the error page's content type and context, which this article did not verify at browser level. Fifth, the targets and the demo all run on the Flask/Werkzeug stack; other stacks change Server headers and error-page shapes, and the conclusions generalize only to similar shapes.

Three open questions remain. What similarity algorithm and threshold should define a soft-404 baseline — too loose leaks real files, too tight resurrects false positives — needs a corpus of real SPA samples to calibrate. After SQLi gains baseline filtering, will targets whose pages echo the same error stack with and without injection lose true positives to the filter? That needs a dedicated target shape to measure. And at what granularity should a not-covered list be reported — endpoints, parameters, or a check-by-endpoint matrix — is a question to settle after one real round of using it.

6. References

Web-Vuln-Scanner source (version measured here): github.com/Ezra-Zhao/Web-Vuln-Scanner — key files: scanner/cli.py (pipeline assembly, per-check failure isolation), scanner/scope.py (authorization guard), scanner/crawler.py (crawl and field of view), scanner/checks/xss.py, scanner/checks/sqli.py, scanner/checks/headers.py, scanner/checks/sensitive_files.py (the four checks and their TODOs), demo/vulnerable_app.py (demo target).
Experiment scripts and raw data for this article: data/2026-10-15-vuln-scanner-bench.py and data/2026-10-15-vuln-scanner-bench.json (per-check results, request counts, authorization probes, and the JSON reflection probe for all six targets), archived with this article; every number can be re-checked by re-running the script.
OWASP Testing Guide and OWASP ASVS: the conventional framing for injection testing and security headers; used as background orientation only, no specific figures quoted.

Author: Guangyi Zhao. The code, experiments, and text in this article are clean-room work by the author: the subject is an open-source scaffold project maintained by the author, every target is a locally built application, and all scanning happened on 127.0.0.1 loopback addresses. No third-party system was probed, and no customer information, customer systems, or third-party non-public data are involved. Every number comes from the measurements listed at the top of this article, and no scaffold state is presented as a mature product.

How Big Should the Worker Pool Be? Measuring the Concurrency Model of a Go Diagnostics Engine

Guangyi Zhao — Mon, 19 Oct 2026 04:52:36 +0000

Author: Guangyi Zhao
Date: 2026-10-13
Subject: Go-Diagnoser-Engine (open source, github.com/Ezra-Zhao/Go-Diagnoser-Engine), Go 1.24, zero third-party dependencies
Test setup: Linux x86_64 VM, 2 vCPUs, server and load client on the same host

1. Problem and background

Running diagnostics against a fleet of nodes has two failure modes callers hate. Wait synchronously and one slow check hangs the request. Spawn goroutines without limits and a traffic spike takes the service itself down before any node gets diagnosed. So I built Go-Diagnoser-Engine as an asynchronous service: POST /api/v1/diagnose submits a job and returns a job ID, GET /api/v1/jobs/{id} is polled for structured results. Execution sits behind a bounded worker pool fed by a bounded queue, and the checks inside one job fan out in parallel.

The structure is easy to write. The numbers are the hard part. How many workers? How deep a queue? Does a job with three checks cost three times the latency of a one-check job, or roughly the same? When the queue fills up, do you block or reject — and can you predict the rejection count in advance? This article answers those questions by actually loading the service, then goes back to the source to judge which design choices the data supports and which ones only looked reasonable while I was typing them.

One disclosure up front, because everything else depends on it: the project is a scaffold, and its README says so. The three registered checks (connectivity, disk_usage, service_health) are simulated — each sleeps a random 50–200 ms and passes with 90% probability, and every result is labeled SIMULATED. The real probes (ICMP/TCP, disk scraping, /healthz queries) are still TODOs in the code. So this article measures the concurrency skeleton, not diagnostic ability. The throughput, latency, backpressure, and race behavior below are real measurements; swap in real probes and some of the numbers will move. Section 5 spells out which conclusions break first.

2. Method and architecture

The codebase is 523 lines across 6 Go files. go.mod contains a module path and a Go version and nothing else — no third-party dependencies. cmd/server is the entrypoint: it reads PORT, WORKERS, and QUEUE_SIZE, wires the parts together, and owns graceful shutdown. internal/api holds the net/http routes, registered with Go 1.22 method-and-pattern syntax. internal/engine is the worker pool and check registry. internal/store is a concurrency-safe in-memory job store. pkg/models holds the shared types.

Four concurrency decisions, all deliberate:

First, a fixed worker pool instead of a goroutine per request. Workers are long-lived goroutines; the pool size defaults to runtime.NumCPU() and is overridable via WORKERS. The concurrency ceiling is a deployment parameter, not something that drifts with traffic.

Second, a bounded queue with fail-fast submission. Submit uses a non-blocking select: if the queue is full, the API returns 503 immediately instead of parking the HTTP request. The cost is pushed to the caller, who must retry. Whether that trade is fair is a data question, answered in section 3.3.

Third, per-job fan-out. runJob starts one goroutine per check and collects results into a preallocated slice indexed by check position — each goroutine writes only its own slot, so there is no mutex and no result channel to drain. A sync.WaitGroup joins them, and each check runs under its own 30-second context timeout so one slow probe cannot stall the whole job.

Fourth, cooperative shutdown. On SIGINT or SIGTERM, main closes the HTTP server (15-second budget), then engine.Shutdown cancels the engine context and waits for workers to drain, also capped at 15 seconds.

Method: I copied the repository verbatim into a scratch directory and built it (8.7 MB binary; 11 MB with -race). A Python load script, archived with the raw results alongside this article, ran four experiments. A: 30 one-check jobs and 30 three-check jobs submitted sequentially, comparing end-to-end latency. B: 60 three-check jobs submitted concurrently at pool sizes 1, 2, 4, and 8, measuring wall time and completions per second. C: queue squeezed to 5 with a single worker, a burst of 30 concurrent submissions, counting 202s against 503s. D: the race-detector build under 100 concurrent jobs with aggressive polling, plus a SIGTERM shutdown check.

3. Measurements

3.1 Fan-out is real: three checks cost 19% more, not 200%

Experiment A (30 jobs per row, 4 workers):

Configuration	Mean end-to-end	p50	p95	Mean per-check latency
One check (connectivity only)	139.8 ms	151.5 ms	198.6 ms	135.6 ms
Three checks (all)	166.5 ms	169.9 ms	201.9 ms	126.0 ms

Tripling the checks raised mean end-to-end latency by 19%. Had the fan-out secretly been serial, three checks would have landed near three times a single latency, around 400 ms. The numbers also agree with theory. A single check is uniform on 50–200 ms; a parallel job finishes with its slowest check, whose expected value is 50 + 150 × 3/4 = 162.5 ms. Measured: 166.5 ms, within 2.5%. Per-check means (135.6 and 126.0 ms) also sit close to the uniform expectation of 125 ms. This experiment validates two things at once: the fan-out is genuinely parallel, and my description of the simulated delay distribution was accurate.

Where did the extra 27 ms (166.5 minus 139.8) go? I did not take segmented timings, so this is a labeled guess, not a finding: timer wakeups landing together, plus lock traffic when results are stored, are the plausible contributors. What matters more is the boundary of the claim — the fan-out dividend comes from checks waiting on each other. Make the checks CPU-bound and this arithmetic has to be redone.

3.2 Throughput scales almost linearly — do not celebrate yet

Experiment B (60 three-check jobs per row, 20 client threads submitting and polling):

Workers	Wall time	Completion rate	Speedup vs 1 worker
1	9.71 s	6.2 jobs/s	1.0
2	4.98 s	12.0 jobs/s	1.9
4	2.58 s	23.2 jobs/s	3.7
8	1.34 s	44.9 jobs/s	7.2

Eight workers on a 2-vCPU box delivering 7.2× the single-worker rate looks too good, and it is — for the right reason. A simulated check does no work at all; it parks on a timer. Worker count is therefore pure concurrency, and the rate is simply workers divided by per-job time. The single-worker prediction is 1 / 0.1625 ≈ 6.15 jobs/s; measured 6.2.

What this actually proves is narrow but worth having: the pool has no hidden serial bottleneck — neither the queue nor the store's locking ate the scaling. It proves nothing about real probes. Add CPU-heavy parsing or classification and a 2-vCPU box will saturate around 2–4 workers, possibly regressing beyond that. Make the probes network-bound and the bottleneck moves to the targets or to connection counts. Extrapolating this line into production claims is the most common way load tests like this one get abused, so the line stops here.

3.3 Backpressure you can compute in advance

Experiment C used the tightest settings: 1 worker, queue depth 5, one burst of 30 concurrent submissions. Result: 6 accepted with 202, 24 rejected with 503. The 6 is not luck. Total admission capacity is workers plus queue depth — one job running, five waiting — and the seventh submission onward failed fast. No request was parked. All six accepted jobs were later polled to completion; nothing accepted was silently dropped.

The gap is on the response side. A 503 from this API carries one error string: no Retry-After, no queue depth, nothing a client can schedule a retry against. For a machine-facing API, fixing that outranks adding any new check.

3.4 Races and shutdown

Experiment D ran 100 jobs against the -race build with submissions, execution, and polling all hammering the same in-memory store. The race detector reported nothing, and all 100 jobs completed. The code supports the result: every store access goes through one sync.RWMutex and reads return copies; fan-out writes land in disjoint slice slots, and the WaitGroup join establishes the happens-before edge before results are persisted.

One honest footnote. My first run of this experiment saw only 92 of 100 jobs finish inside the polling budget. The cause was my harness, not the server: the -race build answers each request more slowly, and my poll loop gave up too early. Widening the budget to 20 seconds and rerunning produced 100 out of 100. I am leaving this in because it is exactly how load tests lie — the observer's limit masquerading as the system's failure.

Shutdown: SIGTERM on an idle server produced the full sequence (shutting down, all workers stopped, bye) in about 1.0 second. That test had no jobs in flight. What shutdown does to in-flight work was not measured — but reading the code turned up a real problem, covered next.

3.5 A small defect found along the way: unstable result order

With all checks selected, the order of the results array is not stable. In a sample of 20 jobs, two orders appeared (connectivity, disk_usage, service_health in 17; service_health first in 3). The cause sits in resolveChecks: for the "all checks" case it ranges over a map, Go map iteration order is randomized, results are filled by that list's indices, and the randomness leaks straight into the API response. Clients must match results by name, never by position. The fix is a one-line ordered registry, but the lesson generalizes: any map inside your service is a randomness source waiting to surface at the API boundary.

4. Analysis and trade-offs

Four judgments I would reuse, each tied to evidence above:

Make the concurrency ceiling explicit, not traffic-dependent. A fixed pool plus a bounded queue turns admission capacity into a computable constant — workers plus queue depth — and the measured 503 boundary matched that constant exactly. Alerting, capacity planning, and rate policy all need a number like that. A goroutine per request is faster at low load and has a ceiling nobody can state.

Async submission buys a predictable submit path. A submit request only validates, creates a record, and enqueues; the HTTP layer's 10 s read and 30 s write timeouts can never be touched by a slow check. The price is client-side polling and, until server-sent events exist (they are on the roadmap, not in the code), delayed completion awareness.

Pick the simplest fan-out primitive that is actually correct. An indexed slice plus a WaitGroup beat a result channel here: less code, no collection loop, and a clean race-detector run to back it. The precondition is that checks share no writes and have no ordering dependencies. The day checks need to hand intermediate results to each other, this structure gets replaced, not stretched.

Zero dependencies is a deliberate constraint with an expiry date. Standard library only — ServeMux routing from Go 1.22, crypto/rand job IDs — keeps the build and audit surface small at 523 lines. The missing production pieces (metrics, rate limiting, structured logging) are equally obvious. For a scaffold this is the right call; the call expires the day real probes land.

Three spots in the code do not survive scrutiny, in order of severity:

Shutdown semantics disagree with job status. engine.Shutdown cancels the context first, and the worker loop selects between the cancellation case and the queue — when both are ready, the choice is random. A queued job may be abandoned with its status stuck at queued forever. Worse, an in-flight job whose checks abort on cancellation still flows into SetResults and gets marked done, indistinguishable from a job that ran to completion. With an in-memory store the process exits and none of this matters; the roadmap's durable store turns it into real semantic debt. Abandoned jobs need an explicit terminal state.

The store never evicts. Every job record lives in the map for the life of the process. Fine for a scaffold; a long-running service needs a TTL or a capacity cap. This one is not a risk, it is a certainty with a date.

Submit has a narrow shutdown race. Its select includes the context-done case, the queue-send case, and a default. If shutdown begins while the queue still has room, the send case can win and enqueue a job nobody will ever run. Rare, but precisely the kind of rare that costs an afternoon when it fires. Checking the context once before the select closes it.

5. Limitations and open questions

The largest limitation is the one from section 1: every load in this article is a timer. Linear throughput scaling therefore certifies the skeleton, not the probes; the fan-out latency dividend assumes checks that wait; memory and GC behavior under this load would be meaningless to report, so they are not reported.

Method limits, stated plainly: one batch per configuration (30 jobs per row in A, 60 in B, a single burst in C), no repeated trials and no confidence intervals. Client and server shared a host, so polling cost excludes any network. The box is a shared 2-vCPU VM; treat the digits as magnitudes and ratios, not as constants of nature.

Three open questions gate the next round of code. Once real probes exist, should pool size be tuned to CPU or to what the targets can absorb — those two tuning curves need separate load runs. How should queue depth, the 503 rate, and client retry policy interact — that needs a retrying client in the loop. And when the durable store lands, the shutdown and terminal-state semantics from section 4 get fixed first, followed by a shutdown test with jobs actually in flight.

6. References

Go-Diagnoser-Engine source (the version measured here): github.com/Ezra-Zhao/Go-Diagnoser-Engine — internal/engine/engine.go (pool and fan-out), internal/engine/checks.go (simulated checks and registry), internal/api/router.go (routes and the 503 path), internal/store/store.go (in-memory store), cmd/server/main.go (wiring and shutdown).
Raw data and load scripts archived with this article in the project ledger's data directory (2026-10-13-go-diagnoser-bench.json, bench.py, race_shutdown.py); every number above can be reproduced from them.
Go documentation: net/http ServeMux method and pattern routing (Go 1.22+), sync.WaitGroup, and the happens-before definitions in the Go memory model (pkg.go.dev, go.dev/ref/mem).

Author: Guangyi Zhao. All code, experiments, and text in this article are clean-room work by the author: the system under test is the author's own open-source scaffold, load was generated in a self-hosted environment, and no client information, client systems, or third-party non-public data are involved. All figures come from the measurements described above; the simulated nature of the workload is stated in the text and no simulated result is presented as real diagnostic performance.

The Privacy Bill Comes Due Before the Real Model Arrives: Field Constraints of Emotion-Guard, a Kindergarten Emotion-Recognition Demo

Guangyi Zhao — Mon, 19 Oct 2026 04:44:11 +0000

Author: Guangyi Zhao

1. The problem

Emotion-Guard is my open-source demo of an idea that came out of real work: run emotion recognition in a kindergarten, watch for sustained negative mood, and nudge a teacher to check in on a child before distress turns into an incident — or before a stressed adult does something worse. The motivation is protective. The subjects, however, are three to six years old. They cannot consent; their parents sign for them; and the record being created is about their faces and their feelings, accumulated day after day.

This article is not about making the model more accurate. There is no real model yet — the only classifier that runs in this repository is a simulator. What I did instead was audit the code line by line and run it, end to end, on synthetic data, asking a more basic question: what does this system already owe, in deployment terms, before accuracy even enters the picture? Four small things turned out to carry the weight: how often the alert rules fire on perfectly ordinary days, how long history is kept by default, whether reports carry children's names, and whether downstream code can still tell simulated data from real data. The code currently handles none of the four well — and those are precisely the things regulators look at. Article 5(1)(f) of the EU AI Act has, since 2 February 2025, prohibited using AI to infer emotions in workplaces and education institutions, with only a narrow medical-or-safety exception. In the US there is no equivalent federal ban, but the 2025 COPPA amendments name biometric identifiers, facial templates included, as children's personal information requiring verifiable parental consent before collection plus a written retention policy and a ban on keeping data longer than reasonably necessary; Illinois' BIPA demands written notice and a written release, a public retention and destruction schedule, and gives individuals a private right of action. In much of the world, a system like this does not get to exist because it works. It has to prove it deserves to exist first.

One scientific caveat frames everything below. Barrett and colleagues' 2019 review in Psychological Science in the Public Interest covered more than a thousand studies and concluded that facial movements are neither reliable nor specific enough to confidently infer emotion — a smile is not proof of happiness, a scowl is not proof of anger. Facial signals are weak signals. Thresholds, human review, and retention all have to be designed for a system that is often wrong.

2. Method and architecture

The subject is the repository as shipped — 970 lines of Python, package plus examples and tests, unmodified. The pipeline has five stages. detector.py finds faces (a real OpenCV Haar detector, or a mock for the demo). classifier.py classifies emotion behind a swappable protocol, but only MockEmotionClassifier actually runs: it draws from preset distributions, while FERClassifier raises NotImplementedError and survives only as a docstring describing how fer or DeepFace would be wired in, with a seven-to-four class mapping (happy / calm / sad / angry). classroom.py stores every observation per person and computes mood shares. alerts.py runs three rules: sustained negative mood (at least 4 of the last 6 observations sad/angry for a warning, all 6 for critical), teacher spike (a single angry observation in the last 6 triggers a warning), and sudden swing (adjacent happy-then-angry within 20 minutes triggers a watch). dashboard.py renders a console view plus JSON and HTML reports that list, per person, name, ID, mood shares, and alerts.

First, a correction I owe the reader. I have called this system "multimodal." The audit does not support that. Searching the repository for audio, voice, and transcript returns zero hits. The only working input modality is the face. Everything below treats it as the single-modality system it is.

Eight experiment groups, all reproducible from data/2026-10-18-emotion-guard-bench.py and its JSON output:

Baseline: the shipped test suite, plus one scripted school day — 8 children, 1 teacher, 24 twenty-minute slots, 216 observations, with three scripted beats (child C-04 slides into afternoon distress, C-02 swings once, teacher T-01 flashes anger once) and everything else filled by the mock distributions.
Null-background false alarms: no scripted beats at all, only the shipped typical_child and typical_teacher distributions (15% negative rate for children, 5% angry for the teacher), 30 simulated days, alerts evaluated slot by slot as the demo does. Seeds for this group are SHA-256-derived, for reasons section 3.3 will make clear.
Privacy surface audit: static checks of the data model (any retention, deletion, expiry, anonymisation, or consent machinery?) and a dynamic test of whether the report's simulated label survives contact with real-flagged data.
Retention volume: extrapolated from a 147-byte JSON row per observation.
Truncation: cut each person's history to the last 6 observations and see what each rule loses.
Confidence: the same all-sad sequence scored at confidence 0.99 versus 0.51.
Swing edges: an intervening calm observation, and a 40-minute gap.
Reproducibility: the same script in five separate processes, then three more with PYTHONHASHSEED=0.

3. Measurements

3.1 Baseline: tests pass, and most swing alerts are not the story

All 14 shipped tests pass. Evaluated once at end of day, the scripted day produces between 3 and 8 alerts (why a range, see 3.3). One of them is constant: the sustained-negative critical for C-04. The scripted distress is caught — that part works. Swing watches, however, number 2 to 7, and only one is the scripted C-02 swing. The rest are ordinary children whose random adjacent draws happened to go happy, then angry. On the demo day itself, most of this rule's output is background noise wearing an alert's clothing.

The teacher spike shows the opposite failure. In end-of-day evaluation across five separate-process runs it appeared in only one: the scripted anger sits at slot 15 and has slid out of the last-6 window by day's end. examples/demo.py therefore evaluates slot by slot, and its comment says exactly why — end-of-day evaluation misses mid-day spikes. That design choice is correct, but it has a mirror image: any offline replay of stored logs will systematically miss teacher spikes and, as section 3.5 shows, systematically resurrect stale swings.

3.2 Null background: on calm days, the alerts are not calm

Thirty days, eight children a day, nobody distressed, nobody scripted — just the factory distributions rolling dice. Sustained-negative warnings fired at 33 slot evaluations; 15 of 240 child-days, 6.3%, saw at least one "sustained negative emotion, please check in" warning. Critical never fired: six consecutive negative draws at a 15% base rate is genuinely rare, and the rule deserves credit for that. The teacher rule does not. Teacher spike warnings fired on 23 of 30 days, 76.7%. The rule triggers on a single angry draw inside a six-observation window, the teacher's shipped angry rate is 5%, and slot-by-slot evaluation gives the dice dozens of chances a day. A warning most days is not a property of the classroom; it is a property of the arithmetic.

Translated: deploy these thresholds as shipped and, in a class where every child is fine every day, staff still get a sustained-distress warning roughly once per sixteen child-days, and a "your teacher showed anger, consider a rotation" warning two days out of three. Alert fatigue, in this design, is not a training problem. It is the rules' expected output.

3.3 Reproducibility: the demo's seed is not a seed

Why is the baseline an interval? Because examples/simulate.py derives per-slot seeds with Python's built-in hash(), and string hashing is randomised per process via PYTHONHASHSEED. Same script, same master_seed=7: five processes produced alert totals of 4, 5, 6, 8, and 8, and the first six observations' emotion sequences were different all five times. With PYTHONHASHSEED=0, three runs agreed exactly (8 alerts each). The module docstring promises "seeded, reproducible." Within one process, yes. Across runs, no. For a demo that is a shrug; for a system whose logs might later be replayed in a review or a dispute, it is disqualifying — the same "deterministic" day names different children. The fix costs one line: derive seeds with a stable hash. The false-alarm group in 3.2 already does this.

3.4 Privacy surface: the switches that should exist, don't

The static audit is short because there is so little to find. The Observation record holds the person (ID, name, role), a timestamp to the second, the emotion, a confidence, and a simulated flag. One genuine credit: it never stores the face image. Frames live only inside the detect-and-classify call; what persists is the verdict. That is the single best privacy decision in the codebase.

Everything downstream of that decision is missing. ClassroomSession has no purge, delete, expire, or anonymise method; searching the package for retention or delete returns zero hits. History grows without bound, and mood_summary computes fractions over all of it — which is to say, the system builds a longitudinal emotional profile per named child by default, and keeps it for exactly as long as nobody restarts the process. Person has no consent field, and ingest checks only that a person was registered, not that anyone agreed to anything. The pre-collection consent gate that COPPA and BIPA are built around does not exist in the data model. Reports are per-person by construction: ID, name, role, observation count, four mood shares, and every alert with its timestamp. One demo day serialises to 2,163 bytes in the run sampled here (it wobbles slightly with alert count), essentially all of it about identifiable individuals.

Then the dynamic test found something worse than an omission. I fed a fresh session a single observation explicitly flagged simulated=False — the situation the day a real model gets wired in. build_report still returned simulated: True, because that value is a hard-coded constant in dashboard.py, unrelated to the data. The console dashboard likewise prints "all data below is SIMULATED" unconditionally. In the demo that is harmless. The morning someone connects a real classifier and forgets those two lines, the system will be labelling real children's emotional records as simulated — and provenance labels are the first thing an audit reads.

3.5 Truncation: retention can be sized to the rules — with one exception

Cut every history to the last six observations and re-run. Sustained negative: identical output, critical included. Teacher spike: identical. Both rules only ever read the last six, so every observation beyond six is, for them, storage without function. The exception is the swing rule, which scans the entire history with zip(history, history[1:]). A happy-angry pair from early morning still produces a watch in an end-of-day evaluation over full history; truncated to the last six (by then all calm), it vanishes. So the rule with the worst noise profile (3.1, 3.2) is the only one demanding unbounded retention. That is the worst trade in the codebase. Mood shares are the honest counterpoint: they are defined over all history, so truncation changes their meaning. If you want longitudinal trends, say so, and run them under consent and a stated retention period. If you don't, report classroom aggregates only.

Volume, by contrast, is a non-issue. At 147 bytes a row, the demo cadence (216 observations a day) accumulates about 0.95 MB in 30 days and 5.7 MB in 180. Even at one observation per person per minute (nine people across eight hours, 4,320 a day), it is roughly 19 MB a month and 114 MB in half a year. Disks are cheap. What is not cheap is what each 147-byte row contains: a name, a minute, a feeling. Retention policy has to govern identifiability and time resolution, not megabytes.

3.6 Confidence is ignored, and swings only see neighbours

Two small experiments finish the rules' portrait. The same six sad observations at confidence 0.99 and at 0.51 produce byte-identical alerts — critical either way. AlertEngine never reads confidence when deciding; it only formats the number into the teacher-spike message text. Worse, the mock's confidence is a uniform random draw between 0.62 and 0.97, independent of the emotion drawn, so in the demo every confidence figure is decoration. Wire in a real model without adding a confidence gate and low-quality guesses will count toward "sustained distress" exactly as much as solid ones. And the swing rule only inspects adjacent pairs: happy-calm-angry does not fire, nor does happy-to-angry across a 40-minute gap. At a 20-minute sampling cadence, a real transition with any intermediate state is invisible to it — which is the same fact as 3.1's noise dominance, seen from the other side: the rule detects dice adjacency, not emotional change.

4. What I would change, in order of cost

Stable seed derivation. One line. Without reproducibility, log-based review and appeals are theatre.
Derive the simulated label from the data: summarise the flags at report level, carry provenance on each alert. The day simulated and real streams mix, downstream code must be able to tell them apart.
Size retention to the rules. Six observations per person fully serve two of three rules. Give the swing rule a bounded window (two hours, say) or retire it; do not feed the noisiest rule an infinite memory. Make per-person longitudinal shares an explicit opt-in feature that switches on consent and expiry machinery.
Split reporting into two layers. The classroom layer is aggregates only: class-wide mood shares and per-rule alert counts. I prototyped it — same day, 199 bytes against the original 2,163, alert counts and classroom shares preserved exactly, at the cost of removing all nine names and every per-person curve. The individual layer keeps only what an alert needs, under rotating pseudonyms, with the name mapping held offline.
Two gates on alerting: a confidence floor below which observations do not count toward sustained-negative totals, and a frequency budget (at most one warning per child per day; anything beyond that folds into a digest). Section 3.2 is the argument: without these, the rules manufacture warnings on 6.3% of child-days and 76.7% of teacher-days out of pure chance.
Consent as a first-class field. Person carries consent status and time; ingest refuses anyone without it. A gate at the ingestion point holds up better than a paragraph in a policy document.
Downgrade the language. Output from this system is "worth a look," never "this child is distressed." The current alert copy already says "recommend a caring check-in," and that restraint needs to survive into the product name and the sales pitch. Barrett et al. settle the matter: presenting these outputs as emotional truth is overclaiming.

On the law, an engineer's reading only, not legal advice: in the EU, education-sector emotion inference starts from prohibition, the safety exception has to be argued case by case with the burden on the deployer, and a child-protection framing like this project's may have an argument — it is not a free pass. In the US, the gates are COPPA's consent and retention rules, BIPA-style written releases and destruction schedules, and state student-privacy law. Both regimes point the same way: consent and retention design first, system second.

5. Limitations and open questions

The false-alarm figures are conditional on the mock's factory distributions — 15% negative for children, 5% angry for the teacher — which were chosen to make a demo legible, not measured from real preschoolers. Different base rates move 6.3% and 76.7%; they do not rescue the teacher rule's single-hit sensitivity, nor make critical common at low base rates. No real classifier was benchmarked: real model errors are not independent dice rolls but correlate with age, ethnicity, and lighting, so real-world false alarms will arrive in clusters, and alert fairness across groups needs its own measurement, which this article does not have. Nothing here measures what happens after an alert — teacher response rates, how fast trust in the system erodes, the cost of a missed true event. Those field numbers, not lab numbers, decide whether such a system should exist. The legal discussion is an engineering reading; the boundaries of the EU safety exception and COPPA's reach over purely on-premise, offline systems have not been reviewed by counsel. The reproducibility experiment is small (five processes, three controls), but the effect is all-or-nothing, which is enough for the qualitative claim.

6. References

Emotion-Guard repository (Guangyi Zhao, MIT). Key files: emotion_guard/classifier.py (mock distributions, FER stub), emotion_guard/alerts.py (three rules; defaults window=6, negative_threshold=4, critical_threshold=6, swing_window_minutes=20), emotion_guard/classroom.py (unbounded history, mood shares), emotion_guard/dashboard.py (hard-coded simulated, per-person report), examples/simulate.py (_slot_seed), examples/demo.py (slot-by-slot evaluation comment).
Raw data for this article: data/2026-10-18-emotion-guard-bench.json; script: data/2026-10-18-emotion-guard-bench.py (false-alarm group uses SHA-256-derived seeds, stable across processes).
Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M., & Pollak, S. D. (2019). Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements. Psychological Science in the Public Interest, 20(1), 1–68. https://journals.sagepub.com/doi/10.1177/1529100619832930
Regulation (EU) 2024/1689 (AI Act), Article 5(1)(f); European Commission, Guidelines on Prohibited Artificial Intelligence Practices (2025-02-04), Section 7.1.
FTC, COPPA Rule amendments (final rule announced 2025-01-16): biometric identifiers as personal information; written retention policy and retention limits.
Illinois Biometric Information Privacy Act, 740 ILCS 14/15 (written notice and release; public retention and destruction schedule) and 14/20 (private right of action, statutory damages).

Author: Guangyi Zhao. The system, code, and experiments described here are the author's own clean-room work. All experimental data is synthetic; no real children's images or personal information were used, and no client information is involved.

9 One-Page Cheatsheets for Commands You Keep Looking Up (ffmpeg, git, docker, curl, jq, regex) + 2 Bonus Lists

Guangyi Zhao — Thu, 15 Oct 2026 21:20:37 +0000

I keep a notes file of commands I had to look up twice. At some point the file got long enough that I turned it into proper one-page cheatsheets — one topic per repo, nothing that I haven't personally run and verified.

The cheatsheets

FFmpeg — 28 commands: transcoding, CRF compression, audio extraction, the two-step GIF palette workflow, lossless cutting/joining, scaling, watermarks, RTMP streaming.
https://github.com/Ezra-Zhao/ffmpeg-cheatsheet

Git — 30 commands: daily workflow, the three reset modes explained plainly, rebase (plus the golden rule), stash, and a rescue kit built around reflog and bisect.
https://github.com/Ezra-Zhao/git-cheatsheet

Docker — 28 commands: images, container lifecycle, volumes and networks, Dockerfile essentials, Compose, cleanup, and a debugging checklist for containers that won't start.
https://github.com/Ezra-Zhao/docker-cheatsheet

curl — 29 commands, API-debugging oriented: auth headers, JSON payloads, timing output with -w, retries, cookies, proxies.
https://github.com/Ezra-Zhao/curl-cheatsheet

Regex — 30 patterns and techniques, with the engine differences (PCRE vs Python vs JavaScript) called out instead of hidden.
https://github.com/Ezra-Zhao/regex-cheatsheet

jq — 30 filters: select, map, group_by, aggregation, and curl | jq pipelines. Every filter was executed against real JSON before it went in.
https://github.com/Ezra-Zhao/jq-cheatsheet

Terminal superpowers — 14 modern tools that replace cd, ls, cat, grep and friends.
https://github.com/Ezra-Zhao/terminal-superpowers-cheatsheet

Adobe free alternatives — 16 free tools mapped to what they replace.
https://github.com/Ezra-Zhao/adobe-free-alternatives-cheatsheet

Desubscribe — 10 common subscriptions that add up to about $1,260 a year, each paired with a free self-hosted replacement.
https://github.com/Ezra-Zhao/desubscribe-cheatsheet

Why translate them?

Every repo ships the English cheatsheet plus READMEs in 中文, Español and Português. Spanish- and Portuguese-speaking developers are among the fastest-growing groups on GitHub, and I wanted the sheets to be readable for them from day one.

Corrections are genuinely welcome — open an issue or PR on any repo and I'll fix wrong commands fast.

Claude Code vs Codex CLI vs Cursor: An Honest Field Guide for Working Developers

Guangyi Zhao — Wed, 07 Oct 2026 01:04:24 +0000

Claude Code vs Codex CLI vs Cursor: An Honest Field Guide for Working Developers

I use all three. Not because I enjoy collecting tools, but because they genuinely think differently — and picking the wrong one for a task costs real time.

Last week I sat down and turned my notes into a proper cheatsheet repo: ai-coding-cheatsheet. Every command, shortcut, and config option in it was checked line by line against the official docs on 2026-10-06, and anything I couldn't verify got deleted rather than guessed at. These tools move fast, so treat anything version-sensitive as "check the official docs."

This article is the companion piece: not the command list, but the judgment — what each tool is actually for, and the habits that make them work.

Claude Code: the terminal assistant with a memory

Claude Code's defining idea is that the AI should remember your project. That memory lives in CLAUDE.md: put one at the project root (committed, shared with the team), keep personal preferences in ~/.claude/CLAUDE.md, and keep local-only notes in CLAUDE.local.md — which belongs in .gitignore. An AGENTS.md file is read too, so the two conventions can coexist.

The second defining idea is permission control. Claude Code asks before it acts, and you tune how much: plan mode makes it propose before touching anything, and the blunt instruments like --dangerously-skip-permissions exist for trusted environments only. Inside a session, Shift+Tab cycles the permission modes, and /init scaffolds a CLAUDE.md draft for you to refine by hand.

Two habits matter most. First, write verifiable instructions in your memory files — "run npm test before committing" beats "write good code" every time. Keep each file under about 200 lines; long workflows deserve their own skill file instead. Second, for anything big, start in plan mode: let the AI lay out which files it will change, nod, and only then let it type. One task, one session — don't pile unrelated work into a single conversation.

Codex CLI: the agent that separates "what" from "when"

Codex CLI's signature design decision is splitting two questions most tools blur together: what the agent is allowed to touch (sandbox) and when it has to ask you (approval). The sandbox (read-only / workspace-write / danger-full-access) and the approval policy (on-request by default, or never) are independent knobs. Once you see it, you can't unsee how useful that is.

Its superpower for automation is codex exec "fix the failing tests" — non-interactive, made for scripts and CI. It reads AGENTS.md from the project root on every run (build commands, code standards, forbidden actions), accepts an image in the first prompt for things like error screenshots, and can hand work off to the cloud. The VS Code plugin shares the same AGENTS.md, config, and MCP servers as the CLI, so your setup follows you into the editor.

The honest caveat: the flags with "dangerously" in the name mean it. Only bypass approvals and sandboxes in an isolated environment you can afford to rebuild.

Cursor: the IDE where AI is a first-class citizen

Cursor's whole pitch fits in three shortcuts: Cmd/Ctrl+K edits the selected code inline, Cmd/Ctrl+L opens a chat with your selection already in context, and Cmd/Ctrl+I hands the wheel to the Agent for multi-file rewrites. Learn those three and you know most of the tool.

Context control is explicit: @Files, @Codebase (semantic search across the repo), @Docs, @Web, @Git, or just @ a filename. Project rules live in .cursor/rules/ as one .mdc file each, with four attachment modes — Always, Auto Attached, Agent Requested, Manual — so rules can follow file patterns instead of living in one giant document. And the Agent itself has three postures: Ask (read-only, good for exploring unfamiliar code), Agent (edits files, you confirm each diff), and Plan (proposal first). One honest note from my cheatsheet: Cursor's interface changes fast, so for anything UI-related, check the official docs.

Side by side

	Claude Code	Codex CLI	Cursor
Home turf	Terminal, any editor	Terminal, scripts & CI	Inside the IDE
Getting started	`claude` to chat; `/init` scaffolds project memory	`npm install -g @openai/codex`, then `codex`	Install, learn the three shortcuts
Permission model	Permission modes (`plan` → auto), tuned per session	Sandbox and approval as separate knobs	Per-diff confirmation in Agent mode
Project memory	`CLAUDE.md` (`AGENTS.md` also read)	`AGENTS.md`, auto-read every run	`.cursor/rules/` + `@` references
Best for	Long-lived terminal workflows with team-shared conventions	Scripted and automated agent runs, cloud handoff	Day-to-day editing where you live in the editor

Five habits that work across all three

Plan before typing. Every tool here has a plan mode. Use it for anything bigger than a typo fix — a 30-second plan review saves 30-minute cleanups.
Write the machine-readable rules down. CLAUDE.md, AGENTS.md, .cursor/rules/ — whichever your tool reads, the payoff is the same: stop re-explaining your build commands and taboos every session.
Keep secrets out of the chat. Keys, tokens, private keys, customer data — never paste them in. API keys belong in environment variables, and .env belongs in .gitignore, never in the repo.
Respect the "dangerously" flags. They exist for isolated environments you can rebuild — not your laptop.
Small steps, visible diffs. One task per session, git diff before moving on, and make the AI show test results — "run the tests and paste the output" beats "make sure it works."

I'm still learning these tools, and they change fast — if anything above is out of date or just wrong, I'd genuinely like to know. The full command-level cheatsheet (Chinese, Spanish, and Brazilian Portuguese versions included) lives here:

https://github.com/Ezra-Zhao/ai-coding-cheatsheet

I'm building in public at github.com/Ezra-Zhao — corrections welcome.