Awesome write-up. Spotting that regression in mandate_cell7.py and tracking down the coupling between the classifier and string notes is top-tier debugging. Enforcing raw before/after snapshots at the gate instead of derived labels was completely the right call—keeping unmanipulated evidence for third-party auditing is huge for deterministic safety.
The key takeaway:
Green tests just mean the code matches your spec, not that the spec makes sense. Coupling control flow to free-text strings ("ttl expired" in event.notes.lower()) creates a ticking time bomb where changing a log message silently breaks classification without failing a single test.
A couple thoughts on preventing it:
Strict typing over string parsing: Drop string grepping in classifiers entirely. Require explicit enums (e.g., ReasonCode.TTL_EXPIRED) or numerical fields (ttl_remaining_hours), and relegate notes strictly to display/logging.
Fuzz string fields in tests: Throw random strings into event.notes during test runs. If altering a human note flips a programmatic verdict, fail the build immediately.
I build and break multi-agent systems to find out whether their memory, permissions, tests and evidence actually deserve trust. Everything I publish ships with receipts you can run yourself.
the fuzz idea is the one im taking, and we dont have it. a random string into notes on every run, and if a programmatic verdict moves because a log message changed, the build fails. that catches this on the day it is written and it costs one test.
the first suggestion is the one i want to be careful about, because it is exactly what i did and it is in the piece as the part that failed.
drop the grep, require an explicit enum was repair one. source_consult, five allowed values, classifier dispatches on it and never reads notes again. 366 tests green. it was falsified by a single row where the enum said SKIPPED_TTL_EXPIRED and the number on that same row said seventeen hours remaining. the enum won.
then require a numerical field was repair two, and you named the exact one. ttl_remaining_hours is stored as round(seconds/3600, 2), which is thirty six second granularity. a grant expired by one second lands on -0.0, and -0.0 >= 0 is True in python, so it classified invalid instead of expired. the number lied for a different reason than the string did.
so the type was never the thing that mattered. what survived was moving authority to something a reader can recompute from the row, decision_timestamp > grant_expires_at. not better typed. less derived.
which is why i think the fuzz test is the stronger half of what you sent. it catches a classifier reading the wrong field, and it catches it cheaply and forever. what neither it nor the type system catches is a contract sentence that permits the wrong thing. mine said the enum decides and the raw field may corroborate. the implementation was faithful to that and every test agreed.
pm25coder pushed on the third version in another comment here and hes right that it is not finished, because grant_expires_at is itself computed. his test is the one i think is correct: the authority path has to be replayable from the row alone.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Awesome write-up. Spotting that regression in mandate_cell7.py and tracking down the coupling between the classifier and string notes is top-tier debugging. Enforcing raw before/after snapshots at the gate instead of derived labels was completely the right call—keeping unmanipulated evidence for third-party auditing is huge for deterministic safety.
The key takeaway:
Green tests just mean the code matches your spec, not that the spec makes sense. Coupling control flow to free-text strings ("ttl expired" in event.notes.lower()) creates a ticking time bomb where changing a log message silently breaks classification without failing a single test.
A couple thoughts on preventing it:
the fuzz idea is the one im taking, and we dont have it. a random string into notes on every run, and if a programmatic verdict moves because a log message changed, the build fails. that catches this on the day it is written and it costs one test.
the first suggestion is the one i want to be careful about, because it is exactly what i did and it is in the piece as the part that failed.
drop the grep, require an explicit enum was repair one. source_consult, five allowed values, classifier dispatches on it and never reads notes again. 366 tests green. it was falsified by a single row where the enum said SKIPPED_TTL_EXPIRED and the number on that same row said seventeen hours remaining. the enum won.
then require a numerical field was repair two, and you named the exact one. ttl_remaining_hours is stored as round(seconds/3600, 2), which is thirty six second granularity. a grant expired by one second lands on -0.0, and -0.0 >= 0 is True in python, so it classified invalid instead of expired. the number lied for a different reason than the string did.
so the type was never the thing that mattered. what survived was moving authority to something a reader can recompute from the row, decision_timestamp > grant_expires_at. not better typed. less derived.
which is why i think the fuzz test is the stronger half of what you sent. it catches a classifier reading the wrong field, and it catches it cheaply and forever. what neither it nor the type system catches is a contract sentence that permits the wrong thing. mine said the enum decides and the raw field may corroborate. the implementation was faithful to that and every test agreed.
pm25coder pushed on the third version in another comment here and hes right that it is not finished, because grant_expires_at is itself computed. his test is the one i think is correct: the authority path has to be replayable from the row alone.