The Leak Nobody Notices
It started in the backlog: #2637/brand#16. Not a task name—just a truncated segment of checklist and prompt text welded into an issue title. It blended into the ticket flow at first, but pattern recognition kicked in.
Then another: #2644/brand#17. Same pointless fragment, same symptom. These titles were syntactically valid for the API but semantically invalid for a human-facing interface.
The failure was not “agent drift” in general. It was a boundary-validation failure: structurally acceptable internal text crossed into a surface artifact without being validated for human meaning.
Where the Structure Fails
The framing is:
internal representation → boundary transformation → human-facing artifact
Tracking the leak led back to make_issue_title in tasks/issue_title_builder.py. The boundary transformation accepted checklist prompt scaffolding and emitted it as issue/PR titles. Nothing crashed. API calls succeeded. But API success did not mean interface success.
The defect was precise: boundary logic validated structure (string present, request accepted) but not usefulness (title meaningful to a reviewer scanning work history).
Verifying the Data Model Assumption
Inspection of the current task model and title builder path showed description is the available human-facing label consumed at the boundary. The failure was that boundary logic did not consume that canonical human label and instead allowed truncated checklist/prompt scaffolding to pass through as title text.
The implemented correction is therefore structural at the boundary: use description as the source for title generation and treat checklist/prompt-shaped strings as leakage to be flagged even when they are API-valid.
What the Tests Actually Assert
In tests/test_title_generation.py, the implemented assertions check concrete leakage signals:
- title equals or begins with raw checklist/prompt tokens that match the observed leak shape (e.g.,
#2637/brand#16) - boundary output matching known raw/truncated checklist leakage patterns emits a warning even when PR/issue creation succeeds
- when a valid
descriptionis present, generated title uses that human-facing label rather than checklist/prompt scaffolding
Why This Distinction Matters
This was not a loud outage; it was interface degradation. The lesson is narrow and practical:
- internal scaffolding can be structurally valid
- boundary code can pass CI and API calls
- human-facing artifacts can still be wrong
That is exactly what happened here. The system did what the API accepted, not what people needed to read.
The Real Impact
The fix adds a guardrail at the boundary where internal representation becomes public artifact. It does not make the system flashy; it makes it legible.
For now, checklist-prompt leakage is detected and surfaced instead of silently shipped as titles.
Top comments (1)
The distinction you draw — API-valid but interface-invalid — is the one most boundary checks skip, and it's expensive to learn late. I hit the same shape with identifiers that round-trip cleanly through the API while carrying internal scaffolding no reader can parse; the fix that actually stuck wasn't a smarter generator but a validator that refuses to emit when the human-facing label is missing, instead of falling back to whatever string was nearest.
Two things I'd be curious about: does the guardrail ever block a real submission in practice, or only warn, and did you back-fill the historical titles that already leaked or let them stand as evidence of the old boundary? The second one matters more than it looks — silently rewriting issued titles breaks links those tickets advertise.