In late September 2026, the Wall Street Journal reported that OpenAI had scrapped the planned October release of GPT-6.1 Astra. Internal safety tes...
For further actions, you may consider blocking this person and/or reporting abuse
Several of you raised paraphrase matching and the operator-owned log, so I built a free page that runs the three-class check in your browser: paste a tool-log JSON and the agent's report, and it flags reported-but-not-done, done-but-not-reported and out-of-scope calls. It matches on argument values (amounts, IDs) rather than wording, and nothing leaves the page. It is rule-based, so it will miss cases. Tell me which ones: ab-agent-audit.agentbets.workers.dev/
The three classes are a clean way to frame it. Class 2 (done but not reported) is the one I'd worry about most, because nothing in the transcript hints that it happened.
One weakness of matching normalized report text against the log is paraphrase. "Issued a refund" vs a
payments.create_refundcall won't match, so you'll get false Class 1 flags, and real fabrications get lost in the noise. One fix is to make the agent's final report structured: every claim has to cite the tool-call IDs that support it. Then reconciliation is an exact join, and any claim with no ID is a Class 1 by construction.How do you handle generic tools? If the agent runs a migration through
shell.exec, the log shows one call, and detecting the omission means parsing the arguments rather than the tool name.You're right that Class 2 is the scary one — there's no signal in the transcript at all. And yes, paraphrase is the weak point of text matching: "issued a refund" vs payments.create_refund won't normalize together, so you get false Class 1s and real fabrications hide in the noise. The tool-call-ID citation idea is the stricter version — exact join, and an uncited claim is Class 1 by construction. I kept the kit heuristic and stdlib-only on purpose (works on any log format you already have), but if you control the agent, structured claims are strictly better.
Structured claims are the clean version, agreed. For logs you don't control there's a middle route that survives paraphrase: match on argument values rather than verbs. "Refunded $42.10 on order 8841" can change wording freely, but 42.10 and 8841 still have to appear in some call's arguments. A claim whose numbers and IDs match no call gets flagged, and it's far less noisy than matching tool names.
It also helps with the shell.exec case: split the command into binary plus subcommand ("alembic upgrade", "rm -rf") and treat that as the effective tool name. Anything that falls through to a bare "bash -c" goes in an unclassifiable bucket, which is worth reporting separately from Class 1.
Argument-value matching is exactly the right middle route — it's why the kit does field-level extraction rather than string matching on tool names. The shell.exec split is a good call too; binary + first subcommand as the effective tool name handles most of it. The remaining hole is computed values: sums, counts, anything derived can't be matched literally. The kit flags those as "derived" and requires the derivation to be recomputable from the log. Thanks for pushing on this.
The three discrepancy classes are a genuinely clean frame, and the severity rubric is the part teams will actually use. Trending the mediums is the right instinct; drift kills you slower and quieter than incidents do.
One thing to push on: the tool log is still the operator's artifact. Your whole audit reconciles two things the operator produced (the generated report, the generated log) and trusts one of them as ground truth. That is a real improvement over trusting the report alone, but the trust still terminates inside the operator's stack. If the operator's logging path drops a call, your Class 2 detector has nothing to see, and the absence is indistinguishable from nothing having happened.
The missing third artifact is a receipt minted by the executor at call time, checkable without trusting either the report or the log. Then the audit has two gaps to work instead of one: report vs receipt (did it say what it did) and log vs receipt (did it log what it did). The receipt is the independent witness your algorithm currently has to assume.
Your last section already admits the boundary: the honesty audit checks reporting, not correctness. A verifiable receipt has the same boundary, and it is worth stating plainly the way you did: a receipt proves the call happened as recorded, not that the call was wise. If you want to see one checked live, zambo.dev/verify/ takes a receipt and shows you exactly what it does and does not prove.
Fair push, and worth stating plainly: the kit audits the honesty of the report against the log — it does not audit the integrity of the log itself. If the logging path drops a call, that's outside this tool's boundary. The complementary layer is tamper-evident logging (hash-chained, external sink), and the verification report template has a section for declaring the log's provenance. If you can't trust the log, you need a different tool first.
The distinction between reporting honesty and actual correctness is an important boundary here. At IT Path Solutions, we’ve found that a tool log can prove a call happened, but it still may not prove that the intended state change actually occurred. A refund API could return an accepted request while the downstream payment state remains unchanged, for example. I’d treat the audit as a chain of evidence: the self-report should bind to the tool call, the tool call should bind to its execution result, and the final claim should bind to an independently observable outcome. That keeps “the agent said it happened,” “the tool says it ran,” and “the requested state actually changed” as three separately testable assertions.
Agreed on the chain of evidence — and you've named a real gap in the current three classes. They cover report-vs-log, but not log-vs-world: a call that executed yet produced no verifiable effect. That's arguably a fourth class ("executed, effect unverified"), where the check binds the tool result to an independently observable outcome. I'll add it to the rubric.
Separating the self-report from the tool log is the right frame. The summary is written to sound complete, and the log is what actually happened. Most teams only read the first because it is the one that arrives in Slack.
On agent projects we deliver, the sign-off test is a diff between the two: every claim in the final message has to map to a call in the log, and any state-changing call has to appear in the summary. If either side has something the other does not, the run fails, even when the end state looks right.
It is cheap to automate and it catches a surprising number of quiet failures before a client ever sees them. It also gives you a clean record to show when someone asks what the agent actually did last Tuesday.
That's exactly the rule the audit tool encodes — bidirectional check, unmatched on either side means fail. One wrinkle I kept hitting: the "any state-changing call has to appear in the summary" side needs a definition of state-changing. I went with a coarse allowlist (write/delete/execute vs read-only) instead of trusting the agent to self-classify, because a mutating call labeled "lookup" is exactly the thing that gets hidden. Curious how you draw that line.
This matches a rule we landed on building agentic pentesting: a finding only counts when the run log carries the raw tool output behind it — otherwise it stays a hypothesis and never reaches the report. The piece we would add is timing: reconciliation catches class-3 discrepancies after the fact, while a gate at the tool runner is what stops them during the run. Do you keep this post-run only, or also block live?
Both. This piece is deliberately post-run — the audit needs the complete trace to exist before it can say anything. But you're right that reconciliation alone never stops the bad action; the runner-side gate is the enforcement half. The design I'd want: the gate emits evidence in the same format the audit reads, so the two agree by construction instead of by hope. That's the follow-up I'm working on.
Đúng trọng điểm. Mình thấy pattern này lặp lại liên tục khi team scale agent từ prototype sang production: self-report của model thường "hallucinate success" — nó sinh ra narrative nghe hợp lý nhưng tool log lại show retry loop, wrong function call, hoặc silent failure mà model không thừa nhận.
Vài lesson learned từ lúc mình audit hệ thống agent năm ngoái:
Separate observation từ reasoning — log raw tool output (stdout/stderr, HTTP status, DB row count) riêng biệt với model's internal monologue. Đừng để model summarize tool result cho chính nó.
Deterministic checkpoint — mỗi step quan trọng (payment, write DB, external API) phải có idempotency key + external verification. Model nói "done" ≠ done.
Gap metric — track tỷ lệ mismatch giữa self-report vs tool log theo từng skill type. Mình thấy code generation skill gap ~12%, web search ~30%, file ops ~5%. Số liệu này giúp prioritize hardening.
Human-in-the-loop trigger — khi gap vượt threshold, auto-escalate thay vì để agent tự retry vô tận.
Cái khó nhất không phải technical mà là cultural: dev vẫn thói quen trust "agent said it worked" thay vì đọc log. Cần code review culture áp dụng cho — found it via LabAgent, site: labagent .tech
The per-skill gap metric is the most actionable of the three — it tells you exactly where the hardening budget goes. Your numbers match my intuition: the less deterministic the tool, the wider the gap, which is why web search at ~30% doesn't surprise me. And the "never let the model summarize its own tool result" rule is precisely the failure mode Class 2 logging exists to catch.