DEV Community: guanguan li

The explanation was right. The policy ID was wrong.

guanguan li — Thu, 01 Oct 2026 04:24:27 +0000

This is a submission for the Kaggle Benchmarking Challenge.

What I Benchmarked

A support model told me the correct fee, named the applicable policy in its explanation, and then put a different policy in source_ids. A person reading the explanation and software consuming the fields would receive inconsistent guidance.

Support Boundary Bench asks whether a model can select the right support decision and the evidence that justifies it. All policies, products and fees are fictional. No real customer data or actions are involved.

Responses must contain five JSON fields: decision, source_ids, missing_fields, conflict_ids, and answer_text. Decisions must be answer, clarify, or handoff. Labels stay outside the model prompt.

I prepared 10 development cases and froze 30 evaluation cases in 15 pairs. Each pair changes evidence order, a required fact, the event date, source authority, or an untrusted instruction. Some changes should alter the response; others should leave it unchanged.

A pair earns one point only when both cases pass every structural check. The score is passed pairs divided by 15. Format failures count against it; provider failures stop the suite and produce no numeric capability score. Explanation quality is a separate review dimension.

Models Tested

I selected openai/gpt-5.4-mini-2026-03-17 and google/gemini-3.7-flash from the available Kaggle models. Inputs, prompts, labels and scoring rules were identical. Temperature used the SDK default; no seed was specified; each case allowed one attempt.

After the first comparison revealed date-related failures, I recorded a plan for one complete GPT replication on all 30 unchanged cases, before scheduling it. This checks repeatability on the same cases, not generalization to a new holdout.

Run	Valid contract	Structurally correct / assigned	Pairs passed	Request cost, USD
GPT baseline	30/30	26/30	12/15	0.02076075
GPT planned replication	29/30	26/30	12/15	0.02058525
Gemini baseline	30/30	30/30	15/15	0.06399675

These runs had no missing cases or provider errors. Costs are exported request metrics, not a project invoice. Publication packaging runs are kept separate from this experiment table. On October 1, rebuilding version 4 to fix platform task selection required fresh runs: Gemini passed 30/30 cases and 15/15 pairs (USD 0.06229425); GPT passed 27/30 cases and 12/15 pairs (USD 0.02086425). Both returned 30 valid contracts. The public leaderboard displays these version 4 runs, not the historical rows above.

Findings

Correct decisions can conceal incorrect evidence

GPT selected the correct decision type in 30/30 baseline cases, but passed all structural fields in only 26/30. All four failures involved policy dates. A decision-only metric would miss them.

Here is v2-temporal-2-a from the planned replication:

Item	Evidence or output
Event date	June 14, 2026
Policy `te-2-a`	58 fictional credits; ends June 15, exclusive
Policy `te-2-b`	73 fictional credits; begins June 15, inclusive
Expected source	`te-2-a`
Model source_ids	`["te-2-b"]`
Model explanation	Identifies 58 credits from `te-2-a` and says `te-2-b` does not apply yet

The inclusive-start/exclusive-end rule was explicit in the prompt. Mechanical label derivation and AI-assisted review checked the dates; neither is being presented as human approval.

In the replication, three temporal responses again explained the right policy while returning wrong source fields. Two case IDs failed in both GPT rounds; other failures changed. This is a repeated pattern in this suite, not a universal failure rate.

The same total score can hide different failures

GPT scored 12/15 pairs twice. The baseline had four valid responses with wrong structural fields. The replication had three such responses and one contract failure: hand-off instead of the required handoff.

That response described the conflict, but its enum was invalid for the machine interface. I did not normalize it after seeing the result. This is an interface failure, not evidence that the model misunderstood the conflict. Replication decision accuracy is 29/29 among valid responses, not 30/30; assigned-case and pair denominators still include the invalid response.

Gemini passed all 15 pairs in its original evaluation run. The small sample, shared templates and unequal repetitions do not establish a general model ranking.

Verify the evaluator too

An earlier source file hard-coded GPT. A job labeled Gemini therefore still called GPT. The importer detected the identical actual model IDs and refused the comparison. That extra GPT run and its USD 0.02112975 cost remain separate; they were not relabeled or selected to replace a first result.

The corrected entry point uses platform-injected kbench.llm. Import checks the requested model against recorded evidence. For each reported run I verified 120 child-file hashes, all 30 recorded prompts, frozen input/label/scorer hashes, and agreement between the numeric parent result and independent scoring. Earlier development failures remain separate too.

Human review update — October 2, 2026

I reviewed the 11 structurally failed responses from the GPT baseline, planned replication and publication runs, one at a time, using AI-prepared Chinese translations/summaries, policy tables, output fields and suggested judgment options. I confirmed each judgment in the conversation; the decisions are linked to the original response hashes.

Five responses had correct explanations and amounts but incorrect policy citations. Five also contained date-applicability or explanation errors (including one with the wrong amount). One correctly identified a policy conflict but used the invalid decision value hand-off instead of handoff.

This was an AI-assisted, non-blind review by one participant, not independent expert validation. These were selected failures across repeated runs of the same cases, not a representative sample. Full label review and review of the other responses remain incomplete. The frozen scorer, original outputs and reported scores have not changed.

What changes for a support workflow

Validate source IDs against product and event date before downstream software relies on them. A fluent explanation and valid decision enum are insufficient. This benchmark does not prove that adding such a validator improves customer outcomes; that requires another experiment.

Full label and explanation review remains incomplete; the targeted review above covers only the 11 selected failures. Development and evaluation share synthetic templates. Next I would test unseen policy templates and date boundaries, then evaluate a predeclared validation intervention. Tuning on these same cases would make them development data.

My Benchmark

Explore Support Boundary Bench on Kaggle.

Version 4 task and results · Backing notebook and evidence outputs.

Built with the Kaggle Benchmarks SDK. Frozen inputs, labels, scorer and raw per-case evidence are included in the backing notebook and its ZIP outputs. Codex assisted with implementation, synthetic data, analysis and writing; this does not replace human review.

PolicyTrace: Explain the fee. Know when to stop.

guanguan li — Wed, 30 Sep 2026 05:41:06 +0000

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content.

What I Built

A customer disputes an old fictional card fee. The statement was issued in July, but the charge posted in June. There are older and newer policies, a special-customer rule, and scoped clarifications. Which rule applied when the charge was posted, and when must support stop rather than promise a waiver?

PolicyTrace is an English-first, Chinese-supported support agent. A model routes the question, Sanity Context supplies policy records, deterministic checks handle effective dates, customer conditions, conflicts and authority, and a model confirms the citation set. Missing facts trigger a question. An unresolved conflict or waiver request produces an unsent human handoff. There is no refund or waiver execution tool.

Demo

Open the public evidence viewer and video. Try your own fictional bill in the guest sandbox: enter a fictional question or choose a posting date, customer type and request. It evaluates a frozen synthetic Sanity Context snapshot in the browser; it is not a live MCP or model run. The snapshot and verifier are available for inspection. The viewer replays saved results; it does not invoke a live model. The current 44-second narrated judge-sandbox walkthrough shows the missing-facts question and a fictional dated case using a frozen Sanity Context snapshot; it does not call live MCP or a model. The earlier 56-second local UI walkthrough shows the native Studio-backed August conflict, reviewed clarification and waiver boundary. Both videos are edited demonstrations, not uncut recordings. The separate cloud edit is documented in the raw trace below. The cloud-change trace and native-mode decision trace document the newer path.

The key scenario compares the same disputed premium-customer fee before and after a scoped clarification is reviewed. Before review, both applicable policies remain visible and the case goes to a human. After review, the clarification permits a sourced fee explanation, while a waiver request still goes to a human.

A Real Sanity Edit Changed the Answer

I added native Sanity Studio schemas for policy versions and policy clarifications, with explicit effective dates, customer scope, amounts, source IDs, authority and review fields. Six fictional documents are published in the project's production dataset. A separate Sanity Context MCP dataset endpoint exposes only those synthetic document types. The new Studio-backed option reads them through MCP and maps them into the existing deterministic policy verifier. It does not call a model; the original model-assisted Agent still uses the file-backed Context Knowledge Base.

For a reproducible cloud update check, I queried a June charge before and after a real, revision-guarded edit to the published synthetic policy: 40 → 41 → 40 fictional credits. Each read came through the same MCP-backed decision code; the last edit restored the original content. The cited source fingerprint changed with the edit and returned to its original value after restoration. Inspect the saved three-step evidence. This demonstrates that native Sanity content can change a historical-bill explanation without changing code. It is a controlled synthetic test, not a live banking outcome or an autonomous approval.

The Studio-backed August premium-customer example also preserves three applicable policy IDs and the clarification ID. Without the reviewed clarification it hands off; with it, the system can explain the fictional 23-credit fee, while a waiver still requires a human. The server validates a reviewed content hash before applying any clarification. Missing fields, incomplete retrieval, or changed approval content stop the answer.

Code

Download the current v6 source ZIP. The earlier v5 package passed 55 existing offline tests plus four native-adapter contract tests after independent extraction; the v6 archive adds the user-challenge fix and three focused regression tests. Its ZIP integrity and limited credential-pattern checks passed. It includes the FastAPI app, bilingual UI, original Agent path, native Studio schemas/adapter, synthetic fixtures, comparison scripts and saved traces. Live local operation requires the evaluator's own Sanity and model credentials; none are distributed.

How I Used Sanity

The original evaluated path used Sanity Context MCP to read structured records in a file-backed Knowledge Base: policy versions and clarifications. Effective-date ranges, customer conditions, amounts, source IDs, revisions and authority information affect the decision. An old posting date excludes a later policy; conflicting premium-customer rules trigger a handoff; a reviewed, scoped clarification changes what can be explained but never grants waiver authority.

The new native Studio path models those records as Sanity document types and reads the published documents via a separate Context MCP dataset endpoint. This is why the 40 → 41 → 40 edit changes the decision while code stays fixed. The Studio path is deterministic, so I do not claim that the model-assisted Agent itself was rerun against the edited native dataset. The earlier seven model-assisted cases and ten-case comparison use the file-backed snapshot.

A server-side reviewed content hash controls clarification approval; a content entry cannot approve itself by naming a reviewer. Source failures, unknown conditions and mismatched model citations fail closed. Customer facts are synthetic input, not identity verification.

Sanity Project Details

Project ID: 7d711ssk
Native Studio: PolicyTrace Policies
Native dataset: production; Context MCP endpoint: policytrace-structured
Original file-backed Context Knowledge Base ID: kbuLoYcmxK7A
Products, bills, customers and policies are fictional.

Evaluation and Limits

An actual Qwen3-8B → Sanity Context → policy checks → model citation-confirmation run passed seven developer-authored cases after a citation-prompt correction. The earlier first-case citation failure is retained. In ten predeclared fictional cases using one Sanity snapshot and the same Qwen3-8B model, PolicyTrace met the recorded decision, amount/null, valid nonduplicated citation and no-action checks in 10/10; a full-context direct-model baseline met them in 1/10 and a simple keyword Top-3 baseline in 0/10. Frozen cases and raw results are in the package.

I also tested 15 declared synthetic boundary cases against one actual Sanity Context MCP snapshot, covering effective-date edges, missing customer evidence and waiver handoff; three separate FastAPI calls each made a fresh MCP read. The first boundary run was 13/14 because my expected answer for an unverified standard customer was wrong: the policy correctly requested evidence. I retained that initial failed expectation, corrected the test and added the verified counterpart; the rerun was 15/15. Reproduction scripts are public. These tests use the deterministic Studio path, not the model-assisted Agent.

A user then posed a new fictional two-part question in Chinese: “Why does my card have an annual fee, and how can it be waived?” without a posting date, customer type or supporting record. The first run asked only for the date and missed the waiver intent. I preserved it, fixed the deterministic intake, and reran the unchanged question through the Studio API with a fresh MCP read. It now requests the missing facts and says waiver decisions require a human; no fee amount or approval is invented. Try the frozen-snapshot guest sandbox or inspect the scoring protocol. This is one user-authored case, not a representative benchmark.

The 15 boundary cases and ten-case comparison were written by the developer, not independently sampled or blinded. One later user-authored question is reported separately. The keyword baseline is simple, not a strong RAG implementation. The system has task-specific date and authority checks, so the methods have different capabilities. This experiment cannot establish production accuracy, a statistical win rate or superiority over other entries. No real bank connection or measured business return exists.

Built with AI-assisted development and testing. The author reviewed the design, code, results and limitations. No Agent Session transcript is submitted.