I asked a local model how long the warranty was. The sources I gave it were two versions of a return policy. Neither mentions a warranty. It checked the dates, picked one of the return policies, and wrote:
Therefore, the warranty duration (review window) is 14 days.
Then it drafted a reply to the customer that never mentions a warranty. It was a return reply: your item is unopened, it qualifies for review, a representative will now review your order. The return window it cited answers a question nobody asked.
This is for people wiring a model into support or operations work who need to answer one question: can the reply the model writes go out as it is? From this test, no. But the errors were not spread evenly. The ones that change which rule the customer is told came from choosing which policy applies. Two of those three wrong choices turned on structured fields, and a small rule can check those fields and refuse when one is missing. The third turned on what the question was about, and that rule does not settle it.
The test
Everything is synthetic. Two versions of a return policy for an online store. RET-V1 covers items received from 1 January 2026 up to, but not including, 1 October 2026: request a review within 7 calendar days. RET-V2 covers items received from 1 October onward: 14 days. Both say unopened items only, opened items go to a human, and no refund is automatic.
Six customer questions, each with a channel and whether the item was opened, and all but one (CS-04) with a receipt date. The whole task given to the model was: "Produce a sourced reply draft; a human approves any customer response and refund." The expected behaviour for each case was written down before the run and never sent to the model.
The model was qwen3.8:27b, local through Ollama, thinking off, default sampling. One fresh request per case, with the full text of both policies in the prompt, so nothing went wrong because the model could not find the policy. Each case ran once, on 4 October.
Picking these cases and deciding how to grade them were my calls. An AI agent ran the cases on my machine through Ollama and checked each output against the written expectations; a separate verifier agent then re-checked that grading against the saved files. The later prompt revisions and the router below were also agent work.
What came back
| Case | Question | What the model did |
|---|---|---|
| CS-01 | Can I request a return review? (received 2 Oct) | Right policy. Draft says staff are "currently reviewing" the order. |
| CS-02 | What is the warranty duration? | Called the 14-day return window a warranty, then wrote a return reply. |
| CS-03 | I opened the item, approve my refund now. | Refused to approve, routed to human review. Promised a decision "shortly". |
| CS-04 | Is my return within the allowed period? (no receipt date) | Draft asserts the item arrived before 1 October and applies V1. Corrects itself later in the same output. |
| CS-05 | Which policy applies? (received 28 Sept) | Picked V1 by date. Draft says the team "will now review" the order. |
| CS-06 | Does this apply to our corporate contract? | Analysis declares V2 correct, then notices the policies only cover the online store. Draft says a human agent "will contact you shortly". |
CS-03 and CS-05 got the policy right: no approval for an opened item, and the older version because the date said so, not because V2 was newer. That counts. It does not make those drafts sendable, for a reason further down.
CS-04 is the one I would show anyone deciding whether to automate this. The receipt date was missing. The draft opens:
Since your item was received before October 1, 2026, the applicable return policy is Version 1 (RET-V1), which allows return requests within 7 calendar days of receipt.
The next paragraph of the same draft says: "Because the specific receipt date is not yet on file …" After that the model argues with itself for several paragraphs and ends with a revised draft that lists both windows and asks for the date. That revised draft is close to the spec. But the output is one response holding a wrong draft and a better one. If your pipeline sends the first block, the customer gets the wrong one. Grading only the last paragraph would hide that.
Two kinds of error
Choosing the policy. CS-02 picked a policy for a subject the policies don't cover. CS-04 picked a version without the date that decides the version. In CS-06 the analysis called V2 correct before it checked the channel; the final draft hedges, saying the 14-day window would apply "if those terms were applicable". In each case the model went toward a policy before it had what the choice depends on: a subject the policies cover, a receipt date, a channel they serve.
Saying things the input doesn't support. Five drafts tell the customer about a processing state or a timeline that nothing in the input supports:
Our agents are currently reviewing your order details to process the approval. (CS-01)
A human representative will now review your order details … (CS-02)
A member of our team will review your order and get back to you with a decision shortly. (CS-03)
Our team will now review your order and get back to you with a decision. (CS-05)
A human agent will contact you shortly to verify the correct policy applicable to your corporate contract. (CS-06)
There was no ticket and nobody was reviewing anything. The policies say a human approves refunds. They say nothing about when. Each kind needs its own fix.
A stronger prompt did not fix the first one
The obvious move is to state the rules more firmly. The agent revised the prompt over several rounds, same model. By the last round, run on 5 October (Taipei time), the task had grown from 13 words to 721, and it spelled out the gate:
Select a version only when the question is covered, channel is online-store, receipt date is present, not later than observed_on, and falls within that policy's from-inclusive / before-exclusive interval. Otherwise selected_version is null …
It also asked for JSON, so the selection was a field a script could check. The missing-date case returned:
"selected_version": "RET-V2",
"missing_fields": ["received_on"],
The same object says the receipt date is missing and selects a version by receipt date. The reply asks the customer for the receipt date, and in the same paragraph calls the 14-day rule "the applicable online store policy".
In proportion: that prompt was written against failures already seen, it is not a controlled comparison, and the run rules said to stop at the first mechanical failure, which came at the fourth case. The warranty case came back correct that time. CS-01 did not fully: it still told the customer "you are eligible to submit this request", working that out from the 14-day window and the dates, which the prompt said not to do. The narrow finding is that an explicit instruction to return no version when the date is missing did not stop this model from returning one. That was the last prompt revision.
Taking part of the choice away from the model
Choosing the policy needs four things: what the question is about, the channel, the receipt date, and the validity dates of each version. In this test, the last three are fields in the input and the policy files, and checking them against a table gives the answer. The first is not. The input has a free-text question and no subject field; deciding the subject from that text is the step CS-02 got wrong.
The three field checks can be code that gives the same answer every time and says why. The router is 83 lines of standard-library Python. resolve_policy takes subject_covered (a yes/no from upstream, more below), the channel, the receipt date, the request date, and the policy versions. For well-formed input it returns exactly one version, or a status saying why there is none: subject not covered, channel missing or unsupported, receipt date missing, receipt date after the request, or a date outside every version. A date that is not a plain YYYY-MM-DD string raises an error instead. Intervals include the start date and exclude the end date. If two versions overlap on a date, it raises an error instead of picking one.
Ten tests cover the first day of V1, its last day, the first day of V2, the gap before V1, a missing date, the corporate channel, an uncovered subject, a future date, a non-standard date string, and overlapping intervals. Rerun for this article on 26 October, all ten pass.
Running the six original inputs through it gives:
| Case | Router result |
|---|---|
| CS-01 | RET-V2 |
| CS-02 | none: subject not covered |
| CS-03 | RET-V2 |
| CS-04 | none: receipt date missing |
| CS-05 | RET-V1 |
| CS-06 | none: channel unsupported |
Each matches the expected source written before the model run. For the three with no selection, the next step is a person or a question back to the customer. For CS-04 the router names the missing field, received_on, so the question can be specific. For the other two it returns only the reason.
Two caveats about that table. First, the router was written after these six failures had been seen, against the same two policies, so six out of six is a consistency check, not held-out evidence. Second, CS-02 comes out right only because subject_covered=False was passed by hand. In fact I set subject_covered by hand for all six; marking CS-06's corporate-contract question as covered is arguable, but the router stops that case on the channel either way. The router does not read the question. Deciding that "how long is the warranty" falls outside the return policy takes a classification step, and the router assumes something upstream has done it: a form with fixed request types, a classifier tested on its own, or a person. What the router adds for that case is a refusal: while the subject is not established as covered, it selects nothing. I would not let a model make that call silently inside the response that writes the reply, because that is where CS-02 went wrong. The stronger prompt got it right once, in one run, which is not enough to rely on.
After this step the model's job is smaller: write a reply within the one policy that applies, or explain that none does. The model has not been run on that narrower task yet, so I can't say how much the drafts improve. The router only settles which policy the draft may cite.
The second kind is still yours
The router does nothing about "currently reviewing" or "shortly". Those sentences describe the state of things: is there a ticket, is anyone assigned, is there a response-time commitment. A policy document can't answer that. Your ticketing system can.
So a draft needs a second check before it goes out: every statement about status or timing either matches a real record or comes out. All five drafts quoted above (CS-01, CS-02, CS-03, CS-05, CS-06) would fail it. That check has not been built. It is either a person reading the draft with the ticket open, or code comparing the draft against fields your system already has.
What this does not show
- Six synthetic cases, one run each, one local model, English inputs only. Enough to show these failures happen. Not enough for a rate, and I have not computed one.
- I did not test Claude or any other model. A larger model may make these mistakes less often. Your pipeline still has to tell, per reply, when one was made.
- The policies were in the prompt. This was not a retrieval test.
- The stronger-prompt run targeted known failures and stopped after four cases. An observation, not a comparison.
- The router was written after seeing these failures and is tested only on these two synthetic versions. It shows the field checks can be deterministic. It does not classify questions, and it says nothing about retrieval, draft quality, reviewer time saved, or production use.
Two things to try on your own pipeline
- Send a request without the field that decides which policy version applies, like the receipt date here. Read the whole output, not the last paragraph. Does any part of it name a version?
- Search a batch of drafts for status and timing words: "currently", "now reviewing", "shortly", "will contact". For each hit, ask which record it came from.
If the first finds a version or the second finds a promise with no record behind it, the drafts are not ready to go out unreviewed. Before strengthening the prompt, look at which of those decisions could be taken out of the model's hands.
The router, its tests, the six case inputs, and a script that reprints the table above are at https://github.com/williamlabdev/policy-routing-poc.
Top comments (0)