I work with the Auto-Respond Team. This article describes a general testing pattern we use around AI lead handling. The code uses fake services and fake leads. No customer data is involved.
A model can write a convincing quote that is completely wrong.
That is the uncomfortable part of using generated text in a sales workflow. The wording may look polished while the number came from nowhere, the service needs an inspection, or the customer asked for work outside the configured scope.
I do not think a larger prompt fixes this. Price rules belong in ordinary code, where they can be reviewed and tested.
Put the decision before the prose
The model should not decide whether a quote is allowed. Give that job to a small deterministic function. The model can turn an approved decision into natural language after the policy check passes.
Here is a deliberately small TypeScript shape:
export type PriceRule = {
service: string;
mode: "fixed" | "range" | "inspection";
min?: number;
max?: number;
currency: "USD";
policyVersion: string;
};
export type QuoteRequest = {
service?: string;
requestedAmount?: number;
details: string[];
};
export type QuoteDecision =
| { action: "quote"; min: number; max: number; reason: string }
| { action: "ask"; question: string; reason: string }
| { action: "handoff"; reason: string };
The useful constraint is in the return type. There is no free-form price field that a caller can populate from model output.
Keep the policy function boring
Boring is a compliment here.
export function decideQuote(
request: QuoteRequest,
rules: PriceRule[],
): QuoteDecision {
if (!request.service) {
return {
action: "ask",
question: "Which service do you need?",
reason: "service-missing",
};
}
const rule = rules.find(
(candidate) => candidate.service === request.service,
);
if (!rule) {
return {
action: "handoff",
reason: "service-not-configured",
};
}
if (rule.mode === "inspection") {
return {
action: "handoff",
reason: "inspection-required",
};
}
if (rule.min === undefined || rule.max === undefined) {
return {
action: "handoff",
reason: "invalid-price-rule",
};
}
if (
request.requestedAmount !== undefined &&
(request.requestedAmount < rule.min ||
request.requestedAmount > rule.max)
) {
return {
action: "handoff",
reason: "requested-amount-outside-configured-range",
};
}
return {
action: "quote",
min: rule.min,
max: rule.max,
reason: `approved-by-${rule.policyVersion}`,
};
}
This function does not try to sound human. It answers one question: may this workflow present a configured price or range?
Test the table, not the prompt
Prompt snapshots are useful, but they are a weak price-control test. A wording change can make a snapshot noisy without changing policy. A table gives each business rule a name and an expected decision.
import { describe, expect, it } from "vitest";
import { decideQuote, type PriceRule } from "./quote-policy";
const rules: PriceRule[] = [
{
service: "drain-cleaning",
mode: "range",
min: 120,
max: 220,
currency: "USD",
policyVersion: "pricing-2026-07",
},
{
service: "main-line-repair",
mode: "inspection",
currency: "USD",
policyVersion: "pricing-2026-07",
},
];
const cases = [
{
name: "quotes a configured range",
request: {
service: "drain-cleaning",
details: ["kitchen sink is slow"],
},
expected: "quote",
},
{
name: "asks when the service is missing",
request: { details: ["please send a price"] },
expected: "ask",
},
{
name: "hands off an unknown service",
request: {
service: "roof-replacement",
details: ["two-story house"],
},
expected: "handoff",
},
{
name: "hands off work that needs an inspection",
request: {
service: "main-line-repair",
details: ["yard is wet near the cleanout"],
},
expected: "handoff",
},
{
name: "rejects a requested number outside the configured range",
request: {
service: "drain-cleaning",
requestedAmount: 25,
details: ["another company said it would be cheap"],
},
expected: "handoff",
},
] as const;
describe("quote policy", () => {
it.each(cases)("$name", ({ request, expected }) => {
expect(decideQuote(request, rules).action).toBe(expected);
});
});
The fifth case is easy to miss. A lead may mention a price, ask for a discount, or paste a number from another site. That number is context, not an approved rule.
Treat model output as untrusted input
If a model extracts service or requestedAmount, validate the result before calling decideQuote.
A schema validator helps with shape. It does not prove that the extracted value is true. Keep the original lead text available for review, and prefer a handoff when the extraction is uncertain.
I also store four pieces of audit data with the decision:
- the normalized service identifier
- the price-policy version
- the decision reason
- the final allowed action
Do not log the whole conversation by default. Tests rarely need a real phone number, address, or customer name. Synthetic fixtures are easier to share and much safer to debug.
Test the sentence after the number is approved
Once the decision returns quote, the model can draft the customer-facing response. The draft still needs a final assertion.
For a range of $120 to $220, extract every currency amount from the generated message and fail the response if any amount falls outside the approved range. Also fail if the message turns a range into a guaranteed final price.
That gives you two separate tests:
- Was a quote allowed?
- Did the generated sentence stay inside the allowed decision?
The split is worth the extra code. When a test fails, you know whether the problem is policy selection or wording.
The practical boundary
In Auto-Respond's AI answering workflow, the safe product boundary is that AI may quote configured prices or ranges based on business rules. It must not invent or guarantee a final price. The same boundary works in a custom system: deterministic policy first, generated prose second.
If your current test says only expect(message).toContain("$"), replace it with a decision table. You will catch the failures that polished demo transcripts hide.
Top comments (0)