Last Monday I shipped the shipping-tool prompt from Part 9 to 5% of live traffic. It had won the pairwise gate 24 to 7 with 9 ties, the tool discipline diff was clean, and the money-path cases read fine by hand. I went for lunch confident.
At 6:42pm the canary dashboard disagreed. P95 latency was up 38%. Tool calls per conversation had climbed from 3.1 to 5.4. The reworded tool description I was so proud of had taught the agent to call the shipping tool twice per turn, and in real conversations, which run much longer than my 40 test cases, every extra call doubled the wait. I rolled the split back to zero in nine minutes, and the numbers returned to baseline by 7:15.
The 40-case dataset from Part 8 could not have caught this. Its transcripts are short by design. The pairwise judge from Part 9 could not have caught it either, because it judges two responses, not the whole session cost. Only traffic could, and traffic only talks to you if you route a slice of it first.
This part is the production runbook I promised at the end of Part 9: canary traffic splits, automatic fallback when a model degrades, and cost caps that stop a prompt regression from becoming a bill regression. The agent is the same e-commerce assistant from Parts 1 through 9: nine tools, conversation memory, the supervisor, and the human-in-the-loop checkout gate. I have been building production AI agents with Spring Boot and Spring AI for over a year, and every number below is from the rollout as I actually run it.
Why rollout discipline is now the safety layer
Every gate in this series has lived before traffic. Part 6 proves the code is bug-free, Part 8 proves the answers are good on a fixed dataset, Part 9 proves a change beats its predecessor in a controlled comparison. None of them prove a change survives real users, because real users do things your dataset never imagined: they write messages in mixed Bengali and English, they ask about orders from three months ago, they argue with the agent about the cached price from Part 4.
The wider industry is moving in the same direction, and it makes the gap bigger, not smaller. Anthropic measured that Claude Code users approve 97% of permission prompts, a rate it says suggests most click through without reviewing each command, and in a 1,053-tester study its auto mode caught 89% of planted dangerous commands where humans caught 13.6%. It is making auto mode the default for new sessions on Pro, Max, and Team plans from August 14, per its own announcement. Whatever you think of that trade, it describes the same shift: agents act with less per-call human oversight, so the mechanics around the release decide what reaches users, not the review in the IDE. If your agent runs unattended, your canary and your fallback are the reviewer.
Canary splits: Route a slice, watch it, trust it
The principle is boring on purpose. Two versions of the agent exist at the same time, both built from the same components as the production agent. A router sends a small percentage of conversations to the candidate and the rest to the baseline. You watch the candidate cohort against the baseline cohort, and you promote only when the candidate stops losing.
Spring AI gives you the two clients. The ChatClient reference shows the pattern I use: the auto-configured prototype ChatClient.Builder produces one bean per configuration, and you inject them by name with @Qualifier.
@Configuration
public class AgentRoutingConfig {
@Bean("baselineAgent")
ChatClient baselineAgent(ChatClient.Builder builder) {
return builder
.defaultSystem(SYSTEM_PROMPT_V1)
.build();
}
@Bean("candidateAgent")
ChatClient candidateAgent(ChatClient.Builder builder) {
return builder
.defaultSystem(SYSTEM_PROMPT_V2)
.build();
}
}
Both beans share the same tool registry, the same memory wiring, and the same advisors as the production agent from Parts 1 through 9. The only difference is the system prompt, and in this agent the tool descriptions live inside the system prompt, so a tool-description change like the shipping prompt from Part 9 is a system-prompt change. If you change two things between the clients, the canary cannot tell you which one moved the numbers.
The router is where the discipline lives. The important detail is stickiness: the same conversation must stay on the same version for its whole life, because the agent's memory (Part 2) is per-conversation and per-version. A customer who asks a question, gets an answer from the candidate, then refreshes and hits the baseline, will experience a different agent mid-conversation. So I route on a hash of the conversation id, not on a per-message coin flip.
@Service
public class CanaryRouter {
private final Map<String, ChatClient> agents;
private final CanaryProperties props;
public CanaryRouter(Map<String, ChatClient> agents, CanaryProperties props) {
this.agents = agents;
this.props = props;
}
public ChatClient forConversation(String conversationId) {
int bucket = Math.floorMod(conversationId.hashCode(), 100);
if (bucket < props.candidatePercent()) {
return agents.get("candidateAgent");
}
return agents.get("baselineAgent");
}
}
agent.canary.candidate-percent=5 in application.properties, and a restart flips the split without a deploy. CanaryProperties is a small @ConfigurationProperties(prefix = "agent.canary") holder with a single int field, candidatePercent(), so the split comes from configuration, not code. That is the other rule: the ladder is 5, 10, 25, 50, 100, each step held for at least a day, and every step is a config change, never a code change. Code changes restart the experiment. I skip rungs only when the cohort numbers stay flat.
The candidate cohort is a cohort, not a sample. Compare the candidate against the baseline on the same slice of time: error rate, p95 latency, tool calls per conversation, refusal rate, and the Part 8 metrics sampled from live logs. The cohort comparison is what saved me on the shipping-prompt day. The nightly harness would have flagged the tool discipline drop the next morning. The canary flagged it at 6:42pm, hours after the 5% step, because the candidate's p95 had drifted from the baseline's by a margin the cohort report was built to catch.
Rollback is automatic and it is a config flip. My triggers: error rate exceeds the baseline by one percentage point for ten minutes, p95 exceeds 1.5x baseline for ten minutes, or any money-path conversation (checkout, refund, shipping) fails the Part 8 review. Any trigger sets candidate-percent to 0 and pages me. I do not want to be woken up to make a judgment call at 2am; I want to be woken up after the decision is made, to investigate.
The shipping prompt went back to the drawing board. The narrowed description, one that said the tool resolves a region and the delivery estimate and that it is called once per turn, re-ran the Part 9 gate, then climbed the ladder again. It took five days to reach 100%: two days at 5%, one at 10%, one at 25%, then straight to full, skipping 50 because the cohort numbers stayed flat. Traffic is the final reviewer, but it reviews one slice at a time.
Fallback: Surviving a model that misbehaves
Canaries protect you from your own changes. Fallback protects you from everything else: a provider outage, a model that gets worse after an upstream update, a rate limit at peak hour. I split this into two failure classes, because they need different machinery.
Hard failures are exceptions: 5xx responses, timeouts, rate limits. The fix is a decorator around the model. Spring AI's ChatModel interface is small: call(Prompt) returns a ChatResponse, and stream(Prompt) returns a Flux<ChatResponse>. That interface is the seam. I wrap the primary model with a backup model and a small circuit state: three consecutive failures open the circuit for 60 seconds, during which every request goes to the backup, and a successful probe closes it again.
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.chat.prompt.Prompt;
import reactor.core.publisher.Flux;
import java.time.Duration;
public class FallbackChatModel implements ChatModel {
private final ChatModel primary;
private final ChatModel backup;
private final CircuitState circuit = new CircuitState(3, Duration.ofSeconds(60));
@Override
public ChatResponse call(Prompt prompt) {
if (circuit.isOpen()) {
return backup.call(prompt);
}
try {
ChatResponse response = primary.call(prompt);
circuit.recordSuccess();
return response;
} catch (RuntimeException ex) {
circuit.recordFailure();
return backup.call(prompt);
}
}
@Override
public Flux<ChatResponse> stream(Prompt prompt) {
if (circuit.isOpen()) {
return backup.stream(prompt);
}
return primary.stream(prompt)
.onErrorResume(ex -> {
circuit.recordFailure();
return backup.stream(prompt);
});
}
}
The circuit state is a small counter class that encodes the whole policy: failures increment, three failures opens the circuit, a success resets it.
// imports: java.time.Duration, java.time.Instant
class CircuitState {
private final int failureThreshold;
private final Duration openDuration;
private int failures;
private Instant openedAt;
CircuitState(int failureThreshold, Duration openDuration) {
this.failureThreshold = failureThreshold;
this.openDuration = openDuration;
}
boolean isOpen() {
return openedAt != null
&& Instant.now().isBefore(openedAt.plus(openDuration));
}
void recordSuccess() {
failures = 0;
openedAt = null;
}
void recordFailure() {
failures++;
if (failures >= failureThreshold) {
openedAt = Instant.now();
}
}
}
Spring AI itself does not ship a circuit breaker, so this wrapper is the honest option; the Spring Cloud Circuit Breaker integration gives you the annotation-driven alternative. One streaming caveat from Part 3: if the primary fails mid-stream, the user has already seen partial text, and switching models mid-sentence makes the answer worse, not better. My fallback only engages at request start. A mid-stream failure completes with what it has, and the observability layer from Part 4 records the truncation.
Silent degradation is worse, because nothing throws. A model can get noticeably worse without a single exception: more refusals, less tool discipline, hallucinated prices. The only detector is measurement, which is why Part 8 exists. The nightly harness stays the arbiter, and I add one rule: two consecutive nights below a metric threshold triggers the same automatic action as the breaker, a config flip that routes traffic to the backup model and pages me. Part 8 said one night is noise and three nights is a signal. Two nights is the compromise for degradation, because every night a degraded model runs costs you customers and money.
The outage that sold me on this: a provider 5xx wave hit at 2:14pm on a Tuesday. The breaker opened about 40 seconds later, once three requests had failed. Traffic ran on the backup model for 26 minutes while the provider recovered. Error rate for that window stayed at 0.4%, where the same window a week earlier, without fallback, had run at 2.1% during an identical incident. Customers noticed slightly slower answers. They did not see errors.
Cost caps: Stop a prompt regression from becoming a bill regression
The canary caught the shipping prompt's latency before it reached everyone. The cost cap exists for the same reason, because a regression that is too subtle for latency can still empty the budget. My rule of thumb: a prompt change that adds one tool call per conversation, at the traffic the agent now handles, moves the daily model bill by more than 20%. The bill is the last number anyone checks and the first one finance asks about.
Spring AI tracks tokens on every response, and the usage handling reference shows the exact access pattern:
ChatResponse response = chatClient.prompt(prompt).call().chatResponse();
Usage usage = response.getMetadata().getUsage();
long turnTokens = usage.getTotalTokens(); // prompt + completion for this call
Long cacheRead = usage.getCacheReadInputTokens(); // null when the provider has no caching
Three caps sit on top of that one line, and they are the difference between a surprise bill and a managed one.
Per-turn cap. The maxTokens option already limits a single completion, but the expensive input is the prompt: with conversation memory, every turn re-sends the history. The per-turn number to watch is total tokens, not completion tokens.
Per-conversation cap. Long conversations are where cost escapes. I keep a running total per conversation, and when it crosses 6,000 tokens, the agent switches to a cheap model for the rest of that conversation, or hands off to a human when the conversation is money-path. The polite handoff is a product decision, and the Part 7 approval gate gives you the hook to hang it on.
Per-day global cap. One number: the trailing seven-day average daily bill times three. Crossing it flips all traffic to the cheap model for the rest of the day and pages on-call. It fired twice in the first month, and both times the cause was the same: a prompt change that made the agent more conversational, and therefore longer-winded, with zero effect on the metrics anyone was watching. The judge scored the answers as better. The cap scored them as 31% more expensive. The bill landed at 8% over instead of 31%, because the cap caught it at noon.
Prompt caching is the second lever, and it is provider-specific. The usage reference lists which providers report cache reads: Anthropic and OpenAI read cached input, Google Gemini does too, while DeepSeek, Mistral, and Ollama report nothing. For long conversations, the cache read number is the one that tells you whether your repeated history is actually cheap. If you run local models via Ollama, there is no cache metric to lean on, so the per-conversation cap matters more, not less.
The runbook
If you take nothing else from this part, take this list. It is the whole rollout, compressed.
- Route conversation-sticky, not per-message. Hash the conversation id. A customer must never meet two versions of your agent in one conversation.
- Climb 5, 10, 25, 50, 100. Each step is a config change and holds at least a day, and I skip rungs only when the cohort is flat. Code changes restart the experiment.
- Compare cohorts, not absolutes. Candidate vs baseline on error rate, p95, tool calls per conversation, and refusal rate, plus the Part 8 metrics sampled from live logs, on the same slice of time.
- Roll back automatically. Error rate plus one point for ten minutes, or p95 at 1.5x baseline for ten minutes, flips the split to zero and pages you after the fact.
- Wrap the model for hard failures. Three failures opens the circuit for 60 seconds. Fallback engages at request start, never mid-stream.
- Detect silent degradation by measurement. Two consecutive nightly runs below threshold flip traffic to the backup. One night is noise.
- Cap tokens at three levels. Per turn, per conversation (6,000 for mine), and per day (3x the seven-day average). The bill is a metric.
- Watch cache reads alongside totals. If your provider reports them, they tell you whether long conversations are actually cheap.
What I would do differently, if I restarted the series: capture getMetadata().getUsage() on every call from Part 1 on. I retrofitted the usage pipeline in this part, which meant the first cost cap ran on a week of partial data. Usage is the cheapest observability there is, and it should have been a column in the Part 4 dashboards from day one.
What does your rollout runbook look like? What was the last change your traffic caught that your tests did not? I read every response.
I write about Java, Spring Boot, and AI agents every week. Subscribe, it's free.
Bookmark this one. You will need the runbook the day your pairwise winner meets real traffic.
Top comments (1)
Canaries and cost caps are exactly the kind of production details agent demos usually skip. Model fallback is not just a reliability feature either; it changes behavior, latency, and cost at the same time, so treating it as an observable release path makes sense.