Replace a frontier model with a fine-tuned small open model on one narrow, high-volume ancillary task, after a prompted but untrained open model already looks close and an online test holds your hard-resolution rate. Fergal Reid described that swap for query canonicalization: a fine-tuned 14 billion parameter Qwen model took over a GPT-4.1 summarization step. By his estimate the bill could have been like a quarter million dollars a month on that task alone, and the replacement saved almost all of that inference spend.
Fergal Reid is chief AI officer at Intercom. This episode aired in February 2026.
When is one LLM call worth leaving the frontier model?
Move a call off the frontier API when it is ancillary, repeated on every request, and a well-prompted but untrained open model is already close before you train it. Fin, Intercom's customer service agent, still runs a frontier model on the hardest prompt, the one that answers the end user. That prompt runs on Claude Sonnet 4, with a move to a newer Sonnet under way. Smaller prompts sit around it. The long-running first step summarizes and canonicalizes the user question before retrieval, because people ask in colloquial ways, wording varies, and the question can be inaccurate. Retrieval models are powerful, and still a lot less powerful than a modern LLM, so that pre-step improved retrieval and then the final answer.
GPT-4 or Claude Opus was unnecessary for that canonicalization. GPT-4.1 was the cost-performance choice, and monthly spend on that one summarization task became large. The swap lined up with hiring for post-training, with causality running both ways, the gap between open-weight and closed-weight models around the time of DeepSeek, and the product maturing. Qwen 3 back-tested well on the simpler ones. Before any training, they run a production test of a well-prompted but untrained model, to see whether it is even close. Only then did they post-train the 14 billion parameter model, pick up finer control, and cut almost all of that inference spend. If the untrained open model is far from the frontier model on your task, this is the wrong cut.
I don't remember exactly, but it could have been like a quarter million dollars a month on just that summarization task.
Fergal Reid, Chief AI Officer at Intercom, on Chain of Thought ep 49
How Intercom Cut $250K/Month b… - Chain of Thought | AI Agents, Infrastructure & Engineering - Apple Podcasts
Podcast Episode · Chain of Thought | AI Agents, Infrastructure & Engineering · February 26 · 54m
podcasts.apple.com
saved almost all of those, you know, several hundred thousand a month on inference just by doing that one thing.
Fergal Reid, Chief AI Officer at Intercom, on Chain of Thought ep 49
Where should you look if resolution rate is the goal?
Look at the systems around the core model. Reid called Fin a cluster of 10 or 15 different machine learning problems. Resolution rate was about 30 percent at launch and is heading toward 70. The vast majority of that improvement was not in the core frontier model. Surrounding work did the lifting: the retrieval model, the rank model, a custom re-ranker built almost from scratch on ModernBERT, with their own cost function, and canonicalization in front of retrieval. A percentage point on one part, then another, adds up.
The retrieval model trains heavily on direct user feedback. When a user confirms an article resolved the question, the retriever learns that this sort of article fits that sort of question. Training only the answer model leaves the surrounding stack, which he said produced the vast majority of the gain, untouched. A small-model swap is a cost and control move on a task you already understand. Keep that separate from the resolution climb, which came from the surrounding stack.
Why can offline evals pick the wrong winner?
They miss production confounders, and latency is the one Reid called out. For a long time the team believed lower latency was a better end-user experience. They still believe that. Measurement on the metric they optimize went the other way.
But we now know that increased latency almost always leads to more resolutions.
Fergal Reid, Chief AI Officer at Intercom, on Chain of Thought ep 49
Increased latency tends to lead to more hard resolutions, which cuts against a pure deflection story. A backtest that scores answer quality on a fixed set will not show users behaving differently because the reply took longer. Nothing beats testing in production. Every backtest they have built can deviate from the production trend. They care about a tenth of a percentage point of resolution, so the experiment has to be highly statistically powered. Fin resolves more than a million conversations successfully a week, and he thought possibly a good bit over that. The switch from GPT to Claude Sonnet was a multi-million end-user interaction A/B test.
Two labels keep the test from rewarding a bot that merely ends the chat. A soft resolution means Fin answered, treats the answer as correct, offered a human, and the user left without saying anything. A hard resolution means the user confirms it, with something like thanks or a clear yes. Hard resolutions tend to run at about 30 to 40 percent of the soft-resolution rate. Hard resolution is the North Star, the ground truth. Ship the candidate when soft resolutions rise and hard resolutions do not fall. The ratio of soft to hard resolutions should be roughly constant for every model change. If soft climbs and hard slips, you are building a deflection machine.
How do you tally hard and soft resolutions per arm?
Assign each conversation to one arm, run that arm's canonicalizer, then count one final outcome per conversation as soft or hard and compare hard resolutions per soft resolution. Keep the split stable for a conversation so retries do not cross models.
A minimal sketch to adapt, not a drop-in library.
import hashlib
import math
EXPERIMENT_ID = "canonicalizer"
def assign_arm(conversation_id: str) -> str:
key = f"{EXPERIMENT_ID}\n{conversation_id}".encode("utf-8")
bucket = int.from_bytes(hashlib.sha256(key).digest()[:8], "big") % 100
return "small_ft" if bucket < 50 else "frontier"
def _rate_se(count: int, n: int) -> tuple[float | None, float | None]:
if not n:
return None, None
rate = count / n
return rate, math.sqrt(rate * (1 - rate) / n)
def tally(outcomes: list[dict]) -> dict:
# One finalized outcome per conversation; a later record replaces an earlier one.
final = {o["conversation_id"]: o for o in outcomes}
arms: dict[str, dict] = {}
for o in final.values():
arm = assign_arm(o["conversation_id"])
row = arms.setdefault(arm, {"eligible": 0, "soft": 0, "hard": 0})
row["eligible"] += 1
if o["hard_confirm"]:
row["hard"] += 1
elif (o["answered"] and o["bot_judged_correct"] and o["offered_human"]
and o["user_left_silently"] and not o["escalated"]):
row["soft"] += 1
for row in arms.values():
n = row["eligible"]
row["soft_rate"], row["soft_rate_se"] = _rate_se(row["soft"], n)
row["hard_rate"], row["hard_rate_se"] = _rate_se(row["hard"], n)
soft = row["soft"]
row["hard_per_soft"] = (row["hard"] / soft) if soft else None
return arms
Read hard_per_soft beside the raw rates. A cheaper arm that lifts soft resolutions and drops that ratio has not cleared the bar Reid described. Power the test for a tenth of a percentage point if that is the difference you would act on. A small offline win on a summarization rubric is not that test.
FAQ
How big was the canonicalization bill?
Reid estimated it could have been like a quarter million dollars a month on that summarization task. Replacing GPT-4.1 with the fine-tuned model saved almost all of that inference spend.
Did the small model replace Fin's main answer model?
No. The hardest prompt still runs Claude Sonnet 4, with a move to a newer Sonnet under way. The Qwen fine-tune was for easier surrounding tasks, canonicalization among them.
Can you train on hard resolutions live?
Not in their setup. Reinforcement learning stays offline, because the label needs a human and they do not run open-loop RL in production. A resolution model proxies the signal, mixed with checks such as answer length and bullet-point count, and other open-weight models as judges. The release candidate then goes to a production A/B test. For customer service the signal can change month to month, not day to day, so offline LLM training is enough. The retrieval model is what learns straight from in-production confirmation that a question was resolved.
Takeaway
- Pick one ancillary, high-volume task with a stable shape, such as canonicalizing a query before retrieval.
- Keep the hardest prompt on a frontier model.
- Run the open model prompted but untrained in production before you spend on a fine-tune.
- Score soft and hard resolutions separately. Hold the soft-to-hard ratio roughly constant.
- A/B test online. Offline scores miss confounders such as latency, and a tenth of a percentage point can be the difference you care about.
The full conversation, with the transcript, is on Chain of Thought.
Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.
Drafted with AI assistance from the episode transcripts.
Top comments (0)