A worksheet to compare API bills and self-hosted open models

Your CFO will ask whether switching to an open weights model beats the API bill. The honest answer is arithmetic on one task at a time, plus the people and GPUs to post-train and run it. Intercom’s Chief AI Officer, Fergal Reid, puts a real number on what that looks like when the task is narrow enough: roughly $250,000 a month on query canonicalization (summarization before retrieval), later replaced for almost all of that inference spend by a fine-tuned 14-billion-parameter Qwen model. The hardest user-facing prompt in the same product still runs on a frontier model. That split is the pattern worth copying. A wholesale stack swap is not.

Which line items belong on the worksheet?

Split the problem into API spend you can measure today and self-host spend you have to estimate.

Hosted API (monthly):

• requests_per_month × tokens_in_per_request / 1,000,000 × price_per_million_input
• The same calculation for output tokens
• Add caching, batching, and retries if they move your meter

Self-hosted inference (monthly):

• Peak and average tokens_per_second you need at p95 latency
• GPU count from vendor math or your own load test (placeholder: gpu_hours_per_month × $/gpu_hour)
• Remember that for some architectures, active parameters per token differ from total parameters in memory. A mixture of experts model can use fewer active parameters per step while still needing enough GPU memory to hold the full set of experts, which affects hardware choice and whether you quantize.

Post-training and operations (monthly):

• Training runs: fine-tunes, evals, failed experiments (placeholder: training_gpu_hours × $/gpu_hour)
• Engineering and applied science headcount allocated to the pipeline (salary and recruiting as well as cloud)

Reid is explicit that if you count only inference, you get the decision wrong. Intercom’s AI group grew from about 10 people to 55 in roughly two and a half years, with plans to double again, while their standard training setup is described as one node of H200s, with the approach moving from LoRA-style methods toward distributed full fine-tuning and reinforcement learning for harder problems. Cheaper tokens come with a headcount bill.

When does the API side lose on paper?

Use this decision rule before you fine-tune: the task should be high volume, repetitive, and well specified, with a clear input-output contract. Query canonicalization fit that profile. Reid dates wider viability of open-weight models for core ancillary work to around DeepSeek’s release, when the gap to closed models narrowed enough to try; Qwen 3 models back-tested well on Intercom’s workloads, with the large pre-training cost borne by whoever published the weights. In October 2026 Reid wrote on LinkedIn that the post-training pipeline is the important piece, not any one base model, and that Intercom has since switched to Nvidia and Google base models. Your worksheet should include a row for “base model refresh” so you are not married to one checkpoint.

On the API side, fill in Reid’s order-of-magnitude anchor only if your task is comparable: ~$250K/month for one summarization job at their scale. Your placeholders should dominate otherwise.

What do you add that the API quote hides?

Self-hosting buys control and unit economics on that one job. It does not win every prompt.

And we were able to go and take that model and train that model, post train that model to do that summarization task. and able to replace that and saved almost all of those, you know, several hundred thousand a month on inference just by doing that one thing.

Fergal Reid, Intercom Chief AI Officer

You still need evaluation discipline Reid describes: internal back tests (including real customer hallucination reports), plus A/B tests in production on resolution metrics, not leaderboard scores alone.

it's so easy for a researcher or a scientist trying to optimize a benchmark to get good at teaching the model how to do the kind of task that's in the benchmark and then suffer generalization failure.

Fergal Reid, Intercom Chief AI Officer

Budget time for scientists who can run randomized experiments, not only engineers who wire up endpoints. Reid’s team hires applied scientists with ML backgrounds and treats production A/B tests as the gold standard, with enough traffic to detect small moves in resolution rate.

For a quick sanity check on whether your traffic profile is even worth modeling, here is a sketch (placeholders only, not Intercom’s stack):

# Sketch: monthly cost comparison for ONE task
api_cost = (tokens_in + tokens_out) / 1e6 * blended_price_per_million
self_host_infra = gpu_hours * cost_per_gpu_hour
post_train = training_gpu_hours * cost_per_gpu_hour
people = fte_allocated * loaded_monthly_cost_per_fte

total_api = api_cost
total_self = self_host_infra + post_train + people

# Switch only if total_self < total_api AND quality gates pass
# (backtests + production A/B; not public benchmarks alone)

Where should you read next?

The open source AI topic on Chain of Thought collects many episodes on weights, ecosystems, and production tradeoffs if you want more than one company’s story. For a curated path through those conversations, the best episodes on open source AI collection is a sensible next stop after you have your own worksheet filled in.

Do not let “open source is cheaper” or “frontier is better” substitute for profiling your traffic. Teams that save real money find one or two narrow, high-volume tasks, move those, and leave everything else where it already works.

Checklist

• Pick one task with measured monthly API tokens and cost, not a platform-wide mandate.
• Model inference, training GPU time, and allocated team cost on the same time horizon (monthly or annual).
• Confirm the open-weight base model passes your back tests; treat public benchmarks as hints, not proof.
• Plan for base model churn; invest in a post-training pipeline that survives checkpoint swaps.
• Ship only after production A/B tests on metrics you actually care about, alongside the cheaper bill.

The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.