DEV Community

Cover image for Small Models, Big Results: The Case for Dumb AI
VelocityAI
VelocityAI

Posted on

Small Models, Big Results: The Case for Dumb AI

A 7B model fine-tuned on 50,000 customer support tickets beat GPT-4 on the company’s own resolution-quality benchmark by 8 percentage points. It ran on a single A10G GPU, cost $0.12 per thousand resolutions, and responded in 340ms. GPT-4 cost $18 per thousand resolutions and took 1,800ms. The company switched. Six months later, customer satisfaction was up 11%, cost per resolution was down 94%, and the team had cut its AI infrastructure spend from $40K a month to $2.3K.

That story is not unusual. It’s the norm in every domain where someone has bothered to measure. And yet the industry keeps pouring billions into bigger generalists, keeps publishing benchmarks that favor them, and keeps telling itself that scale is the only path. The 7B model that beat GPT-4 is invisible in the discourse because the discourse is not measuring what the 7B model actually does well.

Here’s the claim I want to test: the obsession with scale is a misallocation of capital so large that it is reshaping the industry’s competitive structure in ways that will be obvious in five years and invisible now. The returns to scale are real but concentrated in a narrow band of capabilities. The returns to specialization are real and concentrated in the capabilities that most products actually need. The industry has been optimizing for the wrong band.

The Economics Nobody Publishes
Let me start with the numbers, because the numbers are the argument.

A frontier model training run costs $50M to $200M. Inference on that model costs $5 to $30 per million tokens. A fine-tuned 7B model costs $500 to $5,000 to train (depending on the base model and the data) and $0.10 to $0.50 per million tokens on commodity hardware. The ratio is between 10x and 100x on inference cost, and between 10,000x and 100,000x on training cost.

That ratio is not a small inefficiency. It’s the entire economic structure of the industry. Every product that uses a frontier model at scale is paying a 10x to 100x premium for capability it may not need. Every product that could be built on a small model is not being built, because the default is to reach for the biggest model available.

The industry’s default is expensive, and the default is invisible because it’s the default. The cost is not visible in the model card. It’s visible in the invoices, and the invoices are not published.

The Sacred Cow: “Bigger Models Are More Capable”
Here’s the belief I want to dismantle, and it’s the assumption that underlies every product decision in the industry.

The claim goes like this: bigger models are more capable. They score higher on benchmarks, they handle more edge cases, they generalize better. The capability difference justifies the cost difference. The teams that use the biggest model are the teams that are serious about quality.

This is true in aggregate. It is not true at the task level, and the task level is where products live.

A frontier model is more capable on average. But “average” is a statistical artifact. It’s the mean of a distribution over tasks, and the distribution is not uniform. The frontier model’s advantage is concentrated in tasks that require broad knowledge, open-ended reasoning, and long-context understanding. On narrow, well-scoped tasks with domain-specific data, the advantage shrinks to zero or reverses.

The evidence is not new. It’s been accumulating for years:

DistilBERT (66M) retained 97% of BERT-base’s performance on GLUE at 40% of the size. The 3% gap was concentrated in the hardest tasks, not the common ones.

Phi-3 (3.8B) matched or beat models 10x its size on reasoning benchmarks, using a data curriculum that emphasized quality over quantity.

Gemma 2 (2B and 9B) outperformed several 13B+ models on instruction-following evals, driven by architecture improvements and distillation.

Mistral 7B matched Llama 2 13B on most benchmarks at launch, and the gap has narrowed with better fine-tuning.

A 2023 study on customer support found that a fine-tuned 7B model outperformed GPT-4 on domain-specific resolution quality by 8 percentage points.

The pattern: on narrow tasks with domain data, small models win. On broad tasks with no domain data, large models win. The industry’s products are mostly narrow tasks with domain data. The industry’s models are mostly broad tasks with no domain data. The mismatch is the misallocation.

The contrarian take: “bigger models are more capable” is true on the benchmark, not on the product. The benchmark measures the average over a distribution of tasks that nobody’s product actually uses. The product uses a specific task, and the specific task is where the small model wins.

Why Small Models Win on Narrow Tasks
Let me be concrete about the mechanism, because “small models are better” is too vague to be useful.

  1. The task is narrow, so the model can be narrow.

A customer support model doesn’t need to know about Moby-Dick or the history of the Ottoman Empire. It needs to know the company’s product, the company’s policies, and the company’s tone. A 7B model fine-tuned on that data has all the knowledge it needs. The frontier model has that knowledge plus a million things it doesn’t need, and the extra knowledge is a liability, because it creates surface area for hallucination and off-tone responses.

  1. The data is domain-specific, so the fine-tuning is the advantage.

A fine-tuned model learns the specific patterns of the domain: the vocabulary, the edge cases, the tone. The frontier model has to infer these patterns from the prompt, and the inference is imperfect. The fine-tuned model has them baked in, and the baking is the advantage.

  1. The latency matters, so the small model wins on UX.

A 7B model responds in 340ms. A frontier model responds in 1,800ms. The difference is perceptible to users. The small model feels instant. The frontier model feels slow. The UX difference is real, and it’s the kind of difference that shows up in retention metrics.

  1. The cost matters, so the small model wins on unit economics.

A 7B model costs $0.12 per thousand resolutions. A frontier model costs $18. The difference is 150x. The difference is the difference between a product that can be priced for the mass market and a product that can only be priced for enterprises. The small model is the difference between a viable business and a niche one.

  1. The failure modes are legible, so the small model wins on trust.

A 7B model fails in patterned ways. It fails on specific tasks, specific inputs, specific edge cases. The failures are fixable. A frontier model fails in diffuse ways, which makes the failures harder to debug and harder to trust. The legibility is the advantage.

Put together, these advantages are structural, not incidental. They don’t depend on the tool getting better. They depend on the task being narrow, the data being domain-specific, the latency mattering, the cost mattering, and the trust mattering. These are the conditions of most products, and the small model wins under these conditions.

What “Dumb AI” Actually Looks Like in Practice

I want to be concrete about what “small models, big results” looks like in practice, because the abstract version is too easy to dismiss.

Case 1: Customer support.

A fine-tuned 7B model with retrieval over the company’s knowledge base. Cost: $2.3K/month. Latency: 340ms. Resolution quality: 8 points higher than GPT-4. The team that built it spent three weeks on data curation and fine-tuning. The result was a 94% cost reduction and an 11% satisfaction increase.

Case 2: Code completion.

A fine-tuned 3B model on the company’s internal codebase. Cost: $400/month. Latency: 90ms. Acceptance rate: 42%, up from 28% with the general-purpose model. The team that built it spent two weeks on data curation and fine-tuning. The result was a 50% increase in acceptance rate and a 3x reduction in cost.

Case 3: Document classification.

A fine-tuned 1B model on the company’s document taxonomy. Cost: $80/month. Latency: 40ms. F1: 0.94, up from 0.81 with the general-purpose model. The team that built it spent one week on data curation and fine-tuning. The result was a 16-point F1 increase and a 20x reduction in cost.

Case 4: Medical coding.

A fine-tuned 7B model on the company’s coding guidelines. Cost: $1.2K/month. Latency: 280ms. Accuracy: 94%, up from 87% with the general-purpose model. The team that built it spent four weeks on data curation and fine-tuning. The result was a 7-point accuracy increase and a 15x reduction in cost.

The pattern: in each case, a small fine-tuned model outperformed a large general-purpose model on the specific task, at a fraction of the cost. The fine-tuning was the advantage. The data curation was the advantage. The specialization was the advantage.

Why the Industry Doesn’t Do This

I want to be direct about the incentives, because they explain why the misallocation persists.

  1. The benchmark rewards scale.

The benchmarks that the industry uses to evaluate models are dominated by broad, open-ended tasks. The models that score highest on these benchmarks are the largest models. The benchmark is the signal, and the signal rewards scale. The signal does not reward specialization, because specialization doesn’t show up on the benchmark.

  1. The fine-tuning is hard.

Fine-tuning a small model on domain data is not hard in principle, but it’s hard in practice. It requires data curation, evaluation design, and iteration.A 7B model fine-tuned on 50,000 customer support tickets beat GPT-4 on the company’s own resolution-quality benchmark by 8 percentage points. It ran on a single A10G GPU, cost $0.12 per thousand resolutions, and responded in 340ms. GPT-4 cost $18 per thousand resolutions and took 1,800ms. The company switched. Six months later, customer satisfaction was up 11%, cost per resolution was down 94%, and the team had cut its AI infrastructure spend from $40K a month to $2.3K.

That story is not unusual. It’s the norm in every domain where someone has bothered to measure. And yet the industry keeps pouring billions into bigger generalists, keeps publishing benchmarks that favor them, and keeps telling itself that scale is the only path. The 7B model that beat GPT-4 is invisible in the discourse because the discourse is not measuring what the 7B model actually does well.

Here’s the claim I want to test: the obsession with scale is a misallocation of capital so large that it is reshaping the industry’s competitive structure in ways that will be obvious in five years and invisible now. The returns to scale are real but concentrated in a narrow band of capabilities. The returns to specialization are real and concentrated in the capabilities that most products actually need. The industry has been optimizing for the wrong band.

The Economics Nobody Publishes
Let me start with the numbers, because the numbers are the argument.

A frontier model training run costs $50M to $200M. Inference on that model costs $5 to $30 per million tokens. A fine-tuned 7B model costs $500 to $5,000 to train (depending on the base model and the data) and $0.10 to $0.50 per million tokens on commodity hardware. The ratio is between 10x and 100x on inference cost, and between 10,000x and 100,000x on training cost.

That ratio is not a small inefficiency. It’s the entire economic structure of the industry. Every product that uses a frontier model at scale is paying a 10x to 100x premium for capability it may not need. Every product that could be built on a small model is not being built, because the default is to reach for the biggest model available.

The industry’s default is expensive, and the default is invisible because it’s the default. The cost is not visible in the model card. It’s visible in the invoices, and the invoices are not published.

The Sacred Cow: “Bigger Models Are More Capable”
Here’s the belief I want to dismantle, and it’s the assumption that underlies every product decision in the industry.

The claim goes like this: bigger models are more capable. They score higher on benchmarks, they handle more edge cases, they generalize better. The capability difference justifies the cost difference. The teams that use the biggest model are the teams that are serious about quality.

This is true in aggregate. It is not true at the task level, and the task level is where products live.

A frontier model is more capable on average. But “average” is a statistical artifact. It’s the mean of a distribution over tasks, and the distribution is not uniform. The frontier model’s advantage is concentrated in tasks that require broad knowledge, open-ended reasoning, and long-context understanding. On narrow, well-scoped tasks with domain-specific data, the advantage shrinks to zero or reverses.

The evidence is not new. It’s been accumulating for years:

DistilBERT (66M) retained 97% of BERT-base’s performance on GLUE at 40% of the size. The 3% gap was concentrated in the hardest tasks, not the common ones.

Phi-3 (3.8B) matched or beat models 10x its size on reasoning benchmarks, using a data curriculum that emphasized quality over quantity.

Gemma 2 (2B and 9B) outperformed several 13B+ models on instruction-following evals, driven by architecture improvements and distillation.

Mistral 7B matched Llama 2 13B on most benchmarks at launch, and the gap has narrowed with better fine-tuning.

A 2023 study on customer support found that a fine-tuned 7B model outperformed GPT-4 on domain-specific resolution quality by 8 percentage points.

The pattern: on narrow tasks with domain data, small models win. On broad tasks with no domain data, large models win. The industry’s products are mostly narrow tasks with domain data. The industry’s models are mostly broad tasks with no domain data. The mismatch is the misallocation.

The contrarian take: “bigger models are more capable” is true on the benchmark, not on the product. The benchmark measures the average over a distribution of tasks that nobody’s product actually uses. The product uses a specific task, and the specific task is where the small model wins.

Why Small Models Win on Narrow Tasks
Let me be concrete about the mechanism, because “small models are better” is too vague to be useful.

  1. The task is narrow, so the model can be narrow.

A customer support model doesn’t need to know about Moby-Dick or the history of the Ottoman Empire. It needs to know the company’s product, the company’s policies, and the company’s tone. A 7B model fine-tuned on that data has all the knowledge it needs. The frontier model has that knowledge plus a million things it doesn’t need, and the extra knowledge is a liability, because it creates surface area for hallucination and off-tone responses.

  1. The data is domain-specific, so the fine-tuning is the advantage.

A fine-tuned model learns the specific patterns of the domain: the vocabulary, the edge cases, the tone. The frontier model has to infer these The work is unglamorous, and it’s not the kind of work that gets published or funded. The frontier model is a one-line API call. The fine-tuned model is a project. The project is the hard part, and the hard part is the thing that the industry avoids.

  1. The frontier model is the safe choice.

If you use the frontier model and it fails, nobody blames you. You used the best model available. If you use a small model and it fails, you get blamed. You chose the smaller model. The safe choice is the frontier model, even when the safe choice is more expensive and less effective.

  1. The frontier model is the fundable choice.

Investors fund frontier models. The narrative is compelling. The scale is impressive. The small model is not a narrative. It’s a product. The product is not fundable in the same way, because the product doesn’t have the story.

  1. The frontier model is the default.

The default is the thing you do without thinking. The default in the industry is to reach for the biggest model available. The default is invisible, and the invisibility is the problem. The teams that question the default are the teams that find the small model wins.

The result is a market that is systematically mispricing capability. The frontier model is priced at a premium that the small model doesn’t need. The premium is paid by the products that use the frontier model, and the products that use the small model are the ones that are growing.

A 7B model fine-tuned on 50,000 customer support tickets beat GPT-4 on the company’s own resolution-quality benchmark by 8 percentage points. It ran on a single A10G GPU, cost $0.12 per thousand resolutions, and responded in 340ms. GPT-4 cost $18 per thousand resolutions and took 1,800ms. The company switched. Six months later, customer satisfaction was up 11%, cost per resolution was down 94%, and the team had cut its AI infrastructure spend from $40K a month to $2.3K.

That story is not unusual. It’s the norm in every domain where someone has bothered to measure. And yet the industry keeps pouring billions into bigger generalists, keeps publishing benchmarks that favor them, and keeps telling itself that scale is the only path. The 7B model that beat GPT-4 is invisible in the discourse because the discourse is not measuring what the 7B model actually does well.

Here’s the claim I want to test: the obsession with scale is a misallocation of capital so large that it is reshaping the industry’s competitive structure in ways that will be obvious in five years and invisible now. The returns to scale are real but concentrated in a narrow band of capabilities. The returns to specialization are real and concentrated in the capabilities that most products actually need. The industry has been optimizing for the wrong band.

The Economics Nobody Publishes
Let me start with the numbers, because the numbers are the argument.

A frontier model training run costs $50M to $200M. Inference on that model costs $5 to $30 per million tokens. A fine-tuned 7B model costs $500 to $5,000 to train (depending on the base model and the data) and $0.10 to $0.50 per million tokens on commodity hardware. The ratio is between 10x and 100x on inference cost, and between 10,000x and 100,000x on training cost.

That ratio is not a small inefficiency. It’s the entire economic structure of the industry. Every product that uses a frontier model at scale is paying a 10x to 100x premium for capability it may not need. Every product that could be built on a small model is not being built, because the default is to reach for the biggest model available.

The industry’s default is expensive, and the default is invisible because it’s the default. The cost is not visible in the model card. It’s visible in the invoices, and the invoices are not published.

The Sacred Cow: “Bigger Models Are More Capable”
Here’s the belief I want to dismantle, and it’s the assumption that underlies every product decision in the industry.

The claim goes like this: bigger models are more capable. They score higher on benchmarks, they handle more edge cases, they generalize better. The capability difference justifies the cost difference. The teams that use the biggest model are the teams that are serious about quality.

This is true in aggregate. It is not true at the task level, and the task level is where products live.

A frontier model is more capable on average. But “average” is a statistical artifact. It’s the mean of a distribution over tasks, and the distribution is not uniform. The frontier model’s advantage is concentrated in tasks that require broad knowledge, open-ended reasoning, and long-context understanding. On narrow, well-scoped tasks with domain-specific data, the advantage shrinks to zero or reverses.

The evidence is not new. It’s been accumulating for years:

DistilBERT (66M) retained 97% of BERT-base’s performance on GLUE at 40% of the size. The 3% gap was concentrated in the hardest tasks, not the common ones.

Phi-3 (3.8B) matched or beat models 10x its size on reasoning benchmarks, using a data curriculum that emphasized quality over quantity.

Gemma 2 (2B and 9B) outperformed several 13B+ models on instruction-following evals, driven by architecture improvements and distillation.

Mistral 7B matched Llama 2 13B on most benchmarks at launch, and the gap has narrowed with better fine-tuning.

A 2023 study on customer support found that a fine-tuned 7B model outperformed GPT-4 on domain-specific resolution quality by 8 percentage points.

The pattern: on narrow tasks with domain data, small models win. On broad tasks with no domain data, large models win. The industry’s products are mostly narrow tasks with domain data. The industry’s models are mostly broad tasks with no domain data. The mismatch is the misallocation.

The contrarian take: “bigger models are more capable” is true on the benchmark, not on the product. The benchmark measures the average over a distribution of tasks that nobody’s product actually uses. The product uses a specific task, and the specific task is where the small model wins.

Why Small Models Win on Narrow Tasks
Let me be concrete about the mechanism, because “small models are better” is too vague to be useful.

  1. The task is narrow, so the model can be narrow.

A customer support model doesn’t need to know about Moby-Dick or the history of the Ottoman Empire. It needs to know the company’s product, the company’s policies, and the company’s tone. A 7B model fine-tuned on that data has all the knowledge it needs. The frontier model has that knowledge plus a million things it doesn’t need, and the extra knowledge is a liability, because it creates surface area for hallucination and off-tone responses.

  1. The data is domain-specific, so the fine-tuning is the advantage.

A fine-tuned model learns the specific patterns of the domain: the vocabulary, the edge cases, the tone. The frontier model has to infer theseWhat To Actually Do

Three moves, concrete enough to start on Monday.

  1. Test the small model on your actual task before you scale up.

If your team’s default is to reach for the biggest model available, run a controlled comparison. Take a 7B model with good fine-tuning and compare it to the frontier model on your real workload. Measure quality, latency, and cost. You will often be surprised. If you’re not surprised, you now have evidence for the frontier model.

A rough starting point:def compare_models(task, small_model, large_model, test_set):
results = {"small": [], "large": []}
for example in test_set:
small_output = small_model.generate(example.prompt)
large_output = large_model.generate(example.prompt)
results["small"].append({
"output": small_output,
"quality": measure_quality(small_output, example.expected),
"latency": measure_latency(small_model, example.prompt),
"cost": measure_cost(small_model, example.prompt),
})
results["large"].append({
"output": large_output,
"quality": measure_quality(large_output, example.expected),
"latency": measure_latency(large_model, example.prompt),
"cost": measure_cost(large_model, example.prompt),
})
return results

The point is not the specific metrics. The point is the practice. Measure the specific task, not the benchmark. The specific task is where the small model wins.

  1. Invest in data curation.

The small model’s advantage is the fine-tuning, and the fine-tuning is the data. The teams that invest in data curation will have a structural advantage. The teams that don’t will keep using the frontier model and paying the premium.

  1. Question the default.

The next time your team reaches for the biggest model, ask why. The answer may be “because it’s the best.” The better answer is “because we measured and it’s the best for our task.” If the team can’t answer with the measurement, the team doesn’t know. The measurement is the thing that separates the default from the decision.

The Mental Model

Here’s the lens I want you to keep.

Model quality is not a function of scale. It’s a function of effective capability per dollar, per millisecond, per task. The frontier model has the highest effective capability on the benchmark. The fine-tuned small model has the highest effective capability on the task. The task is where the product lives. The benchmark is where the discourse lives.

The interesting question is not whether big models are better. They are, on the benchmark. The interesting question is whether the industry will figure out that the benchmark is the wrong target, and that the task is the right target. The teams that figure this out will ship better products, cheaper, faster, and be structurally advantaged as costs rise. The teams that don’t will keep paying the premium and wondering why the margins are thin.

The 7B model that beat GPT-4 is not a curiosity. It’s a preview of where the industry is going, whether the industry knows it or not.

What’s your read? If you’ve compared a fine-tuned small model to a frontier model on a real task, what did you find? And if you’re building products, what’s the default model you reach for, and have you questioned it? Drop it in the comments. I’m genuinely curious whether the small-model wins are as consistent in practice as the economics suggest.

Top comments (1)

Collapse
 
useaifix profile image
AIFix •

A useful pilot is to compare models on a held-out set of real tickets and track resolution quality, escalation rate, latency, and cost together. That keeps the cost win tied to the cases users actually care about.