DEV Community: Tyler Edwards

Think smaller: why specialist SLMs beat frontier models in production

Tyler Edwards — Tue, 22 Sep 2026 11:46:42 +0000

Most agent tasks do not need a frontier model. The subtasks inside a production agent are largely repetitive and narrowly scoped, and a small language model of 1B to 8B parameters, fine-tuned on your production data, beats a frontier model on cost, latency and accuracy for that job. Serving a 7B runs roughly 10 to 30x cheaper than a 70B+ model.

Originally published at overmindlab.ai.

Frontier models are overkill for most agent tasks. Specialised small language models beat them on cost, latency, and accuracy.

The AI industry has spent three years in an arms race measured in billions of parameters, and somewhere along the way everyone agreed that the answer to every problem was a bigger model.

Bigger clusters, bigger bills, bigger teams to manage the complexity. The implicit assumption is that general intelligence at scale is the end state and everything else is just waiting to catch up.

Which is obviously great for the foundational teams pushing this agenda, were it not for the fact that the overwhelming majority of problems we throw AI at don't need otherworldly intelligence and infinite compute to solve.

Llama, Phi, Qwen, Mistral: models that would have been considered frontier-grade two years ago are now freely available, fine-tuneable, and deployable on hardware you control. The performance gap that once justified the cost and lock-in of proprietary giants is now much slimmer, so the question is no longer "which frontier model should we use?" but rather "do we need a frontier model at all?"

For most tasks, the honest answer is no.

Why is a frontier model overkill for most agent tasks?

Using a frontier model on every task is like using a flamethrower to light a candle. Technically works, but your eyebrows and your budget are gone.

The majority of subtasks in a deployed agentic system are repetitive, scoped, and non-conversational. Extract this field, classify this ticket, format this output, none of these require a model trained on the sum of human knowledge. They require a model that is fast, predictable, cheap to run, and tuned to behave exactly as expected in a well-defined context.

In fact, on well-defined, repetitive tasks, specialised SLMs do not just cost a fraction of what frontier models cost, they outperform them. A model built specifically for one thing, trained on real examples of that thing, is better at that thing than a model trained to do everything.

Which small models have beaten frontier models on benchmarks?

And this holds even as the tasks get harder. Qwen2.5-Math-7B, running Microsoft's rStar-Math reasoning framework, scores 90% on the MATH benchmark, beating OpenAI's o1-preview. John Snow Labs' MedS, an 8B medical model, was preferred over GPT-4o by practising physicians in blind evaluation across every dimension: factuality, clinical relevance, and conciseness. LlaSMol, a 7B chemistry model, achieves 93% exact match on molecular prediction tasks where GPT-4 scores under 5%. On Berkeley's function-calling leaderboard, an 8B model held the top spot at the time of writing, beating GPT-4o and Claude 3.5 Sonnet.

Specialist model	Params	Benchmark	Score	Model it beats	Source
Qwen2.5-Math-7B with rStar-Math	7B	MATH	90%	OpenAI o1-preview	rStar-Math, Microsoft
John Snow Labs MedS	8B	Blind physician review on factuality, clinical relevance and conciseness	Preferred over GPT-4o on every dimension	GPT-4o	John Snow Labs expert review
LlaSMol	7B	Molecular prediction, exact match	93%	GPT-4, under 5%	LlaSMol, OSU NLP
Top open function-calling specialist	8B	Berkeley function-calling leaderboard	Top of the leaderboard at the time of writing	GPT-4o and Claude 3.5 Sonnet	Berkeley function-calling leaderboard

What does open-weights actually unlock for production AI?

The capability trajectory of small open models has been striking, yet the capability of these models is not the most exciting piece, what open weights actually unlock is control.

A frontier model you do not own, cannot audit, and cannot modify is a dependency, not infrastructure and certainly not your 'technical moat'. When it changes, your system changes. When it halts, your system halts. When the provider decides to deprecate a version or reprice the API, you absorb it. The case for specialised small models is not just that they are better. It is that they are yours.

The physical dimension matters too. Frontier models run in data centres, behind APIs, with latency measured in hundreds of milliseconds and costs that compound with every token. In an agentic system where a single workflow might invoke a model thirty or forty times, that compounds fast. Serving a 7B SLM is roughly 10 to 30x cheaper in latency, energy, and compute than serving a 70 to 175B LLM. Small models can run on-premise, inside the security perimeter, on the device generating the data. A fraud detection model running locally, fine-tuned on that institution's own transaction patterns, makes faster decisions with no external dependency and no data crossing a network boundary. The performance is better, the risk profile and compliance conversation are entirely different.

Even more exciting still (if you can believe it) is that a model small enough to run on a phone can be fine-tuned not just to a specific workflow but to a specific person. An agent that lives on your device and learns from how you actually work is a categorically different thing to one calling to a data centre every time you ask it something. We are moving from general intelligence in the cloud to personal intelligence in your pocket. That shift is bigger than most people are currently treating it.

Which fits your job, a frontier API or a specialist SLM?

	Frontier model behind an API	Specialist SLM you own
Best at	Open-ended reasoning and conversation	Repetitive, scoped, non-conversational subtasks
Serving cost	Compounds with every token, and one workflow may invoke the model thirty or forty times	Roughly 10 to 30x cheaper at 7B than at 70 to 175B
Latency	Hundreds of milliseconds, over the network	No network hop, runs where the data is
Where it runs	A data centre, behind an API	On-premise, inside the security perimeter, or on the device
Control	Version deprecations and repricing land on you	Yours to audit, modify and version
Failure surface	Vast and constantly shifting, answered with hand-written guardrails	Narrow scope, tighter output distribution, predictable failures
Setup effort	Write a prompt, get an answer, iterate, ship	Instrumentation, curation, fine-tuning jobs, evals, versioning, a deployment you can swap

Production is the best training data you have

Every invocation of a model in an agentic workflow is a natural source of high-quality training data. The prompts are well-defined, the expected outputs are narrow, and whether the workflow succeeded or failed is a clean signal. A listener at the model call interface, logging inputs, outputs, and downstream outcomes, accumulates exactly the dataset you need to continuously fine-tune a specialist model for that task, as the system runs. The production environment is generating its own improvement data in real time.

This is where the SLM case becomes compelling beyond economics. You instrument the calls, cluster the patterns, fine-tune the specialists, and the system gets better with every run. What determines whether you actually close that loop is whether you have the infrastructure to capture it, label it, and act on it.

That loop is what we build. Overmind turns your production traces into specialised models you own, automatically trained, benchmarked, and served.

Why are specialist models easier to supervise?

AI security has become a pretty bloated term, so stick with me here. The version of it that actually matters is not about firewalls or prompt injection. It is about knowing what your model is doing, and knowing when it is doing something it should not. Specialised models make that question answerable. When a model has a narrow, well-defined scope, you know what it should be doing, which means you know when it is doing something else. The output distribution is tighter, the failure modes are more predictable, and the feedback loop from production behaviour back to model improvement is shorter and more reliable.

General models are a different problem entirely. The failure surface is vast, constantly shifting, and practically impossible to enumerate in advance. Guardrails are the industry's current answer, which is to say: a list of rules written by humans trying to statically account for an infinite and ever-changing set of ways a frontier model can go wrong.

Are small specialist models harder to train?

Now I would love to just dunk on frontier models for the rest of this post, but there is a real counter-argument worth addressing. Calling an API is easy. You write a prompt, you get an answer, you iterate, you ship. The feedback loop is fast and the overhead is minimal. Building SLM-first infrastructure is a different proposition entirely.

Fine-tuning a specialist model requires training data, which requires instrumentation, which requires someone to own it. You need pipelines to capture production traces, processes to curate and label them, and infrastructure to run fine-tuning jobs on a cadence that actually moves the needle. You need model versioning, evaluation harnesses, and a deployment setup that lets you swap models without taking down the system. None of this is impossibly hard, but none of it is free either.

Why this matters to us

Everything in this post is, admittedly, not a neutral observation. We built Overmind on a specific bet: that the teams who win the next phase of agentic AI will be the ones who own their models, not the ones who rent the most powerful ones.

The infrastructure to do that, fine-tuning pipelines, production observability, continuous improvement loops that do not require a dedicated ML team to run, that is what we are building. Not just because it is an interesting technical problem, but because without it the agentic future we are all so excited about, the one where AI runs diagnostics in hospitals or runs locally on your phone and adapts to you as an individual, simply cannot happen.

So we are building that future.

Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own, automatically trained, benchmarked, and served. If you are deploying agentic AI, get in touch.

FAQ

Are small language models better than frontier models?

For repetitive, scoped tasks they are. A model trained specifically for one job, on real examples of that job, outperforms a generalist on cost, latency and accuracy. Frontier models still win on open-ended reasoning and conversation.

What is a specialist SLM?

A small language model (typically 1B to 8B parameters) fine-tuned on a narrow task using your own production data. Llama, Phi, Qwen and Mistral are common open-weight bases.

How much cheaper is a 7B SLM compared to a 70B+ LLM?

Roughly 10 to 30x cheaper in latency, energy and compute when self-hosted, and the gap widens at high request volumes.

Can I run a specialist SLM on-device?

Yes. 7B and 8B models run on modern phones and laptops, which enables per-user fine-tuning, offline operation, and no data leaving the device.

What do you need in place before you can run specialist models in production?

Four things. Instrumentation that captures production traces, a process to curate and label them, somewhere to run fine-tuning jobs on a regular cadence, and model versioning plus evaluation harnesses so you can swap a model without taking the system down. None of it is impossibly hard, but none of it is free either.

Tyler Edwards is Co-Founder and CEO of Overmind. He writes about agent infrastructure, fine-tuning, and what it takes to ship AI that actually improves in production. Connect on LinkedIn.

Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own. Get started.

Prompt engineering vs fine-tuning: which one do you need?

Tyler Edwards — Tue, 08 Sep 2026 17:01:47 +0000

Every LLM team hits the same fork eventually, either keep steering the prompt or start training the model. Here's how to read the signals before you burn a sprint finding out the hard way.

Originally published at overmindlab.ai.

A perspective on when to tinker with the prompt, and when to train your own model.

Most teams building LLM applications start the same way. They write a system prompt, test it against a few examples, and ship. Prompt engineering is cheap, fast, and reversible. Fine-tuning used to be something only teams with a training pipeline and a GPU budget could afford. Things have changed but the order of operations hasn't.

Both techniques steer model behaviour. Prompting changes what you say to the model. Fine-tuning changes the model itself.

Prompt engineering vs fine-tuning, side by side

	Prompt engineering	Fine-tuning
What changes	The input	The model's weights
Where it lives	The context window	The model
Cost	Seconds, no infrastructure	GPU time and a training pipeline
Data needed	None, or a handful of examples	Thousands of labelled runs
Reversibility	Instant	Only by retraining
Survives a model swap	Yes	No
Ceiling	Plateaus quickly	High, bounded by data quality

Why prompting is the right first move

Prompt engineering means editing what enters the context window. No adjustment of the model's weights. You're changing the input and reading the output.

For a new agent, that's almost always the correct starting point. You learn what the model can already do before you spend anything on training. Most of the behaviour changes you want, prompting will give you.

How to tell which regime you're in

This isn't a one-time decision so much as a loop. You stay in prompt-engineering mode until the eval curve goes flat, then check whether you actually have what fine-tuning needs, a narrow task and real data.

Ship prompt changes until the eval goes flat, then fine-tune.

Where you are	What you're seeing	What to do next
New behaviour needed	A gap between what the agent does and what it should do	Ship a prompt change
Eval still moving	Each change shifts the score	Keep prompting
Eval flat for 3-4 changes, task is narrow, a few thousand labelled runs in hand	Plateau, and you have the ingredients	Fine-tune
Eval flat, but the task is broad or the labelled runs aren't there	Plateau without the ingredients	Build the dataset first

Where prompting hits its ceiling

Prompting always runs out of road eventually, and the symptoms show up in predictable order:

The plateau. Each prompt tweak moves the eval less.
Whack-a-mole. The instruction that fixes today's failure mode causes tomorrow's.
Prompt bloat. The system prompt keeps growing, slows the model down, and gets partially ignored.
Paying rent. You're re-sending the same correction on every single call.

At that point, prompting stops being cheap. Engineering hours are the priciest line item in most LLM pipelines, and it's easy to burn a month nudging a prompt that plateaued weeks ago.

When to fine-tune

Fine-tuning updates a model's weights on examples of the behaviour you want until the model does it by default, without being told every time. It's worth trying when these conditions are met:

The task is narrow. Fine-tuning sharpens a model on one job, not general capability.
You have real data. A few thousand labelled runs, ideally sampled from production traffic rather than synthetic examples.
Cost or latency matters. A smaller, fine-tuned model can outperform a much larger general-purpose one carrying a bloated prompt, at a fraction of the inference cost.

Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own. Get started.

Open-weights LLMs vs frontier APIs: when to rent, when to own

Tyler Edwards — Mon, 07 Sep 2026 14:54:01 +0000

Most AI products start on a frontier API and stay there until the bill, the latency or legal forces a rethink. Here's the practical case for when a self-hosted open-weights model is the better call, and the cost-crossover logic behind it.

Originally published at overmindlab.ai.

Own your intelligence, or rent the API?

Developers start building AI products by tokenmaxxing. Calling frontier APIs and paying per token.

Then the bill shows up. Or the agent feels slow. Or legal asks where the data goes. Suddenly "just use the API" might not be the right strategy.

Underneath every AI product is one choice. Rent a frontier API, or own a model. Renting means closed weights, so someone else trains it, serves it, and sets the price. Owning means picking the model, post-training it and running self-hosted or third-party inference.

This piece is the practical guide to open vs closed models.

The trade-offs

Dimension	Closed models (frontier API)	Open models (open-weights)
Capability	Best capability out of the box	Strong, and tunable to your task
Infrastructure	None needed	Runs on infrastructure you own or buy
Cost	Costs more with every token	Drops hard in cost after the upfront work
Latency	Adds a network trip and a shared queue	Serves on dedicated hardware, with lower time to first token
Data	Sends prompts outside your network	Stays on your own servers or in your own cloud
Lock-in	Locks you to the vendor's pricing and retirement schedule	Leaves you your stack and your weights
Switching	Swap with one line of code	Needs a retrain or a re-quantise to swap

LLM inference cost: when open-weights beat the API

Renting feels cheap at the start, especially when showered with credits.

Then usage grows. The bill grows and a painful chunk of that spend is frontier prices for work a smaller specialised model could do.

Owning the model changes that. You pay upfront for a LoRA / QLoRA fine-tune, inference setup and some evals. After that, per-token cost is a fraction of the API. Somewhere on the volume axis the lines cross. That crossover point is where a self-hosted open-weights LLM stops being a science project and starts being the cheaper path.

This is not theory. A LoRA fine-tune of a 7B-8B open-weights model on a few thousand trajectories often costs a few hundred dollars.

Beyond cost: latency, data residency, and vendor lock-in

Your agent feels sluggish

A small model you serve has no hop to someone else's cluster and fewer parameters per forward pass. If your AI agent makes dozens of tool calls per task, 200ms vs 2s is not a benchmark flex, it is whether the product feels alive. Lower time to first token (TTFT) compounds across the whole agent loop. Users notice the speed. They leave when it isn't there.

Your data cannot leave the building

When you rent, prompts and completions leave your perimeter. For fintech, healthcare, or anyone under data-residency, GDPR, or HIPAA rules, that is not a preference. It is a hard no. A private LLM, self-hosted or VPC-hosted open-weights, is not an optimisation. It is how you ship at all.

The vendor moved the goalposts

Renting means their roadmap, their rate limits, their deprecation calendar. When they sunset the model your agent was tuned around, you re-tune on their schedule. Owning means your inference stack and your weights. Still a dependency, but one you control.

When to use open-weights vs a frontier API

Own (open-weights, self-hosted) when:

volume is high and climbing
the task is narrow
latency is a product problem
data cannot leave your perimeter
the model is your edge
you know what you are building

Rent (frontier API) when:

volume is low or spiky
the task is broad and messy
latency is fine
data has no residency limits
raw frontier capability matters most
you are still finding out

FAQ: fine-tuning open-weights vs using a frontier API

Do I need a team of ML engineers to fine-tune an open-weights LLM?

No. LoRA / QLoRA plus a managed inference host (Fireworks, Together, Modal, or vLLM on your own GPUs) is within reach of one strong engineer. The hard part is not GPUs. It is good training data and honest evals.

Which open-weights base model should I start with?

Pick a family that fits your licence and task, such as Llama, Qwen, Mistral or DeepSeek. Start small. A 7B-8B model, fine-tuned on your task, handles more production work than most teams expect.

What if a better frontier model ships next month?

If you rent, you get the upgrade (until the API changes). If you own, you decide whether to re-tune. Here is the part people miss. A small model trained on your production trajectories often still wins on your cost, latency, and task metrics. That is the scoreboard that pays the bill, not the leaderboard screenshot.

Is my data really at risk on a frontier API?

For most consumer SaaS a zero-retention, no-training contract covers you. For regulated data, get an explicit guarantee in writing or run a private, self-hosted LLM. Do not guess.

Fine-tuning on production data is the real moat

Owning only works if your model is actually better at your task than the rented one. Downloading open-weights does not get you there. Fine-tuning on your production data does. Traces, tool-call trajectories, preference pairs, real evals from real usage.

Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own. Get started.

How do you turn traces into a training dataset?

Tyler Edwards — Fri, 04 Sep 2026 10:17:42 +0000

If you're running agents in production, you already have a training dataset sitting in your traces and most teams never look. Here's the four-stage pipeline for turning that raw telemetry into something you can actually fine-tune on.

Originally published at overmindlab.ai.

Every agentic system can create its own training data.

Every agentic AI system you run is quietly writing its own training set, and most teams never use it.

When an agent runs, it emits a stream of inputs and outputs: the prompt, the tool calls, the reasoning and the final answer. That stream is a trace, and once you standardise how you capture it (the OpenTelemetry GenAI semantic conventions is the default schema here) you have raw telemetry. A training dataset is narrower: a curated set of runs, scored against a spec, designed for fine-tuning jobs.

Building a dataset like this is a four-stage pipeline.

Four stages, then a weekly rebuild.

Here's a worked example, using a refund agent, to show what each stage actually involves.

Step 1: Capture every run as a trace

If your agent is instrumented at all, you already have this part. Every run generates a trace: the goal it was given, every model call, every tool invocation, every observation from the environment, and the eventual outcome. That trace is the raw material everything downstream depends on.

The industry is converging on a shared vocabulary for this. OpenTelemetry's GenAI working group has been building standard span attributes for LLM calls since 2024, covering model name, token counts, latency, and tool execution as first-class fields rather than something each team invents from scratch. That matters more than it sounds like it should, because a trace schema you have to redesign every time you switch observability vendors is a trace schema nobody trusts enough to build a dataset on top of.

This is also where a tool like Overmind tends to sit. Its SDK wraps the model call interface directly, so a single overmind.init() call captures every LLM invocation across OpenAI, Anthropic, Google Gemini and Agno, logging inputs, outputs, latency, token counts and errors without extra plumbing on your side. The point isn't the SDK itself. It's that capture has to be automatic and total, or the sampling and labelling stages downstream never get the raw material they need.

Step 2: Sample, because you cannot label everything

You should not try to label every run. Most of what an agent does in a given week is unremarkable, and reviewing all of it teaches a labelling team nothing it didn't already know. The job at this stage is picking which runs are worth a human's attention.

Three strategies cover most cases:

Sampling strategy	What it gives you	When to use it
Random	An honest baseline of what the agent does on an ordinary day	Every cycle, as the control slice you compare everything else against
Stratified	Deliberate coverage of cases you already care about, such as a specific refund reason, a customer tier, or a tool that keeps timing out	When a known segment matters more than the average run
Failure-weighted	The most signal per run, because the runs that went wrong carry the most information	When you have an error flag or a satisfaction score to sort on

Out of the refund agent's 50,000 weekly runs, a reasonable pull is around 500: a random slice plus every run that hit an error or a low satisfaction score. Whatever strategy you pick, run it on a cadence, weekly to start, rather than treating it as a one-off export you remember to do after something breaks in production.

Step 3: Label against what good actually looks like

This is the step almost everyone skips, and it's the one that decides whether the dataset is worth anything.

The instinct is to label by gut: skim a run, decide it feels fine, move to the next one. That works at a hundred runs and falls apart completely at ten thousand, mostly because "feels fine" means something slightly different to every reviewer and drifts over time even for the same reviewer. The alternative is writing down, explicitly, what good looks like, and scoring every sampled run against that spec.

You're not starting from nothing here. Your agent's own codebase already encodes most of what it's supposed to do: the outputs it should produce, the tools it's allowed to call, the checks it runs before acting, the paths it should and shouldn't take. Read the code first and most of the labelling criteria fall directly out of it.

For the refund agent, that read produces a checklist along these lines:

Criterion	What it requires	Example violation
Refund within terms	Amount within the order value and the 30-day window	Refunded a 90-day-old order
No invented terms	Only cites the published refund terms	Quoted a returns rule that does not exist
Escalate disputes	Hands chargebacks off to a human	Auto-refunded a disputed charge

Automating this scoring step is where LLM-as-judge techniques have become the practical default, since manually reviewing thousands of runs against a rubric doesn't scale. The approach has real limits worth knowing before you lean on it: research comparing LLM judges against human-labelled relevance data found strong rank correlation but only fair agreement on exact labels, and accuracy drops sharply on the more nuanced categories rather than the easy pass/fail calls. A survey of LLM-as-judge methods notes it emerged specifically because manually assessing helpfulness in training data got too expensive to do at scale by hand, which is exactly the tradeoff a labelling pipeline is making. Overmind runs this scoring as evaluators against a rubric you write. Six evaluator kinds cover it, from a deterministic check to an LLM judge, and every run is scored against a baseline before the result counts.

The runs that fail a criterion, whatever the customer clicked afterward, are the highest-value training data you have. The labels encode your judgment about what good looks like, and that judgment is the one thing no generic tool can supply for you. Do the first pass yourself for the first month. It's the fastest way to find out what your criteria actually are, as opposed to what you assumed they were when you wrote the checklist.

Step 4: Build the training set

Labelled runs aren't training examples yet. The last step shapes them for whatever method you're about to run, and the method decides the shape.

	Supervised fine-tuning (SFT)	Reinforcement learning (RL)
Which runs you keep	Runs that passed every criterion	Sampled runs with their criterion scores attached
What a row holds	An input-output pair of the behaviour you want repeated	The run plus the scores, used as a reward signal
What the model learns	To imitate a fixed set of good examples	To produce runs that score higher
What labelling has to produce	A pass/fail verdict per criterion	A usable score per criterion, not just pass/fail

Either way, the format needs to be consistent and machine-readable, with the goal and outcome attached to every run.

This is also the stage where the case for smaller, specialised models gets concrete. Overmind's own research argues that every model invocation inside an agentic workflow is a natural source of high-quality training data, precisely because the prompts are narrow and well-defined and the pass/fail signal is clean, unlike open-ended chat data. A team that instruments its model calls, clusters the resulting patterns, and fine-tunes a specialist model on them ends up with a system that improves with every production run instead of one that's frozen at whatever a general-purpose model happened to learn at pretraining time.

Which method you feed, supervised fine-tuning or RL, is its own separate decision. The dataset from steps 1 through 3 is what feeds either one.

Why this is the hard part

None of these four steps is exotic on its own. What makes the whole thing difficult is that it never stops. Production keeps producing new traces, your criteria keep getting sharper as you find edge cases you didn't anticipate, and the dataset has to be rebuilt against what users actually did this week, not what they did last quarter. The dotted feedback line on that diagram at the top is the entire job.

It's also exactly the gap most observability tooling leaves open. Datadog's own writeup on GenAI tracing gets at this directly: teams are encouraged to promote interesting production traces into curated, version-controlled "golden" datasets and layer evaluation metadata on top, which is essentially this same capture-to-label pipeline described from the observability side. An observability platform hands you the traces and stops there. Stitching production traces to behavioural training data to a deployed, improved model is work that mostly happens in spreadsheets and one-off scripts today, and it's the specific gap platforms like Overmind are built to close, running the optimise-evaluate-accept loop end to end instead of leaving it as a manual export.

Owning that labelled dataset matters because differentiation in agentic AI increasingly lives in proprietary behavioural data, not in which foundation model you call. Anyone can capture traces. The labelled dataset built from them, tuned to your own definition of correct, is the thing actually worth owning.

Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own. Get started.

What are the different types of fine-tuning?

Tyler Edwards — Tue, 01 Sep 2026 15:56:43 +0000

Picking a fine-tuning method usually comes down to one thing, whether you can write the ideal output yourself or can only judge one when you see it. Here's a practitioner rundown of SFT, RFT with GRPO, distillation and LoRA/QLoRA, and how to choose between them for your own task.

Originally published at overmindlab.ai.

Supervised fine-tuning (SFT)

SFT is where most people start, and most people should. You give the model input/output pairs and train it until it reproduces that pattern. It's teaching by worked example: here's the prompt, here's the answer, do more of this.

You'll typically need a few thousand examples for a narrow task. The catch is that SFT only works if you know what "good" looks like.

LoRA and QLoRA

LoRA doesn't compete with SFT or RFT.

Full fine-tuning updates every weight in the model. That's slow, and it eats memory. LoRA freezes the base model and trains a small set of adapter weights on top, which gets you most of the benefit for a fraction of the compute. QLoRA pushes this further by quantising the frozen base model down to 4-bit before training the adapter. That's the reason a 7B model now fits on one consumer GPU instead of a rack of them.

Tools worth knowing:

Unsloth is open source and runs LoRA/QLoRA 2 to 5x faster with up to 80% less VRAM, using custom kernels and 4-bit quantisation. It's become the default starting point for solo builders.

Tinker, from Thinking Machines Lab, is a managed LoRA API. It handles GPU scheduling and checkpointing but still lets you control the actual training algorithm. It launched in October 2025 and supports both SFT and RFT.

Overmind is a model training platform. It runs LoRA fine-tuning on your agent traces.

Reinforcement fine-tuning (RFT)

This used to just be called "RL," or RLHF if you wanted to impress someone.

SFT needs you to write the correct answer. RFT only needs you to score it. You define a reward, hand it the model's output, and the model learns to produce completions that score higher across many attempts.

Reach for this when good is easier to recognise than to demonstrate. I can't write the perfect customer support reply off the top of my head, but I can tell you whether one resolved the ticket, stayed on brand, and didn't promise something we don't offer. RFT is also the natural fit for agents: tool calls, multi-step retrieval, anything where success depends on the whole trajectory and not any single message.

GRPO (Group Relative Policy Optimization) is the algorithm doing most of the RFT work right now. It came out of DeepSeekMath and got famous when it trained DeepSeek-R1. The trick is sampling a group of responses to the same prompt, then scoring each one against the group average instead of some absolute scale. That kills the need for a separate critic model, which is what made older methods like PPO so expensive to run. DAPO and Dr.GRPO are newer variants that patch specific instabilities GRPO runs into on long chain-of-thought outputs.

Distillation

Distillation is copying a big model's behaviour into a small one. Run your inputs through the big model, collect what it says, then train a smaller model to say the same things. You lose a little quality and gain a lot of speed, plus a much smaller bill.

This is helpful when a frontier model already nails your task but is too slow or too expensive to run at real volume. It's also a big part of why small language models have taken off this past year. Teams distill a specialist model instead of shipping the 400B-parameter original into production.

Picking one

Once you know your task, the decision is mostly mechanical.

Method	What you must supply	When to pick it	Data volume	Relative cost
Supervised fine-tuning (SFT)	Input/output pairs you already know are right	You can write the ideal output yourself	A few thousand examples for a narrow task	Low
Reinforcement fine-tuning (RFT, usually GRPO)	A reward function or judge that can score an attempt	You can recognise a good output but can't write one, or success spans a whole agent trajectory	Prompts plus a scorer, no written answers needed	Highest, you sample many completions per prompt
Distillation	A teacher model that already does the task, plus your inputs	A frontier model nails the task but is too slow or too expensive at volume	As many teacher outputs as you can afford to generate	Medium, teacher inference then a cheap SFT run
LoRA / QLoRA	Whichever method above, plus one GPU	Almost always, unless you specifically need full-weight training	Same as the method it wraps	Lowest, a 7B run over a few thousand examples often costs a few hundred dollars

The same tree in words.

Your situation	Method
A big model already does the job, but it's too slow or too costly	Distillation
You can write the ideal output	Supervised fine-tuning
You can only judge the output, not write it	Reinforcement fine-tuning with GRPO
Any of the three, on one GPU	Run it through LoRA or QLoRA

Data is still the hard part

Every method above needs the same raw material underneath. SFT needs labelled pairs. RFT needs a reward or a judge that can rank attempts. Distillation needs a teacher and your inputs. Pick the wrong method and you lose some efficiency. Show up with bad data and none of them work. No algorithm fixes that for you.

Fine-tuning has gotten cheap. A LoRA run on a 7B model over a few thousand examples often costs a few hundred dollars. But cheap training doesn't fix bad data, and none of these tools hand you a set of real runs labelled, or ranked, against your own definition of good. That's a separate problem, and it's the one that eats the time. We cover it in how do you turn traces into a training dataset and how to train your agent.

FAQ

Is LoRA a replacement for fine-tuning?

No. LoRA is a way to run fine-tuning or RFT cheaper, not a different goal. You still pick SFT or RFT first, then decide whether to run it through LoRA.

When should I use reinforcement fine-tuning instead of SFT?

When you can judge a good output but can't write one yourself, or you're training an agent where success depends on a whole multi-step trajectory rather than one response.

Is distillation cheaper than fine-tuning a small model from scratch?

Usually, if a large model already performs well on your task. You're paying for inference calls to the teacher model instead of building the target behaviour from labelled data by hand.

What is GRPO and why does everyone use it now?

Group Relative Policy Optimization scores a batch of responses against each other instead of an absolute baseline, which removes the need for a separate critic model. It's cheaper to run than older RL methods like PPO, which is most of why it's become the default for RFT.

Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own. Get started.

"So, you have observability. Now what?"

Tyler Edwards — Tue, 01 Sep 2026 13:46:47 +0000

By Tyler Edwards, co-founder and CEO at Overmind. If you run an agent in production, odds are you already have traces piling up in Langfuse or LangSmith. This piece is about the part nobody instruments: what those traces are for.

Observing your agent is table stakes.

In the not-so-distant past there was a brief period when picking an LLM meant picking a vendor. You wrote against their SDK, you paid their bill, and when their servers fell over on a Tuesday afternoon so did yours. Switching was easy enough but did require Ctrl+F'ing your way through the codebase - or asking your coding agent of choice to perform this insultingly simple task for you instead.

Then on the sixth day came the LLM gateway! One endpoint, dozens of models behind it, swap them in and out faster than the UK swaps Prime Ministers (and for international readers here is the context on that one). The gateway is, on its own terms, a genuinely nice piece of engineering, and there is now a number of teams shipping model routers with various bells and whistles: Portkey, LiteLLM, OpenRouter, TrueFoundry, Kong, Cloudflare.

Sitting alongside the gateway, usually paid for separately and instrumented by a different team three weeks later, is the LLM observability stack. Langfuse, Helicone, Arize Phoenix, Braintrust, LangSmith, Datadog now too. Pick your favourite, they mostly do the same thing. They capture every prompt and every completion, stitch them into traces, and present you with dashboards.

Between the two, your AI product is being watched harder than a Premier League VAR decision.

What Is Observability?

LiteLLM shipped to solve a pretty annoying problem: every provider's API was almost-but-not-quite the same. Portkey and OpenRouter commercialised the idea, TrueFoundry and Bifrost followed, and the existing API-gateway crowd (Kong, Cloudflare) extended their products to cover LLMs the moment it became clear LLMs were here to stay whether anyone liked it or not.

The observability side has a similar shape. Langfuse, Helicone, Arize Phoenix, Braintrust all turned up in 2023 to build native LLM observability from scratch. The companies that already did ML observability - Fiddler, WhyLabs, Galileo - pivoted with the arrival of ChatGPT and have spent the last three years insisting they were actually always about LLMs really anyway.

By 2026 both categories had matured enough to start being acquired. Portkey went to Palo Alto Networks in April for somewhere around $120-140M. Helicone went to Mintlify the month before. Langfuse was rolled into ClickHouse's Series D in January. Cisco picked up Galileo and folded it into Splunk.

All Data, No Insight

So you now have hundreds of thousands, possibly millions, of traces composting nicely in a database somewhere. Every prompt, every completion, every tool call, every retry. The gateway has them as individual requests. The observability tool has them stitched into trajectories. Between the two, your AI product is the most thoroughly documented thing in your engineering org.

The gateway, by design, treats each call as a self-contained unit - the only way to route, cache, fall back and bill at scale. It doesn't have a concept of the trajectory because it doesn't need one. As far as the gateway is concerned, each call was a separate transaction.

The observability tool does have a concept of the trajectory. It will stitch the spans together, draw you a lovely waterfall diagram, and tell you exactly how the agent arrived at whatever it arrived at. What it won't tell you is whether what it arrived at was any good - because good is a judgement, and the observability tool is, as its name suggests, there to observe.

Every call along the way can pass every check either tool runs. A 200 from the provider, the right shape of completion, no PII, latency inside SLO, and cost on budget. The agent can still invent a company policy, ignored an instruction from two steps earlier, and make three perfectly correct-looking calls on its way to deleting a production database - and then gleefully reported its action back to you in the same way a dog might excitedly drop a dead bird at your feet.

Where Observability Fails

The observability vendors aren't naive about any of this. Most of them ship some version of trace review - LangSmith has annotation queues, Braintrust has human review alongside LLM-as-judge scoring, Langfuse has dataset workflows, Arize Phoenix ships evals out of the box. Teams use them. Engineers were spending afternoons grading traces and writing scoring functions and tagging failure modes, and the tools make this less painful than it would otherwise be.

What you get at the end of all that effort is a graded slice of traces. A few thousand examples marked good or bad, often with structured scores attached, sometimes with notes. This is useful - for catching regressions, for spotting drift, for arguing with your model provider about why their new release broke something. It is not, on its own, a training or eval dataset, and it is several steps short of a model that has actually improved.

LLMs are intelligently dumb, they will learn an approximation of what you teach them, as such you need to teach them exactly what you want them to do. A graded trace is an observation. A dataset is a curated, balanced, deduplicated set of examples organised around the behaviours you want the model to learn. The difference between them is curation logic, clustering, sampling strategy, and a clear view of what the model is supposed to get better at - none of which the observability tool produces.

And even once you have the dataset, the pipeline from there to a better agent has its own shape. You need to fine-tune a model or train a LORA adapter, eval-gate the result against a held-out slice, deploy the new weights behind your routing layer, and watch the next week of traces to see whether the change actually moved the agent's behaviour or just shuffled the failure modes around. Each of those steps is its own piece of infrastructure.

Bridging The Gap

Some of the vendors have started shipping bolt-ons aimed at closing parts of this. Portkey's Autonomous Fine-Tuning wires gateway logs into a provider's training API and hands off the dataset. LangChain's Engine edits prompts and code from production traces. Braintrust's Loop generates evaluators and datasets from natural-language descriptions of failure modes. Each is a real move in the right direction, and each stops one step short of the model itself. The dataset gets handed off and the trail goes cold.

This is the gap between what's possible and what's shipped. The technology to take a production trace, score it, cluster it, curate it into a dataset, fine-tune a model on it, eval-gate the result and deploy it back through your routing layer - all of that exists, in pieces, sold by different companies with different pricing pages. What doesn't exist yet is a single layer that does the whole loop. You can probably see where i’m going with this but lets not jump ahead.

The Opportunity

Cursor is the example everyone reaches for, and for good reason. They trained their Composer model on production agent traces using online reinforcement learning, and it now beats the frontier at its size on cost and performance. The model that ships to Cursor users is the model trained on what Cursor users actually do. The labelling encodes the judgement. The fine-tune encodes the labelling. The product encodes the fine-tune. It compounds to create a very powerful product.

Doing this used to require a research team and the patience of an academic institution. It doesn't anymore. Open-weights bases - Llama, Qwen, Mistral, DeepSeek - are now genuinely competitive on scoped tasks. A LoRA fine-tune on a few thousand labelled trajectories runs for low hundreds of dollars on Modal or Together or your own GPUs if you're feeling brave. Unsloth and Tinker have made the mechanics broadly accessible to anyone willing to spend a weekend with them, with the small caveat that you do still currently have to be a bit of a genius to use them properly.

What this adds up to is something most teams haven't fully reckoned with yet. The data is sitting in the gateway. The traces are sitting in the observability tool. The curation can be automated. The fine-tuning is cheap. The deployment is a config change. The only thing standing between any reasonably-sized team and a continuously-improving fleet of specialist models trained on their own production traffic is whether anyone has joined the pieces up.

Overmind

And we arrive at the aforementioned point. We built Overmind on the same bet we wrote about in Think Smaller: the teams who win the next phase of agentic AI will be the ones who own their models, not the ones who rent the most powerful ones, and certainly not the ones who own the most expensive plumbing in front of those rented models.

Overmind is the layer that turns the data your gateway and observability stack are already collecting into models that improve because of it. It plugs into the trace store you've already got, runs the curation, builds the dataset, fine-tunes the model, eval-gates the result and deploys the new weights behind your existing routing layer. The gateway keeps doing what it's good at. The observability stack keeps doing what it's good at. Overmind does the bit in between that nobody else is doing.

Keep your gateway. Keep your observability stack. Just stop pretending they're enough on their own.

FAQ

What's the difference between an LLM gateway and LLM observability? Gateways route and rate-limit calls between your app and model providers. Observability tools record traces and let you inspect them. Neither curates the data or trains models.

Why isn't observability enough for production agents? Observability shows you what happened. It doesn't tell you whether the trajectory was good, and it doesn't turn that judgement into model improvements. That gap is the bottleneck.

What does Overmind do that observability tools don't? Overmind sits on top of your trace store, curates the data into training-grade datasets, fine-tunes specialist models, evaluates them, and deploys the new weights behind your existing routing layer.

Tyler Edwards is Co-Founder and CEO of Overmind. He writes about agent infrastructure, fine-tuning, and what it takes to ship AI that actually improves in production. Connect on LinkedIn.

Stay in-the-loop

Join our newsletter to follow as we build supervision for super-intelligence.

If your traces should end as a model you own rather than a dashboard, that loop is what Overmind does: audited datasets from production traces, fine-tuning, a benchmark against the model you run today, and serving. There is a live Langfuse import if your traces already live there.