DEV Community: Tyler Edwards

How to train your agent: 7 patterns from production teams

Tyler Edwards — Wed, 07 Oct 2026 13:21:44 +0000

Training an agent in production runs on seven patterns. Instrument every trace, build evals from real traffic, label those traces into a dataset, accept that prompt engineering has a ceiling, run a LoRA fine-tune, freeze the eval set before you train, then repeat the loop. Most of the effort goes into infrastructure, not the agent.

Originally published at overmindlab.ai.

Learnings from building, deploying and improving agents in production.

If you're running an LLM-powered agent in production, the seven patterns below show up again and again.

What are the seven patterns for training an agent?

#	Pattern	What you do	Tactical advice
1	Instrument every agent trace	Route every agent run into one OpenTelemetry-compatible tracing tool	Keep traces at least 30 days. Tag with user ID, use case and agent version
2	Build evals from real traffic	Sample production traces weekly and turn the failures into new test cases	By week two in production, your primary eval set should come from real traces
3	Turn traces into a labelled dataset	Score traces against pass/fail criteria taken from your codebase, then group failures into a taxonomy	Draft 5 to 20 criteria straight from the code. Label the first month yourself
4	Prompt engineering has a ceiling	Keep iterating while the eval score still moves, then switch lever	Run the eval twice on an unchanged prompt first. Ignore any change smaller than that wobble
5	Fine-tuning doesn't need a research lab	Run a LoRA pass over labelled trajectories on an open-weights base	1,000+ labelled trajectories and a narrow task, compared against your prompt-engineered baseline
6	Freeze the eval set before you train	Lock the eval set before training starts and test inside the agent harness	Every model change runs against the same frozen eval set, every time
7	The self-improvement loop	Repeat capture, label, train, deploy and measure until it runs in hours	Put a cadence on the calendar. Weekly is aggressive, monthly is the floor

Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own, automatically trained, benchmarked, and served. The patterns below are that same loop, whether you build it yourself or not.

How do you instrument an agent trace?

Every agent run generates a trace: the full sequence of inputs, tool calls, retrieved context, intermediate reasoning, and final output. Without that record, you're reconstructing agent behaviour from scattered log files, which is miserable work.

OpenTelemetry has become the default standard for this. Its GenAI semantic conventions set one shared format for what each step of a run records, including model name, token usage, and tool calls. A trace from a LangChain agent then looks the same as one from a raw API call. Most observability platforms now speak this format natively, including Langfuse, Arize Phoenix, Datadog LLM Observability, and LangSmith.

Tactical advice: pick one OpenTelemetry-compatible tracing tool. Route every agent run into it. Keep the traces for at least 30 days. Tag them with user ID, use case, and agent version, so you can filter them later without going back to change your tracing code.

How do you build evals from real production traffic?

Before deploying an agent, run it against some evals. An eval is a scored check of whether the agent did the job. However, when real traffic arrives, so will edge cases.

Production traces can be a real eval source. Sample traces weekly and turn the failures into new test cases. Older eval sets can become a regression check, to confirm old fixes still hold, rather than your source of truth. Langfuse, Braintrust, Arize, and LangSmith all support this workflow directly, turning sampled production data into structured eval sets.

Tactical advice: by week two in production, your primary eval set should be built from real traces, not synthetic ones written before launch.

How do you turn traces into a labelled dataset?

A trace only records what happened. Before you can train or evaluate on it, you need to know whether the run was any good. That judgment is the label, and labels are what a dataset is actually made of. More unlabelled traces will not make it better, so label what you have before you collect more.

Start with your agent's codebase. It already spells out what the agent is meant to do, so turn that into a checklist of pass or fail criteria. Score traces against the checklist and write a note on each one that fails. Hamel Husain calls this error analysis. Group those notes into a failure taxonomy, a fixed list of failure types you can count and track over time.

Once every trace carries a label, and a failure type where it failed, you have a dataset. That is what you fine-tune on, and what you score every later change against. We go through the full pipeline in how do you turn traces into a training dataset.

Tactical advice: draft 5 to 20 pass/fail criteria straight from the code, then refine them as you label. Spend a few hours a week labelling yourself for the first month. Don't outsource the first pass. Your judgment about what "good" looks like only shows up once you've sat with the traces yourself.

When should you stop prompt engineering?

Prompt engineering has a ceiling. Prompt iteration is cheap and fast, so use it for as long as it keeps improving your eval score. Eventually each new instruction fixes yesterday's bug and quietly introduces a new one. The system prompt bloats, latency creeps up, and the model starts ignoring parts of it altogether.

At that point you're choosing between more tweaking or a different lever entirely. Most teams keep tweaking a month longer than they should, because prompt edits feel free. They aren't. Engineering time is the most expensive line item in the whole pipeline. We break the trade-off down in prompt engineering vs fine-tuning.

Tactical advice: track your eval score before and after every prompt change. First run the eval twice on an unchanged prompt to see how much the score wobbles on its own. Ignore any change smaller than that.

Do you need a research team to fine-tune?

Fine-tuning an open-weights model used to mean a research team and a large GPU budget. LoRA freezes the base model and trains a small set of low-rank adapter matrices instead. Those adapters are often under 1% of total parameters. That is why a 70B model can be fine-tuned on a single high-memory GPU, using QLoRA's 4-bit quantisation. Quantisation stores the weights at lower precision, so the model takes up less memory.

Pick an open-weights base, run a LoRA pass over labelled trajectories, meaning recorded agent runs you have already scored. Then serve the result on Fireworks, Together, or Modal. Unsloth, OpenPipe, and Predibase handle most of the pipeline for you. The method choice is a separate question, covered in what are the different types of fine-tuning.

Tactical advice: if you have 1,000+ labelled trajectories and a narrow task, run a LoRA fine-tune. Compare it against your prompt-engineered baseline on the same eval set.

Why should you freeze the eval set before you train?

The eval set you score a fine-tune against has to be the same one you scored the baseline against, and it has to be locked before training starts. Change the criteria and the training run in the same week and you cannot tell which one moved the number.

Test the model inside the agent harness, not on its own. A model that scores better in isolation can still make the agent worse, because the agent depends on tool call formats, output structure, and instruction following that a benchmark score does not capture.

Tactical advice: every model change runs against the same frozen eval set, every time. Models regress for all sorts of reasons, and a fixed scoreboard is the only way you find out.

The self-improvement loop

The first full pass through capture, label, train, deploy, and measure takes weeks. By the fifth pass, most of the friction is gone and the loop runs in hours.

That compounding is the whole point. A team that runs this loop five times in a quarter ends up with an agent that matches their business needs at a fraction of frontier API cost. A team that runs it once, ships, and calls it done is roughly where it started six months ago.

This is essentially Overmind's product thesis. It connects your repository and traces, then proposes changes through its optimiser. It tests those changes against your labelled data and only surfaces the ones that actually move your score. The loop from evidence to shipped improvement stays short, instead of becoming its own six-week project.

Tactical advice: pick a cadence and put it on the calendar. Weekly is aggressive but doable. Monthly is the floor. Anything slower and you're not really running a loop, you're running occasional cleanup.

Infrastructure is more work than the agent

Any one of these seven steps is a real job on its own. Strung together, they're several jobs. Most teams find the surrounding infrastructure costs more engineering time than the agent.

Here is every tool named above, grouped by the job it does.

Job	Tools named in this post
Trace format standard	OpenTelemetry GenAI semantic conventions
Tracing and observability	Langfuse, Arize Phoenix, Datadog LLM Observability, LangSmith
Eval sets from production traces	Langfuse, Braintrust, Arize, LangSmith
Fine-tuning pipeline	Unsloth, OpenPipe, Predibase
Serving a trained model	Fireworks, Together, Modal
The whole loop as one system	Overmind

Overmind closes that gap. It reads your agent's codebase to understand what it does. Then it handles trace capture, labelling, fine-tuning, deployment, and measurement as one connected system, instead of five separate vendor integrations.

If you're past the prototype stage and want the loop running without building it from scratch, it's worth a look.

FAQ

How many labelled examples do you need to fine-tune an agent?

If you have 1,000 or more labelled trajectories and a narrow task, a LoRA fine-tune is worth running. Below that, keep labelling and keep iterating on the prompt.

Can you fine-tune a 70B model without a GPU cluster?

Yes. LoRA trains a small set of adapter matrices, often under 1% of total parameters, and QLoRA's 4-bit quantisation shrinks the frozen base model further. That puts a 70B fine-tune on a single high-memory GPU.

Should you test a fine-tuned model on its own or inside the agent?

Inside the agent harness. A model that scores better in isolation can still make the agent worse, because the agent depends on tool call formats, output structure, and instruction following that a benchmark score does not capture.

How often should you run the training loop?

Weekly is aggressive but doable, monthly is the floor. Slower than that and you're running occasional cleanup rather than a loop, so the compounding never arrives.

Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own. Get started.

Secret Agent Overmind: diagnose an AI agent with code, traces, evals and datasets

Tyler Edwards — Wed, 30 Sep 2026 10:24:05 +0000

To diagnose an AI agent you need four records joined, not four dashboards. Read the codebase for what the agent can do, traces for what it actually did, dataset rows for what it will be trained and tested on, and eval scores for how well it scored. The answer to "why did it do that" lives in the gaps between them.

Originally published at overmindlab.ai.

Your agent is the most documented system in your engineering org and yet you can't answer simple questions.

No, Christopher McQuarrie has not asked me to direct the next instalment of Mission: Impossible, though I am keeping a keen eye on my inbox. Today, we are talking state secrets!

That is also not true. But I did spend many years building stuff and things (the OFFICIAL term) for British Intelligence, and the single most valuable thing I took away was not state secrets. Instead, it was a deeply troubling obsession with data.

"Is this person a credible threat?" is the canonical example, and a very difficult one to answer. Intelligence analysts are basically superhumans, capable of finding links between vast amounts of varied data, and one of the ways they do that is by layering context.

Data, data, everywhere.

That data is vast: signals intelligence, human intelligence, financial records, travel manifests, satellite imagery, and a mountain of open-source material. Some of it is a tidy database, some of it is a photograph, and some of it is words scrawled on a piece of paper. The scale is hard to picture: by the NSA's own figures, it "touches" around 29 petabytes of data a day, which is equivalent to the entire Library of Congress being inspected nearly 3,000 times over, daily.

I hear you shouting, "Well Tyler, big tech companies have access to lots of data, so why is MI5 so special?" Beyond the legal implications and the fact that we don't (yet) live in a Blade Runner-style dystopia, the reasons are structural. Broadly speaking, any one organisation only ever holds a single thin slice of the picture: what you searched, what you spent, or where you travelled. You can know someone's entire spending history in perfect detail and still have no idea what any of it means. An intelligence agency exists precisely to sit above all those slices and join them, which is the part almost nobody else is mandated, or built, to do. The fusion, not the volume, is the whole point.

This is the core function of an intelligence agency: not simply to collect the data, but to surface the hidden correlations in vast pools of information. Which brings us, as everything eventually does, to AI agents.

Context really is king

Your agent is the most thoroughly documented system in your engineering org. Every prompt, every completion, every tool call, every retry is captured somewhere: traces in your observability tool, scores in your eval suite, rows in your dataset. And yet you still can't answer simple questions. Why did it do that? Where did the behaviour come from? Who gave it a gun? The records are all there, sitting in separate systems, and nobody joins them up.

Each of those systems holds one thin slice, and each slice has a blind spot the next one covers.

Layer	What it holds	What it can tell you	What it can't
Codebase	The tools the agent can call and their contracts, its input and output schema, the prompts, the expected output	What the agent could do	Whether it ever did
Production traces	Every prompt, completion, tool call and retry	What the agent actually did	What it was built to be able to do
Dataset rows	Format, schema, per-column distribution, label space, cohorts, hygiene problems	What you are about to evaluate or train on	Whether that covers the agent's real surface
Eval scores	Scores against your eval suite	That a behaviour was wrong	Why it was wrong

So we did exactly that. We built a context layer that joins all these things together: one grounded picture of how your agent is put together and what it actually does, assembled from your codebase and traces, then turned loose to power Overmind. Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own, automatically trained, benchmarked and served. Getting there starts with knowing what the agent actually is.

What is a Capability Card?

We start at the codebase, because that's where the ground truth lives. Before Overmind touches a single trace, it reads the code the agent runs on and compiles a Capability Card: the tools the agent can call and their real contracts, its input and output schema, the prompts, and the expected output. That's the agent at its raw logical level, taken from the source rather than inferred from behaviour. Skip it, as most tooling does, and a trace is just a behaviour with nothing behind it, leaving you to reverse-engineer intent from whatever the trace happened to capture.

What Overmind reads from the code	What it pins down
The tools the agent can call, with their real contracts	The full set of actions available to it, not only the ones a trace happened to capture
Input and output schema	The shape the agent is written to accept and return
The prompts	The instructions in the source, not a reconstruction from behaviour
The expected output	The result the code is written to expect

Then we bring in the data and the traces. Every record of what the agent did is a context source: production traces, dataset rows, eval scores, all built into the same shape. For a dataset, that layer is the spine: format, schema, per-column distribution, label space, cohorts, hygiene problems, derived straight from the data. If you have not got traces flowing yet, start there, and if the word "agent" is doing a lot of undefined work in your org, we pulled it apart here.

How do you know whether your data covers your agent?

The codebase context is what makes that data useful. With the Capability Card in hand, Overmind knows the full surface of the agent and can measure the data against it: how much of that surface the data actually covers, which behaviours are well represented, which tools never appear, and which paths the agent takes in production that no dataset row touches. A pile of traces tells you what happened. The codebase tells you what could happen, and the gap between the two is the thing you need to see. That delta is what decides whether the data is good enough to evaluate against or train on, or whether you're about to write evals for a fraction of the agent or fine-tune on the same narrow slice of its total behaviours.

Overmind's analysis of a PostHog Agent. The same view in text:

What the console shows	For the PostHog agent
Agent	PR Approval Agent (StampHog)
Discovered from	The `posthog/posthog` repository
Tags	ci, github, security
Models it calls	Seven, including Claude Sonnet 4.6, GPT 4.1 Mini, o3, Gemini 2.5 Flash Lite and Claude Haiku 4.5
Score	50%
Trace coverage	1 of 2 runs traced
Eval status	Eval metrics not approved
Flow graph	Input, Agent and Read into conversation_summarizer, title_generator, session_summarizer and llm_traces_summarizer, ending at Output

Nothing in that card was filled in by hand. Seven models, three tags and a flow graph, read out of a public repository, with the coverage number sitting next to it so you can see how much of that surface anyone has actually watched run.

Codebase tied to data tied to traces is the working surface for everything else we do. The data preprocessing reads from it, the evals are written against it, the fine-tuning sets are curated out of it, and the agent optimisation is steered by it. That's the line between a tool that can tell you a behaviour was wrong and one that can tell you why, as well as how to fix it.

So, what do you know about your agent?

So much, maybe too much. But not in a useful way.

That is what Overmind is built for: not more raw evidence, but the relationships between it. Code, traces, datasets, evals, and training runs joined into one grounded view of the agent, so you can see what is really happening and where the gaps are.

FAQ

Why can't I tell why my AI agent did something?

Because the evidence is split across systems that never meet. Traces live in your observability tool, scores in your eval suite, rows in your dataset, and the agent's actual definition in your repository. Each one answers a different question, and no single one of them answers "why".

Are production traces enough to debug an agent?

No. A trace tells you what happened, not what could have happened. Without the codebase you cannot tell whether a behaviour was one of many available paths or the only one, which tools the agent never reached for, or which parts of its surface nobody has ever watched run.

What is agent coverage analysis?

Measuring your traces and dataset rows against the agent's full capability surface, taken from the code. It tells you which behaviours are well represented, which tools never appear, and which production paths no dataset row touches. That delta decides whether your data is fit to evaluate against or train on.

How is this different from an observability tool?

Observability records behaviour. Joining that record to the codebase, the datasets and the eval scores is what turns a wrong behaviour into an explained one, and then into a fix you can ship.

Tyler Edwards is Co-Founder and CEO of Overmind. He writes about agent infrastructure, fine-tuning, and what it takes to ship AI that actually improves in production. Connect on LinkedIn.

Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own. Get started.

Unwrapping the wrappers: who actually trains their own models, and why

Tyler Edwards — Tue, 29 Sep 2026 10:54:39 +0000

Cursor, Intercom, Replit and Perplexity all train specialist models on their own production data. Each picked one narrow, high-volume task, trained on traffic no competitor can access, and shipped a model that matches or beats frontier APIs on that job at lower latency and cost. None of them tried to out-build OpenAI.

Originally published at overmindlab.ai.

Application layer companies, often labelled as 'ChatGPT wrappers', are starting to train their own models. Specialist models, post-trained or fine-tuned on a dataset their competitors cannot touch: their own production traffic.

Four recent examples:

Cursor trained Composer, the model inside its coding agent, with online RL on production agent traces. Delivering frontier-level coding at a fraction of the latency and cost.
Intercom trained Fin Apex, a customer-service model post-trained on billions of support interactions. It bought them control of the resolution-rate metric at scale.
Replit trained a 7B code-repair model, a fine-tune of DeepSeek-Coder-Instruct-v1.5. It bought them a fast, narrow specialist trained on IDE data nobody else has.
Perplexity trained Sonar, its search-answer model, a fine-tune of Llama 3.3 70B served on Cerebras hardware. It bought them sub-second, citation-heavy answers at search scale.

Which companies train their own models?

The same four, with the base model, the method and the measured result in one view.

Company	Model	Base model	How they trained it	Measured result	Source
Cursor	Composer	Not disclosed, a mixture-of-experts architecture	Online RL on production agent traces, inside a harness mirroring Cursor's real tool space; new checkpoints ship several times a day	Generates tokens several times faster than comparable frontier models, landing close to the top tier on internal coding benchmarks	Cursor, Composer 2 report
Intercom	Fin Apex 1.0	Not disclosed	Post-trained on the company's own support interaction data, billions of interactions	73.1% benchmark resolution rate against 71.1% for GPT-5.4 and Claude Opus 4.5; one enterprise customer went from 68% to 75% overnight	Intercom, VentureBeat
Replit	7B code-repair model	DeepSeek-Coder-Instruct-v1.5 (7B)	Fine-tuned on IDE telemetry, scoped to roughly 20 diagnostic categories	Matched or beat GPT-4 and Claude 3 on real-world fixes inside that use case	Replit
Perplexity	Sonar	Llama 3.3 70B	Fine-tuned for factual grounding, citation quality and readability, served on Cerebras wafer-scale hardware	Roughly 1,200 tokens per second, and matched or beat larger frontier models on user satisfaction in Perplexity's own A/B tests	Perplexity, Cerebras

Where we sit, for the record. Overmind is the model training platform for AI teams, and the pattern running through these four case studies is the one it is built for. The case studies themselves are other people's work, cited at source.

Cursor: reinforcement learning on live agent traces

Cursor didn't fine-tune once and ship. The Composer family of models run on real-time RL: model checkpoints go to production, real user interactions become the reward signal, and an improved checkpoint ships several times a day. The model is a mixture-of-experts architecture trained inside a harness built to mirror Cursor's actual tool space, including file edits, semantic search, and terminal commands. The reward signal reflects how developers really use the product rather than a synthetic benchmark, according to Cursor's technical post on Composer.

Cursor reports Composer generates tokens several times faster than comparable frontier models while landing close to the top tier on internal coding benchmarks, a tradeoff explored in more depth by the Composer 2 technical report. Nobody else has Cursor's trace data, which means nobody else can train Cursor's model, even with an identical base checkpoint and an identical training recipe.

What it bought them: coding performance competitive with larger models, at preferential latency and cost, plus a moat provided by their own proprietary data.

Intercom: post-training on the metric the product is sold on

Fin launched in 2023 on top of GPT-4. Fin 2 moved to Claude in late 2024. Then Intercom's roughly 60-person AI research group, led by Fergal Reid, built Fin Apex 1.0, a model post-trained on the company's own support interaction data, as described in the Fin Apex launch post.

One large enterprise customer saw its resolution rate jump from 68% to 75% overnight after the switch to Apex, a 22% cut in unresolved conversations according to Intercom's own numbers. Independent coverage from VentureBeat puts Apex's benchmark resolution rate at 73.1%, against 71.1% for both GPT-5.4 and Claude Opus 4.5.

Model	Benchmark resolution rate
Fin Apex 1.0	73.1%
GPT-5.4	71.1%
Claude Opus 4.5	71.1%

Intercom's own model documentation breaks the pipeline into separate retrieval, reranking, and generation models rather than one monolithic system, detailed on the Fin CX models page.

Worth noting: Intercom hasn't abandoned rented models. Apex is layered on top of retrieval and routing components, and the company still uses third-party models elsewhere. This architecture is common as companies balance performance, latency and cost.

Replit: a narrow 7B specialist trained on IDE telemetry

Replit's bet was the opposite of "bigger is better." The team fine-tuned a 7B open-weights model, DeepSeek-Coder-Instruct-v1.5, chosen after testing several code-focused base models and documented in its code-repair write-up.

The model targets a specific use case, roughly the 20 diagnostic categories responsible for the majority of everyday coding errors, and the fine-tuned 7B checkpoint matched or beat GPT-4 and Claude 3 on real-world fixes in that use case. Small open source models can beat much larger general-purpose models at many tasks.

Perplexity: search-optimised inference at 1,200 tokens per second

Perplexity fine-tuned Meta's Llama 3.3 70B into Sonar, tuned specifically for factual grounding, citation quality, and readability in its default search mode, as the company explains in its Sonar launch post. Sonar runs on Cerebras wafer-scale inference hardware, which Perplexity and Cerebras jointly announced pushes throughput to roughly 1,200 tokens per second, an order of magnitude faster than comparable models on conventional GPU inference.

In Perplexity's own A/B testing, Sonar matched or beat larger frontier models on user satisfaction inside search, while running far cheaper and faster than calling those models directly.

Why build a specialist model instead of prompting a frontier one?

Calling a frontier model for every request is sometimes the equivalent of running a data centre to toast bread. It works, but it's the wrong tool for a narrow, high-volume, well-defined job. Across these four companies, three conditions kept showing up:

A narrow task, not a general one. Small specialised models are excellent at one narrow task. Don't ask them to be broadly intelligent.
Proprietary production data no competitor can access. Cursor has Cursor's traces. Intercom has Intercom's support conversations. Your production traffic is the one dataset nobody can buy.
Enough volume that per-token latency and cost actually move the P&L. Fine-tuning a model can be expensive. It only pays off once inference volume is high enough that shaving cost per call compounds.

None of these companies want to out-build OpenAI or Anthropic on general intelligence. Each trained a model that wins one specific task, using the one dataset it uniquely owned.

What does the traces-to-training loop look like?

Stage	What it produces
Production runs	Traces of what the agent actually did
Labelled dataset	Those traces, curated and labelled
Fine-tuned model	A specialist trained on that dataset
Redeploy	The specialist back in production, generating the next round of traces

Capture production traces, turn them into a labelled dataset, train a model on it, and ship that model back into production. That's where the application layer is going. We break the middle two stages down in how do you turn traces into a training dataset and how to train your agent.

FAQ

Do companies that train their own models still use frontier APIs?

Usually yes, alongside. Intercom hasn't abandoned rented models. Fin Apex sits on top of retrieval and routing components, and the company still uses third-party models elsewhere. A mix of owned and rented models is the common architecture, chosen to balance performance, latency and cost.

Do you need a large model to beat a frontier one?

Not on a narrow task. Replit's 7B fine-tune of DeepSeek-Coder-Instruct-v1.5 matched or beat GPT-4 and Claude 3 on real-world fixes inside its target diagnostic categories. Perplexity's Sonar, a 70B fine-tune, matched or beat larger frontier models on user satisfaction inside search.

What data do you need to train a specialist model?

Your own production traffic. Cursor used live agent traces, Intercom used its support interactions, Replit used IDE telemetry, and Perplexity tuned against its own search behaviour. The common ingredient is a dataset the product generates itself, which is the one thing a competitor cannot buy.

When does training your own model not pay off?

When any of the three conditions is missing. A broad task rather than a narrow one, no proprietary data sitting behind it, or volume too low for per-call savings to compound. Fine-tuning costs money, so it only pays back once inference volume is high enough.

Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own. Get started.

The anatomy of an AI agent: model, loop, tools, memory

Tyler Edwards — Wed, 23 Sep 2026 11:50:46 +0000

An AI agent is a language model wrapped in a loop that plans, calls tools and iterates toward a goal across many steps. Five components make one up. A model that picks the next action, a harness that runs the loop, tools that reach outside the model, memory that carries state, and a trajectory, the record of the whole run.

Originally published at overmindlab.ai.

An agent has a model, loop, tools and memory.

In 2024, most AI products were chatbots. A single LLM inference call, one prompt in, one completion out. In 2026, the default architecture is an AI agent, an agentic system that plans, calls tools, and iterates toward a goal across many steps.

	Chatbot (the 2024 default)	AI agent (the 2026 default)
Model calls per task	One	Many
Shape	One prompt in, one completion out	Plans, acts, observes, repeats
Tools	None	APIs, databases, code execution
Execution path	A single step	Varies run to run, non-deterministic
What you get back	The first model response	An outcome, plus a trajectory

Instead of returning the first model response, teams wrap the LLM in an agent harness, expose tool calling (APIs, databases, code execution), and let the system work toward an objective. Frameworks such as LangChain, Agno, CrewAI, and the OpenAI Agents SDK exist to make it easier to build the harness.

This article decomposes an AI agent into its core components. We define each part of the AI harness and walk through one complete agent run. Worth stating where we sit. Overmind is the model training platform for AI teams, and the trajectory, component five below, is the raw material it works from.

What are the five components of an AI agent?

Strip an agentic system down and you get five components. Four you assemble and the fifth is what they produce.

Model. Selects the next action given goal and state.
Harness. Runs the loop of reason, act, observe, repeat.
Tools. How the agent interacts with external systems.
Memory. How the agent persists and retrieves state.
Trajectory. The record of one complete agent run.

Component	What it does	Assembled or produced	How you change it
Model	Selects the next action given goal and state	Assembled	Swap it, or train and fine-tune it
Harness	Runs the loop of reason, act, observe, repeat	Assembled	Rewrite prompts, context rules and stop conditions
Tools	How the agent interacts with external systems	Assembled	Add, remove or redescribe tools
Memory	How the agent persists and retrieves state	Assembled	Change what stays in the context window, change retrieval
Trajectory	The record of one complete agent run	Produced by the other four	Not directly. It changes when the other four change

What is the model in an AI agent?

The language model is the only component that reasons, and the only one you can train or fine-tune. Conditioned on the goal and the observations so far, it chooses the next action. Most teams start on a frontier model API and some migrate to an open-weights model they host themselves or run through a third-party provider.

More on that trade-off in open-weights LLMs vs frontier APIs.

What is an agent harness?

The harness assembles what the model sees, reads the action it picks, calls the matching tool, and feeds the result back in. It also decides everything the model does not. Which tools exist and how they are described, what stays in the context window, what happens when a tool call fails or comes back malformed, and when to stop, whether that is the goal being met, a step limit, or an error.

This is what agent frameworks give you. It is the part of an agent you can change immediately.

What is tool calling?

Tools are how the agent reaches outside the model, whether that is querying a database, calling an HTTP API, running code, or sending an email. OpenAI shipped function calling in June 2023, letting the model emit a structured request to invoke a function instead of prose. Anthropic open-sourced the Model Context Protocol (MCP) in November 2024 to standardise how agents connect to those tools, and now everything is an MCP server.

What is agent memory?

Memory is how the agent carries state across loop iterations. Short-term memory is the context window, the running transcript of the goal, prior steps, and tool outputs. Longer-term memory is retrieval, pulling documents or past runs from outside the window (often via RAG or a vector store). Without memory, the agent loses its way.

What is an agent trajectory?

The trajectory, also called the agent trace, is the full ordered record of a single run. The goal, every LLM call, every tool invocation, every observation, and the final outcome. It is the agent's stack trace. You do not build it; the other four components produce it. It is also the only unit of analysis at which you can honestly say whether the agent succeeded. Capture it.

One agent run, start to finish

Watch a support agent handle a refund. The user goal enters, the agentic loop runs, an outcome exits.

The user goal arrives. Refund this customer.
The model calls get_order(id); the tool returns the order details.
The model checks the order against the refund policy.
The model calls issue_refund(amount); the tool returns a confirmation.
The model reports back. Refund done.

Four model calls, two tool calls, one outcome. That entire sequence is the agent trajectory. Change the goal, the tool responses, or the model's sampling behaviour and the next run takes a different path. Agents are non-deterministic. The same prompt does not guarantee the same execution path.

What does each layer of the AI stack actually control?

Each layer of a typical AI stack controls an aspect of the agent. Almost none control the whole run.

Layer	What it sees	Can it update the model?
LLM gateway	One request and response at a time	No
Agent observability	The full trace, after the fact	No
Agent evals	A score for the run	No
Closed training loop	Judged trajectories fed back into training	Yes

Three of the four tell you what happened. Only the last one changes what happens next time. More on that gap in so, you have observability. Now what?.

How do you improve an AI agent?

Improving an AI agent means operating at the level of the trajectory, not the individual LLM call. Capture the full run, judge it against your definition of success, and feed judged trajectories back into the system.

There are two distinct ways to improve an agent.

Improve the harness. Rewrite prompts, tighten tool definitions, change the context rules and stop conditions.
Improve the model. Retrain it on the failure modes in the traces, then benchmark the result against the model you serve today.

This is the loop Overmind is built to close.

FAQ: agents, harnesses and trajectories

What is the difference between an AI agent and a chatbot?

A chatbot is one model call. One prompt in, one completion out. An agent runs many model calls inside a loop, calls tools between them, and keeps going until the goal is met or it stops. The comparison table above sets the two side by side.

Is an AI agent the same thing as a workflow?

No. In a workflow, you decide the order of steps in advance. In an agent, the model decides the next action at each turn given the goal and what it has observed so far. That is why two runs of the same agent can take different paths and a workflow cannot.

Why are AI agents non-deterministic?

Because the model chooses each action, and its sampling behaviour, the goal, and the tool responses all vary. Change any of them and the next run takes a different path. This is why a single run tells you very little and the trajectory is the unit you have to judge.

Do I need an agent framework to build an AI agent?

No. Frameworks such as LangChain, Agno, CrewAI, and the OpenAI Agents SDK exist to make the harness easier to build, not to make it possible. The loop is reason, act, observe, repeat, and plenty of teams write their own.

Which part of an agent should you change first when it fails?

Start with the harness, because it is the part you can change immediately. Prompts, tool descriptions, context rules, stop conditions. Go to the model when the same failure keeps showing up across trajectories after the harness is clean.

Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own. Get started.

Think smaller: why specialist SLMs beat frontier models in production

Tyler Edwards — Tue, 22 Sep 2026 11:46:42 +0000

Most agent tasks do not need a frontier model. The subtasks inside a production agent are largely repetitive and narrowly scoped, and a small language model of 1B to 8B parameters, fine-tuned on your production data, beats a frontier model on cost, latency and accuracy for that job. Serving a 7B runs roughly 10 to 30x cheaper than a 70B+ model.

Originally published at overmindlab.ai.

Frontier models are overkill for most agent tasks. Specialised small language models beat them on cost, latency, and accuracy.

The AI industry has spent three years in an arms race measured in billions of parameters, and somewhere along the way everyone agreed that the answer to every problem was a bigger model.

Bigger clusters, bigger bills, bigger teams to manage the complexity. The implicit assumption is that general intelligence at scale is the end state and everything else is just waiting to catch up.

Which is obviously great for the foundational teams pushing this agenda, were it not for the fact that the overwhelming majority of problems we throw AI at don't need otherworldly intelligence and infinite compute to solve.

Llama, Phi, Qwen, Mistral: models that would have been considered frontier-grade two years ago are now freely available, fine-tuneable, and deployable on hardware you control. The performance gap that once justified the cost and lock-in of proprietary giants is now much slimmer, so the question is no longer "which frontier model should we use?" but rather "do we need a frontier model at all?"

For most tasks, the honest answer is no.

Why is a frontier model overkill for most agent tasks?

Using a frontier model on every task is like using a flamethrower to light a candle. Technically works, but your eyebrows and your budget are gone.

The majority of subtasks in a deployed agentic system are repetitive, scoped, and non-conversational. Extract this field, classify this ticket, format this output, none of these require a model trained on the sum of human knowledge. They require a model that is fast, predictable, cheap to run, and tuned to behave exactly as expected in a well-defined context.

In fact, on well-defined, repetitive tasks, specialised SLMs do not just cost a fraction of what frontier models cost, they outperform them. A model built specifically for one thing, trained on real examples of that thing, is better at that thing than a model trained to do everything.

Which small models have beaten frontier models on benchmarks?

And this holds even as the tasks get harder. Qwen2.5-Math-7B, running Microsoft's rStar-Math reasoning framework, scores 90% on the MATH benchmark, beating OpenAI's o1-preview. John Snow Labs' MedS, an 8B medical model, was preferred over GPT-4o by practising physicians in blind evaluation across every dimension: factuality, clinical relevance, and conciseness. LlaSMol, a 7B chemistry model, achieves 93% exact match on molecular prediction tasks where GPT-4 scores under 5%. On Berkeley's function-calling leaderboard, an 8B model held the top spot at the time of writing, beating GPT-4o and Claude 3.5 Sonnet.

Specialist model	Params	Benchmark	Score	Model it beats	Source
Qwen2.5-Math-7B with rStar-Math	7B	MATH	90%	OpenAI o1-preview	rStar-Math, Microsoft
John Snow Labs MedS	8B	Blind physician review on factuality, clinical relevance and conciseness	Preferred over GPT-4o on every dimension	GPT-4o	John Snow Labs expert review
LlaSMol	7B	Molecular prediction, exact match	93%	GPT-4, under 5%	LlaSMol, OSU NLP
Top open function-calling specialist	8B	Berkeley function-calling leaderboard	Top of the leaderboard at the time of writing	GPT-4o and Claude 3.5 Sonnet	Berkeley function-calling leaderboard

What does open-weights actually unlock for production AI?

The capability trajectory of small open models has been striking, yet the capability of these models is not the most exciting piece, what open weights actually unlock is control.

A frontier model you do not own, cannot audit, and cannot modify is a dependency, not infrastructure and certainly not your 'technical moat'. When it changes, your system changes. When it halts, your system halts. When the provider decides to deprecate a version or reprice the API, you absorb it. The case for specialised small models is not just that they are better. It is that they are yours.

The physical dimension matters too. Frontier models run in data centres, behind APIs, with latency measured in hundreds of milliseconds and costs that compound with every token. In an agentic system where a single workflow might invoke a model thirty or forty times, that compounds fast. Serving a 7B SLM is roughly 10 to 30x cheaper in latency, energy, and compute than serving a 70 to 175B LLM. Small models can run on-premise, inside the security perimeter, on the device generating the data. A fraud detection model running locally, fine-tuned on that institution's own transaction patterns, makes faster decisions with no external dependency and no data crossing a network boundary. The performance is better, the risk profile and compliance conversation are entirely different.

Even more exciting still (if you can believe it) is that a model small enough to run on a phone can be fine-tuned not just to a specific workflow but to a specific person. An agent that lives on your device and learns from how you actually work is a categorically different thing to one calling to a data centre every time you ask it something. We are moving from general intelligence in the cloud to personal intelligence in your pocket. That shift is bigger than most people are currently treating it.

Which fits your job, a frontier API or a specialist SLM?

	Frontier model behind an API	Specialist SLM you own
Best at	Open-ended reasoning and conversation	Repetitive, scoped, non-conversational subtasks
Serving cost	Compounds with every token, and one workflow may invoke the model thirty or forty times	Roughly 10 to 30x cheaper at 7B than at 70 to 175B
Latency	Hundreds of milliseconds, over the network	No network hop, runs where the data is
Where it runs	A data centre, behind an API	On-premise, inside the security perimeter, or on the device
Control	Version deprecations and repricing land on you	Yours to audit, modify and version
Failure surface	Vast and constantly shifting, answered with hand-written guardrails	Narrow scope, tighter output distribution, predictable failures
Setup effort	Write a prompt, get an answer, iterate, ship	Instrumentation, curation, fine-tuning jobs, evals, versioning, a deployment you can swap

Production is the best training data you have

Every invocation of a model in an agentic workflow is a natural source of high-quality training data. The prompts are well-defined, the expected outputs are narrow, and whether the workflow succeeded or failed is a clean signal. A listener at the model call interface, logging inputs, outputs, and downstream outcomes, accumulates exactly the dataset you need to continuously fine-tune a specialist model for that task, as the system runs. The production environment is generating its own improvement data in real time.

This is where the SLM case becomes compelling beyond economics. You instrument the calls, cluster the patterns, fine-tune the specialists, and the system gets better with every run. What determines whether you actually close that loop is whether you have the infrastructure to capture it, label it, and act on it.

That loop is what we build. Overmind turns your production traces into specialised models you own, automatically trained, benchmarked, and served.

Why are specialist models easier to supervise?

AI security has become a pretty bloated term, so stick with me here. The version of it that actually matters is not about firewalls or prompt injection. It is about knowing what your model is doing, and knowing when it is doing something it should not. Specialised models make that question answerable. When a model has a narrow, well-defined scope, you know what it should be doing, which means you know when it is doing something else. The output distribution is tighter, the failure modes are more predictable, and the feedback loop from production behaviour back to model improvement is shorter and more reliable.

General models are a different problem entirely. The failure surface is vast, constantly shifting, and practically impossible to enumerate in advance. Guardrails are the industry's current answer, which is to say: a list of rules written by humans trying to statically account for an infinite and ever-changing set of ways a frontier model can go wrong.

Are small specialist models harder to train?

Now I would love to just dunk on frontier models for the rest of this post, but there is a real counter-argument worth addressing. Calling an API is easy. You write a prompt, you get an answer, you iterate, you ship. The feedback loop is fast and the overhead is minimal. Building SLM-first infrastructure is a different proposition entirely.

Fine-tuning a specialist model requires training data, which requires instrumentation, which requires someone to own it. You need pipelines to capture production traces, processes to curate and label them, and infrastructure to run fine-tuning jobs on a cadence that actually moves the needle. You need model versioning, evaluation harnesses, and a deployment setup that lets you swap models without taking down the system. None of this is impossibly hard, but none of it is free either.

Why this matters to us

Everything in this post is, admittedly, not a neutral observation. We built Overmind on a specific bet: that the teams who win the next phase of agentic AI will be the ones who own their models, not the ones who rent the most powerful ones.

The infrastructure to do that, fine-tuning pipelines, production observability, continuous improvement loops that do not require a dedicated ML team to run, that is what we are building. Not just because it is an interesting technical problem, but because without it the agentic future we are all so excited about, the one where AI runs diagnostics in hospitals or runs locally on your phone and adapts to you as an individual, simply cannot happen.

So we are building that future.

Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own, automatically trained, benchmarked, and served. If you are deploying agentic AI, get in touch.

FAQ

Are small language models better than frontier models?

For repetitive, scoped tasks they are. A model trained specifically for one job, on real examples of that job, outperforms a generalist on cost, latency and accuracy. Frontier models still win on open-ended reasoning and conversation.

What is a specialist SLM?

A small language model (typically 1B to 8B parameters) fine-tuned on a narrow task using your own production data. Llama, Phi, Qwen and Mistral are common open-weight bases.

How much cheaper is a 7B SLM compared to a 70B+ LLM?

Roughly 10 to 30x cheaper in latency, energy and compute when self-hosted, and the gap widens at high request volumes.

Can I run a specialist SLM on-device?

Yes. 7B and 8B models run on modern phones and laptops, which enables per-user fine-tuning, offline operation, and no data leaving the device.

What do you need in place before you can run specialist models in production?

Four things. Instrumentation that captures production traces, a process to curate and label them, somewhere to run fine-tuning jobs on a regular cadence, and model versioning plus evaluation harnesses so you can swap a model without taking the system down. None of it is impossibly hard, but none of it is free either.

Tyler Edwards is Co-Founder and CEO of Overmind. He writes about agent infrastructure, fine-tuning, and what it takes to ship AI that actually improves in production. Connect on LinkedIn.

Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own. Get started.

Prompt engineering vs fine-tuning: which one do you need?

Tyler Edwards — Tue, 08 Sep 2026 17:01:47 +0000

Every LLM team hits the same fork eventually, either keep steering the prompt or start training the model. Here's how to read the signals before you burn a sprint finding out the hard way.

Originally published at overmindlab.ai.

A perspective on when to tinker with the prompt, and when to train your own model.

Most teams building LLM applications start the same way. They write a system prompt, test it against a few examples, and ship. Prompt engineering is cheap, fast, and reversible. Fine-tuning used to be something only teams with a training pipeline and a GPU budget could afford. Things have changed but the order of operations hasn't.

Both techniques steer model behaviour. Prompting changes what you say to the model. Fine-tuning changes the model itself.

Prompt engineering vs fine-tuning, side by side

	Prompt engineering	Fine-tuning
What changes	The input	The model's weights
Where it lives	The context window	The model
Cost	Seconds, no infrastructure	GPU time and a training pipeline
Data needed	None, or a handful of examples	Thousands of labelled runs
Reversibility	Instant	Only by retraining
Survives a model swap	Yes	No
Ceiling	Plateaus quickly	High, bounded by data quality

Why prompting is the right first move

Prompt engineering means editing what enters the context window. No adjustment of the model's weights. You're changing the input and reading the output.

For a new agent, that's almost always the correct starting point. You learn what the model can already do before you spend anything on training. Most of the behaviour changes you want, prompting will give you.

How to tell which regime you're in

This isn't a one-time decision so much as a loop. You stay in prompt-engineering mode until the eval curve goes flat, then check whether you actually have what fine-tuning needs, a narrow task and real data.

Ship prompt changes until the eval goes flat, then fine-tune.

Where you are	What you're seeing	What to do next
New behaviour needed	A gap between what the agent does and what it should do	Ship a prompt change
Eval still moving	Each change shifts the score	Keep prompting
Eval flat for 3-4 changes, task is narrow, a few thousand labelled runs in hand	Plateau, and you have the ingredients	Fine-tune
Eval flat, but the task is broad or the labelled runs aren't there	Plateau without the ingredients	Build the dataset first

Where prompting hits its ceiling

Prompting always runs out of road eventually, and the symptoms show up in predictable order:

The plateau. Each prompt tweak moves the eval less.
Whack-a-mole. The instruction that fixes today's failure mode causes tomorrow's.
Prompt bloat. The system prompt keeps growing, slows the model down, and gets partially ignored.
Paying rent. You're re-sending the same correction on every single call.

At that point, prompting stops being cheap. Engineering hours are the priciest line item in most LLM pipelines, and it's easy to burn a month nudging a prompt that plateaued weeks ago.

When to fine-tune

Fine-tuning updates a model's weights on examples of the behaviour you want until the model does it by default, without being told every time. It's worth trying when these conditions are met:

The task is narrow. Fine-tuning sharpens a model on one job, not general capability.
You have real data. A few thousand labelled runs, ideally sampled from production traffic rather than synthetic examples.
Cost or latency matters. A smaller, fine-tuned model can outperform a much larger general-purpose one carrying a bloated prompt, at a fraction of the inference cost.

Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own. Get started.

Open-weights LLMs vs frontier APIs: when to rent, when to own

Tyler Edwards — Mon, 07 Sep 2026 14:54:01 +0000

Most AI products start on a frontier API and stay there until the bill, the latency or legal forces a rethink. Here's the practical case for when a self-hosted open-weights model is the better call, and the cost-crossover logic behind it.

Originally published at overmindlab.ai.

Own your intelligence, or rent the API?

Developers start building AI products by tokenmaxxing. Calling frontier APIs and paying per token.

Then the bill shows up. Or the agent feels slow. Or legal asks where the data goes. Suddenly "just use the API" might not be the right strategy.

Underneath every AI product is one choice. Rent a frontier API, or own a model. Renting means closed weights, so someone else trains it, serves it, and sets the price. Owning means picking the model, post-training it and running self-hosted or third-party inference.

This piece is the practical guide to open vs closed models.

The trade-offs

Dimension	Closed models (frontier API)	Open models (open-weights)
Capability	Best capability out of the box	Strong, and tunable to your task
Infrastructure	None needed	Runs on infrastructure you own or buy
Cost	Costs more with every token	Drops hard in cost after the upfront work
Latency	Adds a network trip and a shared queue	Serves on dedicated hardware, with lower time to first token
Data	Sends prompts outside your network	Stays on your own servers or in your own cloud
Lock-in	Locks you to the vendor's pricing and retirement schedule	Leaves you your stack and your weights
Switching	Swap with one line of code	Needs a retrain or a re-quantise to swap

LLM inference cost: when open-weights beat the API

Renting feels cheap at the start, especially when showered with credits.

Then usage grows. The bill grows and a painful chunk of that spend is frontier prices for work a smaller specialised model could do.

Owning the model changes that. You pay upfront for a LoRA / QLoRA fine-tune, inference setup and some evals. After that, per-token cost is a fraction of the API. Somewhere on the volume axis the lines cross. That crossover point is where a self-hosted open-weights LLM stops being a science project and starts being the cheaper path.

This is not theory. A LoRA fine-tune of a 7B-8B open-weights model on a few thousand trajectories often costs a few hundred dollars.

Beyond cost: latency, data residency, and vendor lock-in

Your agent feels sluggish

A small model you serve has no hop to someone else's cluster and fewer parameters per forward pass. If your AI agent makes dozens of tool calls per task, 200ms vs 2s is not a benchmark flex, it is whether the product feels alive. Lower time to first token (TTFT) compounds across the whole agent loop. Users notice the speed. They leave when it isn't there.

Your data cannot leave the building

When you rent, prompts and completions leave your perimeter. For fintech, healthcare, or anyone under data-residency, GDPR, or HIPAA rules, that is not a preference. It is a hard no. A private LLM, self-hosted or VPC-hosted open-weights, is not an optimisation. It is how you ship at all.

The vendor moved the goalposts

Renting means their roadmap, their rate limits, their deprecation calendar. When they sunset the model your agent was tuned around, you re-tune on their schedule. Owning means your inference stack and your weights. Still a dependency, but one you control.

When to use open-weights vs a frontier API

Own (open-weights, self-hosted) when:

volume is high and climbing
the task is narrow
latency is a product problem
data cannot leave your perimeter
the model is your edge
you know what you are building

Rent (frontier API) when:

volume is low or spiky
the task is broad and messy
latency is fine
data has no residency limits
raw frontier capability matters most
you are still finding out

FAQ: fine-tuning open-weights vs using a frontier API

Do I need a team of ML engineers to fine-tune an open-weights LLM?

No. LoRA / QLoRA plus a managed inference host (Fireworks, Together, Modal, or vLLM on your own GPUs) is within reach of one strong engineer. The hard part is not GPUs. It is good training data and honest evals.

Which open-weights base model should I start with?

Pick a family that fits your licence and task, such as Llama, Qwen, Mistral or DeepSeek. Start small. A 7B-8B model, fine-tuned on your task, handles more production work than most teams expect.

What if a better frontier model ships next month?

If you rent, you get the upgrade (until the API changes). If you own, you decide whether to re-tune. Here is the part people miss. A small model trained on your production trajectories often still wins on your cost, latency, and task metrics. That is the scoreboard that pays the bill, not the leaderboard screenshot.

Is my data really at risk on a frontier API?

For most consumer SaaS a zero-retention, no-training contract covers you. For regulated data, get an explicit guarantee in writing or run a private, self-hosted LLM. Do not guess.

Fine-tuning on production data is the real moat

Owning only works if your model is actually better at your task than the rented one. Downloading open-weights does not get you there. Fine-tuning on your production data does. Traces, tool-call trajectories, preference pairs, real evals from real usage.

Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own. Get started.

How do you turn traces into a training dataset?

Tyler Edwards — Fri, 04 Sep 2026 10:17:42 +0000

If you're running agents in production, you already have a training dataset sitting in your traces and most teams never look. Here's the four-stage pipeline for turning that raw telemetry into something you can actually fine-tune on.

Originally published at overmindlab.ai.

Every agentic system can create its own training data.

Every agentic AI system you run is quietly writing its own training set, and most teams never use it.

When an agent runs, it emits a stream of inputs and outputs: the prompt, the tool calls, the reasoning and the final answer. That stream is a trace, and once you standardise how you capture it (the OpenTelemetry GenAI semantic conventions is the default schema here) you have raw telemetry. A training dataset is narrower: a curated set of runs, scored against a spec, designed for fine-tuning jobs.

Building a dataset like this is a four-stage pipeline.

Four stages, then a weekly rebuild.

Here's a worked example, using a refund agent, to show what each stage actually involves.

Step 1: Capture every run as a trace

If your agent is instrumented at all, you already have this part. Every run generates a trace: the goal it was given, every model call, every tool invocation, every observation from the environment, and the eventual outcome. That trace is the raw material everything downstream depends on.

The industry is converging on a shared vocabulary for this. OpenTelemetry's GenAI working group has been building standard span attributes for LLM calls since 2024, covering model name, token counts, latency, and tool execution as first-class fields rather than something each team invents from scratch. That matters more than it sounds like it should, because a trace schema you have to redesign every time you switch observability vendors is a trace schema nobody trusts enough to build a dataset on top of.

This is also where a tool like Overmind tends to sit. Its SDK wraps the model call interface directly, so a single overmind.init() call captures every LLM invocation across OpenAI, Anthropic, Google Gemini and Agno, logging inputs, outputs, latency, token counts and errors without extra plumbing on your side. The point isn't the SDK itself. It's that capture has to be automatic and total, or the sampling and labelling stages downstream never get the raw material they need.

Step 2: Sample, because you cannot label everything

You should not try to label every run. Most of what an agent does in a given week is unremarkable, and reviewing all of it teaches a labelling team nothing it didn't already know. The job at this stage is picking which runs are worth a human's attention.

Three strategies cover most cases:

Sampling strategy	What it gives you	When to use it
Random	An honest baseline of what the agent does on an ordinary day	Every cycle, as the control slice you compare everything else against
Stratified	Deliberate coverage of cases you already care about, such as a specific refund reason, a customer tier, or a tool that keeps timing out	When a known segment matters more than the average run
Failure-weighted	The most signal per run, because the runs that went wrong carry the most information	When you have an error flag or a satisfaction score to sort on

Out of the refund agent's 50,000 weekly runs, a reasonable pull is around 500: a random slice plus every run that hit an error or a low satisfaction score. Whatever strategy you pick, run it on a cadence, weekly to start, rather than treating it as a one-off export you remember to do after something breaks in production.

Step 3: Label against what good actually looks like

This is the step almost everyone skips, and it's the one that decides whether the dataset is worth anything.

The instinct is to label by gut: skim a run, decide it feels fine, move to the next one. That works at a hundred runs and falls apart completely at ten thousand, mostly because "feels fine" means something slightly different to every reviewer and drifts over time even for the same reviewer. The alternative is writing down, explicitly, what good looks like, and scoring every sampled run against that spec.

You're not starting from nothing here. Your agent's own codebase already encodes most of what it's supposed to do: the outputs it should produce, the tools it's allowed to call, the checks it runs before acting, the paths it should and shouldn't take. Read the code first and most of the labelling criteria fall directly out of it.

For the refund agent, that read produces a checklist along these lines:

Criterion	What it requires	Example violation
Refund within terms	Amount within the order value and the 30-day window	Refunded a 90-day-old order
No invented terms	Only cites the published refund terms	Quoted a returns rule that does not exist
Escalate disputes	Hands chargebacks off to a human	Auto-refunded a disputed charge

Automating this scoring step is where LLM-as-judge techniques have become the practical default, since manually reviewing thousands of runs against a rubric doesn't scale. The approach has real limits worth knowing before you lean on it: research comparing LLM judges against human-labelled relevance data found strong rank correlation but only fair agreement on exact labels, and accuracy drops sharply on the more nuanced categories rather than the easy pass/fail calls. A survey of LLM-as-judge methods notes it emerged specifically because manually assessing helpfulness in training data got too expensive to do at scale by hand, which is exactly the tradeoff a labelling pipeline is making. Overmind runs this scoring as evaluators against a rubric you write. Six evaluator kinds cover it, from a deterministic check to an LLM judge, and every run is scored against a baseline before the result counts.

The runs that fail a criterion, whatever the customer clicked afterward, are the highest-value training data you have. The labels encode your judgment about what good looks like, and that judgment is the one thing no generic tool can supply for you. Do the first pass yourself for the first month. It's the fastest way to find out what your criteria actually are, as opposed to what you assumed they were when you wrote the checklist.

Step 4: Build the training set

Labelled runs aren't training examples yet. The last step shapes them for whatever method you're about to run, and the method decides the shape.

	Supervised fine-tuning (SFT)	Reinforcement learning (RL)
Which runs you keep	Runs that passed every criterion	Sampled runs with their criterion scores attached
What a row holds	An input-output pair of the behaviour you want repeated	The run plus the scores, used as a reward signal
What the model learns	To imitate a fixed set of good examples	To produce runs that score higher
What labelling has to produce	A pass/fail verdict per criterion	A usable score per criterion, not just pass/fail

Either way, the format needs to be consistent and machine-readable, with the goal and outcome attached to every run.

This is also the stage where the case for smaller, specialised models gets concrete. Overmind's own research argues that every model invocation inside an agentic workflow is a natural source of high-quality training data, precisely because the prompts are narrow and well-defined and the pass/fail signal is clean, unlike open-ended chat data. A team that instruments its model calls, clusters the resulting patterns, and fine-tunes a specialist model on them ends up with a system that improves with every production run instead of one that's frozen at whatever a general-purpose model happened to learn at pretraining time.

Which method you feed, supervised fine-tuning or RL, is its own separate decision. The dataset from steps 1 through 3 is what feeds either one.

Why this is the hard part

None of these four steps is exotic on its own. What makes the whole thing difficult is that it never stops. Production keeps producing new traces, your criteria keep getting sharper as you find edge cases you didn't anticipate, and the dataset has to be rebuilt against what users actually did this week, not what they did last quarter. The dotted feedback line on that diagram at the top is the entire job.

It's also exactly the gap most observability tooling leaves open. Datadog's own writeup on GenAI tracing gets at this directly: teams are encouraged to promote interesting production traces into curated, version-controlled "golden" datasets and layer evaluation metadata on top, which is essentially this same capture-to-label pipeline described from the observability side. An observability platform hands you the traces and stops there. Stitching production traces to behavioural training data to a deployed, improved model is work that mostly happens in spreadsheets and one-off scripts today, and it's the specific gap platforms like Overmind are built to close, running the optimise-evaluate-accept loop end to end instead of leaving it as a manual export.

Owning that labelled dataset matters because differentiation in agentic AI increasingly lives in proprietary behavioural data, not in which foundation model you call. Anyone can capture traces. The labelled dataset built from them, tuned to your own definition of correct, is the thing actually worth owning.

Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own. Get started.

What are the different types of fine-tuning?

Tyler Edwards — Tue, 01 Sep 2026 15:56:43 +0000

Picking a fine-tuning method usually comes down to one thing, whether you can write the ideal output yourself or can only judge one when you see it. Here's a practitioner rundown of SFT, RFT with GRPO, distillation and LoRA/QLoRA, and how to choose between them for your own task.

Originally published at overmindlab.ai.

Supervised fine-tuning (SFT)

SFT is where most people start, and most people should. You give the model input/output pairs and train it until it reproduces that pattern. It's teaching by worked example: here's the prompt, here's the answer, do more of this.

You'll typically need a few thousand examples for a narrow task. The catch is that SFT only works if you know what "good" looks like.

LoRA and QLoRA

LoRA doesn't compete with SFT or RFT.

Full fine-tuning updates every weight in the model. That's slow, and it eats memory. LoRA freezes the base model and trains a small set of adapter weights on top, which gets you most of the benefit for a fraction of the compute. QLoRA pushes this further by quantising the frozen base model down to 4-bit before training the adapter. That's the reason a 7B model now fits on one consumer GPU instead of a rack of them.

Tools worth knowing:

Unsloth is open source and runs LoRA/QLoRA 2 to 5x faster with up to 80% less VRAM, using custom kernels and 4-bit quantisation. It's become the default starting point for solo builders.

Tinker, from Thinking Machines Lab, is a managed LoRA API. It handles GPU scheduling and checkpointing but still lets you control the actual training algorithm. It launched in October 2025 and supports both SFT and RFT.

Overmind is a model training platform. It runs LoRA fine-tuning on your agent traces.

Reinforcement fine-tuning (RFT)

This used to just be called "RL," or RLHF if you wanted to impress someone.

SFT needs you to write the correct answer. RFT only needs you to score it. You define a reward, hand it the model's output, and the model learns to produce completions that score higher across many attempts.

Reach for this when good is easier to recognise than to demonstrate. I can't write the perfect customer support reply off the top of my head, but I can tell you whether one resolved the ticket, stayed on brand, and didn't promise something we don't offer. RFT is also the natural fit for agents: tool calls, multi-step retrieval, anything where success depends on the whole trajectory and not any single message.

GRPO (Group Relative Policy Optimization) is the algorithm doing most of the RFT work right now. It came out of DeepSeekMath and got famous when it trained DeepSeek-R1. The trick is sampling a group of responses to the same prompt, then scoring each one against the group average instead of some absolute scale. That kills the need for a separate critic model, which is what made older methods like PPO so expensive to run. DAPO and Dr.GRPO are newer variants that patch specific instabilities GRPO runs into on long chain-of-thought outputs.

Distillation

Distillation is copying a big model's behaviour into a small one. Run your inputs through the big model, collect what it says, then train a smaller model to say the same things. You lose a little quality and gain a lot of speed, plus a much smaller bill.

This is helpful when a frontier model already nails your task but is too slow or too expensive to run at real volume. It's also a big part of why small language models have taken off this past year. Teams distill a specialist model instead of shipping the 400B-parameter original into production.

Picking one

Once you know your task, the decision is mostly mechanical.

Method	What you must supply	When to pick it	Data volume	Relative cost
Supervised fine-tuning (SFT)	Input/output pairs you already know are right	You can write the ideal output yourself	A few thousand examples for a narrow task	Low
Reinforcement fine-tuning (RFT, usually GRPO)	A reward function or judge that can score an attempt	You can recognise a good output but can't write one, or success spans a whole agent trajectory	Prompts plus a scorer, no written answers needed	Highest, you sample many completions per prompt
Distillation	A teacher model that already does the task, plus your inputs	A frontier model nails the task but is too slow or too expensive at volume	As many teacher outputs as you can afford to generate	Medium, teacher inference then a cheap SFT run
LoRA / QLoRA	Whichever method above, plus one GPU	Almost always, unless you specifically need full-weight training	Same as the method it wraps	Lowest, a 7B run over a few thousand examples often costs a few hundred dollars

The same tree in words.

Your situation	Method
A big model already does the job, but it's too slow or too costly	Distillation
You can write the ideal output	Supervised fine-tuning
You can only judge the output, not write it	Reinforcement fine-tuning with GRPO
Any of the three, on one GPU	Run it through LoRA or QLoRA

Data is still the hard part

Every method above needs the same raw material underneath. SFT needs labelled pairs. RFT needs a reward or a judge that can rank attempts. Distillation needs a teacher and your inputs. Pick the wrong method and you lose some efficiency. Show up with bad data and none of them work. No algorithm fixes that for you.

Fine-tuning has gotten cheap. A LoRA run on a 7B model over a few thousand examples often costs a few hundred dollars. But cheap training doesn't fix bad data, and none of these tools hand you a set of real runs labelled, or ranked, against your own definition of good. That's a separate problem, and it's the one that eats the time. We cover it in how do you turn traces into a training dataset and how to train your agent.

FAQ

Is LoRA a replacement for fine-tuning?

No. LoRA is a way to run fine-tuning or RFT cheaper, not a different goal. You still pick SFT or RFT first, then decide whether to run it through LoRA.

When should I use reinforcement fine-tuning instead of SFT?

When you can judge a good output but can't write one yourself, or you're training an agent where success depends on a whole multi-step trajectory rather than one response.

Is distillation cheaper than fine-tuning a small model from scratch?

Usually, if a large model already performs well on your task. You're paying for inference calls to the teacher model instead of building the target behaviour from labelled data by hand.

What is GRPO and why does everyone use it now?

Group Relative Policy Optimization scores a batch of responses against each other instead of an absolute baseline, which removes the need for a separate critic model. It's cheaper to run than older RL methods like PPO, which is most of why it's become the default for RFT.

Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own. Get started.

"So, you have observability. Now what?"

Tyler Edwards — Tue, 01 Sep 2026 13:46:47 +0000

By Tyler Edwards, co-founder and CEO at Overmind. If you run an agent in production, odds are you already have traces piling up in Langfuse or LangSmith. This piece is about the part nobody instruments: what those traces are for.

Observing your agent is table stakes.

In the not-so-distant past there was a brief period when picking an LLM meant picking a vendor. You wrote against their SDK, you paid their bill, and when their servers fell over on a Tuesday afternoon so did yours. Switching was easy enough but did require Ctrl+F'ing your way through the codebase - or asking your coding agent of choice to perform this insultingly simple task for you instead.

Then on the sixth day came the LLM gateway! One endpoint, dozens of models behind it, swap them in and out faster than the UK swaps Prime Ministers (and for international readers here is the context on that one). The gateway is, on its own terms, a genuinely nice piece of engineering, and there is now a number of teams shipping model routers with various bells and whistles: Portkey, LiteLLM, OpenRouter, TrueFoundry, Kong, Cloudflare.

Sitting alongside the gateway, usually paid for separately and instrumented by a different team three weeks later, is the LLM observability stack. Langfuse, Helicone, Arize Phoenix, Braintrust, LangSmith, Datadog now too. Pick your favourite, they mostly do the same thing. They capture every prompt and every completion, stitch them into traces, and present you with dashboards.

Between the two, your AI product is being watched harder than a Premier League VAR decision.

What Is Observability?

LiteLLM shipped to solve a pretty annoying problem: every provider's API was almost-but-not-quite the same. Portkey and OpenRouter commercialised the idea, TrueFoundry and Bifrost followed, and the existing API-gateway crowd (Kong, Cloudflare) extended their products to cover LLMs the moment it became clear LLMs were here to stay whether anyone liked it or not.

The observability side has a similar shape. Langfuse, Helicone, Arize Phoenix, Braintrust all turned up in 2023 to build native LLM observability from scratch. The companies that already did ML observability - Fiddler, WhyLabs, Galileo - pivoted with the arrival of ChatGPT and have spent the last three years insisting they were actually always about LLMs really anyway.

By 2026 both categories had matured enough to start being acquired. Portkey went to Palo Alto Networks in April for somewhere around $120-140M. Helicone went to Mintlify the month before. Langfuse was rolled into ClickHouse's Series D in January. Cisco picked up Galileo and folded it into Splunk.

All Data, No Insight

So you now have hundreds of thousands, possibly millions, of traces composting nicely in a database somewhere. Every prompt, every completion, every tool call, every retry. The gateway has them as individual requests. The observability tool has them stitched into trajectories. Between the two, your AI product is the most thoroughly documented thing in your engineering org.

The gateway, by design, treats each call as a self-contained unit - the only way to route, cache, fall back and bill at scale. It doesn't have a concept of the trajectory because it doesn't need one. As far as the gateway is concerned, each call was a separate transaction.

The observability tool does have a concept of the trajectory. It will stitch the spans together, draw you a lovely waterfall diagram, and tell you exactly how the agent arrived at whatever it arrived at. What it won't tell you is whether what it arrived at was any good - because good is a judgement, and the observability tool is, as its name suggests, there to observe.

Every call along the way can pass every check either tool runs. A 200 from the provider, the right shape of completion, no PII, latency inside SLO, and cost on budget. The agent can still invent a company policy, ignored an instruction from two steps earlier, and make three perfectly correct-looking calls on its way to deleting a production database - and then gleefully reported its action back to you in the same way a dog might excitedly drop a dead bird at your feet.

Where Observability Fails

The observability vendors aren't naive about any of this. Most of them ship some version of trace review - LangSmith has annotation queues, Braintrust has human review alongside LLM-as-judge scoring, Langfuse has dataset workflows, Arize Phoenix ships evals out of the box. Teams use them. Engineers were spending afternoons grading traces and writing scoring functions and tagging failure modes, and the tools make this less painful than it would otherwise be.

What you get at the end of all that effort is a graded slice of traces. A few thousand examples marked good or bad, often with structured scores attached, sometimes with notes. This is useful - for catching regressions, for spotting drift, for arguing with your model provider about why their new release broke something. It is not, on its own, a training or eval dataset, and it is several steps short of a model that has actually improved.

LLMs are intelligently dumb, they will learn an approximation of what you teach them, as such you need to teach them exactly what you want them to do. A graded trace is an observation. A dataset is a curated, balanced, deduplicated set of examples organised around the behaviours you want the model to learn. The difference between them is curation logic, clustering, sampling strategy, and a clear view of what the model is supposed to get better at - none of which the observability tool produces.

And even once you have the dataset, the pipeline from there to a better agent has its own shape. You need to fine-tune a model or train a LORA adapter, eval-gate the result against a held-out slice, deploy the new weights behind your routing layer, and watch the next week of traces to see whether the change actually moved the agent's behaviour or just shuffled the failure modes around. Each of those steps is its own piece of infrastructure.

Bridging The Gap

Some of the vendors have started shipping bolt-ons aimed at closing parts of this. Portkey's Autonomous Fine-Tuning wires gateway logs into a provider's training API and hands off the dataset. LangChain's Engine edits prompts and code from production traces. Braintrust's Loop generates evaluators and datasets from natural-language descriptions of failure modes. Each is a real move in the right direction, and each stops one step short of the model itself. The dataset gets handed off and the trail goes cold.

This is the gap between what's possible and what's shipped. The technology to take a production trace, score it, cluster it, curate it into a dataset, fine-tune a model on it, eval-gate the result and deploy it back through your routing layer - all of that exists, in pieces, sold by different companies with different pricing pages. What doesn't exist yet is a single layer that does the whole loop. You can probably see where i’m going with this but lets not jump ahead.

The Opportunity

Cursor is the example everyone reaches for, and for good reason. They trained their Composer model on production agent traces using online reinforcement learning, and it now beats the frontier at its size on cost and performance. The model that ships to Cursor users is the model trained on what Cursor users actually do. The labelling encodes the judgement. The fine-tune encodes the labelling. The product encodes the fine-tune. It compounds to create a very powerful product.

Doing this used to require a research team and the patience of an academic institution. It doesn't anymore. Open-weights bases - Llama, Qwen, Mistral, DeepSeek - are now genuinely competitive on scoped tasks. A LoRA fine-tune on a few thousand labelled trajectories runs for low hundreds of dollars on Modal or Together or your own GPUs if you're feeling brave. Unsloth and Tinker have made the mechanics broadly accessible to anyone willing to spend a weekend with them, with the small caveat that you do still currently have to be a bit of a genius to use them properly.

What this adds up to is something most teams haven't fully reckoned with yet. The data is sitting in the gateway. The traces are sitting in the observability tool. The curation can be automated. The fine-tuning is cheap. The deployment is a config change. The only thing standing between any reasonably-sized team and a continuously-improving fleet of specialist models trained on their own production traffic is whether anyone has joined the pieces up.

Overmind

And we arrive at the aforementioned point. We built Overmind on the same bet we wrote about in Think Smaller: the teams who win the next phase of agentic AI will be the ones who own their models, not the ones who rent the most powerful ones, and certainly not the ones who own the most expensive plumbing in front of those rented models.

Overmind is the layer that turns the data your gateway and observability stack are already collecting into models that improve because of it. It plugs into the trace store you've already got, runs the curation, builds the dataset, fine-tunes the model, eval-gates the result and deploys the new weights behind your existing routing layer. The gateway keeps doing what it's good at. The observability stack keeps doing what it's good at. Overmind does the bit in between that nobody else is doing.

Keep your gateway. Keep your observability stack. Just stop pretending they're enough on their own.

FAQ

What's the difference between an LLM gateway and LLM observability? Gateways route and rate-limit calls between your app and model providers. Observability tools record traces and let you inspect them. Neither curates the data or trains models.

Why isn't observability enough for production agents? Observability shows you what happened. It doesn't tell you whether the trajectory was good, and it doesn't turn that judgement into model improvements. That gap is the bottleneck.

What does Overmind do that observability tools don't? Overmind sits on top of your trace store, curates the data into training-grade datasets, fine-tunes specialist models, evaluates them, and deploys the new weights behind your existing routing layer.

Tyler Edwards is Co-Founder and CEO of Overmind. He writes about agent infrastructure, fine-tuning, and what it takes to ship AI that actually improves in production. Connect on LinkedIn.

Stay in-the-loop

Join our newsletter to follow as we build supervision for super-intelligence.

If your traces should end as a model you own rather than a dashboard, that loop is what Overmind does: audited datasets from production traces, fine-tuning, a benchmark against the model you run today, and serving. There is a live Langfuse import if your traces already live there.