DEV Community: Robert Imbeault

We Got 91% With Kimi. Repeatable 91% Is the Hard Part.

Robert Imbeault — Thu, 24 Sep 2026 16:29:49 +0000

A peak score tells you what an AI system can do. Production cares about what it can do again tomorrow.

We recently got 91% on Terminal-Bench 2.1 using Kimi K3.

Tasks solved: 81/89
Total model cost: $28.72.

This is the part where I am apparently supposed to put the number in very large type, add a trophy emoji, and move on.

Unfortunately, we ran the experiment more than once.

And Kimi does not score 91% every time.

That is where the story gets much more interesting.

91% is a Result. It Is Not a Personality Trait.

It would be very convenient to say: Kimi K3 is a 91% Terminal-Bench model.

It would also be wrong.

We have seen meaningful variance between runs with Kimi and other open-weight models. Sometimes the result is exceptional. Other times it comes down materially. The 91% run was also deliberately different from an official Terminal-Bench submission.
We ran all 89 tasks once. An official submission requires at least five attempts per task. So this was an experimental run, not a judged leaderboard submission. That qualification makes the headline slightly less exciting. It also makes the experiment considerably more useful.

Because the question we actually care about is not “How high can the number go?”

It’s “How much of that performance can we make repeatable?”

Peak Performance and Production Performance Are Different Things

Imagine an agent scores:
91% on Monday.
87% on Tuesday.
83% on Wednesday.
What is its capability?

All three numbers tell you something.

The 91% tells you what the system is capable of under a strong trajectory. The lower results tell you what you might actually have to depend on.

And if I am putting that system into a production workflow, I become considerably more interested in the second number.

Nobody wants an agent whose résumé says “occasionally brilliant.”

Open Models Are Getting Uncomfortably Good

This is what caught our attention.

Even when Kimi comes down from that particularly strong run, the performance range we are seeing from open-weight models inside a well-engineered agent system is becoming surprisingly competitive.

For years, AI performance was mostly discussed as a model-selection problem.

Want better results? Use the better model.

Everything around the model was plumbing. Necessary plumbing, perhaps, but definitely an expensive plumbing.

I think that assumption is starting to break.

As more models move into broadly useful capability ranges, the system around them starts determining how much of that capability actually reaches the user.

And small architectural decisions begin to matter much more than they look like they should.

We Gave Kimi Less to Do

One of the more useful lessons from this run was that better agent design does not always mean giving the model more.

Sometimes it means taking things away.

R-CLI did not use Kimi K3 for everything. Kimi handled the primary reasoning, while delegated subagent work could be routed to DeepSeek V4 Flash.

We also removed 10 tools from the schema because they either conflicted with benchmark constraints or had measured negative effects on performance.

That meant a smaller tool surface, less prompt overhead, and fewer irrelevant decisions.

There is a natural temptation when building agents to keep giving them capabilities.

Another tool.
Another integration.
Another 40,000 tokens of context, just in case.

Eventually your extremely intelligent model is standing in the middle of Home Depot holding 900 tools and wondering which screwdriver you wanted.

Capability is useful.
Choice has a cost.

Context Windows Are Not Storage Units

The same principle applies to context.

Large context windows are useful. That does not mean the goal is to fill them.

In the 91% run, 71.8% of input tokens were served from cache, while median context occupancy was only about 1.5% of the model’s available window. That number is worth sitting with.

The model had a large context window. We mostly did not use it, because the objective is not to make sure the model has seen everything. The objective is to make sure it sees what matters now.
There is a subtle but important distinction between giving a model enough information and making it read the entire filing cabinet before answering every question.

One helps. The other is technically also a strategy.

The Model Did Not Change

None of these decisions changed Kimi’s weights.

We did not make the underlying model smarter. We changed the environment in which it operated.

The harness controlled what context arrived, which tools were available, when work could be delegated, and how much the system should spend. It also used turn timers, budget-aware spend nudges, and a cache-first approach to context. Those choices look small individually. Across hundreds of agent steps, they compound.

A capable model can still underperform if the system keeps giving it irrelevant context or an unnecessarily complicated tool surface.

It can waste enormous amounts of computation following bad trajectories.

It can fail once and never recover. The underlying intelligence may be there.

The harness determines how much of it survives contact with the task.

$28.72 Is Interesting

The full 89-task sweep cost $28.72 which is an impressive number. It is also one I would handle with oven mitts.

Dollar comparisons between AI systems are messy.
Model prices differ.
Caching policies differ.
Commercial agreements differ.

Subscription economics can make apparently simple comparisons considerably less simple. Token consumption gives another useful signal.

Our run used roughly 0.24 million tokens per trial. Several configurations included in our analysis consumed more than one million tokens per trial, while the highest-scoring submission in that comparison used about 2.65 million. That does not make token count a perfect measure either.

But if two systems are attempting similar work and one requires dramatically more model activity to get there, something meaningful is happening in the architecture.

Efficiency is not just model pricing.

It is also how intelligently the system uses the model.

Reliability Is an Engineering Problem

Here is the encouraging part about variance:
It is frustrating.

But some of it creates things engineers can work on.

If one trajectory succeeds and another fails, we can inspect why.

Maybe routing reduces the variance.
Maybe verification catches a weak trajectory before it becomes a failure.
Maybe retry behavior needs improvement.
Maybe a specific behavior can be improved through post-training.
Maybe a different model should handle a particular stage of the task.

The source experiment makes exactly that broader point: inconsistency is not necessarily a model-only problem; the surrounding system can influence how repeatable high performance becomes.

Which means the goal should not simply be:
Make the highest number possible.

It should be:

Make good performance boringly repeatable.

That is a much harder engineering problem.

It is also the one production systems eventually have to solve.

A Benchmark Score Is a Distribution Wearing a Name Tag

This is also why I think AI benchmarks need to mature. A single aggregate number is wonderfully easy to understand. It is also very good at hiding everything underneath it.

Variance matters.
Failures matter.
Retries matter.
Cost matters.

A 91% peak and an 85% repeatable system can tell you very different things.

Neither number is automatically more “real,” because they answer different questions.

One tells you about the ceiling.

The other tells you about reliability.

Production needs both.

The Interesting Competition May Be Moving Above the Model

There will always be another model.
GPT.
Claude.
Gemini.
Kimi.
DeepSeek.

Then six months later something with a name that sounds like either an AI lab or an obscure Pokémon comes out.

The frontier will keep moving, but if several models increasingly occupy the same broad capability range, then the scarce part of the system starts to change.

The advantage shifts toward the teams that can make those models more dependable without wasting enormous amounts of computation getting there.

It moves toward the people who understand how to build systems around intelligence rather than simply purchase access to it.

That is the bet we are making at Backboard.
The 91% run is exciting.
The cost is exciting.
But neither is really the thesis.

The thesis is that open-weight models are getting better quickly, while the difference between an average harness and a great one may only be starting to matter more.

And if that is true, the next major advantage in AI may not come from owning the smartest model.

It may come from making strong performance so repeatable that eventually nobody finds it interesting anymore.

This is a conversational remix of an article I published on Backboard’s blog. Read the original deep dive here.

Same Claude. Different Harness. Very Different Result.

Robert Imbeault — Fri, 18 Sep 2026 15:20:23 +0000

Claude didn’t get smarter. We changed everything around it. Somehow, 6.5 points appeared between them.

We beat Claude Code with Claude.

Which is a slightly ridiculous sentence, but it is also a useful one.

We ran Backboard CLI on Terminal-Bench 2.1 using Claude Opus 4.8 through Amazon Bedrock. Claude Code, using the same model, had a published score of 78.9%.

Our submitted result was 85.4% ± 0.8%.

Same underlying model.
Different system around it.
A 6.5-point difference.

And that is much more interesting to me than saying we found a smarter model.

Because we didn’t.

Claude Did Not Wake Up Smarter That Morning

For the last few years, AI performance has mostly been discussed as a model problem.

Want better results? Use the better model. And obviously, models matter. A lot. But once models become sufficiently capable, another question starts becoming more important:
How much of that capability can your system actually get out of them?

A model working inside an agent does not operate in a vacuum.
It receives context.
It sees tools.
It decides what to do next.
It fails.
It recovers.
It maintains state.
It spends tokens while doing all of this.

The model may be the engine, but there is quite a lot of car wrapped around it.

And occasionally someone remembers that transmissions matter too.

The Harness Is Not Just Plumbing

It is tempting to describe everything around the model as plumbing.
Context management. Tool interfaces. Planning. State. Recovery. Verification.

Necessary, certainly.
Exciting enough for a keynote slide?
Apparently less so.

But those decisions compound.

Give a capable model too much irrelevant context, and you make its job harder.

Give it tools it does not need, and you create more ways to make the wrong choice.

Handle failure poorly and one bad step can ruin an otherwise good trajectory.

Burn too many tokens getting there, and you may technically solve the task while also solving your CFO’s problem of having too much money.

The model did not suddenly become more intelligent because Backboard called it.

The system around the model got more of the existing intelligence into the result.

That distinction is the reason I care about the benchmark.

The Cost Result Was Almost More Interesting

Performance was only half of it.

Our full submitted Terminal-Bench run cost $280.72. At the time of the original article, the current verified leader had scored 83.8% at a reported cost of $552.67.

So our submitted result came in higher, while the reported run cost was roughly 49% lower. That changes the question.

It is easy to ask:
Which model are you using?

It is harder, and much more useful, to ask:
What can you get that model to accomplish reliably, and what does it cost to get there?

Those are different optimization problems.

One produces model comparison charts. The other produces systems you can afford to run.

This Is Why We Benchmark

I have mixed feelings about AI leaderboards.

They are useful and extremely effective at turning otherwise sensible adults into people refreshing a webpage to see whether a decimal moved.

For us, the useful part is not the trophy. It is the failure data.
The submitted evaluation covered 89 tasks, with five attempts per task, for 445 trials under the same agent configuration. We included the failures and errors. The aggregate result was 85.4%.

That was the number we were comfortable submitting.

Every failed task tells us something.

Sometimes the model made a bad decision. Sometimes context was poor.
Sometimes the harness allowed an unproductive trajectory to continue too long.
Sometimes an implementation change that seemed clever turned out to be very clever at making the benchmark worse.

Useful information, all of it.

A benchmark is valuable when it tells your engineers where the system breaks. The leaderboard is what happens afterward.

Same Model Is a Surprisingly Useful Experiment

Comparing AI systems gets messy fast. Change the model, provider, prompt, tools, or context strategy and you’ve changed the experiment.

That’s why I find the Claude comparison interesting.

Same Claude Opus 4.8 on both sides. 6.5 points apart.

I don’t think there’s one clever trick behind that gap. It’s probably a lot of fairly boring engineering choices adding up.

Which, unfortunately, is usually how software works.

Models Will Keep Moving

We care about this for a bigger reason too.

I don’t think companies should have to rebuild their AI stack every time a new model becomes the favorite.

Claude might be the right fit today. Tomorrow it could be something else. Six months from now, it could be a model nobody has heard of yet.

The models will keep changing. That’s normal.

The infrastructure around them shouldn’t have to.

Your context, memory, evaluation, application logic, and the lessons you’ve learned about turning model capability into useful work should be able to stick around.

That’s where I think a lot of the durable value in AI infrastructure will live.

The Model Is a Component, Not the Whole Product

This benchmark does not prove that models do not matter.

That would be a strange conclusion from an experiment powered by Claude Opus 4.8.

It proves something narrower.

Choosing a strong model is not the end of the engineering problem.

There is still a substantial gap between having access to intelligence and building a system that uses it well.

The surrounding architecture determines what context reaches the model, what actions it can take, how failures are handled, and how much the whole thing costs when it finally succeeds.

Those decisions are easy to dismiss individually.

Collectively, they can move a benchmark by 6.5 points.

I Think the Competition Is Moving Up the Stack

Foundation models will keep getting better. They’ll keep changing too.

If several models can solve the same class of problems, simply having access to a good one becomes less of a differentiator.

The harder questions move up the stack:

Can you make it reliable?
Can you use context well?
Can you recover when things go wrong?
Can you tell whether an optimization actually helped?
Can you do all of that without treating tokens like confetti?

That’s the engineering layer we’re betting on at Backboard.

Our Terminal-Bench result is one piece of evidence, not proof of everything. It’s simply a useful controlled comparison:

The model stayed the same. The system changed. The outcome changed with it.

That’s the part worth paying attention to.

This is a conversational remix of an article I published on Backboard’s blog. Read the original deep dive here.

Your AI Has the Memory of a Goldfish. That’s an Architecture Choice.

Robert Imbeault — Tue, 15 Sep 2026 18:00:00 +0000

Most AI assistants are extremely capable and remarkably forgetful.

Give one a complicated problem, and it can reason through it. Come back tomorrow and, depending on the system around it, you may need to explain who you are, what you were doing, and why FINAL_v12_ACTUALLY_FINAL.pdf apparently matters again.

The industry’s answer has mostly been to give the model an external memory.

Store conversations somewhere else.
Index them.
Retrieve the relevant pieces later.
Attach them to the next prompt.
It works.

But it also means the model itself still remembers nothing. We have essentially built a very smart employee who arrives every morning with amnesia and a very organized filing cabinet.

At Backboard, we wanted to test a different idea:
What if some of the memory lived inside the model itself?

We post-trained a model on a LOCOMO conversation set and scored 99.95% on the benchmark.

That number sounds impressive, but it also needs a very large asterisk. And that asterisk is the interesting part.

Memory Is an Architecture Decision

When people say an AI system “has memory,” they can mean very different things.

The most common approach looks something like this:

A conversation happens.
The system stores pieces of it.
Those pieces are embedded and indexed.

Later, when the user asks another question, the system searches for relevant information and includes it in the new request.

This is basically retrieval. The model has not remembered anything. The application has remembered something for the model.

That distinction matters because the memory system now becomes another piece of infrastructure you have to design, secure, operate, monitor, and pay for.

Where does the memory live?
How is it isolated between users?
What gets retrieved?
Who is allowed to retrieve it?
How long is it retained?
How do you delete it?
What happens when retrieval confidently finds exactly the wrong thing?

“Memory” sounds like a feature.

It is actually a collection of architectural decisions wearing a friendly name._

The Filing Cabinet Approach

Retrieval-based memory is useful for good reasons. It is easy to update, and it is flexible.

But every time the model needs that information, the system generally has to retrieve it and put it back into context.

Imagine having an employee who knows nothing about your company, but before every meeting you hand them the relevant sections of the employee handbook, customer history, policy documents, and notes from the last six meetings.

They can absolutely do useful work, but there is a lot of photocopying.

In AI terms, that photocopying is context.

And context costs tokens.

What If the Model Actually Learned Some of It?

There is another option: Instead of storing all long-term knowledge outside the model, you can teach some of it into the model through post-training.

The analogy is closer to what happens with an experienced employee.

At first, they look everything up.
Eventually, they know the terminology.
They know the recurring policies.
They know how the organization tends to work.
They stop checking the manual every time someone asks where the expense form lives.

Until recently, doing something similar with a model was expensive and specialized enough that it was not a realistic memory primitive for most companies.

That is changing.

Techniques such as LoRA make targeted post-training more efficient, while quantization can make adapted models smaller and easier to deploy.

So we wanted to know how far this idea could go.

Then We Got 99.95%

We post-trained a model on conversations from LOCOMO, a benchmark designed to test long-term conversational memory.

The resulting model scored 99.95%.

Essentially perfect recall.

At which point the obvious marketing strategy would have been to put the number in a 72-point font and quietly stop explaining.

Unfortunately, the explanation is the important part.

We trained the model on the same conversation material it was later evaluated on.

On purpose.

So no, this does not prove that we built a universally superior memory model.

It does not prove extraordinary generalization to conversations the model has never seen.

And it certainly does not mean everyone else should throw away their vector database before lunch.

What it demonstrates is something narrower and, I think, more useful:
If a model is taught a body of conversational memory, that memory can be represented parametrically with extremely high recall.

We were testing the ceiling of the mechanism. Not trying to disguise a training set as a magic trick.

Recall and Generalization Are Different Problems

This experiment also exposed something I think AI benchmarks need to handle more carefully. There are at least two different questions hiding inside “Does the model remember?”

Can it recall information it has previously learned?
Can it reason correctly over new conversations it was never trained on?

Those are not the same capability.

A model can be excellent at one and weaker at the other.

So we proposed an extension we call LOCOMO-Δ, where some conversations are held back from training.

That would make it possible to measure parametric recall separately from generalization.

Peak numbers are useful. Understanding what produced them is more useful.

This has become something of a recurring theme in our benchmark work.

Parametric Memory Changes the Economics

The interesting part for enterprise systems is not the benchmark. It is the architecture.

If stable information lives inside a model, the application does not necessarily need to retrieve and resend that information every time it is relevant. That changes the cost structure.

A retrieval-heavy system may repeatedly pay inference cost for the same policies, product knowledge, terminology, or recurring context.

Parametric memory shifts some of that work toward a teaching step. For knowledge that is queried again and again, that can become economically interesting.

It also changes deployment. An adapted model can potentially move with its knowledge.

Put it in a private environment. Run it on-prem. Quantize it and move it onto a capable workstation. Deploy it somewhere without continuous access to the original external memory service.

The memory becomes part of the artifact you deploy.

That is a very different architecture.

It Also Changes the Security Boundary

Another interesting property is external retrieval systems often keep many users’ or tenants’ memories inside shared infrastructure and rely on permissions, filters, metadata, and application logic to keep everything separated.

Those controls can be perfectly reasonable, but they are still controls that have to be implemented correctly.

With separate parametrically adapted models, tenant knowledge can instead reside in separate model weights.

There is no shared retrieval store containing that particular memory.

That gives the architecture a different isolation model.

Not automatically a perfect one. Not magically “secure.”

Just different.

And importantly, auditable in different ways.

Security architecture gets better when teams have multiple well-understood options instead of pretending one pattern is always correct.

Unfortunately, Models Are Bad Databases

There is a catch.

Several, actually.

Once information has been learned parametrically, editing it is not the same as updating a database row. Deleting one specific fact is harder. Auditing individual pieces of information is harder.

If a user invokes a right-to-deletion requirement, “we taught it to the weights” is not the kind of sentence that makes a privacy lawyer relax.

And some information changes constantly.

Yesterday’s transaction, this morning’s inventory, a user preference they changed five minutes ago, the current exchange rate, etc.

You probably do not want to retrain a model every time someone changes their shipping address.

Which leads to the part of this argument that tends to be less exciting on social media.

The Answer Is Both

I don’t think enterprise memory becomes:
RAG is dead. Everything goes into the model.

That would be replacing one overly broad architecture with another one.

The better distinction is about the kind of knowledge being stored.

Stable, high-value, recurring knowledge can be a good candidate for parametric memory.

Things like organizational terminology, durable policies, product knowledge, recurring workflows, or relatively stable tenant context.

Volatile, user-owned, or highly regulated information generally belongs somewhere external where it can be updated, inspected, audited, and deleted precisely.

Transactions.
Recent events.
Changing preferences.
Records governed by deletion requirements.
You use the memory mechanism that matches the information.

A shocking conclusion, I know:
Different data probably deserves different architecture.

AI Memory Is Becoming a Design Space

For a while, retrieval was effectively the default answer to long-term AI memory because it was the practical answer available.

That is starting to change.

As post-training becomes cheaper and model deployment becomes more flexible, memory stops being one pattern and becomes a design space.

Some memory can live in retrieval.
Some can live in application state.
Some can live in databases.
Some can live parametrically inside the model.
And a good system may use all of them.

The question is no longer:
How do we give the AI memory?

It is:
Which things should the model know, and which things should the system look up?

That is a much better architecture question.

And if the last few years of AI infrastructure have taught me anything, it is that the useful questions usually begin once the magic-sounding feature gets decomposed into boring engineering decisions.

Which is inconvenient, but also where most of the interesting work turns out to be.

This is a conversational remix of an article I published on Backboard’s blog. Read the original deep dive here.

AI Won’t Be Priced by Tokens Forever

Robert Imbeault — Thu, 10 Sep 2026 18:00:00 +0000

Nobody wants to buy tokens. They want useful work, and eventually, AI will be priced that way too.

Every new technology seems to begin by billing us for the thing engineers can measure most easily.

Electricity gave us kilowatt-hours.
Telecommunications gave us minutes.
Cloud computing gave us CPUs, memory, storage, and bandwidth.

AI has tokens.

And right now, token pricing makes perfect sense, because tokens are measurable. Providers need to charge for computation, and developers need a way to estimate their bills. Input and output tokens give both sides a usable starting point.

For an industry still figuring itself out, it is a sensible meter, but I don’t think the meter survives as the product.

Because nobody wakes up in the morning thinking:
You know what would really improve the business today? Another 40 million tokens.

People want things done. Tokens are just what gets consumed along the way.

Cloud Went Through the Same Phase

Early cloud computing was very infrastructure-shaped.

Organizations talked about CPU counts, memory allocations, storage volumes, and bandwidth because those were the things they were buying.

Then cloud matured.

Developers stopped thinking quite so much about disks and started buying managed databases. They stopped managing individual servers and started consuming platforms.

The underlying infrastructure did not disappear.

Someone still has to care about CPU utilization at 2 a.m., it just stopped being the thing most customers thought they were purchasing.

The abstraction moved upward, and I think AI is beginning the same transition.

Tokens Are a Cost. They Are Not the Outcome.

Today, almost every major AI platform exposes pricing in terms of input and output tokens.

That makes comparison easy.

Model A costs this much per million.
Model B costs that much.
Model C is cheaper, provided you only read the footnotes after midnight.

But token cost tells us surprisingly little about the value of the system.

A customer does not care how many tokens were required to summarize a meeting, they care whether the summary was useful.

They do not care how many tokens were consumed analyzing a contract, they care whether the important clauses were identified correctly.

Tokens measure computational consumption.

Customers care about completed work.

Those are related. They are not the same thing.

Better AI Can Actually Mean Using Less AI

This is where the economics become interesting.

Some of the most useful improvements in AI systems reduce token consumption rather than increase it.

Better prompts can produce better answers with less context.
Better retrieval can avoid stuffing unnecessary information into every request.
Smarter routing can send straightforward work to smaller, cheaper models.
Quantization can serve more requests on the same hardware.
Post-training can teach recurring organizational knowledge into a model instead of attaching the same context to every prompt forever.

None of those improvements make the customer’s experience worse.

Quite the opposite, because they improve the system while often reducing the amount of computation required. And this creates a slightly awkward incentive.

If your business grows when customers consume more tokens, efficiency can look suspiciously like a revenue problem.

If your business grows when customers accomplish more with less infrastructure, efficiency is the product.

Those are very different businesses.

The Same Million Tokens Can Produce Very Different Value

Imagine two AI systems processing the same workload.

Both consume one million tokens.

One repeatedly sends irrelevant context, retrieves the wrong information, uses an unnecessarily large model, and eventually produces something useful after considerable persuasion.

The other gives the right model the right information and gets to the answer efficiently.

Same number of tokens.
Very different system.

This is why I think token pricing will gradually become less useful as a proxy for value.

The model is only part of the equation. The software around it determines whether it gets useful context or has to sift through irrelevant information. Routing each task to an appropriate model can also avoid expensive computation that adds nothing to the result.

The customer ultimately experiences the system, not the token counter.

Eventually, Enterprises Will Ask Different Questions

As AI becomes a larger part of enterprise infrastructure, I expect buying decisions to move upward from raw consumption.

Organizations will ask different questions that are harder to put on a pricing page than $X / 1M tokens.

Unfortunately for pricing pages, they are much closer to the questions customers actually care about such as security, workload, and productivity.

Local AI Makes Token Pricing Even Stranger

Token pricing becomes less useful as AI moves beyond centralized APIs.

Some workloads will stay in the public cloud, while others will run on infrastructure companies manage themselves. More capable hardware will also make it practical to run models directly on laptops and workstations.

When a company owns the hardware, the cost calculation changes. It’s already paying for the equipment and its operation, rather than receiving a bill for every token. What matters is whether that investment delivers enough useful work to justify the expense.

Engineers will still count tokens to understand performance and spot inefficiencies. But token volume alone won’t tell the business whether it’s getting value from the system.

Tokens Are Going to Become an Implementation Detail

I don’t think tokens disappear.

CPU utilization did not disappear when cloud platforms moved toward managed services.

Storage did not disappear when developers stopped manually provisioning disks.

Bandwidth certainly did not disappear. My monthly bills remain committed to proving that.

Those metrics still matter enormously to the people operating the infrastructure.

Tokens will be the same. Engineers will continue tracking them.
They will matter for cost optimization, context management, inference efficiency, and system design.

But customers will increasingly judge AI by something else:
What did it accomplish, and what did that outcome cost us?

That is a much more meaningful abstraction.

Eventually We’ll Price What People Actually Came to Buy

The most valuable AI systems will not necessarily be the ones producing the largest number of tokens.

They will be the ones delivering the best outcomes with the least friction and the greatest efficiency.

Tokens are useful.
They are measurable.
They are operationally important.
But they are not the product.

Like CPU hours before them, tokens are simply the meter we use until the industry gets better at pricing the thing customers actually wanted in the first place: useful work.

This is a conversational remix of an article I published on Backboard’s blog. Read the original deep dive here.

Your AI Doesn't Need to Pick a Side

Robert Imbeault — Tue, 08 Sep 2026 16:42:15 +0000

Cloud, private cloud, on-prem, on-device, and edge aren’t competing futures. They’re different answers to one question: where should this workload actually run?

Enterprise AI deployment has somehow become a team sport.

Cloud is the future. Local is the future. Edge is the future.

Pick your side. Buy the T-shirt. Defend it on LinkedIn.

I think the framing is wrong.

These architectures aren’t competing destinations. They’re points on a continuum, and the useful question isn’t which one wins.

It’s:

How close does this workload need to be to the data, the person, and the decision?

That question makes most of the deployment debate considerably less dramatic.

The Continuum Is Really About Distance

We’ve seen this pattern before.

Computing started on centralized mainframes because that was where enough compute existed. Then more capability moved to PCs, laptops, and eventually phones.

But laptops didn’t kill data centers. Phones didn’t kill laptops.

New computing layers changed which workloads belonged where.

AI is following the same pattern.

The cloud came first because frontier models require enormous amounts of compute and memory. Centralized infrastructure made that capability available to almost anyone through an API.

For many workloads, that remains the right architecture.

But as AI moves into more parts of an organization, the tradeoffs change.

Think about deployment as a continuum:

Cloud → Private Cloud → On-Prem → On-Device → Edge

Moving along that continuum generally brings intelligence closer to the organization and eventually to the place where the work happens.

That can mean lower latency, greater control, better offline operation, and tighter data boundaries.

It also means more operational responsibility.

Everyone likes sovereignty until someone has to patch the inference server.

There is no universally correct point on the continuum. There is only the right tradeoff for a particular workload.

Why More AI Is Moving Closer to the Work

A few years ago, running capable language models inside an ordinary organization sounded expensive enough to make the cloud decision fairly obvious.

That is changing.

Open models are improving. Hardware is getting better. Quantization and inference optimization are making capable models practical on increasingly modest infrastructure.

That shifts what can run where.

A workload that required a large cloud environment may become practical on-prem. Something that needed a server may eventually fit on a workstation. Something that needed a workstation may move onto a laptop.

The machine gets stronger while the model gets smaller.

Eventually, they meet somewhere in the middle.

That’s particularly interesting for workloads where proximity changes the product itself.

An on-device model can keep working without connectivity and may never need to send sensitive information away from the device.

At the edge, the argument becomes even stronger.

A model running beside factory equipment, medical hardware, industrial sensors, or autonomous machinery operates under very different constraints from an assistant sitting behind a web interface.

Milliseconds matter. Connectivity may be unreliable. Sending every decision across a network can become more than inefficient.

Sometimes it’s simply the wrong architecture.

Hybrid Isn’t a Compromise

One of the stranger assumptions in enterprise AI is that organizations will eventually settle on a single deployment model.

Why would they?

A company might use a frontier cloud model for compute-heavy reasoning while keeping sensitive workloads on infrastructure it controls. Knowledge workers may eventually run useful models directly on their devices, while industrial systems execute inference at the edge.

All at the same time.

That is not architectural indecision. That is architecture.

The deployment decision should follow the workload, not the other way around.

That also means today’s answer doesn’t have to be permanent.

Models will become more efficient. Hardware will improve. Costs will change. A workload that belongs in the cloud today may make much more sense somewhere else two years from now.

Good AI infrastructure should preserve that option.

Ask Where the Workload Belongs

So instead of asking:

Which deployment model wins?

I think developers should ask:

Where should this particular workload run?

Start with the constraints.

Where is the data? How much latency matters? Does the system need to survive without connectivity? What security boundary does it need to stay inside? How much compute does the workload actually require?

Then choose the architecture.

The best AI infrastructure won’t force every workload into one environment. It will make it easier for models, context, policies, and workflows to move as those requirements change.

AI isn’t becoming completely centralized.

It isn’t becoming completely decentralized.

It’s becoming distributed.

And the interesting engineering problem isn’t choosing which side wins.

It’s making sure each workload can run where it makes the most sense.

This is a conversational remix of an article I published on Backboard’s blog. Read the original deep dive here.

Your AI Isn’t Sovereign If It Can’t Leave

Robert Imbeault — Fri, 04 Sep 2026 16:35:00 +0000

Running a model locally is useful. Real sovereignty means retaining control over the model, data, policies, infrastructure, and, crucially, the exit door.

The word sovereign is having a very good year.

Every week, another product, platform, or press release promises “sovereign AI.” Usually, this means an open model running in a particular country, private cloud, or company-owned data center.

Those are meaningful advances. They solve real problems.

But a model does not become sovereign simply because you moved it into the basement and gave it a server rack.

We use the word sovereign at Backboard because it describes an important shift in enterprise computing. But I worry the term is being stretched until it means little more than “the deployment option we happen to sell.”

Sovereignty should mean something more demanding.

Sovereignty Is the Freedom to Decide

To me, AI sovereignty is an organization’s ability to make every meaningful decision about how its AI operates.

Where does the model run?

Which model do you use?

Can you modify or replace it?

Where does the data live?

Who controls the policies, memory, and access?

Can you move the workload without rebuilding the entire application?

That last question is the real test.

You can run a model entirely on infrastructure you control and still be deeply dependent on someone else.

The server may be yours, but the architecture can still belong to your vendor.

If your model cannot be adapted to your business, how sovereign are you?

If essential reasoning is locked behind a third-party API, how sovereign are you?

If your deployment options depend on a vendor’s roadmap, how sovereign are you?

If switching providers requires an 18-month migration, three consulting firms, and a small support group, you probably do not have sovereignty. You have a long-term relationship.

Real sovereignty is not isolation from every external provider. That is neither realistic nor especially desirable.

It is having meaningful control, credible alternatives, and the practical ability to leave.

Infrastructure Is Only One Layer

Most discussions about AI sovereignty begin and end with infrastructure.

Where is the data centre? Which cloud region is being used? Does the workload remain in-country?

These questions matter. But they cover only one layer of the system.

An enterprise AI application may rely on:

A foundation model
Post-training and model weights
Organizational data
Retrieval and memory systems
Policies and permissions
Application logic
Inference infrastructure
Deployment and monitoring tools

Controlling one layer does not automatically give you control over the others.

You can own the hardware while renting the intelligence. You can use an open model while remaining trapped in proprietary infrastructure. You can keep the data in Canada while sending critical parts of the workflow through services you cannot inspect, modify, or replace.

The residency map may look reassuring. The dependency map may tell a different story.

What Happens When the Model Learns Your Business?

Until recently, most enterprises treated foundation models as essentially fixed.

The company’s knowledge lived around the model:

In system prompts
In retrieval systems
In vector databases
In external memory
In application code
Occasionally, in a very large PDF called 'FINAL_v7_revised_ACTUAL_FINAL.pdf'

That architecture made sense when adapting a model was slow, expensive, and technically difficult.

But that assumption is changing.

As post-training becomes faster and more affordable, organizations can increasingly teach models their terminology, policies, workflows, institutional knowledge, and domain expertise.

The model stops being merely something the organization consumes. It becomes an asset the organization shapes and continually improves.

In some cases, it may become a meaningful part of the company’s intellectual property.

That changes the sovereignty question.

If years of institutional knowledge have been incorporated into a model, control over that model is no longer a minor procurement detail. The organization needs to know whether it can preserve, move, improve, audit, and continue operating that capability if a provider, price, licence, or regulation changes.

The smarter the model becomes about your business, the less comfortable you should be with someone else controlling the off switch.

Portability Is the Part Everyone Forgets

A sovereign architecture should let an organization choose between open and proprietary models.

It should allow models to be post-trained using the organization’s own knowledge.

It should allow them to be quantized and optimized for different hardware.

And it should allow the organization to decide where they execute.

Sometimes that will be a public cloud.

Sometimes it will be a private cloud.

Sometimes it will be infrastructure inside the organization’s own facilities.

Sometimes it will be a workstation, laptop, vehicle, factory, or edge device sitting directly in front of the person using it.

The important point is not that one of these options is inherently sovereign.

It is that the organization can choose among them, and change its mind later.

Models will change. Hardware will change. Providers will change. Prices will change. Regulations will change. Someone will rename a product, move three features into a new pricing tier, and call it simplification.

A sovereign architecture preserves the ability to respond.

Yesterday’s infrastructure decision should not become tomorrow’s permanent technical constraint.

Sovereignty Requires an Exit Door

There is a simple way to test an AI system’s sovereignty:
What happens if you need to leave?

Can you export the organization’s data and memory?

Can you preserve the adapted model?

Can you change the inference provider?

Can you move the workload from cloud to private infrastructure?

Can you replace one model without rewriting the application?

Can you continue operating if a vendor changes its terms?

If the answer is no, the system may still be private, secure, compliant, or locally hosted. Those are valuable properties.

But it is not fully sovereign.

It is dependent, just with better geography.

No Single Layer Makes You Sovereign

Not the model.

Not the infrastructure.

Not the deployment.

Not the data.

Sovereignty exists when an organization retains meaningful control across the system, along with the freedom to change it.

That is where I believe enterprise AI is heading.

Not only because regulation may demand it or security teams will insist on it, but because AI is becoming core infrastructure. Organizations will want to own the capabilities that differentiate them without permanently tying those capabilities to today’s model, vendor, or hardware.

The future of enterprise AI is not simply open.

It is not simply local.

It is not simply private.

It is portable, adaptable, governable, and replaceable.

If your AI cannot leave, it is not sovereign. It is just being hosted somewhere you recognize.

This is a conversational remix of an article I published on Backboard’s blog. Read the original deep dive here.

I Think Therefore I Am… A Big Pain in the A$$

Robert Imbeault — Mon, 20 Apr 2026 17:59:09 +0000

If you’ve tried to build anything serious on top of LLMs recently, you’ve probably run into this:

“Thinking” is supposed to make models better.
In practice, it makes your infrastructure worse.

Let’s break down where it actually hurts.

The illusion of “just turn on reasoning”

At a high level, you’d expect something simple:

Turn reasoning on → better answers
Turn reasoning off → cheaper, faster

Reality is messier.

Models sometimes don’t think when you ask them to
Models sometimes overthink trivial prompts, burning tokens for no gain
There’s no consistent behavior across providers

So now you’re not just building a product.
You’re debugging model psychology.

The fragmentation problem nobody talks about

Every provider decided to implement “thinking” differently.

OpenAI → effort levels (low, medium, high)
Anthropic → token budgets (explicit caps)
Google → both… depending on the model version

That’s just inputs.

Outputs are worse:

Some return dedicated thinking blocks
Others return reasoning summaries
Some mix reasoning into standard content structures

There is no standard. No shared schema. No predictable behavior.

So if you’re routing across models, you now need:

Input normalization
Output parsing per provider
Logic to reconcile different reasoning formats

This is where “simple API routing” stops being simple.

Billing is inconsistent too

Even cost modeling breaks.

Some providers expose reasoning tokens explicitly
Some hide it inside total usage
Some introduce provider-specific fields (looking at you, xAI)

Now you’re not just optimizing performance.
You’re building a cost translation layer.

Model switching makes everything worse

Switching models mid-thread sounds great… until you try it.

Even within the same provider:

Different endpoints behave differently (yes, even inside OpenAI)
Input formats change
Output structures change
Reasoning formats change

Now add state:

What context do you carry over?
How do you preserve reasoning continuity?
How do you avoid exploding token usage?

This is where most teams either:

Give up on portability, or
Build a fragile pile of adapters that break every few weeks

What we realized building Backboard

The real problem isn’t reasoning.

It’s lack of abstraction.

Developers shouldn’t have to:

Learn 5 different “thinking” systems
Normalize 5 different response formats
Track 5 different billing models
Rebuild state every time they switch models

So we made a call:

Unify it.

What “unified thinking” actually means

Instead of exposing provider quirks, we abstract them into one model:

A single thinking parameter
Direct control over reasoning budget
Consistent behavior across models
Normalized input and output structures

So you can:

Tune reasoning without caring about provider differences
Switch models without rewriting logic
Keep state intact across everything

And most importantly:

Stop thinking about thinking.

The uncomfortable truth

If you’re building on multiple LLMs and you haven’t hit these issues yet, you will.

The complexity is not obvious at the start.
It compounds as soon as you:

Add a second provider
Introduce reasoning
Try to optimize cost
Or maintain state across sessions

At that point, you’re not building your product anymore.
You’re building infrastructure.

Where this goes long term

Short term:

Abstractions like this save teams weeks or months of engineering time
They reduce cost volatility and debugging overhead

Long term:

The winning platforms won’t be the best models
They’ll be the ones that make models interchangeable and stateful

That’s the real unlock.

If you’re currently stitching together multiple providers, do a quick audit:

How many reasoning formats are you handling?
How portable is your state layer?
How confident are you in your cost predictability?

If the answer isn’t clean, you’re already paying the tax.

This is what we're working on at Backboard.io :)

Why Token Counting in Multi-LLM Systems Is Harder Than You Think

Robert Imbeault — Thu, 16 Apr 2026 14:09:06 +0000

When we set out to build our adaptive context window management component, we ran into a problem that sounds deceptively simple: how do you manage context windows when your system routes requests across multiple LLM providers?

The Core Problem
Each model has its own tokenizer, context window, and pricing rules. The same text is not "the same" across providers. OpenAI might count a prompt as 1,200 tokens; Claude might see it as 1,450. A chat session that fits comfortably in one model can silently exceed limits or cost significantly more in another.

This creates real problems when you switch providers mid-conversation. The new model has to ingest the full conversation history again — but since each model counts that context differently, you can hit:

Unexpected context-window overflow: the conversation that fit before now breaches the limit
Inconsistent truncation: different models truncate at different points, changing what context the model sees
Hard-to-predict routing failures: your router makes decisions based on one token count, but the model uses another

Why a Single 'Token Estimate' Doesn't Cut It
The tempting solution is to maintain a single token count with a safety margin. The problem: OpenAI, Claude, Gemini, Cohere, xAI, and others don't tokenize text the same way. A single estimate will be wrong in both directions — undercount and you risk failures; overcount and you truncate too aggressively, degrading conversation quality unnecessarily.

How We Solved It
The answer is making token counting provider-aware. Instead of a single universal estimate, the context management layer measures each prompt the way the specific target model will measure it. The router uses this measurement before the request is sent.

In practice this means the system:

Knows when a conversation is approaching the edge of a model's context window
Trims or compresses history intelligently, not just blindly chopping from the front.
Avoids expensive overages from miscounted tokens
Keeps model-switching complexity invisible to the end user

The user sees a smooth conversation. The system handles the messy reality that every model speaks a slightly different "token language."

What We're Building Toward
This is one component of a larger routing layer. The goal: switch LLM providers mid-product — based on cost, capability, or availability — without that complexity leaking to users. Provider-aware token counting turns out to be a foundational piece of that.

We're doing this so you won't have to. :)

Your Context Window Is Chaos. We Fixed It.

Robert Imbeault — Tue, 31 Mar 2026 10:47:36 +0000

If you’re routing across multiple LLMs, you probably already know this feeling:

One model happily accepts your massive conversation.
The next model chokes, truncates half the important bits, and hallucinates the rest.

Same app. Same user. Different context window. Chaos.

Backboard.io now includes Adaptive Context Management, a system that automatically manages conversation state when your app moves between models with different context sizes.

ps. if you have keys from any of the frontiers or OpenRouter you can use this for free!

You still get access to 17,000+ LLMs on the platform.

You just don’t have to personally babysit their context windows anymore.

And yes, it’s included for free.

The Problem: Context Windows Are Inconsistent (and Annoying)
In a multi‑model setup, this is what actually happens:

You start on a large‑context model. Everything fits:

system prompt
conversation history
tool calls + tool responses
RAG chunks
web search results
random runtime metadata you forgot you added
Your router decides to send the next request to a smaller‑context model.

Suddenly your carefully curated “state” is too big to fit. Something has to go.

Most platforms respond with:

“Cool, just write truncation and summarization logic that:

prioritizes what matters,
handles overflow nicely,
doesn’t break when you add a new tool,
and works for every model you might ever route to.”
So we all end up writing the same brittle code:

if tokens > limit: drop_old_messages() maybe_summarize() hope_nothing_important_was_there()
In a multi‑model system, that logic gets complicated and fragile fast.

What We Shipped: Adaptive Context Management

Backboard now automatically handles context transitions when models change.

There’s no extra endpoint and no new config. It runs inside the Backboard runtime whenever a request is routed to a model.

When that happens, Backboard:

Looks up the model’s context window.
Dynamically budgets it:
20% reserved for raw state
80% freed via summarization
Within that 20% “raw state” budget, we prioritize:

system prompt
recent messages
tool calls
RAG results
web search context
Whatever fits in that 20% goes through unchanged.

Everything else is handled by intelligent summarization.

You don’t write the logic. You just route between models.

How Intelligent Summarization Works
When we need to compress, we follow a simple rule:

First try the model you’re switching to.

“Hey smaller model, summarize this so you can still understand what’s going on.”
If the summary still doesn’t fit:

We fall back to the larger model that was previously in use to generate a more efficient summary.
This preserves the important parts of the conversation while ensuring the final state always fits within the new model’s context window.

All of this happens automatically during the request and tool calls.

No manual orchestration. No custom jobs. No extra service.

You Should Rarely Hit 100% Context Again
Because Adaptive Context Management runs continuously:

It reshapes and compresses state before you slam into the limit.
It keeps a buffer in the context window instead of riding at 99.9% and hoping for the best.
Mid‑conversation model switches stop being a coin flip on whether something vital gets chopped.
Your job: define the routing logic and features.

Our job: make sure the context window doesn’t quietly wreck them.

You Still Get Visibility: context_usage in msg
This is not a black box.

We expose context usage directly in the msg endpoint so you can see what’s happening in real time.

Example response:

"context_usage": { "used_tokens": 1302, "context_limit": 8191, "percent": 19.9, "summary_tokens": 0, "model": "gpt-4" }

You can track:

how much context is currently used
how close you are to the limit
how many tokens are from summarization
which model is currently managing the context
If you like graphs and dashboards, this gives you the raw data without forcing you to build your own context tracking system from scratch.

The Bigger Idea: Treat Models Like Infrastructure
Backboard’s thesis is simple:

You should be able to treat models as interchangeable infrastructure.

Your state should just move with the user.

That only works if state can move safely between:

cheap and expensive models
long‑context and short‑context models
different providers and pricing tiers
Adaptive Context Management is the safety layer that makes that viable:

You route across thousands of models.
Backboard keeps the conversation state aligned with each model’s constraints.
You don’t write ad‑hoc truncation and summarization logic per model.
You focus on product behavior.

We handle the context window drama.

Adaptive Context Management is free and live today in the Backboard API.

No feature flag. No extra pricing line.

You can start building with it now at:

👉 https://docs.backboard.io

If you’re already routing across multiple models and have horror stories about context windows, I’d love to hear them.

I'm bias but I love this!

Robert Imbeault — Tue, 24 Mar 2026 15:13:56 +0000

Jonathan Murray

Mar 24

I’m Learning AI in Public, and I Think Developers Need to Chill a Bit

#ai #devops #devrel #programming

5 min read

The Hidden Problem With Multi-Model AI Systems: Context Window Mismatch

Robert Imbeault — Tue, 24 Mar 2026 13:37:22 +0000

Notes from building infrastructure for 17,000+ LLMs

One of the promises of modern AI infrastructure is simple:
You should be able to switch models whenever you want.

Different models have different strengths. Some are faster. Some are cheaper. Some reason better. Some support large context windows.

In theory, you route requests dynamically and get the best of each.
In practice, something breaks almost immediately.
Context windows don’t match.

The Moment Everything Breaks

Imagine this common scenario

A conversation begins on a large context model. Maybe something like a 128k context window.
The system prompt is fairly large.
The user has been chatting for a while.
Tools have been called.
A RAG system has pulled in documents.
Everything works.
Then your router decides to switch to a smaller model. Maybe for latency or cost reasons.

Suddenly the entire state no longer fits.
The request fails or the model behaves unpredictably.
This happens because the model’s context window is not just holding messages. It contains the entire runtime state:
system prompts recent conversation turns tool calls and tool outputs RAG results web search context other metadata.

When you exceed the limit, something has to give.
Most teams end up writing custom logic to handle this:
truncating older messages prioritizing certain content summarizing conversation history trying to prevent context overflow

This logic grows quickly and often becomes fragile.
We ran into this problem while building Backboard, which currently routes across 17,000+ LLMs.
So we built a system to handle it automatically.

The Core Idea: Treat Context Like a Budget
The approach we landed on was surprisingly simple.
Instead of filling the entire context window with raw state, we reserve a portion of it as a stable budget.
When a request is routed to a model, we allocate the context window like this:
~20% reserved for raw state
~80% available for summarization

The system calculates how many tokens fit inside that 20% allocation.
Within that space we prioritize the most important live inputs:
system prompt most recent messages tool calls, RAG results, web search context:

Everything else becomes eligible for summarization.

The Summarization Strategy
Once the system identifies which parts of the state cannot fit directly into the context window, it compresses them.
We designed the summarization pipeline around a simple rule:
First try summarizing using the target model.

If the summary still does not fit, fall back to the larger model previously used to generate a more efficient summary.

This helps preserve as much information as possible while guaranteeing the final prompt fits inside the model’s context window.
All of this happens automatically in the runtime.

Avoiding Hard Context Failures
One of our goals was to make context exhaustion extremely rare.
Because the system runs continuously during requests and tool calls, the state is reshaped before the context window is fully consumed.
In practice this means applications rarely hit the absolute context limit of a model.
Developers do not have to constantly monitor token counts or worry about prompt overflow.

Making Context Usage Observable
Even though the system runs automatically, we wanted developers to see what was happening.
So we added context metrics directly to the API response.

Example:

"context_usage": {
 "used_tokens": 1302,
 "context_limit": 8191,
 "percent": 19.9,
 "summary_tokens": 0,
 "model": "gpt-4"
}

This makes it easy to track:
how much context is being used when summarization happens how close you are to a model’s limit which model processed the request

For production systems, this visibility is useful for debugging and optimization.

Why We Think This Belongs in Infrastructure
A lot of AI applications now route between multiple models depending on cost, latency, or capability.
But context window management often ends up as application code.
Our view was that this is infrastructure responsibility, not application responsibility.
Developers should be able to move between models freely without rebuilding state management every time.

Adaptive Context Management
We ended up calling this system Adaptive Context Management.
Its job is simple:
Ensure the conversation state always fits the model being used.
No prompt surgery.
No manual truncation logic.
No context window surprises.

As AI systems move toward multi-model architectures, context management becomes one of the most important reliability problems.

Different models will always have different limits.
The goal is to make those differences invisible to developers.

If you are curious about the architecture behind this or how we tested summarization quality, I’d love to hear how others are approaching context management in multi-model systems.

Adaptive Context Management is now available in Backboard and automatically enabled for users.