DEV Community: llm

The sparse-reward excuse is an authoring problem

Aamer Mihaysi — Wed, 14 Oct 2026 14:37:48 +0000

Every eval conversation I've had this year hits the same wall. Someone wants to measure an agent on a long-horizon task — 40, 60, 100 steps — and someone else says you can't score that densely. Grading every step means a human or a judge model reading the whole trajectory, and that's expensive, slow, and noisy. So you run the task, check the final assertion, and get one bit back. Pass or fail.

I've built evals that way. They're cheap and they're nearly useless. When the agent fails at step 47, you have a terminal error and a transcript, and you spend the afternoon reading it like a detective. The signal you needed was at step 12, and you didn't collect it because collecting it looked expensive.

Here's what I've started to believe: dense scoring was never the expensive part. Authoring was. We've been blaming the wrong cost.

Think about what dense reward actually requires. To say "step 14 was progress," something has to know what progress means at step 14. If all you have is a task prompt and a final assertion, nothing knows that. So you hand the trajectory to a judge model and ask it to guess. That's expensive, and it's guessing, because the judge has no ground truth either. It's reading the same transcript you are, with the same missing information, and charging you per token for the privilege.

Tev makes the other move, and it's the one I keep coming back to. Instead of scoring the trajectory after the fact, you write the hidden ground-truth world spec first — the actual state of the world, the preconditions, the milestones, the terminal conditions — and then you render the environment from that spec. The environment isn't hand-built and then described in a doc that drifts. The spec is the source, and the environment is a projection of it. Once that's true, progress at any step is a function of the spec. You don't judge it. You compute it. Dense per-step rewards fall out as a byproduct, because you already wrote down what "closer" means.

Paper's here if you want the actual method: https://arxiv.org/abs/2610.18868v1

The reason this lands for me isn't the benchmark numbers. It's that it's the same discipline I keep failing to apply in production. I run long-lived agents. They make tool calls, they hold state, they run for minutes at a time. When one goes sideways, my instrumentation tells me it errored. It does not tell me it was drifting at step 9 and unrecoverable by step 20. I've got logs, I've got traces, and I've got no notion of expected state to compare against, so all of it is description, not diagnosis. I can tell you exactly what the agent did. I can't tell you where it went wrong.

The fix isn't a smarter judge model watching the agent in real time. I tried that. It's slow, it costs per step, and it's confidently wrong in exactly the places I need it to be right — the ambiguous middle, where the agent is doing something plausible but off-spec. The fix is writing down the world spec: what the state should look like, which invariants should hold, what a milestone looks like. Then checking observed state against it. That check is cheap. It's a comparison, not an inference. And it gives me a progress trace instead of a postmortem.

So the myth I want to bust is this: dense agent evals are too expensive to run. They're not. They're too expensive to improvise. If you don't have a spec, every step is a judgment call, and judgment calls don't scale — not in money, and not in consistency. If you do have a spec, every step is a lookup.

What I'd steal from Tev, concretely:

Write the world spec before the task prompt. State, invariants, milestones, terminal conditions. If you can't write it, you don't understand the task well enough to eval it, and no amount of judge model will fix that.
Render the environment from the spec. One source of truth. Hand-built environments drift from their descriptions, and then your rewards are scoring a world that doesn't exist.
Emit progress as a byproduct, not a second pass. If scoring is a separate pipeline, it rots the moment the environment changes. If it's a projection of the spec, it stays in sync for free.
In prod, make the same spec your instrumentation contract. Every tool call gets checked against expected state. You get a curve, not a vibe.

Where I'm skeptical: rendering from a spec is only as good as the spec. Under-specify it and you get dense rewards that are confidently wrong, and an agent that optimizes straight into the gap between your spec and reality. That's worse than sparse, because sparse failure is at least honest — it doesn't tell you you're winning. Also, spec authoring is real work. You've moved the cost, not deleted it. But moving it from per-run scoring to one-time authoring is the right trade, because the spec is reusable, reviewable, and versionable. A wrong spec is a bug you can fix. A noisy judge is fog you can't.

The other thing dense rewards buy you is debuggability. Sparse gives you pass/fail and a feeling. Dense gives you a curve, and the curve tells you where the agent stalls. That's the actual product. Not the score — the shape.

Next time someone tells you long-horizon agent evals have to be sparse because dense scoring is too expensive, ask them for the world spec. If they don't have one, that's the cost they're actually paying. They just haven't itemized it.

AI Reasoning Breakthroughs: DeepSeek-R1 and OpenAI's Path to Logic

trillioniar s — Wed, 14 Oct 2026 14:34:37 +0000

Today's AI landscape is shifting from probabilistic word prediction to structured logical reasoning. New models are utilizing reinforcement learning to "think" before they speak, drastically improving performance in math and coding.

The Rise of Reasoning Models

OpenAI's Learning to Reason

OpenAI has introduced new methodologies for LLMs to reason through complex problems. By integrating chain-of-thought processing, models can now self-correct and explore multiple paths to a solution. This reduces hallucinations in technical domains.
Source: OpenAI

DeepSeek-R1 and RL

Incentivizing Logic via Reinforcement Learning

DeepSeek-R1 demonstrates that reasoning capabilities can be incentivized through specialized RL. The model shows a marked improvement in mathematical reasoning by rewarding the process of derivation rather than just the final answer.
Source: arXiv

The Philosophical Shift

LLM Inevitabilism

The concept of "LLM Inevitabilism" suggests that the integration of these models into every layer of software is unavoidable. As reasoning capabilities stabilize, LLMs move from being "chatbots" to becoming the core operating system for digital tasks.
Source: Tom Renner

Open Source and Accessibility

Meta's Open Source Push

Meta continues to advocate for open-source AI as the only viable path to prevent corporate monopolies. By releasing high-performance weights, they enable a global ecosystem of fine-tuning and safety research.
Source: Meta

Infrastructure for Scale

Google DeepMind's Gemini Evolution

Gemini continues to iterate on multi-modal reasoning, integrating native audio and video processing into the reasoning loop. This allows the model to reason about physical space and temporal events more accurately.
Source: Google DeepMind

FAQ

What is Chain-of-Thought processing?

It is a technique where the model generates intermediate reasoning steps before providing the final answer.

How does DeepSeek-R1 differ from standard LLMs?

DeepSeek-R1 uses reinforcement learning to specifically reward the reasoning process, not just the output.

Why is open-source AI important for reasoning?

Open weights allow researchers to inspect the "hidden" reasoning chains to ensure safety and transparency.

Will reasoning models replace traditional coding?

They will likely augment coding by handling architectural logic, while humans focus on high-level design.

How do these models reduce hallucinations?

By verifying internal steps of a logic chain, models can spot contradictions before finalizing a response.

One Key, a Dozen-Plus Models: Switching LLMs in VS Code Chat with Modelflare

Magnus — Wed, 14 Oct 2026 14:32:00 +0000

How to use the Modelflare VS Code extension to put every text model your account can reach into the VS Code chat model picker, behind a single Smart API Key.

Most developers no longer use one model. A fast, cheap model is fine for "rename this variable everywhere." A frontier model earns its price on a gnarly refactor. A third model is the one you trust for reviewing a migration. The problem is that each of those usually lives behind its own extension, its own account, and its own key.

That setup has real costs:

Configuration sprawl. Three providers means three keys in three places, three billing dashboards, and three sets of rate limits.
Switching is slow. If changing models means changing extensions, you stop doing it, and you end up paying frontier prices for trivial edits.
New models lag. Every release means waiting for a provider extension update or editing config by hand.
Outages are your problem. When one provider has a bad hour, the chat panel just errors.

VS Code already has the right UI for this: the chat model picker. What it needs is one provider that can fill that picker with models from many vendors. That is what the Modelflare extension does.

What the extension does

Modelflare for VS Code registers Modelflare as a language model provider for VS Code chat. According to its README:

Text models available to your key are loaded from GET /v1/models, so new models appear without editing the key.
Responses stream, and tool calling works for agent mode.
Routing across groups, fallback and the routing strategy are handled server-side by the Smart API Key.
The key lives in VS Code's secret storage, never in settings.json.
It is bring-your-own-key: no Copilot subscription is required for BYOK models.

The extension ID is modelflare.modelflare-vscode, and it requires VS Code 1.104 or newer.

Step 1: Create a Smart API Key

This is the piece that makes "one key" true. An ordinary Modelflare key is bound to a single routing group and can only call that group's models. A Smart API Key is bound to auto. On every request Modelflare:

collects every routing group your account may use,
keeps only the groups that currently have a channel for the model you asked for,
orders them by the key's strategy, and
sends to the first candidate, moving to the next group if that one runs out of usable channels.

So a model that only one group serves still works, and a group that is momentarily unusable is skipped instead of failing your chat.

To create one:

Sign up or open the console.
Go to API Keys â†’ Create Smart API Key.
Pick a routing strategy:

Strategy	Order of candidate groups
Balanced (default)	Starts in the middle of the price range, works up to the most expensive candidate, then back down through the cheaper ones
Stability first	Highest group multiplier first
Low price first	Lowest group multiplier first

You can change the strategy on the existing key at any time; the next request uses the new order. Quota, expiry date and IP allowlist work as on any other key.

Step 2: Install the extension

From the VS Code Marketplace, search for Modelflare in the Extensions view, or install from the terminal:

code --install-extension modelflare.modelflare-vscode

The same extension is also published on Open VSX, the registry used by VS Code-compatible builds such as VSCodium: open-vsx.org/extension/modelflare/modelflare-vscode. It needs a build based on VS Code 1.104 or newer, the release line with the chat model provider API the extension uses.

A note for early adopters: version 0.1.0 shipped under the ID modelflare.modelflare. From 0.1.1 the ID is modelflare.modelflare-vscode, because the shorter name is taken on the VS Code Marketplace. Commands and settings are unchanged, but VS Code keeps secrets per extension ID, so if you installed 0.1.0, run Manage Modelflare API Key once after updating.

Step 3: Add your key

Either:

open the chat model picker â†’ Manage Models â†’ Modelflare, or
run Modelflare: Manage Modelflare API Key from the Command Palette.

Paste the Smart API Key. It goes into VS Code's secret storage. Then open the model picker again and choose a Modelflare model.

The extension has exactly one setting:

{
  "modelflare.baseUrl": "global"
}

global (the default) points at https://modelflare.dev/v1; china points at https://origin.modelflare.dev/v1; you can also give a custom URL. Most readers never need to touch it.

Worked example: three models, one task

To show why switching matters, here is a realistic loop on a small TypeScript service. All three models were picked from the same dropdown, on the same key, without leaving the chat panel.

The model IDs below are real entries in the Modelflare catalog. Each one was also returned by GET /v1/models for a Smart API Key on the day we wrote this, and that is the same list the extension loads. Which ones appear in your picker depends on what your account can reach.

1. Explore with a fast model. Pick gemini-3.5-flash and ask:

Summarize what src/billing/ does and list every place we compute a rounded price.

Cheap, quick, and good enough to map a folder.

2. Refactor with a frontier model in agent mode. Switch the picker to claude-opus-5-5, switch the chat to agent mode, and ask:

Extract the rounding logic you found into src/billing/round.ts, update every caller, and add unit tests.

This is where streaming plus tool calling matters: the model reads files, proposes edits, and runs through the change set the way agent mode normally does.

3. Get a second opinion. Switch to gpt-6.1-sol and ask:

Review the diff in this workspace. Look for behavior changes in rounding at .5 boundaries.

A different model family reviewing another one's work catches a surprising number of edge cases.

Three vendors, one key, one bill. Every request shows up in the usage logs, billed with the multiplier of the group that actually served it, and the log records the resolved group and strategy (Usage and cost).

For reference, here are a dozen-plus text model IDs that were both in the live catalog and in a Smart API Key's GET /v1/models response on the day we wrote this:

Family	Example IDs
OpenAI GPT	`gpt-6.1-sol`, `gpt-6-sol`, `gpt-5.5`
Anthropic Claude	`claude-opus-5-5`, `claude-sonnet-5-5`, `claude-haiku-5-5`
Google Gemini	`gemini-3.8-flash`, `gemini-3.5-flash`, `gemini-3.1-pro-preview`
xAI Grok	`grok-4.7`
DeepSeek	`deepseek-v4-pro`, `deepseek-v4-flash`
Others	`kimi-k3`, `glm-5.3`, `qwen3.8-max`

Treat this as a snapshot. The picker is the source of truth for your account, and Models & Pricing is the source of truth for prices.

Tips

Refresh after a release. When a new model lands, run Modelflare: Refresh Modelflare Models. The key itself needs no edit: a Smart API Key has no per-model allowlist and always follows your account's current access.

Use the strategy, not a second key. If you want cheaper defaults for a side project and maximum stability for work, change the strategy, or create two Smart API Keys with different strategies. You do not need provider-specific keys.

Do not build your own fallback list. A Smart API Key already walks every group your account can use. Ordered fallback chains are for ordinary keys; see Group failover.

Pick the model per task, not per day. The cost difference between a flash-class model and a frontier model is large. Defaulting to a fast model and switching up only for the hard step is the habit this setup makes cheap.

Read errors literally.

401: the key is missing or invalid. Create a Smart API Key at modelflare.dev/keys.
model_not_found: a regular key only calls its own group's models. If you already use a Smart API Key, that model is not available to your account right now.
403 insufficient_user_quota: top up at modelflare.dev/billing.

Slow model list from mainland China? Set modelflare.baseUrl to china. Since 0.1.2, the extension's public catalog request times out after 8 seconds and falls back to the key's own model list, so the picker no longer hangs.

Want images and video too? The VS Code extension covers text models. For media generation in an agent, see the Media Skill or the @modelflare/mcp-server package.

Where to go next

Install: VS Code Marketplace Â· Open VSX
Source and changelog: github.com/castaly-ai/modelflare-vscode
Smart API Key: docs.modelflare.dev/guides/features/smart-api-key
Models and groups: docs.modelflare.dev/guides/models-and-groups
Text models: docs.modelflare.dev/models/text Â· Live catalog: modelflare.dev/pricing
Platform features: docs.modelflare.dev/guides/features

The model picker was always the right place to choose a model. With one key behind it, choosing is finally cheap enough to do on every task.

Disclosure: I'm part of the Modelflare team, which builds the AI API gateway mentioned in this article.

Claude vs Gemini vs GPT: What a 2026 Study of 26 AI Models Reveals About Long Context

Egor — Wed, 14 Oct 2026 14:28:14 +0000

26 AI models. 6,438 evaluation instances. Contexts up to 1 million tokens. The results challenge how we compare modern language models.

Introduction

Imagine giving an AI assistant an entire software repository.

Thousands of files. Hundreds of modules. API documentation, database schemas, and years of development history.

The model accepts everything without complaining.

You ask it to identify a subtle architectural problem.

It confidently provides an answer.

But is that answer actually reliable?

In 2026, the question is no longer simply whether a language model can accept a million tokens.

The more interesting question is:

How much of that context can the model actually use correctly?

A recent research paper, ATLAS: All-round Testing of Long-context Abilities across Scales, provides some surprising answers.

The researchers evaluated 26 models on 6,438 instances across eight capability dimensions, considering input lengths from 8K to 1M tokens.

One finding stands out:

Gemini 3.1 Pro Preview led at 128K, but Claude Opus 4.6 led at 1M.

The best model depends on how you measure it.

Let's examine what the researchers tested, what their results actually mean, and why these findings matter for developers building real AI systems.

Research note: This article analyzes published 2026 research. It is not an independent benchmark, and it does not claim that the author personally tested all 26 models.

1. The Problem With AI Benchmarks

When comparing language models, we often see a familiar table:

Model	Context window
Model A	128K
Model B	256K
Model C	1M

The implication seems obvious.

Model C can process more information, so it must be better at large-document tasks.

But this reasoning contains a hidden assumption:

That accepting more information means using that information effectively.

Consider two different tasks.

Task A: Simple retrieval

A document contains the sentence:

"The deployment identifier is QZ-7194."

The model is asked to retrieve that identifier.

Task B: Complex reasoning

A repository contains dozens of service definitions, configuration files, and dependency relationships.

The model must identify which changes could introduce a security regression.

The second task requires much more than finding one string.

It may involve combining evidence from many locations, resolving contradictions, and understanding dependencies.

A model that performs well on Task A might struggle on Task B.

That's exactly why evaluating long-context AI requires more than a single benchmark number.

2. ATLAS: A Different Way to Measure Long-Context Intelligence

Research: ATLAS: All-round Testing of Long-context Abilities across Scales

Authors: Deli Huang and colleagues

Published: May 27, 2026

Paper: https://arxiv.org/abs/2605.28079

The ATLAS researchers investigated two major weaknesses in traditional long-context evaluation.

First, model performance can deteriorate as input length increases.

Second, success on foundational tasks such as retrieval does not necessarily transfer to more complex application workloads.

Their framework introduces several methodological improvements.

Evaluating across context lengths

Instead of measuring performance at only one input size, ATLAS considers scores across a predefined context-length grid, from 8K to 1M tokens.

This makes it possible to analyze how performance changes as more information enters the prompt.

A model that performs exceptionally well at 128K may deteriorate more sharply than another model at longer lengths.

Separating foundational abilities from applications

ATLAS distinguishes foundational operations from downstream workloads.

This matters because retrieving information and applying it correctly are different capabilities.

For instance, a model may identify relevant functions in a repository without understanding how they interact.

Measuring balanced performance

The researchers introduce ATLAScore, a harmonic-mean-based aggregate across capability categories.

This is intended to penalize highly imbalanced capability profiles.

A model that dominates one task but struggles on others should not automatically receive the strongest overall evaluation.

The framework also includes uncertainty propagation when computing its final aggregate scores.

Together, these changes provide a more nuanced picture than a single retrieval accuracy number.

3. The Result: The Leader Changes at 1 Million Tokens

One of the most interesting findings concerns model rankings.

According to the ATLAS paper:

Evaluation scope	Leading model
Up to 128K tokens	Gemini 3.1 Pro Preview
Up to 1M tokens	Claude Opus 4.6

This does not mean Claude is universally better than Gemini.

Nor does it mean Gemini is inherently unreliable.

It means the relative ranking changes under the specific evaluation conditions used by ATLAS.

The researchers also reported that seven models changed position by at least two ranking places when the evaluation expanded from the 8K–128K range to the 8K–1M range.

That's important.

Imagine selecting a model for your application after reviewing results obtained only at 128K.

Your production workload, however, frequently processes 600K-token documents.

The benchmark you used may not reflect your actual operating conditions.

And if models deteriorate at different rates, their relative performance at 600K may differ from their relative performance at 128K.

The key insight: model quality is not independent of context length.

A useful benchmark should measure the range your application actually uses.

4. Why Context Length Changes the Difficulty

Let's examine a simplified example.

Imagine this information appears inside a long repository:

service_a.py:
Service A sends requests to Service B.

service_b.py:
Service B uses Database C.

config.yaml:
Database C has public network access enabled.

security_policy.md:
Production databases must not expose public access.

Now ask:

"Does Service A indirectly depend on a database with a potentially unsafe network configuration?"

Answering correctly requires several steps.

First, identify the dependency between Service A and Service B.

Second, identify Service B's database.

Third, find Database C's configuration.

Finally, compare that configuration with the security policy.

No single sentence contains the entire answer.

As the context grows, the model must identify relevant information among more irrelevant material while preserving the connections between different pieces of evidence.

This creates distinct possible failure modes:

Missing one of the necessary pieces of evidence
Retrieving a similar but incorrect detail
Correctly retrieving the evidence but connecting it incorrectly
Producing a plausible conclusion unsupported by the source material

The example is illustrative, not an ATLAS test case.

But it demonstrates why longer context can increase difficulty even when every relevant document fits into the model's advertised context window.

5. Another 2026 Study Reveals a Deeper Problem

ATLAS evaluates long-context capabilities across models and scales.

A separate 2026 study, CorpusQA: A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoning, investigates a different question:

Can AI systems analyze entire collections of documents when the evidence is distributed throughout the collection?

Authors: Zhiyuan Lu and colleagues

Originally published: January 21, 2026

Revised: April 25, 2026

Paper: https://arxiv.org/abs/2601.14952

CorpusQA includes tasks involving financial, educational, and real-estate documents.

Some questions require aggregating numerical information from many documents.

For example, imagine analyzing hundreds of financial reports to identify every company satisfying several accounting conditions.

A traditional retrieval system might find a few relevant documents.

But if answering the question requires analyzing every company, retrieving only five documents is insufficient.

This is a fundamentally different problem from finding one answer-bearing passage.

The published results

CorpusQA evaluated several models at different context lengths.

For Gemini 2.5 Pro, the authors reported:

Context length	Overall accuracy
128K	80.19%
1M	50.73%

These results show a decline of 29.46 percentage points under the study's conditions.

Importantly, CorpusQA evaluated Gemini 2.5 Pro, not Gemini 3.1 Pro.

Its results cannot be directly substituted for the newer model's performance.

But they illustrate how difficult corpus-level analysis can remain, even when a model supports very long inputs.

The study also found that conventional retrieval-augmented generation performed poorly on tasks requiring globally distributed evidence.

The lesson isn't that RAG is useless.

It's that retrieving a handful of relevant chunks is not the same as analyzing an entire document collection.

6. Why Traditional RAG Can Fail on Global Questions

A typical RAG pipeline looks like this:

User question
      |
      v
Vector search
      |
      v
Top-K relevant chunks
      |
      v
LLM
      |
      v
Generated answer

This works well when a question can be answered using a relatively small number of relevant passages.

For example:

"What is the API rate limit?"

A good retrieval system can locate the appropriate documentation paragraph and pass it to the model.

Now consider another question:

"Across all customer reports from the last year, which failure category increased the most?"

The answer requires a global aggregation.

Retrieving only the most semantically similar documents may exclude a large part of the dataset.

No matter how capable the LLM is, it cannot correctly aggregate information it never receives.

This is an information-coverage problem, not merely a language-model problem.

In the CorpusQA evaluation at 1M tokens, standard RAG achieved 17.38% accuracy, compared with 44.95% for the Memory Agent approach using Gemini 2.5 Pro.

This illustrates how different approaches to managing large document collections can significantly affect performance.

A better architecture for some analytical tasks

For questions requiring exact aggregation, a structured processing pipeline may be more appropriate:

Document collection
        |
        v
Information extraction
        |
        v
Structured records
        |
        v
SQL / analytical computation
        |
        v
Evidence verification
        |
        v
LLM-generated explanation

This design separates factual computation from language generation.

The LLM can interpret the user's question and explain the result, while deterministic code handles calculations.

Of course, the approach has its own limitations.

Extraction errors, incomplete schemas, and stale data can still produce incorrect answers.

But it creates opportunities to validate individual steps instead of relying entirely on one enormous prompt.

7. What About Vendor Benchmarks?

Independent research is only one source of evidence.

Model developers also publish technical evaluations.

For example, Anthropic's Claude Opus 4.6 System Card discusses long-context benchmarks such as MRCR and GraphWalks.

Source: https://www.anthropic.com/system-cards

MRCR evaluates the ability to locate and distinguish between similar pieces of information across long inputs.

GraphWalks examines structured reasoning over graph-like information represented within the context.

These evaluations are useful because they test capabilities beyond ordinary question answering.

However, developer-published benchmark results need careful interpretation.

Providers may use different inference settings, reasoning budgets, input limits, filtering rules, or evaluation configurations.

A benchmark score obtained under one provider's setup should not automatically be treated as directly comparable with an independent benchmark.

In particular, the ability to answer a question using a long prompt depends on more than the model checkpoint.

It can also depend on:

Reasoning budget
Tool usage
Prompt construction
Context truncation
Sampling configuration
Dataset filtering
Evaluation metrics

These differences can materially affect observed results.

That's one reason ATLAS is interesting: its evaluation explicitly attempts to compare long-context capabilities across multiple tasks and lengths.

8. What This Means for AI Coding Assistants

Consider an AI coding assistant with access to a large repository.

There are two possible designs.

Approach A: Put everything in the prompt

Entire repository
       |
       v
Large context window
       |
       v
LLM response

This is conceptually simple.

But it can be expensive and may introduce large amounts of irrelevant information.

It also doesn't guarantee that the model will correctly connect all important details.

Approach B: Search, inspect, and verify

User question
      |
      v
Repository search
      |
      v
Relevant files
      |
      v
Dependency analysis
      |
      v
LLM reasoning
      |
      v
Tool-based verification
      |
      v
Final answer

This approach can use search tools, static analysis, tests, and targeted context retrieval.

For some software engineering tasks, those tools may provide more reliable evidence than simply expanding the prompt.

For instance, when checking whether a function is actually called, repository search can identify references.

When checking whether a proposed patch works, an automated test can provide execution evidence.

An LLM can assist with interpretation, but it doesn't have to perform every operation internally.

The broader principle is:

Use the context window as one resource in an engineering system, not as a replacement for the system itself.

9. How I Would Benchmark Modern AI Models

The published results suggest a practical evaluation protocol for comparing models on real development workflows.

I would test each model across four dimensions.

Test A: Context length

Evaluate the same task family at:

8K
32K
128K
256K
512K
1M

Only include lengths genuinely supported by each tested model and API configuration.

Test B: Evidence position

Place answer-relevant information near the beginning, middle, and end of the input.

Measure whether accuracy changes.

Test C: Reasoning complexity

Use several difficulty levels:

Level 1: Retrieve one explicitly stated fact.

Level 2: Connect information from two files.

Level 3: Combine evidence from several documents.

Level 4: Perform global analysis across an entire corpus.

Test D: Tool-assisted reasoning

Compare direct long-context prompting against workflows using retrieval, code execution, or other verification tools.

For every experiment, record:

Metric	Why it matters
Answer accuracy	Measures correctness
Context length	Reveals scale sensitivity
Input/output tokens	Measures resource usage
Latency	Measures responsiveness
Cost	Measures practical efficiency
Evidence completeness	Checks whether necessary information was found

Model versions and inference settings should be fixed and documented.

Tasks should be repeated using multiple independent examples, with uncertainty estimates reported alongside aggregate scores.

This is a proposed experimental design, not a claim that these tests have already been run for this article.

10. Limitations and Research Caveats

There are several reasons to interpret these findings carefully.

First, ATLAS is a May 2026 preprint. Its results are useful research evidence, but they should not be mistaken for a universally accepted model ranking.

Second, the comparison covers a specific collection of 26 models and evaluation settings. It does not establish the performance of every model released later in 2026.

Third, CorpusQA evaluates a different set of models and tasks. Its numerical results cannot be directly compared with ATLAS scores.

Fourth, performance at 1M tokens may involve implementation-specific constraints, including inference settings and context handling.

Finally, benchmark results do not automatically predict real-world performance.

An AI system answering legal questions, writing code, analyzing scientific papers, or processing financial records may require very different capabilities.

The correct model and architecture should be selected through task-specific evaluation.

Conclusion: We Need Better Questions Than "How Many Tokens?"

The rapid growth of context windows is an important technological achievement.

But the 2026 research tells a more complicated story.

The ATLAS study found that model rankings change when evaluation extends toward one million tokens.

CorpusQA demonstrated the difficulty of analyzing large document collections when relevant evidence is dispersed across many sources.

Together, these studies suggest that increasing context capacity alone is not enough to guarantee reliable information processing.

For developers, the practical implications are straightforward:

Don't select an LLM based only on advertised context size.

Evaluate it at the lengths your application actually uses.

Test retrieval and reasoning separately.

Measure the quality of evidence, not only the fluency of the generated answer.

And consider whether structured tools, retrieval systems, or deterministic computation can make your application more reliable.

The future of long-context AI may not be about putting everything into one prompt.

It may be about building systems that know what to retrieve, what to remember, what to compute, and what to verify.

A million-token context window is a capability.

Reliable reasoning over a million tokens is a different achievement.

References

[1] ATLAS — All-round Testing of Long-context Abilities across Scales (2026)

Deli Huang et al.

https://arxiv.org/abs/2605.28079

[2] CorpusQA — A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoning (2026)

Zhiyuan Lu et al.

https://arxiv.org/abs/2601.14952

[3] Claude Opus 4.6 System Card (2026)

Anthropic.

https://www.anthropic.com/system-cards

Discussion

Do you think future AI systems will rely primarily on larger context windows, or will retrieval, memory, and tool-assisted reasoning become more important?

And have you encountered cases where an LLM missed important information even though it was present in the prompt?

I'd be interested in hearing about real-world experiences.

GitHub: https://github.com/coffee1337

Portfolio: https://coffee1337.github.io/

Agentes LLM: diseña una señal de email confiable

Silviu Technology — Wed, 14 Oct 2026 14:24:51 +0000

Un agente LLM puede registrarse, pedir un código y responder “hecho” aunque la verificación no haya terminado. Tal vez leyó un mensaje viejo, interpretó un 200 OK como entrega confirmada o cerró la tarea después de un error. El texto final suena convincente, pero no es evidencia.

En los flujos de email, la señal de finalización debe venir de eventos observables. Este patrón propone separar tres cosas: la intención del agente, la entrega del mensaje y la prueba de que el mensaje correcto fue consumido. Es especialmente útil al combinar modelos LLM con un fake emails generator o con un buzón temporal para pruebas.

El problema: el agente puede acertar por casualidad

Imagina esta secuencia:

El agente crea una cuenta de prueba.
La aplicación solicita el envío de un enlace.
El agente consulta el inbox.
Extrae un token y lo usa.
El evaluador mira la respuesta final.

El paso cinco no debería depender solo de lo que el modelo dice. Si el buzón se comparte entre ejecuciones, un correo atrasado puede hacer que una prueba pase. Si el endpoint responde antes de que la cola entregue el mensaje, el agente puede reintentar una acción que en realidad ya se realizó.

También conviene manejar con cuidado búsquedas ambiguas como tamp mail com. Ese texto puede aparecer en una consulta, pero nunca debe convertirse en configuración, dirección de prueba o criterio para aceptar un mensaje.

Define una señal de finalización observable

Una señal útil combina varios hechos pequeños:

existe una ejecución con un run_id único;
el mensaje llegó al buzón asignado a esa ejecución;
el asunto, destinatario y tipo de evento coinciden con lo esperado;
el token fue usado una sola vez;
la aplicación confirmó el estado final;
el buzón se cerró o expiró con una razón registrada.

En forma de diagrama de palabras:

intención del agente
        ↓
acción de aplicación → evento de entrega → mensaje coincidente
        ↓                                      ↓
respuesta de API ← estado verificado ← consumo del token
        ↓
recibo de ejecución

Diseña el contrato de la herramienta

El agente necesita una interfaz pequeña y predecible:

{
  "create_inbox": {"run_id": "eval-1842", "ttl_seconds": 900},
  "list_messages": {"after": "2026-10-14T14:00:00Z"},
  "read_message": {"message_id": "msg-73"},
  "close_inbox": {"reason": "verified"}
}

Cada acción debería devolver un estado explícito. message_not_found, inbox_expired y rate_limited tienen consecuencias diferentes; un false genérico empuja al modelo a improvisar.

El contrato también debe declarar qué campos son confiables. El asunto normalizado, el message_id, el received_at y el run_id sirven para correlacionar. El cuerpo del email puede contener instrucciones no confiables, así que no debería poder cambiar las reglas de la evaluación.

Para la capa de interfaz, vale la pena estudiar una ayuda de email que no roba el foco: el mismo principio de estados explícitos ayuda a que una UI no presente como éxito algo que aún está esperando.

Aísla el buzón y limita los reintentos

El aislamiento debe ser por ejecución, no por modelo. Dos agentes usando el mismo prompt siguen necesitando buzones separados porque sus tiempos de entrega no serán idénticos.

Asigna un TTL desde fuera del agente y un presupuesto de lecturas. Por ejemplo, permitir seis consultas en tres minutos es más fácil de razonar que “espera hasta que aparezca algo”. Un reintento puede ser válido ante un error de red; no lo es necesariamente ante un asunto que no coincide.

Registra cada intento de lectura con su motivo:

{
  "run_id": "eval-1842",
  "attempt": 3,
  "result": "message_not_found",
  "next_action": "retry_after_backoff",
  "remaining_seconds": 96
}

Si la entrega no sucede a tiempo, termina con delivery_timeout. No conviertas el silencio en una respuesta inventada. Para probar el lado asíncrono sin depender siempre de un proveedor real, las fixtures de email reproducibles en FastAPI ofrecen una referencia práctica para separar datos, estados y limpieza.

Un flujo mínimo con recibo de ejecución

El orquestador puede implementar este recorrido:

Crear run_id y buzón con TTL.
Ejecutar al agente con la herramienta limitada.
Validar el evento de entrega y el mensaje esperado.
Consumir el token y confirmar el estado en la aplicación.
Escribir un recibo con éxito o causa de fallo.
Cerrar el buzón, incluso si el agente fue cancelado.

Un recibo pequeño sería suficiente:

{
  "run_id": "eval-1842",
  "agent_version": "checkout-agent-12",
  "message_id": "msg-73",
  "verification": "passed",
  "tool_errors": 0,
  "closed_reason": "verified"
}

No guardes el cuerpo completo por defecto. Un hash, el asunto y los identificadores de correlación suelen bastar para depurar. Si necesitas conservar contenido, redáctalo antes de enviarlo a los logs y define cuándo se elimina; la retención indefinida se vuelve un problema muy pronto.

Tradeoffs y puntos de comprobación

Un buzón real prueba más integración, pero cuesta más tiempo y añade fallos externos. Un servidor simulado es rápido y determinista, pero puede esconder problemas de entrega. Usa simulación para contratos y un buzón aislado para algunas pruebas de integración.

Antes de evaluar el modelo completo, comprueba por separado:

la aplicación emite el evento correcto;
la herramienta filtra por run_id;
el consumidor rechaza mensajes viejos;
el cierre ocurre después de éxito y después de timeout;
el recibo distingue infraestructura, aplicación y agente.

Así una regresión no queda etiquetada como “fallo del LLM” solo porque el proveedor de email estaba lento. El diagnóstico sale mas limpio y las comparaciones entre prompts son más justas.

Preguntas frecuentes

¿Es obligatorio usar un correo temporal en cada prueba? No. Para lógica de dominio, un mock suele ser suficiente. Usa un buzón aislado cuando quieras probar la integración real de entrega y lectura.

¿Cuántos reintentos debería permitir? Depende de la latencia esperada y del objetivo, pero debe ser un límite externo, visible en el recibo y finito. El agente no debe ampliar su propio presupuesto.

¿Dónde encaja temp mail email en este diseño? Puede ser una etiqueta contextual para encontrar un servicio de prueba en un entorno controlado; el enlace a temp mail email debe mantenerse como una referencia puntual, no como una sustitución del contrato ni de la validación.

La idea central es sencilla: un agente confiable no se evalúa por sonar seguro. Se evalúa por producir una cadena verificable de intención, entrega, consumo y cierre. Con límites claros, la automatización se vuelve repetible.

I finally cut AI context bloat by 80% and the fix was embarrassingly simple

Lars Winstand — Wed, 14 Oct 2026 14:12:07 +0000

A few months ago I watched an agent do something deeply cursed.

It had already fetched the data.
It had already parsed the data.
It had already stored the useful fields.

And then it pasted the entire raw tool response into the next model call anyway.

Full JSON blobs. Logs. Metadata. Useless fields. Everything.

The next step got slower.
The step after that got more expensive.
Then the model started doing that annoying thing where it answers the easy part of the task and quietly ignores the important part.

That was the moment the bug became obvious:

Most “context window problems” are not model problems.
They are state management problems.

Once you see that, you start spotting it everywhere in n8n flows, LangGraph agents, MCP tool loops, and custom OpenAI-compatible orchestration code.

The actual fix

Stop replaying raw tool output into every prompt.

Keep full results in structured state outside the model.
Send only:

a short summary
the exact fields needed for the next step
or the exact evidence slice if verification matters

That one change usually does more for agent quality than switching from one frontier model to another.

The anti-pattern

Here’s the bad version:

result = weather_api.get_forecast("Austin")

messages.append({
    "role": "user",
    "content": f"Use this API response to decide whether to send an alert:\n{result}"
})

If result contains 200 lines of hourly forecast data, provider metadata, timestamps, debug fields, and trace IDs, you just turned your prompt into a junk drawer.

Now the model has to:

find the relevant fields
ignore the irrelevant ones
remember prior steps
plan the next action
not get distracted by noise

Models can do this.
They are just not especially efficient at it.

The better pattern

Store the full result in app state.
Pass only the fields the next step needs.

result = weather_api.get_forecast("Austin")
state["weather_raw"] = result

weather_summary = {
    "temperature_f": result["current"]["temp_f"],
    "conditions": result["current"]["condition"],
    "wind_alert": result["alerts"].get("wind", False)
}

messages.append({
    "role": "user",
    "content": f"Decide whether to send an alert using this data: {weather_summary}"
})

Same workflow.
Smaller prompt.
Less distraction.
Better odds the model focuses on the decision instead of re-parsing junk.

Prompt size is not the same as prompt quality

A lot of builders confuse these two ideas:

the model can fit a long context
the model should receive a long context

Those are not the same.

A giant prompt is often just a badly managed state store wearing a trench coat.

Even when GPT-5, Claude Opus, Grok, Qwen, or Llama can technically accept the full history, performance still gets mushier as stale content piles up. Latency rises. Cost rises. Attention gets diffused.

This is why long-running agents often feel smart for 5 steps and weirdly flaky by step 20.

The 3 patterns, ranked

Approach	What happens in practice
Raw tool output in prompt	Biggest prompt growth. Highest distraction. Latency and cost climb every step.
Summarized tool result in prompt	Usually the best balance. Smaller prompt footprint, but bad summaries can hide edge cases.
Structured state outside prompt	Best default for multi-step agents. Small recurring prompts, cleaner reasoning, better orchestration.

My opinionated take: structured state outside the prompt should be the default.

If you are building anything more serious than a toy bot, replaying raw tool output into every step is lazy architecture.

What should live where?

This is the mental model I wish more agent builders started with:

Short-term conversational state stays attached to the thread.
Long-term facts and tool results live outside the prompt in a database or memory store.
The next model call gets only the smallest useful slice of that state.

That’s it.

Not complicated. Just disciplined.

Real examples

LangGraph: checkpoint state, don’t replay transcripts

LangGraph gets this right.

Instead of stuffing previous tool output back into every prompt, persist graph state and resume by thread.

from langgraph.checkpoint.postgres import PostgresSaver

DB_URI = "postgresql://postgres:postgres@localhost:5432/postgres?sslmode=disable"

with PostgresSaver.from_conn_string(DB_URI) as checkpointer:
    graph = builder.compile(checkpointer=checkpointer)

That is the grown-up version of memory.

The model gets what the next node needs.
Postgres keeps the rest.

n8n: trim memory on purpose

n8n is refreshingly practical here.

If you are building AI workflows in n8n, you already have the tools to avoid context bloat:

Simple Memory
Redis
Postgres
Xata
Zep
Motorhead
Chat Memory Manager

The existence of Chat Memory Manager should tell you everything.

If a workflow platform ships a node specifically to inspect and reduce memory, then “just append to chat history forever” is obviously not the best pattern.

MCP: tool output is app state first, prompt material second

MCP makes this cleaner because tools already have structure.

Example tool definition:

{
  "name": "get_weather",
  "description": "Get current weather information for a location",
  "inputSchema": {
    "type": "object",
    "properties": {
      "location": { "type": "string" }
    },
    "required": ["location"]
  },
  "outputSchema": {
    "type": "object"
  }
}

That should push you toward a simple pattern:

call the tool
store the full structured result
pass only the needed fields into the next prompt

For example:

tool_result = {
    "temperature": "72F",
    "conditions": "cloudy",
    "wind_alert": False,
    "hourly": [...],
    "provider_metadata": {...},
    "trace_id": "abc123"
}

state["weather_result"] = tool_result

prompt_payload = {
    "temperature": tool_result["temperature"],
    "conditions": tool_result["conditions"],
    "wind_alert": tool_result["wind_alert"]
}

The prompt becomes a working note, not a warehouse.

Prompt caching helps, but it does not fix bad context hygiene

This part gets misunderstood a lot.

Anthropic prompt caching is useful when you keep reusing the same stable prefix.

That means things like:

system instructions
tool definitions
fixed documents
repeated scaffolding

It does not magically make a sloppy prompt good.

Claude caching only helps when the prefix is identical, and cache entries last 5 minutes by default. That’s great for repeated prefixes. It does nothing to make a giant raw tool dump more relevant.

If anything, caching can hide bad architecture for a while because the bill hurts less before quality problems become obvious.

Example:

import anthropic

client = anthropic.Anthropic()
response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=1024,
    cache_control={"type": "ephemeral"},
    system="You are an AI assistant tasked with analyzing literary works.",
    messages=[
        {
            "role": "user",
            "content": "Analyze the major themes in Pride and Prejudice."
        }
    ],
)

Use caching for stable prefixes.
Do not use it as an excuse to keep shoving junk into prompts.

A quick before/after refactor

Here’s what this looks like in a normal agent loop.

Before

def run_step(state, user_request):
    search_results = search_docs(user_request)
    crm_data = fetch_customer_record(state["customer_id"])

    prompt = f"""
    User request: {user_request}

    Search results:
    {search_results}

    CRM data:
    {crm_data}

    Decide what to do next.
    """

    return llm(prompt)

This works at first.
Then search_results gets huge.
Then crm_data gets huge.
Then every step drags around more baggage.

After

def run_step(state, user_request):
    search_results = search_docs(user_request)
    crm_data = fetch_customer_record(state["customer_id"])

    state["search_results_raw"] = search_results
    state["crm_data_raw"] = crm_data

    prompt_payload = {
        "user_request": user_request,
        "top_matches": [doc["title"] for doc in search_results[:3]],
        "customer_tier": crm_data["tier"],
        "open_tickets": crm_data["open_tickets_count"],
        "billing_status": crm_data["billing_status"]
    }

    prompt = f"Decide what to do next using this state: {prompt_payload}"
    return llm(prompt)

This version is less fragile.

And when the model needs exact evidence later, you can pull the raw state back in selectively.

When should you keep raw output in the prompt?

There are real exceptions.

Sometimes the next step genuinely needs exact source material.

Keep raw output in-context when you need:

legal or compliance citations
exact stack traces for debugging
code review on full diffs
extraction from messy text
verification against original evidence

But even then, pass the exact slice that matters.
Not the entire upstream universe.

The rule I use now

This is the heuristic that cleaned up most of my agent workflows:

If information is needed for reasoning, summarize it.
If it is needed for retrieval, store it.
If it is needed for verification, pass the exact slice.

That rule has saved me more latency, more cost, and more agent quality than swapping models ever did.

Why this matters even more when you pay per token

This problem gets expensive fast in production.

If your agent runs 24/7 in n8n, Make, Zapier, OpenClaw, or a custom loop, every extra blob you keep replaying becomes recurring cost.

That’s one reason teams end up obsessing over token usage dashboards instead of fixing the architecture.

A cleaner state design reduces prompt bloat at the source.

And if you are using an OpenAI-compatible endpoint with flat monthly pricing, this gets even nicer because you can optimize for agent quality and throughput instead of constantly asking whether one more tool call is going to spike the bill.

That’s the part I like about Standard Compute: it works as a drop-in OpenAI API replacement, so you can keep your existing SDKs and agent code, but stop thinking in per-token panic mode while you clean up orchestration.

The architecture fix still matters.
You just get to apply it without token anxiety hanging over every experiment.

Practical checklist

If your agent feels slower and dumber over time, check these first:

Are you pasting raw tool output into every next prompt?
Are you storing canonical results outside the model?
Are you summarizing tool results before the next reasoning step?
Are you passing exact evidence only when verification is required?
Are you using prompt caching only for stable prefixes?
Are you trimming memory in n8n or your orchestration layer?
Are you treating the model like a reasoning engine instead of a database?

If you fix just those, you will probably get a bigger improvement than another model bake-off.

I went looking for a model fix.

What I found was an architecture fix.

Most “context window problems” are really just us using chat history as a storage layer.

Once you stop shoving raw tool output back into the next prompt, agents get faster, cheaper, and noticeably less confused.

Not because the models changed.

Because we finally stopped making them carry our state management mistakes.

How to Choose the Right AI Software Development Company in 2026?

hoor ali — Wed, 14 Oct 2026 14:10:44 +0000

To choose the right AI software development company, look for a partner that starts with your business problem, has shipped AI to production in your industry, owns data and MLOps end to end, follows a recognized AI risk framework, and prices against outcomes rather than hours. Ask for live references, a paid discovery phase, and a clear plan for what happens after launch.

Almost every business is now experimenting with AI. In McKinsey's 2025 State of AI survey, 88% of respondents said their organization uses AI in at least one function, yet only 7% said it was fully scaled. The gap between a promising pilot and a system that earns its keep is usually the partner you pick. This guide walks through how to close it.

What does an AI software development company do?
An AI software development company designs, builds, and maintains software that uses machine learning, large language models (LLMs), or AI agents to automate decisions and tasks. Typical AI software development services include:

AI strategy and discovery: finding use cases with a measurable return.
Data engineering: cleaning, labeling, and piping the data models depend on.
Custom model and LLM development: fine-tuning, retrieval-augmented generation (RAG), and agent workflows.
AI integration: adding AI features to your existing apps, CRM, or ERP.
MLOps and support: monitoring, retraining, and cost control after launch. This is different from a team that simply uses AI in software development, such as coding assistants that help engineers write code faster. Most good partners do both, but you are hiring for the first.

7 criteria for choosing an AI software development company:

They start with the problem, not the model:
Strong partners ask about your workflow, KPIs, and budget before they mention GPT, Claude, or Gemini. If the first call is a tech demo, expect a solution looking for a problem.
Proven production work in your industry:
Ask for two or three case studies with numbers: hours saved, error rates cut, revenue gained. Then ask to speak with those clients. A pilot that never went live is not a reference.
Data readiness and security:
AI is only as good as its data. The company should assess your data quality early and explain where your data is stored, who can access it, and whether it is used to train third-party models. Look for SOC 2 or ISO 27001 practices and compliance with rules such as GDPR or HIPAA where they apply.
Responsible AI and governance:
Ask how they test for bias, hallucinations, and failure cases. Partners who align with the NIST AI Risk Management Framework can show you a documented process for mapping, measuring, and managing AI risk.
Model-agnostic architecture:
The best model today may not be the best in six months. Prefer a team that can swap LLM providers or run open-source models without a rebuild, so you avoid vendor lock-in and control costs.
MLOps and post-launch support:
Models drift as data changes. Confirm who monitors accuracy, how often models are retrained, and what the support SLA covers. Get the cost of running the system, including API and cloud fees, in writing.
Clear ownership and transparent pricing:
Your contract should state that you own the code, prompts, fine-tuned weights, and data. Pricing should be broken into phases with defined deliverables.

Red flags to watch for:

Guaranteed accuracy figures before seeing your data.
No paid discovery phase; a fixed quote after one call.
Vague answers on data privacy or model training.
Every project uses the same tool, whatever the problem.
No plan for maintenance after launch.

Common engagement and pricing models:
Most AI software development companies offer four engagement models. A paid discovery phase or proof of concept tests feasibility in 2–6 weeks, and it doubles as a test of the team, not just the idea. A fixed-scope project suits well-defined features with clean data, though change requests can get expensive.
Time and materials fit evolving products and R&D, but it needs strong sprint reporting to keep costs visible. A dedicated AI team works best for long-term roadmaps, as long as you have your own product owner to steer it. Whichever model you choose, start with a small, paid discovery phase; it tells you more about how a company works than any sales deck.

Frequently asked questions:

How much does it cost to hire an AI software development company?

Costs depend on scope and data readiness. A discovery phase or proof of concept is usually the smallest commitment, while a production system with integrations and MLOps costs much more. Ask for phased quotes and a monthly estimate of running costs such as API and cloud fees.

How long does an AI software project take?

A proof of concept typically takes 2–6 weeks. A production-ready AI feature often takes 3–6 months, depending on data quality, integrations, and compliance reviews.

What is the difference between AI software development services and using AI in software development?

AI software development services build products that contain AI, such as chatbots, prediction engines, or agents. AI in software development refers to tools that help engineers write, test, and review code faster. A good partner uses the second to deliver the first more efficiently.

Should I build an in-house AI team or outsource?

Outsourcing is faster when you need specialized skills now or are testing a first use case. An in-house team makes sense once AI is core to your product. Many companies start with a partner and hire internally as the system matures.

I verified all 160 official U.S. business-data portals so your agent doesn't have to — here's what I found

bradraylucas-eng — Wed, 14 Oct 2026 14:07:46 +0000

Every agent workflow I build starts with the same wasted motion: the agent searches the web for the official registry, lands on an ad or an aggregator, and burns its run. So I verified the official portals myself — business entity search (50 states + DC + federal), UCC lien search, and professional license verification — and wrote down what agents actually need to know before they start.

Findings that surprised me:

Texas is the only state that charges for a basic business entity search ($1 via its portal) — and it retired the system agents' training data points to (SOSDirect) in September 2026.
UCC search is a different world: 21 of 51 jurisdictions charge, 18 require an account before you can search, and Delaware has no public search at all (state-licensed "Authorized Searchers" only).
License verification has no center: only 21 of 51 jurisdictions have a true central portal. California's main portal excludes contractors and real estate entirely. A "not found" there proves nothing.
In 10+ states the business registry is not run by the Secretary of State at all.
A large share of these portals return 403/bot challenges to plain HTTP clients. If your agent fetches with a simple client, it needs a browser session for these — the datasets flag them per portal.

I packaged the verified data (JSON + CSV, documented schema, per-entry cost/login/access notes, last verified October 2026) as the Source Finder datasets on Gumroad, with free samples so you can check the schema before paying anything: https://github.com/bradraylucas-eng/ai-agent-tools-free-samples · Vol. 1 on Gumroad: https://bradray2.gumroad.com/l/fxnjw

If you build agents that do due diligence, vendor checks, or lead verification, I'd genuinely like to know what other official sources you re-hunt every run — that's what Vol. 4 should be.

— Brad (maker; the datasets are my product, the samples are free)

2026 Frontier MoE VRAM Sizing & Bare-Metal GPU Repatriation Math (DeepSeek-V4.1-Flash, Kimi K2.5, GLM-5.3)

Asif Muhammad Iqbal — Wed, 14 Oct 2026 14:02:57 +0000

Originally published with an interactive 8-Blade HGX Silicon X-Ray simulator at Logic42 Sovereign Engineering Practice.

Most enterprise infrastructure teams sizing private GPU clusters for 2026 Mixture-of-Experts (MoE) models make a costly arithmetic mistake: they size HBM3e VRAM for active parameters per token instead of the full resident checkpoint + concurrent PagedAttention KV cache.

If you try to deploy DeepSeek-V4.1-Flash (552B MoE + 196B Engram, 8B–16B active per token) or Moonshot Kimi K2.5 (1.04T total, 32B active) on an undersized cluster, vLLM will either OOM during weight initialization or evict KV-cache blocks on every concurrent multi-turn agent request.

You can test your exact concurrency and token volume live in our Sovereign MoE Silicon X-Ray & Cloud Repatriation Calculator. Below is the complete engineering math and 2026 reference table.

1. The 3-Component MoE VRAM Sizing Formula

Total cluster VRAM required at production concurrency ($C$ concurrent streams at 32k context) is governed by three additive components plus a mandatory $\ge 6\%$ HBM3e fragmentation buffer:

$$\text{VRAM}{\text{total}} = \left(\text{VRAM}{\text{weights}} + \text{VRAM}{\text{KV}}(C) + \text{VRAM}{\text{CUDA}}\right) \times 1.06$$

Resident MoE + Engram Weights ($\text{VRAM}_{\text{weights}}$): Every routed and shared expert must reside in high-bandwidth GPU memory across your Tensor Parallel (--tensor-parallel-size) or Expert Parallel (--enable-expert-parallel) group. Offloading inactive experts over PCIe Gen5 to host DDR5 RAM degrades token generation latency by $8\times\text{–}14\times$.
Active PagedAttention KV Cache ($\text{VRAM}_{\text{KV}}$): Scales linearly with concurrent streams ($C$) and context window length. Models using Multi-Head Latent Attention (MLA) like DeepSeek-V4.1-Flash compress KV states to ~2.4 GB per 32k stream (FP8 KV), whereas wider architectures consume 3.0 GB–3.8 GB per 32k stream.
CUDA Graph & NCCL Buffer Overhead ($\text{VRAM}_{\text{CUDA}}$): PyTorch activation scratchpads, NCCL AllReduce ring buffers, and CUDA graph capture consume ~3.5% of resident weight memory (minimum 6 GB–20 GB per node).

2. Q4 2026 Frontier MoE & Dense VRAM Sizing Reference Table (at 40 Concurrent 32k Streams)

2026 Frontier Model	Architecture (Total / Active)	Native FP4 / INT4 Weights	FP8 Weights	BF16 Weights	KV Cache @ 40 Streams (32k)	Minimum Recommended HGX Cluster
DeepSeek-V4.1-Flash	`552B MoE + 196B Engram` (`8B/16B` active)	`511 GB` (MXFP4/FP8)	`590 GB`	`1,104 GB`	`96 GB`	8× H200 SXM (1,128 GB)
Moonshot Kimi K2.5	`1.04T MoE` (`32B` active, 384 experts)	`595 GB` (QAT-INT4)	`1,050 GB`	`2,080 GB`	`128 GB`	8× H200 SXM (1,128 GB) (INT4) or 8× B200 (1,440 GB) (FP8)
Zhipu GLM-5.3-Flash	`320B MoE` (`18B` active)	`96 GB`	`185 GB`	`360 GB`	`64 GB`	4× H100 SXM (320 GB) (FP8)
Zhipu GLM-5.2 Flagship	`744B MoE` (`40B` active)	`215 GB`	`410 GB`	`780 GB`	`120 GB`	8× H100 SXM (640 GB) (FP8)
DeepSeek-V4-Pro	`1.6T MoE` (`49B` active)	`480 GB`	`920 GB`	`1,780 GB`	`152 GB`	8× B200 Blackwell (1,440 GB) (FP8)
Zhipu GLM-4.5-Air	`106B MoE` (`12B` active)	`34 GB`	`64 GB`	`124 GB`	`44 GB`	4× L40S (192 GB) or 4× H100 (320 GB)
Qwen 3.8-27B Dense	`27B Dense` (`27B` active)	`16 GB`	`29 GB`	`56 GB`	`36 GB`	2× L40S (96 GB)
Llama 4 Scout	`109B MoE` (`17B` active, 16 experts)	`35 GB`	`68 GB`	`132 GB`	`48 GB`	4× L40S (192 GB)

3. What Bare-Metal GPU Node Rates Include vs. Exclude

When comparing commercial API token spend ($4.60–$7.20 / 1M blended tokens on GPT-5 or Claude 4.5 Sonnet) against dedicated sovereign bare-metal clusters ($13,500/mo for an 8×H100 node or $17,800/mo for an 8×H200 node on a 36-month reserved lease):

Included Inside the Server Chassis BOM:

Host CPUs: Dual Socket AMD EPYC 9654 / 9754 (192–256 physical cores) or Intel Xeon Platinum 8592+
System RAM: 2.0 TB – 4.0 TB DDR5-5600 ECC Registered memory for fast safetensors mmap loading
Local NVMe Storage: 30.7 TB – 61.4 TB PCIe Gen5 NVMe U.3 RAID0 scratch array (55+ GB/s sequential read)
Interconnect Fabric: 900 GB/s – 1.8 TB/s NVLink / NVSwitch backplane + 8× 400GbE RoCEv2 / InfiniBand NDR NICs

Excluded (On-Premises Colocation & Electrical Utility):

If deploying into your own Tier-3/Tier-4 colocation cage rather than a reserved bare-metal cloud lease, budget separately for rack space, Direct Liquid Cooling (DLC), and continuous electrical draw (10.2 kW–14.2 kW IT load per 8-GPU chassis at PUE 1.30, adding roughly $950–$1,450/mo depending on regional industrial kWh tariffs).

Interactive Tool: Run Your Own MoE VRAM & 24-Month TCO Simulation

You can inspect the live 8-Blade HGX GPU Rack X-Ray (G1–G8), test concurrency sliders, view the 24-month cumulative payback curve, and export a 1-page CFO Business Case PDF at:

👉 Launch the Sovereign MoE Silicon X-Ray & Cloud Repatriation Calculator

You can also explore our 4 interactive Adversarial & Failure X-Ray Playbook Hubs:

Character.ai Filed for Bankruptcy. How to Export Your Chats Before the Sale

Alan West — Wed, 14 Oct 2026 13:40:03 +0000

Character.ai filed for Chapter 11 bankruptcy on October 12, and the only bid on the table so far is Disney's $20 million for a license to some of its IP, which doesn't cover the app you chat in. If you have two years of roleplay, a dozen custom characters, or a chat log you actually care about, you have roughly six weeks to export your Character.ai chats before a court decides who ends up with the company and your data.

Character Technologies started a voluntary Chapter 11 process in federal court in Delaware on October 12, 2026. The company expects the court-supervised sale to finish "by the end of November or perhaps early December 2026." If no bid keeps the platform running, the official update says it "will be wound down."

Below I go through what the announcement actually promises, how to request the official export, what the CAI Tools extension gets access to, and how I'd rebuild the characters in SillyTavern + Ollama on your own machine.

What Character.ai's bankruptcy announcement says about your data

The announcement on the Character.ai blog is short. It commits to four things:

The platform stays up and all services remain accessible for now.
Paid subscriptions keep billing as normal, and subscribers get notice before paid features change.
User data stays covered by the existing Privacy Policy.
You can export your data or delete your account from the app or the website.

The FAQ heading literally asks "What happens to user data and chats?" and then the answer never mentions chats. User-created characters don't come up at all.

The buyer side doesn't help much. According to The Next Web, Disney's $20 million opening bid buys a non-exclusive, perpetual license to certain IP and certain legal claims, plus a deal to lend back former Character.ai staff it hired so the platform can keep running safely for a while. Nothing in the coverage says Disney is buying the consumer app. The $20 million is a floor that other bidders can top, and the filing puts assets and liabilities at $50 million to $100 million.

So the floor bid might keep the lights on for a while, and that's all it promises. For the app to survive, somebody has to want a chatbot platform that settled five teen mental-health lawsuits in January and has faced more claims from families and US states since.

In the Hacker News thread, commenters brought up 23andMe and Spirit Airlines, two companies whose data went up for sale once they went under. "Privacy Policy unchanged" describes today. It doesn't tell you what a buyer gets to do with your logs after the sale closes.

How to export your Character.ai data with the official request

Do this first. It takes two minutes. Character.ai's own help center returned a 403 when I tried to load it, so these paths come from third-party guides, and the guides don't agree with each other. Menu names may have moved.

On the web:

Sign in at character.ai.
Open Profile Settings, then Account, then Manage Account & Data.
Pick the export data option.

In the mobile app, look for Export my data in account settings (one guide puts it under Settings → Privacy, another under Legal & Privacy). The official update says export works from both the app and the website, though one guide claims the desktop site doesn't have the option. If you can't find it on the web, use the phone.

Things to know before you rely on it:

There's no per-chat export. You get one archive of the whole account (LLMnesia's guide).
It's a snapshot. Anything you chat after you request it isn't included, so request it again right before you stop using the service.
Delivery reportedly goes to email. One guide says the download link arrives by email, expires (it guesses 24 to 48 hours), and can take anywhere from about an hour to a few days to show up. It also says the ZIP has a folder named after your username with a data/ subfolder holding user.json, character.json and message.json. I haven't seen Character.ai confirm that file list. Open the archive and check for yourself.
Don't spam the button. That same guide says every request queues separately. Assume everyone else read the same headline and is in that queue with you.

The archive is raw JSON. You can grep it, but it won't load straight into another app.

Export characters and chats with CAI Tools (read the permissions first)

To get something another app can import, most people will reach for CAI Tools. SillyTavern's own docs point to it for importing Character.ai chats and bots. According to its Firefox listing, it downloads chats in a "readable" or a "transferable" format and downloads characters as cards. The Chrome listing says JSON and PNG cards, and PNG cards are what SillyTavern imports.

This is what you're agreeing to when you install it, going by the public listings:

	CAI Tools (Firefox, v3.8.2)
Last updated	Oct 9, 2026
Users	5,597
Required permissions	Browser tabs, site data for character.ai and chatbotmanager.net
Optional permissions	character.ai, chub.ai, jannyai.com, chatgpt.com, janitorai.com
License	All Rights Reserved
Paid features	"Some features may require payment" (Patreon)

The GitHub repo says outright: "Code here is not up-to-date." It exists for bug reports. You can't audit the build you install against that source.

So you'd be giving a closed-source extension your logged-in character.ai session, in the same tab as your most private conversations. I still think it's the right call, because nothing else gets your data out in a format another app can load. Just be careful about it:

Install it in a separate browser profile that's only logged into Character.ai.
Export everything in one sitting, then uninstall it.
If you're paranoid (fair), log out of every session afterwards and change your password.
Lots of similarly named "Character.ai Exporter" extensions are going to show up in the stores this month. Don't install any of them unless you can explain who wrote it.

Also open every exported character card and look at the definition field. If it comes out empty, you've saved a name, an avatar and a greeting, and nothing else.

Run your Character.ai characters locally with SillyTavern and Ollama

Once your characters are PNG cards and your chats are files on disk, you don't need Character.ai's servers anymore. The local setup takes about ten minutes.

Step 1 is installing Ollama and pulling a model. On Linux:

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1:8b

Choose a model that fits your RAM or VRAM. The Ollama library has plenty, and roleplay quality is a matter of taste, so try a few.

Step 2 is raising the context window, which matters more than which model you pick. The Ollama FAQ says "By default, Ollama uses a context window size of 4096 tokens." A character definition plus a long chat history will blow through that, and the character will start forgetting who it is. The FAQ's example raises it server-wide:

OLLAMA_CONTEXT_LENGTH=8192 ollama serve

One catch on Linux: the install script already runs Ollama as a systemd service, so that command will fail with "address already in use". Put the variable in the service instead:

sudo systemctl edit ollama
# add under [Service]:
#   Environment="OLLAMA_CONTEXT_LENGTH=8192"
sudo systemctl restart ollama

Ollama binds to 127.0.0.1:11434 by default. Leave it there. The FAQ shows how to set OLLAMA_HOST to 0.0.0.0:11434 to expose it on your network, and if you do that, anyone on the network can talk to the server.

Step 3 is SillyTavern, from the release branch:

git clone https://github.com/SillyTavern/SillyTavern -b release
cd SillyTavern
./start.sh

You'll need git and Node.js installed first. It opens in your browser, by default at http://127.0.0.1:8000.

Step 4: connect the two. In the API connections panel, choose Text Completion, set the API type to Ollama, and enter http://127.0.0.1:11434. Then pick your model from the dropdown.

Step 5: import. Import the PNG card from the character list first, because a chat has to belong to a character. Then open that character, open Manage chat files from the options menu at the bottom left, use the import button there, and pick the exported chat file.

After that, your characters live in a folder you can back up with rsync, put in git, or copy to a USB stick.

Character.ai export checklist

Request the official export now. Request it again before you leave.
Download the email link as soon as it arrives.
Export cards and chats with CAI Tools in a separate browser profile, then uninstall it.
Check that every card actually contains a definition.
Set up SillyTavern + Ollama and import one character to prove the pipeline works.
Decide whether to delete your account before the sale closes. You've got your copy. Whether the buyer also gets one is up to you.

A local 8B model is worse than what Character.ai ran at its peak. I'll take that trade over having my chat logs listed as an asset in a Delaware court.

So which is it for you: would you trust another hosted AI companion app after this, or is local the only safe option now? And if you're staying hosted, what would a company have to promise you about data in a bankruptcy before you'd sign up?

Sources

An update on Character.ai (official announcement)
Hacker News discussion
The Next Web: Character.AI files for bankruptcy as Disney bids $20M
CAI Tools on Firefox Add-ons and GitHub
SillyTavern docs: chat file management and Linux/macOS install
Ollama FAQ

AI agent development cost: what actually moves the number (2026)

Joan Comadran — Wed, 14 Oct 2026 13:35:36 +0000

Here's the short answer people actually want: AI agent development cost is mostly scope, not model fees. The model bill is real and easy to measure. The number that surprises founders is the engineering around the model — the channel, the tools that write to real systems, multi-tenant isolation, evals, and the week you spend hardening the webhook so Meta doesn't double-send every reply.

I don't publish a rate card here, and I'm not going to invent a "market average" with fake precision. What I can do is show what moves the quote, what it costs to run an agent on Claude as of October 2026, and what I learned shipping Claient (WhatsApp lead agent), Adorea.ai, and Dairector. If you're comparing agency cost guides that all land on $30k–$180k, treat those as their sales bands — useful as a sanity check, not as a standard.

The three costs people mix up

Cost	What it is	Who pays
Build	Engineering to ship the agent: product, tools, auth, channel, evals	You, once (or in slices)
Run	Model tokens, hosting, channel fees, observability	You, every month
Own	Maintenance when models deprecate, prompts drift, APIs change	You, forever

Most "AI agent development cost" pages only talk about the first row, then bury run cost in a footnote. In production the second row is where a good architecture either stays cheap or quietly burns the margin — I wrote the token side up in keeping Claude's token bill from ruining your month.

What actually moves the build number

These are the drivers I budget against, in the order that usually bites.

1. How many systems the agent can write to. A read-only FAQ bot over your docs is one shape. An agent that books appointments, creates CRM rows, or sends WhatsApp templates is another. Every write-action needs schemas, retries, idempotency, and a human-approval path when the stakes are high. Tool count is a weak proxy; write count is the real one.

2. The channel. WhatsApp via Meta Cloud API is not "add a chat widget." You need webhook verification, signature checks, idempotency on wamid, and a plan for Meta's retries when your serverless function times out. That alone ate more engineering time than the Claude loop on Claient. Email or an in-app chat is usually cheaper. Voice is its own project.

3. Single-tenant vs multi-tenant. Solving the agent for one business is a focused build. Solving it so any tenant self-serves, with RLS on account_id, encrypted BYOK keys, and one webhook routing by phone_number_id, is a product. Claient only becomes interesting as a business once the second version exists.

4. Evals and observability. Cheap agents skip them. Expensive incidents come from skipping them. Budget time for a golden set of real conversations, cost logging per account, and traces you can read when a lead gets the wrong answer at 23:00.

5. Who owns the model key. Flat product price with your key absorbed (Claient defaults to Haiku) is a different quote from BYOK, where the tenant pays Anthropic directly. BYOK lowers your variable cost and raises onboarding and vault work.

Rough shape from builds I've done: a single-task agent with a couple of tools and one channel is a few weeks of senior work to something clickable; hardening + multi-tenant + billing stretches that into months. I won't turn that into a fake dollar table — hours × whoever you hire is the only honest formula.

Running cost: model rates you can verify (as of Oct 2026)

List prices from Anthropic's pricing page, per million tokens (input / output). They change; check before you budget.

Model	Input	Output	Notes
Claude Haiku 5.5 (≤100K prompt)	$0.10	$0.50	Cheap high-volume tier
Claude Haiku 4.5	$1	$5	What Claient still defaults to in production
Claude Sonnet 5.5	$2	$10	Workhorse for harder reasoning
Claude Opus 5.5	$4	$20	Heavy tasks, drafting, research
Claude Fable 5.1	$10	$50	Top tier

Prompt cache reads are a fraction of input (0.05×–0.1× depending on model; Fable goes lower). Batch is 50% off when you can wait. Client-side tool use is just tokens; Anthropic web search is $10 per 1,000 searches on top.

What that means in practice for a WhatsApp lead agent:

Default to a cheap model for the conversation loop. Claient runs Haiku by default; tenants who want Sonnet/Opus bring their own key.
Cache the static system prompt and tool schemas. Don't resend the full history every turn — use a short verbatim window plus a rolling summary (details here).
Log real USD per account. If you can't see cost per conversation, you can't price the product.

Hosting for a Next.js + Supabase agent is usually noise next to a bad model choice: tens to low hundreds of dollars a month until you have serious traffic. Channel fees matter more — Meta service conversations within 24h on Cloud API direct are free, which is why Claient can sit at €49/month with ~0 € platform cost per customer. A BSP in the middle would have broken that price.

Build vs buy vs hire

Buy a SaaS when your use case is generic (FAQ, basic lead capture, appointment booking templates) and you're fine renting their qualification rules. Fastest path; you pay forever and bend to their product.

Build in-house when you already have a senior engineer who has shipped LLM tool-use, and the agent is core IP. Still budget for the channel and for the month after launch, not just the demo week.

Hire when you need production reliability and don't have that person on payroll. Fit for custom tools, multi-tenant SaaS, WhatsApp/CRM write-paths, or a first LLM product where a wrong architecture is expensive to unwind. Not fit for a slide-deck prototype.

I ship the hire path from Barcelona, remote-friendly: TypeScript, Next.js, Supabase, @anthropic-ai/sdk directly — no LangChain tax unless the project truly needs multi-provider orchestration. Details and process on hire an AI engineer.

When it's worth it / when it's not

Worth building or hiring for when:

Leads or revenue depend on speed (out-of-hours WhatsApp, inbound that dies in an inbox).
The agent must call your tools with real side effects, not just chat.
You need multi-tenant isolation, BYOK, or channel economics a template can't give you.
You're launching a product (not a one-off script) and need cost controls from day one.

Not worth it when:

A Zapier/Make/no-code bot already covers the happy path.
You only need an internal demo for a stakeholder meeting.
Nobody will own evals and model updates after the first deploy.
The "agent" is really a single LLM call with no tools — that's a feature, not a project.

Verdict

If someone quotes you AI agent development cost as one number with no breakdown of channel, write-tools, tenancy and run model, they're selling a vibe. Ask for: scope of write-actions, who pays for tokens, how evals work, and what the monthly run cost looks like at your expected volume on a named model and caching strategy.

My default for a production lead or support agent in 2026: Claude via the official SDK, cheapest model that passes your eval, prompt caching on, conversation memory bounded, channel owned directly. That's the shape behind Claient. The build cost is the engineering to make that boring and safe; the run cost is then a line item you can actually forecast.

Want that built end-to-end — agent, product shell, tracking — talk to me on the AI engineering page. Related reading: Claude vs GPT for AI agents, WhatsApp bot with Claude, ship an AI product in a weekend.

FAQ

How much does it cost to build an AI agent in 2026?

There is no single market price. What moves the number is scope: one channel vs many, how many write-actions the agent can take, multi-tenant isolation, evals, and who owns the channel (WhatsApp, CRM, calendar). A focused single-task agent with a few tools is weeks of senior engineering; a multi-tenant product with billing, dashboards and hardened webhooks is months. Vendor cost guides often quote $30k–$180k+ for production agents — treat those as rough bands from their sales pages, not a standard. Running cost is separate and usually dominated by model tokens.

What does it cost to run an AI agent after you build it?

Mostly model tokens plus hosting. As of October 2026 on Anthropic's list prices, Claude Haiku 5.5 starts at $0.10 / $0.50 per million input/output tokens (prompts up to 100K), Haiku 4.5 is $1 / $5, Sonnet 5.5 is $2 / $10, and Opus 5.5 is $4 / $20. Prompt cache reads are a fraction of input. A WhatsApp lead agent on Haiku with caching and a rolling conversation window stays in the cents-per-conversation range for typical volumes; the same agent on a frontier model without caching does not. Add hosting (often tens to low hundreds of dollars a month on Vercel + a managed Postgres) and whatever the channel charges.

Is it cheaper to buy a SaaS chatbot than to build a custom AI agent?

Usually yes for generic FAQ or lead capture. Buy when a template covers your channel and qualification rules. Build (or hire) when you need custom tools, multi-tenant isolation, a channel you must own (for example Meta Cloud API direct), BYOK, or product economics that SaaS pricing breaks. Claient only made sense as a product because Meta service conversations within 24h are free on Cloud API direct — a BSP fee would have eaten the $49/month price.

When is it worth hiring an AI engineer to build an agent?

When the agent has to touch real systems (WhatsApp, CRM, calendar, payments), when wrong answers cost you leads or trust, or when you need multi-tenant security and cost controls from day one. Not worth it for a one-off internal demo or a FAQ bot a no-code tool already covers. If you want someone who has shipped production agents end-to-end, see the AI engineering page.

Do I need LangChain or an agent framework in the budget?

No. Production agents I ship call @anthropic-ai/sdk directly: a tool-use loop is a few dozen lines. Frameworks help with multi-provider routing or complex graphs; they also hide the provider features (prompt caching breakpoints, server tools) that usually save the most money. Budget for evals, observability and cost logging before you budget for a framework.

Originally published on joancomadran.dev. I'm Joan, an AI engineer in Barcelona — work with me.

Claude vs GPT for AI agents: what I actually ship with (2026)

Joan Comadran — Wed, 14 Oct 2026 13:34:31 +0000

Here's the verdict, so you can leave early if that's all you came for: I build my AI agents on Claude, and I'd still tell most people that the honest answer to "Claude vs GPT for agents" in late 2026 is "it matters less than you think." As of October 2026 the two price ladders mirror each other almost to the cent, both have 1M-token context windows, both enforce tool schemas strictly, and both host web search for you. What's left to choose on is API shape, how much control you get over caching, which hosted tools you need, and which one your evals prefer on your tool schemas.

If you want a default: pick Claude if your agent lives on long, growing context and you care about controlling exactly what gets cached. Pick GPT if you're building a voice agent, want hosted file search over your documents, or your company already runs on OpenAI or Azure contracts.

One disclosure before the table, because comparison posts rarely make it: every agent I run in production is on Claude. I've used OpenAI's API, but I don't have a shipped GPT agent to compare like-for-like, so the GPT column below comes from OpenAI's docs and pricing pages, not from war stories. The Claude column comes from code I can show you.

Claude vs GPT at a glance (as of October 2026)

All prices are list prices per million tokens (input / output), taken from the Anthropic and OpenAI pricing pages in October 2026. These change often; check before you budget.

	Claude (Anthropic)	GPT (OpenAI)
Current lineup	Fable 5.1, Opus 5.5, Sonnet 5.5, Haiku 5.5	GPT-6 Astra, GPT-6.1 Sol, GPT-6 Luna
Top tier	Fable 5.1: $10 / $50	GPT-6 Astra: $10 / $50
Between top and mid	Opus 5.5: $4 / $20	—
Workhorse tier	Sonnet 5.5: $2 / $10	GPT-6.1 Sol: $2 / $10
Cheap, high-volume tier	Haiku 5.5: from $0.10 / $0.50 (prompts up to 100K tokens)	GPT-6 Luna: from $0.10 / $0.50 (short context)
Context window	1M tokens on all current models	1.05M tokens on the flagship three
Long-context billing	Full 1M at the standard rate (except Haiku 5.5 above 100K)	Separate, higher long-context rate
Max output	128K tokens	128K tokens
API you build agents on	Messages API (`tool_use` / `tool_result` blocks)	Responses API (required for tool calling on Astra and Sol)
Schema-enforced tool calls	`strict: true` on tool definitions	`strict: true`; Responses tries to make tools strict by default
Prompt caching	Explicit `cache_control` breakpoints or automatic; cache reads at 10% of input or less	Discounted cached input (5–10% of input on the flagship three)
Batch discount	50%	50%
Hosted tools	Web search ($10 / 1K searches), web fetch, code execution, tool search, MCP connector, computer and browser use	Web search ($10 / 1K calls), file search, hosted shell / code interpreter, computer use, MCP, tool search
Voice / realtime	Text and image in, text out	Dedicated realtime and voice models

Look at the price rows for a second. Not long ago, a "Claude vs GPT" post was half about cost. Now the cheap tiers start at the same price, the workhorse tiers are identical, and the top tiers are identical. Rate cards aren't where the difference is anymore.

Where they actually differ for agent builders

The API shape

On Claude, an agent turn is a Messages API call. If the model wants a tool, the response has stop_reason: "tool_use" and one or more tool_use blocks; you run them and send back a user turn full of tool_result blocks. On GPT, the newest models require the Responses API for tool calling: the output is a list of items, tool calls come back as function_call items, and you return function_call_output items, optionally chaining with previous_response_id.

Neither is better in the abstract. They are different enough that "we'll just swap the model later" is a week of work, not a config change, once you've built retries, caching, cost logging and error handling around one of them.

Caching control

This is the one that decided it for me. Claude lets you mark exactly where the cacheable prefix ends with cache_control breakpoints (or let the API place them automatically). In a conversational agent, where the system prompt is static and the history grows every turn, being able to say "cache everything up to here" is the difference between cost that grows with the square of the conversation and cost that grows roughly linearly. I wrote up the full story in keeping Claude's token bill from ruining your month.

OpenAI discounts cached input heavily too, so this isn't "Claude caches, GPT doesn't." It's about how much you get to steer it.

Long context billing

Both vendors offer roughly a million tokens of context. The difference is the bill: Anthropic charges the full 1M window at the standard rate on its current models (Haiku 5.5 aside), while OpenAI lists a separate, higher long-context price on Astra, Sol and Luna. If your agent stuffs whole codebases, contracts or long transcripts into context, model that difference before you pick. If your prompts are a few thousand tokens, it doesn't matter.

Hosted tools

Both now run web search on their side and support MCP. OpenAI has hosted file search with managed storage and a full line of realtime voice models; if you're building a phone or voice agent, that alone probably decides it. Anthropic has server-side web search and web fetch, code execution, a tool-search tool for large tool sets, and an MCP connector that talks to remote MCP servers straight from the Messages API.

What I actually run in production (and why)

None of my agents use LangChain or the Vercel AI SDK. Every one calls @anthropic-ai/sdk directly, because the provider-specific features are exactly the ones I'm relying on. Here are two, straight from the repos.

Claient: a multi-tenant WhatsApp agent on Haiku

Claient answers and qualifies WhatsApp leads for small businesses (the build story is here). The model choice is gated by who pays:

// With the platform key (we pay), only Haiku is served.
// Tenants who bring their own key (BYOK) can pick Sonnet or Opus.
export const MODELO_BASE = "claude-haiku-4-5-20251001";

export function modeloEfectivo(opts: { apiKey?: string | null; model?: string | null }) {
  if (!opts.apiKey) return MODELO_BASE;
  return opts.model || getClaudeConfig().model;
}

The reply runs in a tool-use loop with three booking tools (check availability, book, reschedule). Two guards matter more than the model: a hard cap of five loop iterations, and a time budget, because a serverless function killed mid-loop means the lead got a message but the webhook didn't finish, and Meta's retry sends it twice:

const MAX_ITER_TOOLS = 5;
const MARGEN_VUELTA_MS = 20_000; // SDK timeout 15 s + 1 retry must fit

for (let i = 0; i < MAX_ITER_TOOLS; i++) {
  if (i > 0 && !hayMargen(opts.deadline)) break; // fall back instead of dying mid-turn
  const respuesta = await claude.messages.create({ model, max_tokens, system, messages, tools });
  if (respuesta.stop_reason === "tool_use") {
    // run each tool_use block, push the tool_result blocks, loop again
    continue;
  }
  return textOf(respuesta);
}

Lead qualification is a separate call that forces a tool, tool_choice: { type: "tool", name: "registrar_lead" }, with nullable fields and an enum for lead temperature (cold, warm, hot). Forcing a tool is the most reliable way I know to get structured data out of a conversation, and GPT can do the same thing with tool_choice plus strict mode. The cache breakpoints sit on the system prompt and the latest turn, and the history sent is a window of the last 16 messages plus a rolling summary. Every call logs input, output, cache-read and cache-write tokens and their USD cost per tenant, with a monthly spend cap.

An honest footnote: Claient still pins Haiku 4.5, which is now a legacy model. Haiku 5.5 lists at a tenth of the price. That's the obvious thing to test next, and it's also a good example of why you want evals: "cheaper and newer" is a hypothesis until your own conversations say otherwise.

Pollitics: two models, two jobs

For pollitics.es, a weekly job drafts candidate questions about the week's news for a human to review. It's split into two Claude calls on purpose:

Research on Sonnet 5.5 with Anthropic's server-side web search (max_uses: 10, effort medium). Most of the cost here is input tokens from the pages it reads, so the cheaper model does the reading.
Drafting on Opus 5.5 with JSON-schema structured output (effort high), where editorial nuance matters and the input is small.

It's two calls, not one, because structured output doesn't carry the search citations, so the research comes back as text with URLs and the drafter works from that. The code also handles pause_turn (the server-side search loop paused, send it back to continue) and refusal stop reasons, and the whole thing has to fit inside a 150-second Edge Function. A deterministic review step then flags anything that breaks the editorial rules before a person approves it.

So why Claude?

Not because GPT couldn't run either of these. The reasons are concrete and a little boring:

Cache breakpoints matched the architecture. Static script plus growing history is exactly the shape explicit cache_control is good at.
Tiering within one vendor. Haiku for every message, Sonnet for reading the web, Opus for the writing that matters. One SDK, one usage format, one cost table. Dairector's script doctor runs on Claude too.
BYOK stays simple. Tenants paste one kind of key. Supporting two providers' keys, models and error classes would double the surface for very little customer value.
Compounding. Timeouts, retries, error classification (AuthenticationError vs transient 429/5xx), cost logging: all written once, against one SDK. That's the real lock-in, and it's mine, not the vendor's.

When to use which

Choose Claude when:

Your agent is conversational or long-running and context grows every turn, so caching control pays for itself.
You need very long context and don't want a long-context surcharge on the mid and top tiers.
You want one vendor with a clear cheap / workhorse / heavy ladder to route tasks across.

Choose GPT when:

You're building a voice or phone agent and want first-party realtime models.
You want hosted file search over your documents without running your own vector store.
Your organisation already has OpenAI or Azure OpenAI contracts, data agreements and monitoring in place. That's a perfectly good reason; procurement is engineering too.

Either way:

Build a small eval set from real conversations (twenty is enough to start) and run both on your actual tool schemas before you commit.
Put the model call behind one module of your own so a future switch touches one file.
Cap loop iterations and spend from day one. Neither vendor will do that for you.

If you're weighing this for a real product and want someone who has shipped agents end-to-end to make the call with you, that's what I do. For the rest of the stack around the model, see how I ship an AI product in a weekend.

FAQ

Is Claude or GPT better for building AI agents in 2026?

Neither wins across the board. As of October 2026 the price tiers mirror each other (Sonnet 5.5 and GPT-6.1 Sol are both $2/$10 per million tokens; Haiku 5.5 and GPT-6 Luna both start at $0.10/$0.50), both offer 1M-token context, 128K max output, strict schema-enforced tool calls and hosted web search. Claude is the better default if you want explicit control over prompt caching and long context billed at the standard rate; GPT is the better default for voice agents, hosted file search, or teams already on OpenAI or Azure. Run your own eval on your own tool schemas before committing.

Is Claude more expensive than GPT for agents?

Not anymore at list price. As of October 2026 the mid tiers (Claude Sonnet 5.5 and GPT-6.1 Sol) are both $2 input / $10 output per million tokens, the cheap tiers (Claude Haiku 5.5 and GPT-6 Luna) both start at $0.10 / $0.50, and the top tiers (Claude Fable 5.1 and GPT-6 Astra) are both $10 / $50. Claude Opus 5.5 sits in between at $4 / $20. Both offer 50% off for batch work and discounted cached input. Real cost differences come from how much context you resend and how well you cache it, not the rate card.

Can I switch an agent from Claude to GPT later?

Yes, but it is not a one-line change. The Claude Messages API uses tool_use and tool_result content blocks; OpenAI's Responses API, which GPT-6 Astra and GPT-6.1 Sol require for tool calling, uses function_call and function_call_output items. Caching, usage fields, error classes and hosted tools also differ. Keep the model call behind one small module of your own so a switch touches one file, and keep an eval set so you can tell whether the switch was an improvement.

Which model does Joan Comadran use for production AI agents?

Claude, called directly through the Anthropic TypeScript SDK with no agent framework. Claient, the WhatsApp lead agent, runs Claude Haiku 4.5 by default with tool use for appointment booking and forced tool calls for lead extraction, and lets tenants bring their own key to use Sonnet or Opus. The weekly-question pipeline for pollitics.es uses Claude Sonnet 5.5 with server-side web search for research and Claude Opus 5.5 with JSON-schema structured output for drafting.

Do I need LangChain or another framework to build an agent with Claude or GPT?

No. A tool-use agent is a loop: call the model with tool definitions, execute any tool calls it returns, send the results back, and stop when it answers in text. Both official SDKs make that a few dozen lines, and both vendors now offer optional helpers (Anthropic's Tool Runner, OpenAI's Agents SDK). A framework can help with many providers or complex orchestration, but it also hides the provider-specific features, like cache breakpoints and server tools, that most of the cost and reliability wins come from.

Originally published on joancomadran.dev. I'm Joan, an AI engineer in Barcelona — work with me.