DEV Community: Conor Bronsdon

Log what your tools put in the context window before blaming the model

Conor Bronsdon — Tue, 13 Oct 2026 23:46:44 +0000

Ask an agent for one person's sales calls since February 1 and it can come back with calls that person wasn't on, even though the right ones were in its context. The data reaching the model causes that, and a live demo on Chain of Thought in March 2026 showed how.

Michel Tricot, CEO and co-founder of Airbyte, ran the query against the Gong API directly. The API couldn't filter calls by user, so the agent paged through every call over roughly a month and a half. He said that path normally takes about three minutes, that it often returns more than 45 calls because the bloated context leaves the agent unable to filter down to the right person, and that it consumed almost 30,000 more tokens than the path through a context store. Those are his rough numbers from one demo, not a benchmark, but the shape is common: the tool handed the model a filtering job the application should have done.

That's context poisoning. Wrong, stale, or badly shaped data reaches the window and the agent reasons off it. Nothing malicious has to happen.

Because as everyone is saying, the problem is not in the model today, the problem is in the data that you provide.

Michel Tricot, CEO and co-founder of Airbyte, on Chain of Thought ep 52

Log the insertion point

Instrument the spot where a tool result becomes model-visible text. The HTTP response size isn't enough, since anything you filter or transform in between changes what the model sees.

For each tool call, record the tokens the serialized result added and the IDs of the records it delivered. After the run, record how many of those records the answer used. Keep the exact payload available (under your normal data-access controls) so you can read what the model read.

Illustrative sketch, not a production library or anyone's actual implementation. count_tokens is a hypothetical helper using your model's tokenizer.

import json


def append_tool_result(call_id, tool, records, messages, audit):
    content = json.dumps({"records": records}, separators=(",", ":"))
    messages.append({"role": "tool", "tool_call_id": call_id,
                     "content": content})
    audit[call_id] = {
        "tool": tool,
        "tokens_added": count_tokens(content),
        "returned_ids": {r["id"] for r in records},
        "ids_used": None,  # unknown until someone checks the answer
    }


def record_usage(call_id, verified_ids, audit):
    entry = audit[call_id]
    entry["ids_used"] = len(entry["returned_ids"] & set(verified_ids))

Call the wrapper after any shaping, so the logged content is exactly what the model got. Fill verified_ids from checked citations or a review that maps claims to records. A citation shows what the model says it used, which isn't always what it relied on, so leave the field unknown until someone has checked.

Look at precision, not only size

A big result isn't automatically a bad one. Comparing forty calls needs forty calls.

The warning sign is a big result full of records outside the scope of the question. Near-duplicates crowd the window, and the model can hold the right record without weighting it. In a trace that reads like a reasoning failure, which sends teams off to evaluate models when the input is the problem.

For a failed run, compare the delivered IDs with the records the task needed, and check any wrong claim against neighbouring records in the payload. Then rerun the task with a shaped result, same model and prompt, and check correctness and coverage as well as whether the window got smaller.

Filter before the model sees it

Tricot described doing that work on the retrieval side:

So the first thing is a lot of this is done on the back end to make sure that we only return what we believe is the most relevant information.

Michel Tricot, CEO and co-founder of Airbyte, on Chain of Thought ep 52

Push the task's constraints to the source where it supports them: fetch by ID, filter by user, apply the date range. Return only the fields the question needs, plus the identifiers you need for attribution. When the source can't express the query, filter in application code or query a store synced for retrieval. Don't send every page to the model because the upstream API happens to paginate.

Projection is per task. A call-listing tool can return compact metadata. A tool that answers questions about what someone said on a call needs the relevant transcript text, and trimming that away saves tokens while making the answer unsupportable.

Decide which reads have to be live

A synced store raises a freshness question. Tricot's own setup is hybrid: search the store to discover and understand, then go to the API for a specific object when the task needs the freshest version.

And APIs are very good at retrieving one object as long as you know what this object is.

Michel Tricot, CEO and co-founder of Airbyte, on Chain of Thought ep 52

Make that a tool policy instead of something the agent improvises.

Illustrative sketch. fetch_record is a hypothetical adapter that reads one authorized record by ID.

FIELDS = ("id", "owner_id", "started_at", "title")


def get_call(call_id, require_live):
    source = "live_api" if require_live else "synced_store"
    record = fetch_record(source, call_id, fields=FIELDS)
    return {f: record[f] for f in FIELDS}

The final projection controls what reaches the window even if the upstream response carries more. Expose sync timestamps so the staleness decision is visible. One catch: refreshing records you discovered in the store doesn't find records created since the last sync. A task that needs a complete, current list needs a discovery path that is current too.

Takeaway

For the next failed agent run:

Read the exact tool payloads that went into the context.
Log tokens added and records the answer used, per tool call.
Filter and project before insertion, not after.
Set freshness rules for discovery and for single-record reads.

The longer explainer, with the episode clips, is on Chain of Thought.

Drafted with AI assistance from the episode transcripts, then edited by me.

Test document extraction before an agent acts on it

Conor Bronsdon — Tue, 13 Oct 2026 23:45:54 +0000

An agent can follow its instructions perfectly and still pay the wrong amount, because the parser misread a table in the invoice. The agent never sees the PDF, only the extraction. If that step is wrong, everything downstream is confidently wrong too.

Jerry Liu, co-founder and CEO of LlamaIndex, made this point in a June 2026 episode. A parser that represents a table wrong, or hallucinates at a meaningful rate, puts a workflow below the accuracy bar you can operate with, and the downstream agent workflow gets disrupted. His bar for the use cases he sees:

And so a lot of these require really high accuracy, 95% plus, especially in more sensitive use cases, with human-in-the-loop review on some of the extracted data.

Jerry Liu, co-founder and CEO of LlamaIndex, on Chain of Thought ep 61

He was describing what those customers need, so set your own bar per field from what a wrong value costs you. The practical move is to treat extraction as a pipeline stage with its own tests, measured on the fields your agent consumes.

Label a small set of real documents

Pick the document types you plan to support and hand-label a few dozen real examples of each. Include the ugly layouts along with the clean ones. Hold some back and don't look at them while you tune prompts or rules.

Label fields, not documents. For an invoice, the amount due and the due date may need a much higher bar than a free-text note, and a single overall score hides that. Decide the missing-value rules up front: a field that isn't in the document has an expected value of None, while a field the extractor fails to return counts as wrong.

Compare values in a canonical form. Normalize dates and currency, but be careful: stripping punctuation from an ID can turn two different IDs into the same string.

Illustrative sketch, not a production library or anyone's actual implementation. extract_fields(doc) is a hypothetical function that returns a dict.

from collections import defaultdict


def evaluate(cases, extract_fields):
    scores = defaultdict(lambda: {"correct": 0, "tested": 0, "misses": []})
    for case in cases:
        predicted = extract_fields(case["document"])
        for field, expected in case["gold"].items():
            s = scores[(case["doc_type"], field)]
            s["tested"] += 1
            if field in predicted and predicted[field] == expected:
                s["correct"] += 1
            else:
                s["misses"].append((case["id"], predicted.get(field)))
    return {
        key: {**s, "accuracy": s["correct"] / s["tested"]}
        for key, s in scores.items()
    }

Report accuracy per document type and field, with the tested count next to it. A perfect score on eight invoices says very little. Keep the misses, too. Seeing which cell got confused for which gives you something concrete to fix, and rerunning the held-out set after each change shows you what regressed.

Turn the scores into a review gate

An offline score tells you how the extractor does on labelled documents. It can't tell you whether a new document's values are right, because production documents arrive without labels. So use the eval to decide what may run unattended, then check each incoming document against that.

Condition	What happens
Document type has no eval coverage	Hold the whole document for review
Field's measured accuracy is below its bar	Review that field for that document type
Required field missing or fails a format check	Block the actions that depend on it
Values break a business rule (line items don't sum to the total)	Show the values and the source region to a reviewer

Scope the hold to what depends on the bad value, so one disputed total doesn't freeze unrelated work. Show reviewers the extracted value next to the source passage or image region. Store the correction and feed adjudicated examples back into the labelled set, keeping a separate held-out set. Sample some auto-accepted documents too, since errors that pass every rule are the ones you won't otherwise see.

Store extracted facts with their review status

Extraction output often ends up as memory, and a wrong value saved for later can outlive the run that produced it. Liu's take on memory is deliberately simple. He described it as a form of persistent context and pointed to files as a starting point:

And so this concept of having memory just be a pool of files on your file system is kind of nice to think about because it's pretty easy to represent and store.

Jerry Liu, co-founder and CEO of LlamaIndex, on Chain of Thought ep 61

He was clear that he hasn't spent a ton of time building more complicated memory systems, so treat it as a place to start. For an extraction pipeline, it suggests a simple rule: keep candidates and accepted facts in separate places. Each record carries its source reference, its review status, and the extractor version that produced it. An agent can write candidates but can't promote them to accepted. When a reviewer corrects a value, mark the old one superseded instead of overwriting it, so you can trace where it came from.

Takeaway

Before an agent acts on extracted data:

Label real documents per type and define comparison rules per field.
Report per-field accuracy with sample counts and keep the misses.
Route uncovered document types and weak fields to a human.
Store extracted facts with their source, status and extractor version.

The full conversation, with the transcript, is on Chain of Thought.

Drafted with AI assistance from the episode transcripts, then edited by me.

Open weights in production: five builder takeaways from Chain of Thought

Conor Bronsdon — Tue, 13 Oct 2026 21:52:05 +0000

Open weights are easy to cheer for and hard to run profitably. The questions that decide it are what you pay per task, what you must prove to a customer, and how much of the system lives outside the weights. A leaderboard alone cannot settle them. Below is one practical takeaway per episode from the Chain of Thought collection on open source AI. (To follow new episodes on YouTube, Apple Podcasts, Spotify or by email, use the subscribe page.)

What changed the week DeepSeek R1 landed?

When DeepSeek changed the AI race overnight, the argument on the show was recorded while the news was still settling. That timing matters for builders: treat the episode as a snapshot of how people were reframing cost and openness.

The practical read is the price floor. DeepSeek's R1 release forced a rethink of what an open model was allowed to cost and what that implies for how models get built next. Export controls and "Agents as a Service" came up as second-order questions, but the builder move is simpler. Re-run your inference budget assumptions and your roadmap for agentic features, because the conversation also tied heavier agent stacks to the need for serious evaluation of agent behavior beyond single-turn demos.

If you are adding tools, memory, or multi-step flows, plan how you will measure agent performance before you scale traffic. The episode pairs the shock of R1 with agentic evaluation as the guardrail layer that complexity demands.

When does one summarization bill justify fine-tuning Qwen?

How Intercom cut $250K/month by ditching GPT for open models is the most concrete open-weights-in-production walkthrough on the show. Intercom was spending $250K a month on a single summarization task on GPT, then moved to a fine-tuned 14B Qwen model and saved almost all of that spend.

Do not copy "swap model, save money" without copying the surrounding system. Fergal Reid's story for Fin is that resolution moved from about 30% to nearly 70%, and most of that lift came from custom re-rankers, retrieval models, and query canonicalization rather than from swapping the core frontier LLM alone. Benchmarks can lie in production (Goodhart's law shows up explicitly in the episode), so Intercom leaned on A/B tests with soft versus hard resolution definitions.

Decision rule I'd steal: isolate the task with a brutal monthly line item, then ask whether a smaller open model plus fine-tuning beats API spend. If the task is narrow (summarization, canonicalization), run production tests beyond benchmarks before you commit. Remember the latency wrinkle Reid describes: slower responses can raise resolution rates, which breaks the instinct to optimize only for speed.

What does an open ecosystem bet mean if you are not picking the GPU?

AMD's challenge to NVIDIA: the open ecosystem bet is hardware-heavy, but the builder lesson is lock-in. Anush Elangovan frames AMD's MI350 series on CDNA 4 architecture, including 20 petaflops of FP4 performance he calls mind blowing, as part of a full-stack push paired with ROCm 7 and AMD Developer Cloud with Day Zero support for frontier models.

Sharon Zhou's half of the conversation is about making contribution real: a happy path for community work, teaching, and listening to developers and treating open source as more than a press release. For application teams, the takeaway is to know your escape hatches. If training or serving depends on one vendor's CUDA-only path, your open-model strategy can still die on infrastructure bills or portability. Ask whether your stack has a credible second home before you bake in assumptions.

How should you ship an on-machine open agent without ignoring safety?

AI, open source, and developer safety with Block's Rizel Scarlett centers on Goose, Block's on-machine open source agent built to automate engineering work, including starting with migrations for Block employees. Block also collaborated with Anthropic on MCPs, which is a reminder, in my reading, that open agents still sit in an ecosystem of other people's work. What it gives you is the ability to inspect and adapt more of the stack.

Rizel Scarlett ties open source agent tooling to developer safety, including psychological safety for junior developers, plus responsible and ethical use and community involvement when building developer tools. Practical procedure: default sensitive work to on-machine agents where policy allows, document what the agent can touch, and treat openness as a reason to involve the community in review while keeping your guardrails.

What should enterprises weigh before open versus proprietary?

The state of AI: open-source models and enterprise trust was the show's first episode, and it frames open versus proprietary as an enterprise trust question more than a benchmark contest. Writer CEO May Habib talks about implementing generative AI at scale and the challenges of productionizing GenAI.

I'd start checklists with trust artifacts before parameter counts: who is liable for output, how humans stay in the loop, and what breaks in deployment that demos never saw. Regulation and election-era policy show up in the episode framing, but the builder question I take from her segment is whether your customer can rely on the system in production, open weights or not.

Optional sketch for comparing paths on one task (illustration only, not Intercom's pipeline):

# Sketch: per-task economics + trust gates (not production code)
def should_explore_open_weights(task):
    gates = {
        "monthly_api_spend": task.api_usd_per_month,  # e.g. one bad summarization line item
        "task_narrow": task.is_single_purpose,
        "eval_in_prod": task.has_ab_not_only_benchmarks,
        "trust_requirements": task.enterprise_checklist_done,
    }
    return gates["monthly_api_spend"] > 0 and gates["eval_in_prod"]

Checklist

Re-budget inference after open-model price shocks (R1-scale events) and add agent evals before you scale tool-using flows.
For high-spend single tasks, model Intercom's pattern: fine-tuned open weights plus retrieval, re-ranking, and canonicalization, with production A/B tests instead of leaderboard faith.
Map hardware and software lock-in (CUDA-only paths vs open stack options like ROCm and developer cloud Day Zero support) before you commit training pipelines.
Ship open agents like Goose with clear on-machine boundaries, MCP integrations where needed, and explicit safety practices for junior-heavy teams.
For enterprise buyers, lead with trust and productionization lessons (the episode's framing) before open source ideology.

More on this topic, with the related episodes, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.

When your LLM plan moves a robot, clamp every command first

Conor Bronsdon — Thu, 08 Oct 2026 14:30:08 +0000

A bad completion in chat is annoying. A bad completion on a robot arm can cause a collision. Physical AI closes the loop with the real world: sensor input in, actions out, and mistakes land in a room rather than a chat window. That shift is why I'd treat the model as a planner, not as the final authority on motion.

Why does talking to a robot feel different from a chat app?

Paige Bailey of Google DeepMind puts the current demo tier plainly: Gemini already runs on robots. The hardware story is modest on purpose. Stanford's open Pupper dog is 3D printable, runs on a Raspberry Pi, and takes spoken instructions through Gemini APIs. Bailey's account is that the APIs invoke the robot's components on device without her describing a separate layer written to translate intent into motion. Natural language in, the model picks which robotic components to call, same general API shape as a web app.

That pattern is what multimodal AI looks like in the wild: camera feeds and speech as inputs alongside text. Commodity boards matter because the expensive part moves toward the model while the body gets cheaper. Bailey forecasts more robots on Raspberry Pi class hardware and alternatives that ship faster and cost less. DeepMind also runs a robotics trusted tester program; she names Enchanted Tools, Boston Dynamics, and the Figure Team as testing or using Gemini models for robotics. Since the episode, Google has offered its Gemini Robotics reasoning model to developers as a preview in the Gemini API, and Boston Dynamics runs an earlier version in a live inspection feature for Spot. The models that directly drive robot motion are still limited to early-access partners.

But the pupper can do all sorts of things. So you can say like, hey, pupper, shake my hand. Hey, Pupper, follow me. Hey, Pupper, do the spider or, like, go swimming. Or, hey, Pupper, tell me a joke. Like, all of these things are are possible today just with the Gemini APIs by themselves running and invoking all of the different robotic components on device on a Raspberry Pi. Like, how cool is that thing?

Paige Bailey, Google DeepMind, Chain of Thought ep 47

I'm excited that this runs on hobby hardware. I'm still skeptical that production stacks should copy the demo architecture verbatim.

What should sit between the model and the motors?

My rule: keep the model's job at the level of intent and component choice. Keep execution deterministic.

The model might output something like { "action": "set_joint_velocity", "joint": 2, "rad_per_s": 1.4 }. Your action layer should parse that schema, clamp every numeric field to safe ranges, reject unknown actions, rate limit commands, and only then call hardware drivers. If the model hallucinates a joint index or asks for impossible speed, the room never finds out the hard way.

Prompt engineering alone does not give you that guarantee. Bailey's Pupper story is about the model invoking components directly through Gemini APIs. For your stack, add an explicit validator even if the vendor demo skips talking about one.

Sketch only, not copied from any guest pipeline:

# Sketch: deterministic gate between model JSON and hardware
SAFE = {"set_joint_velocity": {"joint": (0, 5), "rad_per_s": (-0.5, 0.5)}}

def execute(plan: dict, hw) -> str:
    action = plan.get("action")
    if action not in SAFE:
        return "rejected: unknown action"
    limits = SAFE[action]
    joint = plan.get("joint")
    vel = plan.get("rad_per_s")
    if joint is None or vel is None:
        return "rejected: missing fields"
    jmin, jmax = limits["joint"]
    vmin, vmax = limits["rad_per_s"]
    if not (jmin <= joint <= jmax and vmin <= vel <= vmax):
        return "rejected: out of range"
    hw.set_joint_velocity(int(joint), float(vel))
    return "ok"

Run the model against a fake hw that logs commands. Only plug in real drivers after fuzzing bad plans.

Where is the field headed beyond robot dogs?

Bailey's enthusiasm tilts toward labs, not living room gadgets. She points at Periodic Labs using AI to control robotics and run material science work: design experiments, run them, and evaluate component choices for semiconductor work. She frames the win as tedium and human error on repetitive bench work, not raw speed.

And then I I I'm also really excited about companies like Periodic Labs, which are using AI to to control robotics, but also to to have those robots do really interesting material science work. So they can design experiments, run the experiments, test out, you know, the likelihood of different component parts being successful for the creation of semiconductors.

Paige Bailey, Google DeepMind, Chain of Thought ep 47

The glossary entry on physical AI still applies: automated scientific experimentation, where a model both designs an experiment and physically runs it, is a harder target and mostly unbuilt. That is exactly where plan versus execute separation matters most. A wrong hypothesis wastes money. A wrong pour or grind wastes samples.

Do you need Google's whole stack to start thinking clearly about this?

No. The lesson from the Pupper demo is architectural: general multimodal models on cheap hardware, language as the interface. Google's TPU story matters upstream: controlling the chip, the XLA compiler and the models is part of why Google can price its model tiers so far apart. You call an API rather than a chip.

For background on how multimodal models show up across episodes, the multimodal AI topic page collects the full conversations.

Checklist

Treat model output as a proposed plan. Parse JSON or tool calls, then clamp, reject, and log before any driver call.
Copy safe limits from the robot's datasheet or your lab procedure into code, not into the prompt.
Test in simulation or a stub hardware object first. Feed malformed and adversarial plans until rejection is boring.
Let the model choose the component or skill; let your code decide the numeric command that reaches the hardware.
Read vendor demos as capability proofs. Ship your own deterministic action layer anyway.

The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.

Five agent episodes, and the build decision to take from each

Conor Bronsdon — Wed, 07 Oct 2026 20:31:12 +0000

An agent loop can look fine in a notebook and fall apart across sessions, teams, and production data. The Chain of Thought collections group episodes for people shipping agents as working systems. Below is one build decision per episode from that agents set, tied only to what those guests argue on the show.

Why isn't my framework the moat anymore?

Jerry Liu helped turn LlamaIndex into one of the most installed AI frameworks of the last three years, then argued on The AI Framework Era Is Over: Why Context Is the Moat that the framework era is over. His point for builders: as agent loops and harnesses get good enough, the scaffolding stops being where you win. Context quality is what compounds.

Practical move: budget engineering time for the context layer (what gets retrieved, parsed, and kept accurate) before you spend it swapping orchestration libraries. If your use case is document heavy, Jerry treats very high parse accuracy as the bar for legal, insurance, and financial work (he cites 95%+ as the real target in those verticals). If the agent reads bad inputs, I would not expect prompt polish to rescue it.

What should I build so the agent remembers the next session?

Richmond Alake, Director of AI Developer Experience at Oracle, treats memory as the layer that decides whether an agent works across sessions, and says most teams under-build it. On Agent Memory: The Last Battleground in the AI Stack, he separates memory engineering from prompt engineering and context engineering as its own discipline.

Two failure modes he calls out often: the wrong mental model, and deleting data instead of forgetting it. Production memory should decay and deprioritize (he points to relevance, recency, and importance style scoring from the Generative Agents line of work) instead of hard-deleting them. He maps working, episodic, semantic, and procedural memory to how you segment what goes in the window versus what lives in stores. Files are fine for prototypes; he argues databases win in production.

When do many agents beat one, and what breaks?

Mikiko Chandrasekhar of MongoDB frames multi-agent systems as a reliability problem first. On Mastering Multi-Agent Systems, she says agents offer serious capability, but mastering reliability is what decides whether agentic systems work in production. Her recommendation is to treat agents as software products: observability, evaluations, and guardrails are part of the product for non-deterministic systems.

Add agents only when the task needs split roles or parallel work, because coordination adds failure modes. Plan for chat logs, vector search, and debugging trails before the second agent joins.

What layers am I stacking when I call something an agent?

João Moura, founder and CEO of CrewAI, uses The Emerging AI Agent Stack to map what enterprises assemble when they move past curiosity: orchestration, provisioning, authentication, and measurement for large numbers of agents in production. Knowledge-work automation at scale needs measurability, reliability, and trust in his framing; a demo that completes once is the start.

The episode talks about provisioning, authentication, and measurement for hundreds or thousands of agents. Before you add agents, list what must exist at that scale: identity, access, metrics, and interoperability between components. If you cannot name how you will measure success and catch regressions, you are still in experiment mode. João and his co-panelists also stress that non-deterministic multi-agent setups multiply failure modes, so testing has to go beyond what worked for traditional services.

How do I tell an agent from a fancy workflow?

The earliest anatomy episode in this set is still the clearest line in the sand. On Got Agents? Agentic Workflows & Architecture, João Moura draws the line at agency: a fixed if-this-then-that flow is not an agent. Brian Raymond and Bob van Luijt sit in the same conversation on data management, deployment, and generative feedback loops, but that definition is the decision rule.

Use this before you rename pipelines:

# Sketch: agency check before you call it an "agent"
def is_agentic(step_plan, can_replan: bool, tool_choice: str) -> bool:
    if step_plan == "fixed_dag" and not can_replan:
        return False  # workflow with an LLM step, not an agent
    if tool_choice == "model_picks_per_turn":
        return True
    return can_replan

If every branch is predetermined, you likely need better workflow tooling and evals more than an autonomous label on the README.

Checklist

Invest in context and document accuracy where Jerry Liu says the moat moved; treat framework churn as secondary.
Name memory engineering explicitly: prefer controlled forgetting over deletes, and plan stores past file-based prototypes (Richmond Alake).
Add multiple agents only with product-grade observability and evals; treat reliability as the gate (Mikiko Chandrasekhar).
Stack provisioning, auth, and measurement before you scale agent count (João Moura on the agent stack).
Refuse the agent label when the system is a fixed if-this-then-that flow (João Moura, EP 2).

More on this topic, with the related episodes, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.

How to write tool descriptions agents won't misread

Conor Bronsdon — Wed, 07 Oct 2026 14:30:09 +0000

An agent picks a tool by matching the current task to each tool's name and description, then it writes the arguments for that call. Chip Huyen names the two ways that goes wrong: an invalid tool, or the right tool with the wrong parameter. Write each description so the right choice and the valid inputs are obvious, score that selection on its own, and make the agent ask before it runs anything that can change or delete data.

This post draws on two Chain of Thought episodes. Chip Huyen, then at Tép Studio (formerly NVIDIA), and Vivienne Zhang of NVIDIA, discussed agent evaluation in episode 5 (Dec 2024). Angie Jones, then VP of Engineering for AI Tools and Enablement at Block, described deploying Goose with MCP in episode 48 (Jan 2026).

How does an agent choose a tool at each step?

It reads the task against the tools it has been given, picks one (or answers with no tool), and generates the call. Chip Huyen treats an agent as tools plus a plan. Web search, a calculator, and a retriever are tools. Given a task, a constraint, and that set, the agent is supposed to produce a strategy, then carry it out one call at a time. After a call returns, the next step uses that result.

The call itself is where selection goes wrong. A weather function is not a bare call weather. The model has to pull out a location and a time, and those values have to be valid. A search string, a JSON payload, and function arguments fail the same way: the tool name can be right while the fields are not.

"The other is whether it can call the right tool. So a lot of time, we'll try to call like an invalid tool. Or it will try to use the correct tool, but not with the right parameter."

Chip Huyen, then at Tép Studio (formerly NVIDIA), on Chain of Thought ep 5

Practical Lessons for GenAI Ev… - Chain of Thought | AI Agents, Infrastructure & Engineering - Apple Podcasts

Podcast Episode · Chain of Thought | AI Agents, Infrastructure & Engineering · December 4, 2024 · 48m

podcasts.apple.com

So a description has two jobs. It says when this tool is the right choice, and it says which arguments the call needs. Protocols like MCP, which Angie Jones uses with Goose, put many systems behind one client. That only helps if each server's tool cards stay distinct. One shared interface does not fix two cards that sound the same.

Why do overlapping descriptions break selection?

The model picks the wrong tool, or it picks the right one and still sends bad arguments. Either way the rest of the task follows a bad step. Chip Huyen's method is to map how the application will fail, then put a metric on each of those points. For agents she uses three checks:

Can it finish the task inside the constraints? Her example is a two-week trip on a $5,000 budget.
Does it call the right tool, and are the parameters valid?
Is the sequence of actions short enough to stop, rather than looping on and on?

Keep those scores separate. A single "accuracy" number hides a wrong tool behind a lucky final answer, and it hides a correct tool that was called with garbage arguments. If two cards could both match the same sentence in the user request, rewrite them until a reader can point at one card and say why the other does not apply. State the data the tool touches, the case it is for, and what it returns.

How do you keep a destructive tool from running unchecked?

Mark it on the tool, and require a person to approve that call. Angie Jones does this in Goose at the tool level, not per model. The annotation tells the agent the tool can change, edit, or delete things. A separate toggle lets the agent run tools that are not destructive, and forces a question when it hits one that is.

"annotating tools that say like this tool is this word is strong, destructive. But that just means, uh, it can change things. It can edit or it can delete or something like that. So helping your agent understand that this is a destructive tool, right?"

Angie Jones, then VP of Engineering for AI Tools and Enablement at Block, on Chain of Thought ep 48

The same episode covers a second gate: an allowed list of MCP servers. If a server is not on the list, it will not install. In her words, the agent will say nope. Most of the servers on Block's list are built in-house, and those still go through security review, because not every problem is intentionally malicious. Pair that with the destructive flag. The list decides which MCP servers will install. The annotation identifies destructive tools, and the permission toggle requires approval before they run.

"agent, you're allowed to run any tools that are not destructive. But if you run into one, that is, you need to ask my permission first. Like, give me some insight into what this is, what you're trying to do, and I'll. I'll say yes or no as the human in the loop."

Angie Jones, then VP of Engineering for AI Tools and Enablement at Block, on Chain of Thought ep 48

A minimal sketch to adapt, not a drop-in library.

def agent_step(task, allowed_tools):
    if not allowed_tools:
        return no_tool(task)
    scored = [(t, match_score(task, t.desc)) for t in allowed_tools]
    applicable = [pair for pair in scored if pair[1] > 0]
    if not applicable:
        return no_tool(task)
    applicable.sort(key=lambda pair: pair[1], reverse=True)
    if len(applicable) > 1 and applicable[0][1] == applicable[1][1]:
        return flag_overlap(task, applicable[0][0], applicable[1][0])
    best = applicable[0][0]
    args = generate_args(task, best.desc)
    if not valid(best, args):
        return no_tool(task)
    if best.flag == "destructive" and not ask_permission(best.name, args):
        return no_tool(task)
    return run_tool(best, args)

Read that as a decision shape. If the allowed list is empty or no description matches, answer with no tool. If two descriptions tie for the top score, flag the pair for a rewrite instead of guessing. Otherwise take the best match, generate and check the arguments, and run a tool that can change data only after approval.

What should you measure once the agent is in production?

Selection quality on its own, then the same failure points again on live traffic. A benchmark you curate before launch is a start. Vivienne Zhang's point is that production queries are a different set: more complex, full of edge cases. What she has seen work is a loop. Capture user feedback and usage data, find what tripped the LLM, and change the system. In her NVIDIA copilot example, the early failures were missing answers in the data, and chunks cut in the wrong places, so the fix was the data and the chunking, not the model.

"I think what I've seen working is that you simply have to keep iterating. You have to capture the user feedback, the usage data, and systematically troubleshoot what have tripped up your LLM and then improve the system."

Vivienne Zhang of NVIDIA, on Chain of Thought ep 5

For tools, log the choice and the arguments, not only the final reply. When users correct a run, check whether the description was ambiguous, whether a parameter was invalid, or whether the plan never stopped. Vivienne Zhang also describes a familiar handoff: a support bot keeps the query until it cannot handle the complexity, then sends it to a person. Treat that handoff as one more failure point, and run the same evals Chip described on it. Fix the card or the constraint, ship again, and read the next batch of traces.

FAQ

What is the fastest fix for wrong agent tool calls?
Measure whether the right tool was chosen and the parameters were valid (Chip Huyen), then rewrite the descriptions behind the misses.

Should destructive tools run without approval?
No. Annotate tools that can change, edit, or delete things, and use a toggle so the agent asks a person before those calls run (Angie Jones).

How do I start measuring agent reliability without a big benchmark?
List the likely breaks (wrong tool, bad parameters, a plan that never stops) and put one metric on each. After release, use feedback and usage data to see which of those still fire (Chip Huyen; Vivienne Zhang).

Takeaway

Write each tool card with the case it covers, the arguments it needs, and what it returns.
Compare cards side by side and rewrite any pair that could match the same request.
Annotate tools that can change or delete data, and require a yes before they run.
Score task completion, tool-and-parameter validity, and plan length as separate checks.
After release, log choices and arguments, then fix the descriptions that real queries trip.

The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.

Shipping practices from five AI engineering episodes

Conor Bronsdon — Tue, 06 Oct 2026 20:30:17 +0000

A demo is cheap. Shipping it means you can say what you reviewed, what you designed, what you measured, and who owns production when the model is wrong. These five episodes are the set I would hand a builder who already has a demo and needs a practice. Transcripts and chapters for every episode are on the Chain of Thought episodes page.

What do you review when an agent writes the code?

Anush Elangovan, AMD's VP of AI Software, describes an agent-first workflow at real scale, the case where AI writes most of the code. He runs 10 to 12 Claude Code agents in parallel, burns 6.5 billion tokens a week, and rewrote a 25-year-old Slurm replacement in Rust overnight. He argues that normal SDLC is dead. He treats testing as the new code review. He puts the rest in one line: software is just tokens. I would take the testing claim as the operating rule. Once that much code is machine-written, reading every diff will not keep up. Write tests that state the behavior you will accept, run them as the gate, and spend review time on those tests. The overnight rewrite is a reminder that the tokens show up before an old review process is ready. Treating software as tokens only works if you still own the definition of a passing test.

What should you design before you pick a model?

Aishwarya Srinivasan, Head of AI Developer Relations at Fireworks AI, argues that most AI agents are built backwards, starting with models instead of system architecture. The shift she describes runs from picking a model to architecting a system, and from prompt engineering to context engineering. In her account, a production agent needs careful orchestration of multiple models, memory systems, and tool calls, along with evaluation-driven development. Benchmarks alone will not save you. She also covers human-in-the-loop control of autonomy, LLM-as-judge checks, open source models such as DeepSeek v3, and compliance in regulated environments. Before I named a model I would write down the context the agent receives, the tools it may call, the memory that persists, and the eval that fails the build. A person stays in the loop wherever a bad action is expensive to undo. You can swap a model inside a system you have already specified.

How do you get from a vibe check to a shipped feature?

Malte Ubl, CTO of Vercel, describes agents as a new type of software for solving flexible tasks. The episode is about the engineering between a demo and a shipped feature. He points at Vercel's developer-first stack, including the AI SDK and AI Gateway, as a way to move from a quick proof of concept to a trusted, production-ready application. I would hold a team to that order. Name the flexible task in one sentence a user would recognize. Build the proof of concept on the developer surface you are willing to run in production, which is the job he gives that SDK and gateway. Treat the result as shipped when you trust it the way you trust the rest of the product. The vibe check ends when the task has to survive real users.

What should you look at before you trust a metric?

Hamel Husain, of Parlance Labs, says the first question he usually gets is which tools to use, and that it is the wrong question. The episode is about the mindset that separates engineers who ship reliable AI from engineers who chase metrics. He argues that vanity dashboards and a buffet of metrics create a false sense of security, and that this is no substitute for customized evals tailored to domain-specific risks. The discipline he describes is error analysis: look at the data yourself, find real-world failures, draw qualitative notes from logs, and turn those notes into quantitative guardrails. That is the experimentation loop he ties to taking a prototype into production. I would not add a metric until I could point at a failure I had read in our own logs. The first artifact is a short list of domain failures, each with an example and an eval that fails when that failure comes back. Tools can wait.

What does AI leave with the engineering team?

Charity Majors, co-founder and CTO of Honeycomb and the person behind charity.wtf, starts from her article "Generative AI is not going to build your engineering team for you." The episode is about production reliability, and about what AI fixes and what it leaves inside your hardest engineering problems. The question she and I take up is whether generative AI makes it easier to build, lead, and sustain a high-performing team. I would plan as if the article title is still true. Keep a team that can build the system, lead the work, and hold production reliability when the model is wrong or unavailable. AI can sit inside that team. Headcount, leadership, and the reliability of the system stay with the people who own it.

Checklist

Gate agent-written code with tests. Anush Elangovan calls normal SDLC dead and treats testing as the new code review.
Specify context, tools, memory, and evals before you commit to a model. Aishwarya Srinivasan's warning is that benchmarks alone will not save you.
Carry the flexible task from a proof of concept to a trusted production app on a stack you mean to keep, which is how Malte Ubl frames the path from vibe check to a shipped feature.
Read real failures before you trust a dashboard. Hamel Husain calls a buffet of metrics a false sense of security.
Staff and lead the team as if generative AI will not build it for you. Charity Majors ties that limit to production reliability and the hardest engineering problems.

More on this topic, with the related episodes, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.

Checks to run before a small model serves traffic

Conor Bronsdon — Tue, 06 Oct 2026 14:30:07 +0000

A small model is ready for traffic only when you can name the precision you will serve, whether an adapter passed the same evals as a fuller update, which machine will run it, and which live signal would make you pull it.

What does lower precision take away?

Quantization stores weights in fewer bits, often 8-bit or 4-bit instead of 16-bit. The model takes less memory and runs faster. That is how a capable model fits a laptop, a phone, or a cheaper GPU, and how serving cost drops when call volume is high. Networks often tolerate small numerical noise, so a careful cut can stay close to the full-precision model on the job you measured.

Quality is the loss. Push the bit width too low and accuracy drops, and the damage is uneven across tasks. I would treat precision as a dial on your evals: ship the lowest precision that still passes, and keep that file as the one you score. If training stays at full precision and only the server copy is quantized, the pass has to be on the quantized weights. Fergal Reid, chief AI officer at Intercom, said most of what his team does now is full supervised fine-tuning, unquantized, with reinforcement learning on some harder work. Match the number format in the eval to the number format in the deploy.

Is a LoRA adapter enough for this job?

LoRA freezes the pretrained model and trains small low-rank adapter matrices. Full fine-tuning updates every weight and produces another full copy per task. Adapters are tiny, so training is cheaper and faster, and you can store many task adapters instead of many full models. Adapters can be swapped or combined. Quantized LoRA (QLoRA) pushes that training cost lower still. I would start with an adapter when you want a specialized behavior and a full fine-tune is more than you need or can pay for.

Fine-tuning belongs on a repeated task, with examples that look like real use and an eval set that can show whether the change helps. A smaller specialized model can still lose once training and serving are both on the bill.

Reid said they started with LoRA and other parameter-efficient methods. Over time the work shifted toward full supervised fine-tuning, plus reinforcement learning for harder objectives. He also said the harder spend was infrastructure: an internal setup so a scientist can log in and run a distributed job on large GPUs on AWS. He described their standard instance at the time as one node of H200s. If the adapter passes the task evals you trust, stop. If it misses a policy you can already hit with a fuller update on those same evals, spend the full update on that task.

Should this run on a server or on the device?

Edge AI runs the model near the data. A camera that detects objects locally, or a phone that recognizes speech on device, is doing that. Processing close to the source can shorten response time and cut what you send over the network. It can also keep sensitive recordings local, depending on what the rest of the application stores and transmits. The weights still have to fit the device memory, the processing capacity, and the power budget. Smaller models and quantization help only if the task evals still pass. A hybrid split can keep urgent work on the device and send heavier requests to the cloud. The split depends on the application.

Maxime Labonne, head of post-training at Liquid AI, wanted an on-device model that is fast on a phone or a wearable. He argues you have to time the operators on the target hardware. Theoretical speed from the math can fail once the operator is on the device. His team measured on a Samsung phone for that reason, alongside a large set of pretraining evaluations (he cited over 100).

So on a Samsung phone, for example. So we would not be misguided into believing that our our operator is is really good. It's very fast. No. Like, we could measure it and make sure that it's actually working in practice.

Maxime Labonne, Head of Post-Training at Liquid AI, on Chain of Thought episode 43

The same small-model range also has a server job. Labonne said a model on the order of 350 million parameters can be deployed on GPU at scale for high-volume work, including turning unstructured text into a JSON object you specify, in areas such as ecommerce and finance. He also noted that API calls are not always possible, and that a car assistant that depends on online connectivity would not be very useful most of the time. If the call has to succeed offline, or the recording should stay on the device, put the model there and prove it fits. If you need many parallel calls and the data can leave the machine, host it yourself and remove the per-token dependency on an outside provider. Either way, time it on the box you will use.

Labonne is also clear about the ceiling: knowledge depends on parameter count, so distilling into a one-billion-parameter model will not make it as smart as a three-billion one, and small models miss frontier quality on complex workflows. Match the model to the task, then match the task to a machine you measured.

What should you watch after launch?

For each task, take the smallest model that still passes your evals. That is where cost, speed, and quality tend to balance in production. Offline passes are the entry ticket. Reid's team still back-tests on a suite built over time, including hallucinations customers reported in the wild, and they like to try a well-prompted but untrained model before they post-train, to see whether it is even close. Then they A/B a release candidate in production. Their reinforcement learning stays offline. They do not run an open-loop setup in production.

But then nothing beats testing in production. We test everything in production.

Fergal Reid, Chief AI Officer at Intercom, on Chain of Thought episode 49

On their customer-service agent, the ground truth is the hard resolution, where the user confirms the question was resolved. The soft resolution, where Fin answers, offers a human, and the user leaves without saying anything, is the proxy. On a model change he wants soft resolutions up and hard resolutions at least flat. If the proxy rises while confirmed success falls, the system is deflecting. He also argues that increased latency almost always leads to more resolutions, including hard ones, possibly because users read the wait as more work done. A back test that ignores latency can prefer a slower answer for the wrong reason.

You may not have his resolution labels. You do need one confirmed outcome and one proxy, logged with latency, on an A/B before the full ramp. The sketch below is only a gate I would write for myself. It is not a pipeline Intercom or Liquid AI runs.

# Sketch only. Illustrates a go/no-go. Not a vendor client.

def go_live(task_ok, quant_ok, adapter_ok, fits_device, ab_ok, latency_neutral):
    if not task_ok:
        return "hold: task eval failed"
    if not quant_ok:
        return "hold: scored precision is not the file you will serve"
    if not adapter_ok:
        return "hold: adapter missed; compare a fuller update"
    if not fits_device:
        return "hold: weights miss memory or power on the target"
    if not ab_ok:
        return "hold: confirmed outcome moved the wrong way"
    if not latency_neutral:
        return "hold: win is explained by delay"
    return "ramp"

The longer conversations these checks come from sit on the AI engineering topic page, with transcripts and chapters.

Checklist

Score the weights you will serve, at the bit width you will serve.
Start with a LoRA adapter. Move to a full fine-tune only when the adapter misses the same evals.
Time the model on the target phone, or on the server instance you will rent, and confirm it fits memory and power.
A/B against a user-confirmed outcome. Keep the proxy from rising while confirmed success falls.
Log latency next to quality so a slower answer cannot win by accident.

The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.

Five podcast lessons on AI defensibility and go-to-market

Conor Bronsdon — Mon, 05 Oct 2026 20:30:07 +0000

If you are building an AI company, you already feel the gap between a slick demo and a business that survives production. I pulled five episodes from the Chain of Thought collection on defensibility, go-to-market, and market reality. Each section is one builder takeaway you can apply this week, with no requirement to agree with every frame.

For full guest bios, episode lists, and transcripts for everyone mentioned here, start at the Guests page on Chain of Thought.

How do you get from demo to something defensible?

Aurimas Griciūnas, CEO of SwirlAI, frames the founder problem in From Demo to Defensibility: How to Build an AI Business that Lasts as moving from an impressive demo to a company that lasts once the technological moat erodes.

His practical filter: startup success in this era tends to cluster around speed, strong financial backing, or immediate distribution. If you have none of those, do not compensate by stacking shiny tools on a weak core. He warns that prioritizing flashy AI over fundamental engineering leaves gaps competitors can walk through.

What I would steal for a weekly review: ask whether your moat is execution (tight feedback loops, relentless shipping) rather than a model trick. Note where you are betting on LLM leaps versus coding agents and self-improving systems, and whether your roadmap still makes sense if model progress slows.

What should AI-native mean on day one?

Marcel Santilli, Founder and CEO of GrowthX, argues in Building an AI-Native Startup that the win comes from mastering the messy middle, the unglamorous work between idea and durable revenue, more than from chasing the next frontier model.

He describes rebuilding from first principles instead of bolting AI onto an old company shape. GrowthX started with a services focus, and he talks about how AI can augment human talent. He names first-principles thinking and delegation as critical skills, and speed and brand as competitive advantages.

Builder procedure: list every step in your value chain that is still manual or tribal knowledge. Pick one slice to codify (playbooks, evals, handoffs) before you buy another tool. If your pitch is only we use the newest model, you are exposed to the same API everyone else has.

Where is the moat when creation gets cheap?

Bharat Vasan, founder and CEO of Intangible, makes the differentiation argument in Taste Is The New Moat: Why Customer Obsession Wins in the AI Era. When AI makes content and code cheap to produce, standing out comes down to taste and distribution, plus how obsessed you are with real customers.

He calls relentless shipping the ultimate clarifier for a business. Resilience matters as much as raw intelligence when the VC side of the market is brutally competitive. Intangible’s mission (simplifying 3D creative tools with AI) is his example of bridging human vision and machine power.

Decision rule: if a competitor can reproduce your output with the same model stack in a weekend, your moat is probably taste (what you choose to build), distribution (who sees it), or customer depth (how well you solve one painful job). Pick one to deepen this quarter.

Should you bet the company on agentic AI?

Kelly Vaughn, then Director of Engineering at Spot AI, takes a blunt market read in The Agent Bubble Debate. She treats much of the agent craze as overpromise that pushes startups toward expectations they cannot sustain.

The episode contrasts building AI-enabled products with building traditional software, and it pushes back on replacing human teams wholesale (customer service is the cautionary tale). Useful angles for builders include how you construct AI-enabled teams, data governance, and user trust instead of shipping autonomy theater.

Before you rename your product an agent platform, write down the user outcome and the failure mode when the agent is wrong. If the plan is fewer humans with no governance story, Kelly’s warning is worth taking seriously.

Why do enterprise AI projects struggle to show ROI?

Selling into the enterprise means facing buyers who already spent heavily and now ask what return they got. Why Most Enterprise AI Projects Fail to Show ROI walks from hype to measurement with a panel including Alex Klug (HP), Sriram Palapudi (ServiceNow), and Jay Subrahmonia (Accenture).

The thread is familiar: ROI was marketed like a panacea, implementation is hard, and wins depend on picking the right use cases, trust, and explainability. They also stress that ROI is not always a single bottom-line number.

If you sell B2B AI, align your pilot metrics with how the customer prioritizes trade-offs and risk. Alex Klug’s concern from the episode framing still applies: nobody wants to make the news for a deployment that steered customers wrong. Build trust and explainability into the pilot scope, not the phase two slide deck.

Checklist

Score your company on speed, capital, and distribution; if you are weak on all three, invest in engineering fundamentals and feedback loops before new AI tooling (Aurimas).
Codify one messy-middle workflow (services, playbooks, evals) before you chase the next frontier model (Marcel).
Name your moat as taste, distribution, or customer obsession, and ship in front of users to test it (Bharat).
Define agent outcomes, failure modes, and governance; avoid human replacement pitches without a trust story (Kelly).
Match enterprise pilots to prioritized use cases, trust, and explainability, and measure ROI in terms the buyer actually uses (enterprise panel).

More on this topic, with the related episodes, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.

Tools before training for a domain legal agent

Conor Bronsdon — Mon, 05 Oct 2026 14:30:11 +0000

Give the agent its domain tools and a citation check before you spend a training run on a custom model. On CoCounsel Bench, Joel Hron said the prior CoCounsel answered about 25% of questions in one shot, and the rebuilt agent reached north of 70% almost overnight once it had the tools and basic instructions. Record every source that run opened, and reject any citation that is not on the list.

Joel Hron, Chief Technology Officer at Thomson Reuters, covered this on Chain of Thought in August 2026. He leads product engineering across legal, tax, audit, and compliance, including Westlaw, Practical Law, and CoCounsel. CoCounsel is the application. Thomson is the model.

What raised one-shot accuracy before the new model?

The tools, with short instructions for how to use them. Earlier CoCounsel versions were not coding-agent architectures, Hron said. The new CoCounsel decomposes Westlaw and Practical Law into tools the agent can call. CoCounsel Bench is their legal-task set: about 3,000 queries, with more added almost every day.

CoCounsel is built around the Claude Agent SDK and also uses models from OpenAI and Google. Thomson is the later step he described: train the model to use those tools, past skills and system prompts. The first CoCounsel job for Thomson is bulk document review, such as due diligence on large document sets. On tabular analysis, the high-volume review tied to that launch, he said Thomson was outperforming the prior model versions they were using by about five percentage points in accuracy. Frontier models stay in the stack for frontier intelligence work, he said. Thomson does not have to replace Claude, OpenAI, or Gemini for that path to proceed.

But we could answer about 25% of those questions in one shot on the prior version of CoCounsel. And when we built this new version of CoCounsel, almost overnight, we got north of 70% just by giving the agent the tools it needed to sort of do the work and some basic instructions on how to use those tools.

Joel Hron, Chief Technology Officer at Thomson Reuters, on Chain of Thought ep 69

Thomson 1: The New $40M Legal … - Chain of Thought | AI Agents, Infrastructure & Engineering - Apple Podcasts

Podcast Episode · Chain of Thought | AI Agents, Infrastructure & Engineering · August 24 · 54m

podcasts.apple.com

How do you block a citation the agent never opened?

Keep a register of every case inspected and use it to ground the claims.

Hron described a citation ledger: a register of every case inspected during research, used to ground the claims the agent makes. Thomson Reuters has a patent pending on CoCounsel's approach to citation ledgers and citation verification. Westlaw's Litigation Document Analyzer applies a related check to a finished brief. It splits the document into claims, looks for case law that supports each claim, and says so when it cannot find a source.

Fabricated case names are a serious failure, Hron said. They still happen, and he sees them at a pretty low rate, less often than the news suggests. The failures he called harder to catch are misreadings of law the agent did open. Humans remain the primary check. He said there is no automated verification loop for these models like there is with code.

A minimal sketch to adapt, not a drop-in library.

class CitationLedger:
    def __init__(self):
        self.opened = set()

    def record_open(self, source_id):
        self.opened.add(source_id)

    def cites_never_opened(self, answer_cites):
        return [cite for cite in answer_cites if cite not in self.opened]


def accept_or_reject(answer, ledger):
    blocked = ledger.cites_never_opened(answer.cites)
    if blocked:
        return {"status": "reject", "blocked": blocked}
    return {"status": "ok", "cites": list(answer.cites)}

The bad hallucinations that are really difficult to catch are the things that are misinterpretations of the law. ... Or it interpreted this statement from the judge as fact versus opinion. These kinds of things might change the way that the agent operates.

Joel Hron, Chief Technology Officer at Thomson Reuters, on Chain of Thought ep 69

Where does continued pretraining sit relative to the tools?

Continuous pre-training is an earlier training stage, on a selected slice, before agentic reinforcement learning teaches tool use. Fresh case law stays a retrieval problem.

The pre-training text is drawn from Westlaw, Practical Law, Checkpoint (their tax research content), and the Reuters News Archive, after they select what matters most for the jobs. He said less than 10% of that corpus had been applied, and he expects that share to increase. They publish thousands, if not hundreds of thousands, of cases a day, so weights alone are not a practical way to stay current. The model is trained to use Westlaw, and the tax and news tools, for that.

And to this point, we started small there, but less than 10% of that corpus has been applied for continuous pre-training. So that's an element of the training process that we think will increase over time.

Joel Hron, Chief Technology Officer at Thomson Reuters, on Chain of Thought ep 69

FAQ

Did training Thomson produce the move from about 25% to north of 70%?
No. Hron tied that change to the rebuilt CoCounsel agent, the tools, and basic instructions for those tools. Training Thomson to use tools more directly is a later step. The first CoCounsel use he named for Thomson is bulk document review.

Does a citation ledger catch a misreading of a real case?
It grounds claims against cases the agent inspected. Fabricated names still occur at a low rate. Hron said the harder errors are misinterpretations of real law, such as reading a holding too broadly or treating a judge's statement as fact. Those still need a person.

How much of the corpus went into the first continued-pretraining pass?
Less than 10%, on text selected from Westlaw, Practical Law, Checkpoint, and the Reuters News Archive. Hron said that share should grow. He put their publishing pace at thousands, if not hundreds of thousands, of cases a day, so keep a retrieval tool beside the weights.

Takeaway

Score one-shot accuracy on a fixed task set before and after the domain tools exist, as its own change, separate from a training run.
Ship basic instructions with each tool. That is the pairing Hron tied to the one-shot move on CoCounsel Bench.
Log every source the agent opens. Reject an answer that cites a source absent from the log.
Have a reviewer check real cases for an overly broad holding and for a statement treated as fact rather than opinion.
Run continued pretraining on a selected slice after the tools work, and keep retrieval for material that updates faster than a training cycle.

The full conversation, with the transcript, is on Chain of Thought.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.

Threat-model the model endpoint your app exposes to the internet

Conor Bronsdon — Sun, 04 Oct 2026 20:30:07 +0000

You ship a classifier, a moderation layer, or an agent that calls a model on every user request. Treat that endpoint as a machine-learning surface with a different abuse model than a CRUD API. Kris Lovejoy argues attackers can now find individual vulnerabilities faster than ever, and chain low-severity issues into high-impact paths. She also expects a fully autonomous AI-driven attack against an enterprise network within roughly a year and a half of her June 2026 episode. Whether or not that timeline lands, the direction is clear: more automation on the attack side means your production model call needs a threat model alongside its latency budget.

I’d start with five attacks: four that NIST groups under adversarial machine learning, plus model denial of service from OWASP’s list, using plain definitions from the Chain of Thought glossary and one control your app team can own for each.

Did someone poison the model before it reached you?

A backdoor attack compromises the model during training so it looks fine on ordinary inputs and switches behavior when it sees a trigger: a phrase, a pattern, or a small patch on an image. Clean accuracy tests will not catch it, because the failure mode is conditional. Outsourced training, third-party weights, and opaque fine-tunes are the supply-chain version of this risk.

Your control: treat every non-base model artifact (adapter, LoRA bundle, vendor fine-tune) like untrusted code. Pin provenance, document who trained on what, and run trigger-style red-team prompts before you promote a build. You will not “spot the backdoor” with a happy-path eval alone.

Can users slip past your filter at inference time?

An evasion attack does not change the weights. The attacker perturbs an input the model should block (spam, fraud, policy-violating content) until the model misclassifies it while a human still recognizes the intent. NIST ties this to adversarial examples; for LLMs, jailbreaking sits in the same family. Attackers often need only query access, including transfer from another model.

Your control: normalize and constrain inputs before they hit the model (length limits, canonicalization, block known perturbation patterns where you can), and keep a human-readable audit trail when the model clears borderline content. Adversarial training is a training-team problem; your job is to avoid giving the model a single brittle gate with no fallback.

Does your API leak training data through the response shape?

A model inversion attack reconstructs information about training data by querying the model and analyzing outputs. Fredrikson and colleagues showed that confidence scores returned alongside predictions can be enough to recover sensitive attributes or recognizable faces. The fix pattern is to return less.

Your control: default the public API to labels or coarse scores instead of full probability vectors or rich logprobs. If product needs scores internally, serve them on a separate, authenticated path with stricter rate limits and logging. Rounding or bucketing probabilities is boring and effective.

Can an attacker learn who was in the training set?

Membership inference exploits the gap between how a model behaves on records it saw during training versus novel records. In some domains, membership alone is sensitive (for example, participation in a study). Shokri and colleagues demonstrated the attack against commercial ML services; Carlini and colleagues later showed language models can regurgitate verbatim training snippets under query access.

Your control: shrink the inference contract. Return top-1 or top-k classes instead of full distributions where possible, add per-identity query budgets, and flag sessions that systematically probe many similar records. Training-time regularization and differential privacy are upstream; your endpoint still should not hand attackers a high-precision loss oracle.

Can one client drain your GPU budget or your bank account?

Model denial of service covers classic availability hits and “denial of wallet”: unbounded inference that burns compute or pay-per-token spend. OWASP’s LLM Top 10 lists unbounded consumption, including floods of variable-length inputs and inputs that repeatedly exceed the context window. Agent loops that fan out into many model and tool calls raise the same cost risk. Sponge-style inputs can spike latency and energy use on language models.

Your control: enforce budgets outside the model: max input tokens, max output tokens, rate limits per key, queue depth caps, timeouts, and for agents a hard cap on steps and tool calls per task. Monitor spend and latency anomalies the way you monitor 5xx rates.

Why frame it this way now?

Lovejoy’s worry extends past external actors. Agentic workflows are outcome-oriented; guardrails written for linear human processes miss parallel tool use. Insiders with utility access can chain weaknesses you classified as low risk because you optimized for crown jewels.

what we know now is that, you know, attackers are able to find individual vulnerabilities faster than we ever have before. And so that means that we have to get better at detection and better at remediation.

Kris Lovejoy, Global Head of Strategy at Kyndryl, on Chain of Thought ep 62

Now we've gotten to a point where the stuff that we didn't worry about is a big problem.

Kris Lovejoy, Global Head of Strategy at Kyndryl, on Chain of Thought ep 62

An endpoint threat model turns a model purchase into a short table: attack, what leaks or breaks, who owns the control. Training teams own data and weights; you own the HTTP contract, quotas, and what you log.

A sketch of the table as code:

# Sketch: one row per attack class for your inference route
THREATS = [
    ("backdoor", "trigger in prod", "provenance + pre-prod red-team"),
    ("evasion", "bypass at runtime", "input limits + secondary policy check"),
    ("inversion", "reconstruct training", "labels only, no logprobs on public API"),
    ("membership", "was record in training?", "coarse outputs + query budgets"),
    ("model_dos", "cost / availability", "rate limits + agent step/token caps"),
]

def review_endpoint(route: str) -> list[tuple[str, str, str]]:
    return [(route, *row) for row in THREATS]

Checklist

Inventory every route that calls a model, including nested agent tool loops and the main chat API.
For each route, write one line per attack class: asset, attacker goal, your control, owner.
Strip public responses down to the minimum fields product needs; put rich telemetry on internal paths only.
Put token, step, and spend caps in the gateway rather than in prompt instructions.
Re-run the table when you swap weights, add an adapter, or expose a new tool to the agent.

More on this topic, with the related episodes, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.

How to audit every credential your AI agent process can reach

Conor Bronsdon — Sun, 04 Oct 2026 15:42:20 +0000

Check whether your agent already reaches more than you intend. Agents are often wired like classic software, with long-lived keys, broad tool plugins, and a service account that outlives the task. Tyler Akidau, CTO of Redpanda, puts the stall point plainly: enterprises lack the governance layer, not model quality. Before you mint another identity or buy a platform, audit what this process can already touch.

That audit is how you shrink excessive agency in the OWASP sense (LLM06:2025): one bad turn, ambiguous output, or injected instruction should not inherit the power to do real damage. The trigger can be a hallucination, prompt injection, or a compromised tool. You cannot guarantee model behavior, so you limit what behavior can do.

What counts as "reach" for this review?

Reach is anything the agent runtime can read or call, not what you meant to allow in a prompt. Jitender Aswani, of Starburst, frames the data side: a federated query engine can let an agent touch many systems without copying data into one warehouse, but the catalog still must answer where data lives and who may access it. Connecting is not the same as being authorized. Federation can widen blast radius while permissions stay vague.

For one deployment, inventory surfaces together:

Environment variables, secret mounts, and CI injectors tied to the agent pod or worker
Tool and MCP configs (including plugins shipped for dev and never removed)
Shared service accounts the runtime reuses across instances
Log and trace pipelines (tracing can write credentials into plaintext logs)

Shadow AI belongs in the same conversation when teams bypass IT: unsanctioned chatbots, extensions, or personally configured agents, and work pasted into contexts you do not govern. IBM defines shadow AI as unsanctioned employee use without formal IT approval. Microsoft and LinkedIn's 2024 Work Trend Index reported 75% of knowledge workers using generative AI at work and 78% of AI users bringing their own tools. Your official agent is one row in a wider table.

How do I list credentials and what each one grants?

Work runtime by runtime, not "our agent product" in the abstract. For each secret or token the process can read, record four fields: name (or vault path), issuer (IdP, cloud IAM, SaaS admin), grants (API scopes, DB roles, repo permissions), and expiry (or "none" if that is true).

Agents sit near real credentials: tool API keys, connection strings, OAuth tokens. Token leakage is when those secrets escape into model context, user-visible tool output, or plaintext traces. OWASP's guidance on system prompt leakage (LLM07:2025) is blunt: do not treat the system prompt as a secret control, and do not put credentials in it. If your audit finds keys in prompts or skill files, treat that as a leakage risk to fix first.

Sketch for turning logs into a review table (adapt to your vault and orchestrator):

# Sketch only: normalize what one agent worker can read
def audit_row(secret_id, issuer, scopes, expires_at, used_by_tools):
    return {
        "secret_id": secret_id,
        "issuer": issuer,
        "grants": list(scopes),
        "expires_at": expires_at,
        "tools": list(used_by_tools),
    }

# Example: walk the env vars and mounted files the worker can read today
rows = [audit_row("OPENAI_API_KEY", "platform", ["inference"], None, ["planner"])]

Flag rows where grants exceed the current task, expiry is missing, or the same secret appears on multiple unrelated agents.

How do I classify "excess" before I fix it?

OWASP groups excessive agency into excess functionality (tools the job does not need), excess permissions (identity can do more than the task), and related failure modes. Map each audit row to a task statement: "summarize tickets", not "administer Jira".

Tyler Akidau's authorization checklist from the agent identity explainer is a good classifier:

Narrowly scoped to the task, not everything the agent might ever need
Short-lived (billing access at 2 p.m. should not linger an hour later)
Deny capable (agent may read production, never write, even if the human can write)
Intersection aware (agent plus user rights are the intersection, never the union)

you can get a guest badge and you can kind of go anywhere that a human will take you. But even if the human has access to the secret server room that guests aren't allowed in, you're still not allowed to go because you've got the guest badge.

Tyler Akidau, CTO, Redpanda, on Chain of Thought ep 60

Jeetu Patel, Cisco's President and Chief Product Officer, adds a second axis on the same explainer: access to email is not permission to email your board. You need action control, not access control alone. Read without send, draft without publish, recommend trade without execute.

What fixes should I apply, in order of effort?

Start with cheap removals, then tighten time and scope, then move enforcement out of the agent's view.

Remove excess functionality. Drop tools and plugins the task does not use (document readers that can delete, dev MCP servers left enabled). This is where I would start on excessive agency.
Stop sharing one credential across concurrent instances. Akidau notes agents clone easily; one key plus many parallel tasks mixes blast radii. Split identities per instance or per task where your IdP allows it.
Add expiry and rotation. Replace "never expires" rows first. Pair with narrowly scoped grants for the single job.
Encode denies and intersections in policy, not prompts. Tyler Akidau is explicit that a repo guidance file is guidance: if it must not happen, enforce it in infrastructure the agent cannot edit. Prompts and guard models fail under injection and hallucination pressure.
Escalate by stakes. After the software audit, ask what still moves money or irreversible data. Charles Guillemet of Ledger argues that securing a high-stakes agent with permissions alone does not give the guarantees finance-grade abuse demands; hardware enters when attacker incentives are high. You do not need a hardware program to delete a stale plugin, but you should not pretend a prompt cap replaces policy for treasury actions.

For broader episode context on identity, federation, and action boundaries, the AI security topic page collects the full conversations.

The only way to have very strong security guarantees is to have the dedicated hardware. And when it comes to your money, the incentives for the attacker are very high. So hardware will be part of the equation.

Charles Guillemet, CTO, Ledger, on Chain of Thought ep 65

Use that quote as a severity triage rule, not as an excuse to skip the audit.

Checklist

List every secret and token the agent process can read (env, mounts, tools, CI), with issuer, grants, and expiry.
Mark rows that exceed the current task, never expire, or feed logging and tracing without redaction.
Include unsanctioned or parallel agent use in scope so shadow setups are not invisible.
Classify each row against narrow scope, short life, deny rules, and intersection (union is a bug).
Apply fixes in order: remove unused tools, split shared credentials, shorten TTL, then enforce denies outside prompts; reserve hardware-grade controls for high-incentive targets.

The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.