Log what your tools put in the context window before blaming the model

Ask an agent for one person's sales calls since February 1 and it can come back with calls that person wasn't on, even though the right ones were in its context. The data reaching the model causes that, and a live demo on Chain of Thought in March 2026 showed how.

Michel Tricot, CEO and co-founder of Airbyte, ran the query against the Gong API directly. The API couldn't filter calls by user, so the agent paged through every call over roughly a month and a half. He said that path normally takes about three minutes, that it often returns more than 45 calls because the bloated context leaves the agent unable to filter down to the right person, and that it consumed almost 30,000 more tokens than the path through a context store. Those are his rough numbers from one demo, not a benchmark, but the shape is common: the tool handed the model a filtering job the application should have done.

That's context poisoning. Wrong, stale, or badly shaped data reaches the window and the agent reasons off it. Nothing malicious has to happen.

Because as everyone is saying, the problem is not in the model today, the problem is in the data that you provide.

Michel Tricot, CEO and co-founder of Airbyte, on Chain of Thought ep 52

Log the insertion point

Instrument the spot where a tool result becomes model-visible text. The HTTP response size isn't enough, since anything you filter or transform in between changes what the model sees.

For each tool call, record the tokens the serialized result added and the IDs of the records it delivered. After the run, record how many of those records the answer used. Keep the exact payload available (under your normal data-access controls) so you can read what the model read.

Illustrative sketch, not a production library or anyone's actual implementation. count_tokens is a hypothetical helper using your model's tokenizer.

import json


def append_tool_result(call_id, tool, records, messages, audit):
    content = json.dumps({"records": records}, separators=(",", ":"))
    messages.append({"role": "tool", "tool_call_id": call_id,
                     "content": content})
    audit[call_id] = {
        "tool": tool,
        "tokens_added": count_tokens(content),
        "returned_ids": {r["id"] for r in records},
        "ids_used": None,  # unknown until someone checks the answer
    }


def record_usage(call_id, verified_ids, audit):
    entry = audit[call_id]
    entry["ids_used"] = len(entry["returned_ids"] & set(verified_ids))

Call the wrapper after any shaping, so the logged content is exactly what the model got. Fill verified_ids from checked citations or a review that maps claims to records. A citation shows what the model says it used, which isn't always what it relied on, so leave the field unknown until someone has checked.

Look at precision, not only size

A big result isn't automatically a bad one. Comparing forty calls needs forty calls.

The warning sign is a big result full of records outside the scope of the question. Near-duplicates crowd the window, and the model can hold the right record without weighting it. In a trace that reads like a reasoning failure, which sends teams off to evaluate models when the input is the problem.

For a failed run, compare the delivered IDs with the records the task needed, and check any wrong claim against neighbouring records in the payload. Then rerun the task with a shaped result, same model and prompt, and check correctness and coverage as well as whether the window got smaller.

Filter before the model sees it

Tricot described doing that work on the retrieval side:

So the first thing is a lot of this is done on the back end to make sure that we only return what we believe is the most relevant information.

Michel Tricot, CEO and co-founder of Airbyte, on Chain of Thought ep 52

Push the task's constraints to the source where it supports them: fetch by ID, filter by user, apply the date range. Return only the fields the question needs, plus the identifiers you need for attribution. When the source can't express the query, filter in application code or query a store synced for retrieval. Don't send every page to the model because the upstream API happens to paginate.

Projection is per task. A call-listing tool can return compact metadata. A tool that answers questions about what someone said on a call needs the relevant transcript text, and trimming that away saves tokens while making the answer unsupportable.

Decide which reads have to be live

A synced store raises a freshness question. Tricot's own setup is hybrid: search the store to discover and understand, then go to the API for a specific object when the task needs the freshest version.

And APIs are very good at retrieving one object as long as you know what this object is.

Michel Tricot, CEO and co-founder of Airbyte, on Chain of Thought ep 52

Make that a tool policy instead of something the agent improvises.

Illustrative sketch. fetch_record is a hypothetical adapter that reads one authorized record by ID.

FIELDS = ("id", "owner_id", "started_at", "title")


def get_call(call_id, require_live):
    source = "live_api" if require_live else "synced_store"
    record = fetch_record(source, call_id, fields=FIELDS)
    return {f: record[f] for f in FIELDS}

The final projection controls what reaches the window even if the upstream response carries more. Expose sync timestamps so the staleness decision is visible. One catch: refreshing records you discovered in the store doesn't find records created since the last sync. A task that needs a complete, current list needs a discovery path that is current too.

Takeaway

For the next failed agent run:

• Read the exact tool payloads that went into the context.
• Log tokens added and records the answer used, per tool call.
• Filter and project before insertion, not after.
• Set freshness rules for discovery and for single-record reads.

The longer explainer, with the episode clips, is on Chain of Thought.

Drafted with AI assistance from the episode transcripts, then edited by me.