The most common agentic system failure I encounter in production is not a bad prompt. It is not a context overflow. It is a tool that changed without its registration changing.
I have seen this cause weeks of debugging in systems that were working fine until they weren't — and because the failure is silent, teams often spend time looking at the model first.
What Tool Schema Drift Looks Like
Most agentic frameworks let you register tools with a name, a description, and a JSON schema for parameters. The model reads the description to decide when to call the tool. It reads the schema to know what parameters to send.
# Initial registration — looks fine
tools = [
{
"name": "search_entities",
"description": "Search the entity registry by name. Returns a list of matching entities.",
"input_schema": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search term"}
},
"required": ["query"]
}
}
]
Six months later, the underlying function has changed. The team added a required entity_type filter after learning that unfiltered searches returned too many results. They updated the implementation. They forgot to update the registration.
# Current implementation — mismatched with registration
def search_entities(query: str, entity_type: str) -> list[dict]:
# entity_type is now required — "company", "person", "location"
return registry.search(query=query, type=entity_type)
Now the model calls search_entities with {"query": "Samsung"}. The function raises a TypeError because entity_type is missing. Or worse — if you have a default fallback somewhere upstream — it silently runs with the wrong parameters and returns results the model was not expecting.
The model sees a response. It generates text from whatever came back. No exception fires. The output just drifts from what it should have been.
Why This Is Hard to Catch
The failure mode is subtle for two reasons.
First, evals usually test output quality, not the call-response cycle. If your eval checks whether the final answer is correct, a tool that silently misbehaves can still produce plausible-looking output — especially if the task has any ambiguity in what "correct" means.
Second, the description is what the model acts on, not the schema. A schema mismatch causes a runtime error. A description mismatch causes behavioral drift — the model calls the tool in situations where it should not, or does not call it when it should, and there is no error signal.
Both failure modes happen in production. Neither shows up in type checking or unit tests.
Three Checks That Catch Drift Before Production
1. Validate the response shape, not just the call shape
Most frameworks validate that the model produced a well-formed tool call (correct JSON, required fields present). Fewer validate that the tool's response matched what the model was told to expect.
from pydantic import BaseModel, ValidationError
from typing import Any
class EntitySearchResult(BaseModel):
entity_id: str
name: str
entity_type: str
confidence: float
def call_tool_with_validation(tool_name: str, args: dict) -> Any:
raw_response = dispatch_tool(tool_name, args)
# Validate response matches what the model expects
if tool_name == "search_entities":
try:
validated = [EntitySearchResult(**item) for item in raw_response]
return validated
except ValidationError as e:
# Response shape changed — surface this immediately
raise ToolResponseSchemaError(
f"Tool '{tool_name}' returned unexpected shape. "
f"Registration is out of sync with implementation.\n{e}"
)
return raw_response
This makes the mismatch loud. A ToolResponseSchemaError is unmistakable. A silently wrong answer is not.
2. Version the tool description alongside the implementation
Treat the description as part of the implementation contract, not as documentation.
The practical pattern: keep tool schemas in a registry file under version control, and make updating the registry a required step when any tool function signature or return shape changes. Code review that accepts a function change without a registry update is accepting a potential drift.
# tools/registry/v2.py — versioned alongside implementation
TOOL_REGISTRY = {
"search_entities": {
"version": "2.0.0",
"description": (
"Search the entity registry by name and type. "
"Returns matching entities with confidence scores. "
"entity_type must be one of: company, person, location."
),
"input_schema": {
"type": "object",
"properties": {
"query": {"type": "string"},
"entity_type": {
"type": "string",
"enum": ["company", "person", "location"]
}
},
"required": ["query", "entity_type"]
},
"response_schema": {
"type": "array",
"items": {
"type": "object",
"required": ["entity_id", "name", "entity_type", "confidence"]
}
}
}
}
If the interface changes in a backward-incompatible way — new required parameter, different return shape — give it a new name (search_entities_v2) rather than silently updating the existing entry. Agents that depended on the old interface continue to work until they are explicitly migrated.
3. Add canary evals that cover the full call-response cycle
A single eval prompt per tool is enough to catch schema drift in CI.
# tests/evals/test_tool_contracts.py
import pytest
def test_search_entities_contract():
"""
Canary eval: triggers the tool and validates the full cycle.
If search_entities changes its interface, this breaks before production does.
"""
agent_response = run_agent(
prompt="Find all company entities named Samsung in the registry.",
tools=TOOL_REGISTRY,
)
# Check the tool was called
tool_calls = extract_tool_calls(agent_response)
search_calls = [c for c in tool_calls if c["name"] == "search_entities"]
assert len(search_calls) > 0, "Agent did not call search_entities"
# Check the call used the current schema
for call in search_calls:
assert "entity_type" in call["args"], (
"Agent called search_entities without entity_type — "
"tool description may be out of sync with schema"
)
# Check the response matched expected shape
assert agent_response.tool_errors == [], (
f"Tool returned unexpected shape: {agent_response.tool_errors}"
)
This eval does not test whether the final answer is good. It tests whether the tool call cycle completed without a schema mismatch. That is exactly what breaks first when a tool drifts.
The Root Cause
The reason this happens repeatedly is that tool descriptions live outside the usual code review discipline. They are strings in a config or a dict literal. Nobody has a linter that flags "the function signature changed but the description did not."
The fix is not complicated. It is a process discipline problem more than a technical one. The registration is the contract between the model and the implementation. Versioning it like a contract — with change control, backward-compatibility rules, and automated validation — closes the gap that produces silent drift.
The agent is not wrong when it mishandles an unexpected tool response. It was not told the tool changed. If you treat the registration as the contract, you start maintaining it like one.
I work on entity resolution and agentic infrastructure at er-api.hannune.ai. If you have seen this failure mode in your own systems, curious what the trigger was.
Top comments (5)
The split you draw between schema drift and description drift is the part I wish more people internalized. A required-param mismatch at least throws. A stale description just quietly re-weights when the model reaches for the tool, and nothing anywhere fires.
We got bitten by the second kind: a scraper tool's description still said it returned "recent posts" long after the implementation had started returning an empty list on a specific failure path. No exception, valid JSON, the model dutifully summarized nothing, and the downstream report just... got thinner for four days before anyone noticed. The eval passed because the output was well-formed prose.
What eventually caught it wasn't validation, it was a floor assertion — "this tool returning zero rows is an error, not a result." Your
ToolResponseSchemaErrorgets you shape; we needed shape plus plausibility.Curious how you handle the versioning side in practice — do you fail the run outright when the registry version doesn't match the implementation's, or just warn and log?
In practice I use a two-tier response: a major version bump (changed required params, removed fields) fails the run immediately at load time because the model is operating from a schema that no longer matches, and a silent failure there is worse than an abrupt stop. A description-only or minor change gets a logged warning plus a health-check flag so the next deploy surfaces it without blocking the current one. The heuristic is whether the mismatch can produce silently wrong output or just noisy output.
The hand-written Pydantic response validator is a useful tripwire, but it's still another copy of the contract. Nothing structurally binds EntitySearchResult to the function's real return value, so the same drift can reappear one layer later: the implementation changes, and the validator either stays green or starts rejecting valid results. That lowers detection latency, which matters, but it doesn't remove the copy-goes-stale failure mode.
For the schema side I'd rather make the registration a derived artifact. Generate the input_schema from the typed function signature at load time, and build the response model from the return annotation. If entity_type becomes required, the model-facing required list changes with it, so the actor stops being told it may omit the field.
The description side is different. "When should this tool fire?" is semantic. A signature has nothing to say about it, so that half needs deploy-time behavioral probes: small checks pinning which situations should trigger a call and which shouldn't, because no shape check can cover that.
Deriving input_schema from the typed signature at load time is exactly right for structural drift, and I should have pushed that point harder in the article. The description side is where I agree there is no clean mechanical solution, which is why I lean on deploy-time behavioral probes rather than a second static artifact. The two problems are genuinely different in kind, and collapsing them into one validation layer tends to under-solve the semantic half.
A behavioral probe is still a copy of the tool's purpose. If it says "in situation X, this tool should fire", that is another hand-written semantic claim sitting beside the description. The upgrade is the failure direction. A stale description fails quietly because the model's routing prior moves and no exception has to appear. A stale probe can fail at deploy time, on the change that made the assertion wrong. That is the useful property. A probe is allowed to go out of date. It just cannot do it silently, and silence is the entire problem with the description.
That also sets the coverage boundary. Behavioral probes are strongest on the should-fire side, where the important situations can be named. The should-not-fire side is open-ended, and description drift often hurts by making a tool disappear from selection rather than by causing an obviously bad call. A green probe suite says the cases you imagined still route correctly. It says less about the space you did not enumerate.
The extra signal I would add is selection frequency per tool, compared against that tool's own history and keyed to a hash of the description text registered at deploy time. If a description edit re-weights the model's choice, the observable is a call-share shift across the deploy boundary while schema drift checks stay clean and no exception is raised. That signal is derived from traffic already flowing through the system, so it avoids becoming a third prose contract in the same way signature-derived input_schema avoids a second structural contract.
It has a hard limit. Frequency says selection moved. It does not prove the new routing is worse. Traffic mix moves too, so attribution only becomes credible when the shift is pinned to a dated description hash. Versioning the semantic artifact makes description drift measurable, even though it remains unprovable in the way schema drift can sometimes be mechanically ruled out.