Originally published on Medium.
I maintain bcb-br-mcp, an open-source MCP server that gives AI agents the Brazilian Central Bank's public data: th...
For further actions, you may consider blocking this person and/or reporting abuse
Lesson 6 hit home this morning. Our MCP server (worklore) had 79 green tests, and get_story still returned isError on every call in production: a DynamoDB Decimal reached json.dumps, and none of the tests went through tools/call. The fix was the client-shaped harness @arhancanli describes: initialize → tools/list → validate each result against the outputSchema it was just handed. It failed on the first run.
One question on #11: do you also run the manifest-vs-live check against the registry's copy, or only the committed one? That's where we've seen drift.
Thanks for writing the failures down in this much detail. It's rarer than it should be.
Glad lesson 6 got there before it cost you more than a morning. A Decimal that only breaks at json.dumps is the perfect example: no unit test goes near the serialiser, and the client-shaped run hits it on the first call.
On #11: only the committed one, and you've put your finger on the gap. The test pins the file in the repo to the live tools/list/resources/list/prompts/list and to the package identity, so a stale manifest can't be released. But the directory's copy only changes when a publish step runs, and that step is part of my release checklist, not something a test checks. So the drift I described (a listing several minors behind npm) is now prevented by process rather than detected by a measurement. That's the weaker of the two, and the last mile the test doesn't cover. The official MCP Registry is in slightly better shape, because its server.json is published by the same CI job that publishes to npm, but nothing reads it back either.
The fix is the same pattern one step further out: fetch the listing's current copy and diff it against the live server on a schedule, not only at release. Registries don't all expose the copy the same way, which is probably why I stopped at the repo. Your drift report is a good reason not to.
Thank you, this is a generous answer. After reading it I checked my own listing: the registry copy and the live worklore server both say 0.1.0, so a version diff passes cleanly. But the server has changed quite a bit since it was listed: tools/list now answers without a token, and the auth rules moved into its instructions. I never bumped the version. So the drift can hide one level deeper: the version matches, while what the server actually does has moved on. Your scheduled diff is going on my list; I'd just compare more than the version.
That's a sharper version of the drift, and I think you're right that the version is the weakest thing to diff on.
I checked what each copy can actually be compared on. The LobeHub manifest carries the whole surface, so there my test compares tools/list, resources/list and prompts/list field by field, descriptions and schemas included, not the version. The official MCP Registry's server.json carries no surface at all: name, version, packages, remotes. Against that copy the version is the only thing there is to diff, which turns your case around: the check is only as good as the habit of bumping the version whenever the surface moves. Nothing in my pipeline enforces that either; the version goes up because I run the release step, not because a test noticed the surface changed.
Your example also finds a gap in mine: both of your changes live outside tools/list. Instructions come back in initialize, which my manifest test doesn't read, and "answers without a token" is behaviour rather than declaration, so no listing would show it. The cheap fix I can see is to hash the initialize result together with the three lists, commit the hash, and fail CI when it changes without a version bump. That would turn "I never bumped the version" into a red build instead of a silent drift.
I built it. The test hashes the initialize result together with tools/list, resources/list and prompts/list, plus which methods answer without a token. It commits the hash next to the version, and the deploy now refuses to run while it's red.
Then I recorded the surface exactly as the server was when I listed it on 09-15 and ran it against today's code. It failed, and on something worse than I'd described: the registry said 0.1.0 for a server with four read-only tools, and the server behind that number now has six, including two that write on the user's behalf. Bumped to 0.2.0, deployed, and the registry shows it now.
Thank you, that's one comment thread that turned into a real fix.
That replay is the part I'd have skipped, and it's the part that found the real problem. A version diff would have passed forever: same number, but four read-only tools had quietly become six, two of them writing on the user's behalf. That's the case a registry listing exists to rule out.
Both of your additions are going into my own version, which isn't built yet. Hashing which methods answer without a token covers the gap I said no listing could reach. And making the deploy refuse to run, instead of only failing CI, is the right place for the gate. I'll also borrow the replay: record the surface each published version had and run it against today's server, so the first run checks the whole history rather than just the next release.
Lesson 2 is the one that bites hardest in finance: the data and the code each assume a convention, and nothing checks them against each other. I hit the same thing with stock splits. The data stored split factors one way and the code applied them the other way, Apple's 2020 split turned a 124.81 adjusted close into 1,996.92, and every test passed, because no test ran real data through the real code. Your "measure invariants across the whole catalogue" is the fix I'd underline. On lesson 6: I'd also make the tests call tools/list first, the way a real client does, since some clients only validate results against the schemas they've listed. Tests that skip that step can pass while real clients fail.
The split example is a clean statement of the problem: two conventions, each correct on its own terms, and a 16× error that no test could see because the tests never put real data through the real code. The IPCA bug was the same shape. Every unit test used synthetic series where "last minus first" happened to be the right formula, and it took real data (and a model tripping over an absurd number) to show that the code held one convention and the source published another.
On lesson 6, fair point, and it made me go and check what my tests actually do. They cover both halves, but separately. One test connects a real client over an in-memory transport, checks that what tools/list announces is identical to the definitions in code, and calls every listed tool, but it only asserts that the call doesn't break. The output-contract test validates every result against its outputSchema, but it reads the schema from the definitions, not from what was listed. It holds by transitivity (listed = defined, result ⊨ defined), but transitivity is an argument, not a test, and it quietly assumes the client validates the way my validator does. The Inspector uses a different one. Collapsing the two into one client-shaped test (initialize → tools/list → tools/call → validate against the listed schema) is cheap, and it's on my list now.
"Transitivity is an argument, not a test" is exactly the gap.
Two small things that make the collapsed test worth its keep. First, validate with whatever the client you care most about uses rather than your own validator, since that's the assumption you just named; with the TypeScript SDK that means letting its client do the call and the validation, so the test fails the way a user's session would. Second, give it a negative control: a test that deliberately breaks one result (drop a required field, or add one the schema forbids) and asserts the client-shaped test goes red. A test that has never failed on a known-bad result hasn't shown yet that it can.
Both taken, and both are now part of the item. The first one corrects my own plan: I'd written "use a different validator, like the Inspector's", which just swaps one assumption for another. Letting the SDK client make the call and do the validation tests the thing users actually run. I still need to confirm that the client validates structured content by default on the SDK version I'm on. If it doesn't, that's worth knowing too.
The negative control is the part I'd have skipped, and it turns out to matter. On one of my servers every tool publishes a permissive output schema, so a broken result would pass the client-shaped test silently. The control is the only thing that would reveal the test is empty there. Thanks for pushing on this twice.
On your open question: in the TypeScript SDK (I checked 1.30.0 just now), Client.callTool does validate by default, but only for tools whose output schema it has cached, and it only caches them inside listTools(). So a test that calls a tool without listing first gets no validation at all and passes. That's the trap we fell into. Once the schema is cached it rejects two things: structuredContent that doesn't match the schema, and a result with no structuredContent at all from a tool that declares an outputSchema (unless isError is set).
That second check gives you a control that still bites on the server with permissive schemas: in a test build, have one tool drop its structuredContent. However loose the schema is, a client that really validates must throw on that call. If it doesn't, the test is empty.
Confirmed on the v2 SDK my servers run (client 2.0.0) — same code path. The validator comes from the cached tools/list response, and with nothing cached callTool returns whatever it got. I measured it on one real tool, tampering with the result between server and client:
v2 adds a third way that doesn't need the list: pass the tool definition to callTool (toolDefinition option) and it validates against that. Same two throws.
Then I checked my own suites. Every test that calls a tool without listing first asserts values, not the schema, and the schema tests list first and validate on their own. So nothing is empty today. But the change I'd planned — handing validation to the client — would have walked straight into this in any test that skipped the listing. It's now written into the item: list first (or pass the definition), and the negative control is the dropped structuredContent. That's also the answer for the server with permissive schemas: no field to remove there, but a missing structuredContent still makes the client throw.
Thanks — this saved me from a test that would have been green for the wrong reason.
This is one of the most honest "lessons learned" posts I've read on MCP servers — most writeups stop at the protocol, but you went after the actual failure surface: a source lying with HTTP 200, ordering that isn't deducible from the endpoint, and the numeric-semantics bug (last-minus-first on a rate-published index) that only a paid model eval caught. That last one especially — a bug in what the number means, invisible to unit tests — is the kind of thing that should scare anyone building tools for agents to reason over.
The two-transports-one-registry lesson hit closest to home for me. I build RAG/LLM systems and agent tooling myself, and the failure mode you describe — two entry points quietly diverging until one client gets a weaker contract than the other — is exactly the class of bug that's invisible until someone diffs the actual wire surface. Committing a normalized tools/list baseline and diffing it in CI is a pattern I'm going to steal outright.
The provenance section is the one I'd push on further: collecting the real fetch instant via AsyncLocalStorage at the single network call site, then taking the oldest timestamp across a cached answer's dependencies, is a genuinely elegant solution to a problem most people don't even realize they have (a cached answer silently misreporting when it was actually true). Have you run into cases where the "oldest instant" rule itself becomes wrong — e.g. a slowly-changing series where an old cached value is technically still accurate but the provenance block undersells its freshness?
Great writeup — bcb-br-mcp is going on my list of references for how to actually harden an MCP server past the demo stage.
Thank you — and steal the tools/list diff freely; it has caught more than any test I wrote by hand.
On the oldest-instant rule: I haven't found a case where it becomes wrong, because it answers a narrower question than freshness. retrieved_at is "when did the server last read this from the origin", which is a fact about the extraction, not a claim about the data. Whether an old cached value is "still accurate" is something the server can't know without asking the origin again — and for official statistics that's not a safe assumption, since series do get revised after publication. So if the block undersells freshness, it undersells it honestly: the fix is to revalidate, not to move the timestamp forward.
Freshness of the data lives in a different field: data_vintage is the period the numbers cover, and that stays correct regardless of when they were fetched. Keeping the two separate is what lets the oldest-instant rule stay conservative without lying.
Where it does get blunt is a mixed answer — part cached, part fetched just now. The block reports the oldest instant and served_from_cache: false, so it tells you the answer isn't fully cached but not which part is. Per-dependency timestamps would fix that; so far it hasn't been worth the extra weight in every response, but your question is a good nudge to write that trade-off down.
Following up on the provenance thread — I've been thinking about where our two areas actually overlap, and I think there's a concrete, buildable idea here rather than just a general "let's collaborate."
bcb-br-mcp is free and open, which is exactly right for adoption. But I've been working directly in x402 (HTTP 402 micropayments for AI agents — I built an agent using Coinbase AgentKit + a CDP-managed wallet that autonomously pays for API access over x402, settling USDC on Base). The pairing seems almost too natural: a rigorously engineered, high-trust financial-data MCP server is precisely the kind of source an autonomous agent economy would pay small amounts to query reliably — and your reliability work (the tools/list CI diff, the retrieved_at/data_vintage separation, the honest "blunt not wrong" framing) is exactly the trust signal that would justify a paid tier over the dozens of sketchier financial-data APIs out there.
Concretely, I'm picturing:
A paid x402 tier alongside the free one — the free MCP stays exactly as-is for discovery/goodwill, but a small x402-metered endpoint offers something the free tier doesn't: higher rate limits, bulk historical exports, or a provenance-audited "certified" response (with the retrieved_at/data_vintage fields as the actual value-add — most competing data APIs can't make that guarantee at all). I could build the x402 payment layer end-to-end; you already have the hard part (correct, trustworthy data) solved.
A RAG/agent layer on top — a natural-language interface for Brazilian macro data ("how has Selic moved relative to IPCA over the last 3 years") that uses bcb-br-mcp as its tool layer under the hood. This is closer to my daily work (RAG pipelines, LLM agent design) and could be sold as a lightweight SaaS to fintechs/analysts who don't want to write MCP client code themselves.
Multi-source expansion — longer-term, the same reliability patterns you built (ordering detection, provenance, CI-diffed surface) could extend to other countries' central bank/statistics APIs. I do data engineering work (Databricks/Spark-scale ETL) that could productionize ingestion at that scale if it ever made sense to go there.
Option 1 is the one I'd actually want to build first — it's the most directly monetizable, and it plays to both of our strengths without touching the free/open core you clearly care about preserving. Would you be open to talking through it, even just at the "does this make sense" stage?
Thanks for thinking this through concretely — I appreciate it. I'm going to pass on a paid tier, though, for reasons of principle rather than effort: the underlying BCB data is free and open, and I want provenance to be something every response carries, not a premium feature. A "certified" tier would also promise more than the server can honestly guarantee — as I wrote above, retrieved_at is a fact about the extraction, not a warranty on the data.
That said, the server is MIT-licensed, so if a natural-language layer on top of it (your option 2) makes sense for your work, you're completely free to build it — no permission needed. If you do, I'd be glad to hear how the provenance fields hold up in a RAG pipeline.
Lesson 2 has an exact twin in SEC XBRL data. A third-quarter 10-Q reports revenue for both the three months and the nine months under the same concept, and the fourth quarter usually isn't filed as its own number at all: it is the 10-K's annual figure minus the nine-month value. So "the latest revenue" can be a quarter, a year-to-date total or a full year depending on which fact you picked up, and the fix is yours: carry the period's start and end dates with every number, and say when a quarter was derived rather than reported.
The same data hides a second convention: restatements. A quarter comes back a year later as the comparative column in the next 10-Q, sometimes with a different value, so the server has to choose "latest restated" or "as originally reported" and say which. My server defaults to restated, exposes a flag for the original, and puts the period dates, form, filing date and accession number on every row, which is my version of your lesson 7.
Disclosure: I build Equibles, a financial-data MCP server for US companies, so these are scars rather than theory.
Thank you, Daniel — both of your conventions have exact twins in the SGS, and checking them was worth it.
Period. An SGS observation is only {date, value}: the date of a monthly or annual series is the first day of the period, and some series arrive already year-to-date or 12-month accumulated, which only their name says. The tool descriptions now say it. Computed values were already marked as derived, but your point found a hole: the tools that compute across several series were emitting data_vintage: null. Fixed in 1.17.0.
Restatements. The SGS keeps no as-first-published copy, so I can't offer your flag. What I can do is say it: every value is now declared as the version current at retrieved_at.
I ran the same two questions on the sibling IBGE server, and that's where it hurt. Five named indicators returned only .., nine mixed a monthly change with its own year-to-date and 12-month figures, and "GDP per capita" was reading municipal GDP. All fixed, with a live test that calls every indicator and requires a number.
Thanks again for sharing them.
Glad it was useful, and declaring every value as the version current at retrieved_at is the honest answer when the source keeps no first-published copy. The IBGE find is the scary one: "GDP per capita" quietly reading municipal GDP passes every schema check, and only a live call with a known answer catches it. Thanks for coming back with the results.
"The response must say which convention it used" is the lesson I learned from the other side. Our earnings metrics came back as the filing printed them: Oracle's revenue as "$19,345" because the table was in millions, Adobe's as "$6.76 billion" in the same response. Faithful, and useless to compare without guessing a multiplier. We ended up adding numeric, unit and scale next to the string, filled only from what the filing itself says, and scale stays null when it says nothing. In your A/B, did the agents actually read the convention field, or did they mostly take the number and move on?
"Faithful, and useless to compare" is exactly the trap, and scale: null when the filing says nothing is the right call. It's the same rule I ended up with for provenance: null means "not stated", never a guessed default.
On your question, honestly: the A/B can't answer it, because it didn't vary the convention field. The only thing toggled between arms was the retrieval diagnostics block; the convention metadata was present in both. Three of the twelve tasks did carry a convention trap (accumulated IPCA for a year, the same for IGP-M found by name, and a comparison mixing a rate series with level series), and every fixed-answer check passed in every cell. But that says more about where the computation lives than about what the model reads. The server accumulates the monthly rates itself and returns the right number, so a model that ignored the convention field entirely would still get it right.
That's the design choice underneath lesson 2, I think: when the server can apply the convention, it should apply it and then say so, rather than hand over a raw number plus a field and hope the model reads the field. Your case is harder, because you can't normalise scale when the filing is silent. There the field is the only defence, and whether agents read it is a real open question. A clean test would be the one you're implying: same tasks, convention field present vs. stripped, scored on whether the answer states the unit. I haven't run it.
That's the split I'd been missing: apply the convention when the server can and say it did, and keep the bare field only for the cases where it can't. Our script already leans that way. It prints the string and the multiplied number side by side, and leaves the number blank when scale is null, so the gap is at least visible to whoever reads the card.
Printing the string next to the multiplied number, and leaving the number blank when scale is null, is a nice way to keep the gap honest: whoever reads the card sees exactly where the convention stopped. Thanks for sharing how you handle it.
The distinction between a transient HTML response and a missing series is exactly where an MCP server becomes production infrastructure instead of a thin wrapper. I like the bounded retry idea for short queries, and I would also expose the source response shape and retry count to the agent so it can explain stale data instead of inventing certainty. Have you measured whether those diagnostics reduce bad tool calls in long agent runs?
Measured it. Here is the number, as promised.
96 agent sessions on bcb-br-mcp (Claude Sonnet 5 driving the server over stdio): 12 multi-step tasks with fixed answers, A/B with and without the
retrievalblock visible to the model, each at 0% and at 20% injected upstream failures (deterministic, at the fetch layer), 2 runs per cell. Total cost: US$ 2.63.Result on your exact question — does it reduce bad tool calls? No, and for a reason I did not expect. The count of bad calls (repeating a call after a definitive error, schema refusals, insisting on a failing key) was zero in both arms, at both failure levels. The server's bounded retry absorbs the injected failures before the model ever sees an error: what reaches the model is a valid result flagged
unstable: true, not a failure. And Sonnet 5 simply does not repeat or insist. The defence at the network edge zeroes the numerator the field was supposed to reduce.What the field does change is the reporting. When the origin did flicker, the model with
retrievalwarned the user about it in 3 of 6 judged sessions; without the field, in 1 of 12 (and that one was an empty result, not a flicker). Small sample, clear direction — it is exactly what the instruction attached to the field asks for ("say so when citing the number, and do not repeat the call").Two side notes: real sessions were 2–3 steps, not the ~20 I had budgeted for, so a session costs about US$ 0.03; and the run found a Windows-only bug in my A/B runner (the failure-injection preload must be passed to
node --importas afile://URL), fixed in@sbissoli/mcp-evals0.2.1.Raw traces, per-cell summaries and the generated report are in the repo: github.com/SidneyBissoli/bcb-br-mc...
Thanks — that distinction cost me the most, so I'm glad it landed.
Partly yes, partly not yet. On failure the agent already gets the diagnostics: how many attempts were made, the per-attempt budget, the fact that an existing series answers in under 0.5 s while a missing one sits ~30 s before the HTML page, and what to do next (verify the code with the search tool, or retry if the source is down). On success it gets the real extraction timestamp and a served_from_cache flag, but not the retry count or the source's response shape — a success after two HTML pages looks the same as a first-try success. Exposing that in the provenance block is cheap, since the extraction collector already sits on the single network call site.
Measured in long agent runs: no. What I have is a single-turn tool-selection eval (44 prompts, real model) and per-call telemetry that classifies errors by class. I don't have an A/B of "with diagnostics" vs "without" across a multi-step session, and I'd rather say that than guess. The one thing I did observe is the failure mode before the fix: a false "series does not exist" verdict made the agent state it as fact; the current message makes it verify or retry instead. That's an anecdote, not a measurement — a fair thing to measure next.
Follow-up on the "partly yes, partly not yet" — the first half shipped.
Shape of the origin's response + attempt count, on success. Every tool response
of
bcb-br-mcp1.15.0 now carries aretrievalfield inside the provenance block(provenance contract v1.1, shared by my servers):
requestsis how many trips to the origin composed the answer (a 10-year dailywindow is sliced into several),
attemptscounts retries too,anomaliessays whatwas overcome and how many times (
timeout,network,http_4xx,http_5xx,rate_limited,malformed_body— the last one is the "HTTP 200 with an HTML errorpage" case from the article), and
unstableis derived:attempts > requestsor anyanomaly. Two design rules I ended up caring about more than I expected:
null, not{1, 1}.nullmeans "not measured", not "it was clean".The retry/timeout/budget/counting logic moved into a small shared package
(
@sbissoli/mcp-upstream) so the other servers get the same field with the samename and shape instead of seven local reimplementations. Side effect worth naming:
while wiring it, I found that HTTP 429 on this server was being treated as a
deterministic 4xx and never retried. It now retries honouring
Retry-After.Your second question — does it reduce bad calls in long runs? Still "not yet".
The field is now emitted and honest, but I have not measured its effect on agent
behaviour over long sessions. That needs an A/B runner (same task set, with and
without the diagnostic visible to the model) and it is the next item on the list.
I'll come back with a number, not a feeling.
The HTTP 200 with an HTML error page problem is the real one here. A gateway sitting in front of an MCP server has the same failure mode with model calls, a provider returns 200 with a garbage or truncated body and looks healthy from the outside. Classifying by body shape instead of status code is the right instinct, curious if you found any pattern that predicts when the flaky 200 shows up, or is it fully random per call.
Not fully random, as it turns out — but I only learned that by measuring, and the first version of the server got it wrong.
Three populations produce that same HTML page with a 200, and two of them are predictable:
The useful consequence: the two deterministic cases can be separated from the flaky one before spending a retry. A request for the last N observations has no reason to take 30 s (real answers come in 0.18–0.41 s), so it gets a 6 s budget of its own; whatever hasn't answered by then is either case 1 or case 3, and repetition tells them apart — a missing code fails every attempt, a real series comes back on the next one in a few hundred ms. The first version treated the first HTML page as a verdict ("series does not exist") and would happily say the Selic target didn't exist whenever the origin hiccuped. That was the bug worth writing about.
Your gateway analogy holds, I think, with one addition: body-shape classification tells you that this attempt failed, but the shape of the request tells you which failure you're looking at. Both together are what let you retry the right thing and refuse the rest fast.