Originally published on Medium.
I maintain bcb-br-mcp, an open-source MCP server that gives AI agents the Brazilian Central Bank's public data: the SGS time series (Selic, IPCA, exchange rates, GDP, credit), the Focus market-expectations survey and the PTAX reference rates. It runs over stdio from npm and over Streamable HTTP on Cloudflare Workers, has 17 tools, 388 tests, and serves a couple of thousand npm downloads a month plus a hosted endpoint.
None of the lessons below are about the MCP protocol itself. The protocol was the easy part. The hard parts were the data source, the two transports, and the difference between a response that looks right and one that is right. Here they are, roughly in the order they cost me.
1. Your source will lie to you with HTTP 200
The single most expensive class of bug: the upstream API answers a bad request with a success code.
The SGS endpoint, asked for a series code that does not exist, returns HTTP 200 with an HTML page ("invalid request"), after about 30 seconds. A valid series answers in 0.2 to 0.4 seconds. So the naive server waited half a minute and then tried to parse HTML as JSON, and the error message blamed the source for being down.
Worse: the same HTML page occasionally comes back for a series that does exist. I measured it on series 432, the Selic target: 3 HTML pages out of 9 calls within a few minutes, then 20 consecutive good JSON answers. My first fix treated the page as a verdict ("this series does not exist") and raised without retrying. That meant a hiccup at the source made the server assert that the Selic target rate did not exist.
The page proves only that that attempt brought no data. The server now resolves the suspicion by repetition: a code that does not exist fails every attempt; a good series comes back on the next one. And small requests (ultimos/N, at most 20 observations) get their own 6-second budget, because they can never be a legitimate long query — which cuts the 30-second silence without touching the long windows.
Lesson: classify errors by the shape of the body, not by the status code, and never let a single anomalous response become a permanent verdict.
2. "Last minus first" is wrong for half of the interesting series
Price indexes — IPCA, IGP-M, INPC — are published as monthly rates, not levels. Compute the variation of IPCA over 2024 as (last − first) / first and you get +23.81%. The real accumulated inflation was 4.83%. For the IGP-M in 2023 the naive formula gave +252.38% for an index that actually fell 3.18%.
Nobody reported this. A paid evaluation run, where a real model picked tools for 44 Portuguese prompts, caught it by accident, because one answer was absurd enough to notice.
The fix is a per-series decision (level vs chained rate vs already accumulated), and it is partial by construction: some series are recognised by the unit the open-data portal publishes, some by their curated name, some by an explicit list of period rates. The index heads (433, 189…) have no unit at all in the source; only the name identifies them. A code outside the catalogue is treated as a level, and the contract says so.
Lesson: a numeric tool has domain semantics. If your server computes anything, the computation needs to know what kind of number it is holding, and the response must say which convention it used.
3. Ordering is not a property of the endpoint
ultimos/N — "the last N observations" — returns 22 of the 169 curated series newest-first and the other 147 oldest-first. The direction is not deducible from the code or the family: series 4390 comes reversed, series 433 does not. Meanwhile, the date-window endpoint returned ascending order in 151 of 151 series.
The visible symptom was a variation published with the wrong sign: −7.40% for a period in which the series rose 8.00%.
Every observation that enters the server now passes through one sort-by-date function. Nothing else reads data[0] as "the oldest".
Lesson: measure invariants you assume, across the whole catalogue, not on the three series you tested with.
4. Two transports, one registry — or you will ship two products
At the start the server had two independent surfaces: the stdio entry derived JSON Schemas from zod through the SDK, and the Cloudflare Worker re-implemented JSON-RPC by hand with its own copy of the schemas. They had genuinely diverged:
| stdio (npm) | hosted (Worker) | |
|---|---|---|
| version in production | 1.3.5 | 1.3.1 |
| tool description length | ~1,200 chars | ~100 chars |
default, minimum, maximum, minItems, additionalProperties
|
present | missing |
| resource names | one set | a different set |
The agent-readable descriptions I had spent a release writing never reached the hosted endpoint, because there was no deploy pipeline for the Worker. And the HTTP client got a weaker contract than the stdio client, because the JSON Schema was maintained twice.
The fix was structural: one registerAll(server) function is the only place that projects tools, resources and prompts onto an McpServer; stdio calls it once, the Worker calls it per request. The JSON Schemas are the published surface and go on the wire verbatim. A normalised dump of tools/list + resources + prompts is committed as a baseline, and the CI compares the surface before and after every change; the hosted endpoint is deployed on every push, with a production smoke test at the end.
Lesson: if two entry points can describe the same tool differently, one of them is lying to some client right now. Make a single registry and test that the two transports are byte-identical.
5. The runtime will veto your validator
The obvious JSON Schema validator for TypeScript is ajv. Ajv compiles schemas with new Function. Cloudflare Workers forbid new Function — and the failure mode is not a nice error on the offending tool; it is HTTP 500 on the whole /mcp route.
The server uses a non-compiling validator on both runtimes on purpose, so that stdio and hosted validate identically.
Lesson: pick the validator for the most restrictive runtime you deploy to, and use the same one everywhere.
6. A field that can be null must say so, or a validating client rejects everything
The tools return null deliberately where the source does not publish a field — null is the information that there is no data, and the normalisation never omits keys. But an outputSchema that declares "type": "string" while the server emits null violates the spec, and a client that validates (the MCP Inspector does) rejects the entire response, not the one field.
Runtime output validation in the SDK is permissive by design, so nothing at runtime catches this. I added a test that validates the structuredContent of every tool against its own announced outputSchema, with the same validator the server uses on input, and made the null-producing paths the test cases.
Closely related SDK rule: any tool that declares outputSchema must return structuredContent on every success; the check runs before any validator and cannot be disabled. All handlers go through one structuredResult() helper, so this can only be forgotten in one place.
Lesson: the output schema is a promise to a stranger's validator. Test the promise in CI; do not rely on the runtime.
7. Provenance is a contract, and the timestamp is the hard part
Every successful response carries a provenance block: source, series, period, the canonical URL that reproduces the query, the licence (ODbL), and retrieved_at.
The tempting implementation of retrieved_at is new Date() at response time. That is a false statement whenever the answer came from a cache: a search served from the 24-hour metadata cache reports the instant of the original fetch, which may be yesterday — and that is the date with legal weight, because it is when the extraction actually happened. The server collects the real instant at the single network call site through AsyncLocalStorage, aggregates by taking the oldest instant across the accesses that built the answer, and sets served_from_cache only when everything came from cache.
The second subtlety: the same channel that carries the source's numbers verbatim must mark anything the server computed — a variation, a correlation, a deflated series — as derived, with the convention used. A provenance block can match the schema and still lie; there is a dedicated test for the block's truthfulness, with a coverage assertion so that a new tool without a provenance test fails the suite.
Lesson: provenance is not metadata decoration. It is a claim about when and from where each number came, and it needs its own tests.
8. Latency is queue depth, not request count
Daily series are capped at 10 years per request (HTTP 406 above that), and a legal 10-year window takes 10–20 seconds at the source, with a server-side cut around 30 seconds that returns — you guessed it — 200 with HTML. The Worker has a 10-second budget. So the server slices windows into 3-year chunks (~2.6 s each).
Then a comparison of two daily series over 10 years still took 10.7 s, and five series took 10.4 s: the problem was not the number of requests but the depth of the queue per series. Splitting a budget of 10 concurrent requests across the series brought the worst case to ~8.8 s.
One measurement rule saved me from fooling myself here: always measure on a window never requested before. The source serves repeats from cache, and an already-requested window measures 600 ms where the cold one measures 10 s.
9. Hand-curated catalogues rot; verify them against the source
The server ships a curated catalogue of 135 series so that agents can find "IPCA" or "household credit" without knowing a code. When I finally verified every entry against the source, about half of the earlier errors had come from names edited by hand: two series swapped, individuals and companies inverted in a pair of credit series, a block of household-debt series sold as Focus expectations.
Each entry now records where its name came from — transcribed from the portal's dataset (82) or measured (53: no dataset anywhere, name inherited, only periodicity and magnitude checked) — and there is a rule not to "improve" a transcribed name by hand.
The obvious shortcut — validate a code against the open-data catalogue before hitting the network — is wrong too: the catalogue lists 4,261 datasets but 10 of 20 well-known valid codes are absent from it, including 433 (IPCA). Pre-validating would have refused good series.
Lesson: a catalogue is a claim about the source. Store the evidence for each claim, and never let a secondary index veto the primary one.
10. Telemetry that counts by status code is blind to the errors you most need
The Worker keeps usage counters. Reconciliation between the tool-call hook and the HTTP layer was by status: if (status < 400) continue, on the assumption that 200 implies the hook recorded the call. But a schema refusal is answered by the validator before the handler, inside an HTTP 200 — so nobody recorded it and the continue closed the only other door. Calls with malformed arguments were counted neither as calls nor as errors. Measured in production: schema refusal, unknown tool (−32602) and unknown method (−32601) all arrive as HTTP 200.
The fix reconciles by name against the hook's receipt and reads the outcome from the JSON-RPC envelope matched by id. The same defect existed in a sibling server, where it was fixed first.
Lesson: in MCP, protocol-level failures live inside 200s. Instrument the envelope, not the transport.
11. Directory listings drift too
Every directory that lists your server keeps its own copy of your manifest. Mine on one marketplace sat at 1.11.0 while npm was at 1.14.1, with descriptions and schemas from before a vocabulary change — because the manifest file in the repo was not tied to anything. Now a test compares the committed manifest with the live tools/list, resources/list and prompts/list of the real server and with the identity in package.json; a stale manifest fails before a release can happen.
What I would do first, next time
- One registry, two transports, a committed surface baseline. Everything else is cheaper once the surface cannot fork.
- Characterisation tests with mocked fetch for every tool, value by value, before any refactor — they were the gate for replacing the statistics engine, and produced exactly two explained differences.
- An output-contract test (structured output vs announced schema) and a provenance test with coverage assertions.
- A paid eval with a real model, run rarely and on purpose. It found the one bug that no unit test could have, because the bug was in what the number meant.
- Measure the source across the whole catalogue — ordering, caps, timeouts, error shapes — and write the numbers into the repo where the next maintainer (often me, six weeks later) will read them before "optimising" something.
The code is MIT, the data is ODbL from the Banco Central do Brasil, and the repository's CLAUDE.md holds the full list of the sharp edges above with the dates they were measured: https://github.com/SidneyBissoli/bcb-br-mcp
Top comments (33)
Lesson 6 hit home this morning. Our MCP server (worklore) had 79 green tests, and get_story still returned isError on every call in production: a DynamoDB Decimal reached json.dumps, and none of the tests went through tools/call. The fix was the client-shaped harness @arhancanli describes: initialize → tools/list → validate each result against the outputSchema it was just handed. It failed on the first run.
One question on #11: do you also run the manifest-vs-live check against the registry's copy, or only the committed one? That's where we've seen drift.
Thanks for writing the failures down in this much detail. It's rarer than it should be.
Glad lesson 6 got there before it cost you more than a morning. A Decimal that only breaks at json.dumps is the perfect example: no unit test goes near the serialiser, and the client-shaped run hits it on the first call.
On #11: only the committed one, and you've put your finger on the gap. The test pins the file in the repo to the live tools/list/resources/list/prompts/list and to the package identity, so a stale manifest can't be released. But the directory's copy only changes when a publish step runs, and that step is part of my release checklist, not something a test checks. So the drift I described (a listing several minors behind npm) is now prevented by process rather than detected by a measurement. That's the weaker of the two, and the last mile the test doesn't cover. The official MCP Registry is in slightly better shape, because its server.json is published by the same CI job that publishes to npm, but nothing reads it back either.
The fix is the same pattern one step further out: fetch the listing's current copy and diff it against the live server on a schedule, not only at release. Registries don't all expose the copy the same way, which is probably why I stopped at the repo. Your drift report is a good reason not to.
Thank you, this is a generous answer. After reading it I checked my own listing: the registry copy and the live worklore server both say 0.1.0, so a version diff passes cleanly. But the server has changed quite a bit since it was listed: tools/list now answers without a token, and the auth rules moved into its instructions. I never bumped the version. So the drift can hide one level deeper: the version matches, while what the server actually does has moved on. Your scheduled diff is going on my list; I'd just compare more than the version.
That's a sharper version of the drift, and I think you're right that the version is the weakest thing to diff on.
I checked what each copy can actually be compared on. The LobeHub manifest carries the whole surface, so there my test compares tools/list, resources/list and prompts/list field by field, descriptions and schemas included, not the version. The official MCP Registry's server.json carries no surface at all: name, version, packages, remotes. Against that copy the version is the only thing there is to diff, which turns your case around: the check is only as good as the habit of bumping the version whenever the surface moves. Nothing in my pipeline enforces that either; the version goes up because I run the release step, not because a test noticed the surface changed.
Your example also finds a gap in mine: both of your changes live outside tools/list. Instructions come back in initialize, which my manifest test doesn't read, and "answers without a token" is behaviour rather than declaration, so no listing would show it. The cheap fix I can see is to hash the initialize result together with the three lists, commit the hash, and fail CI when it changes without a version bump. That would turn "I never bumped the version" into a red build instead of a silent drift.
I built it. The test hashes the initialize result together with tools/list, resources/list and prompts/list, plus which methods answer without a token. It commits the hash next to the version, and the deploy now refuses to run while it's red.
Then I recorded the surface exactly as the server was when I listed it on 09-15 and ran it against today's code. It failed, and on something worse than I'd described: the registry said 0.1.0 for a server with four read-only tools, and the server behind that number now has six, including two that write on the user's behalf. Bumped to 0.2.0, deployed, and the registry shows it now.
Thank you, that's one comment thread that turned into a real fix.
That replay is the part I'd have skipped, and it's the part that found the real problem. A version diff would have passed forever: same number, but four read-only tools had quietly become six, two of them writing on the user's behalf. That's the case a registry listing exists to rule out.
Both of your additions are going into my own version, which isn't built yet. Hashing which methods answer without a token covers the gap I said no listing could reach. And making the deploy refuse to run, instead of only failing CI, is the right place for the gate. I'll also borrow the replay: record the surface each published version had and run it against today's server, so the first run checks the whole history rather than just the next release.
Follow-up, since I said I'd build it: it's live now on all seven of my servers.
Same shape as yours. A lock file next to the version holds a hash of the initialize result (instructions and capabilities included) plus tools/list, resources/list and prompts/list. It also records which methods answer without a token, on every MCP route, with the API key unset and set. If the surface changes and the version doesn't, the test goes red. The script that rewrites the lock refuses to do it under the old version, and the deploy won't run. I added one step after the deploy: it queries the live endpoint and checks it against the lock. Green tests prove the code; that step proves what's actually being served.
The replay covered 142 published versions. In all seven, the live server matches the version it reports. It also found two things I didn't know: two old releases of one server that don't even start (a broken bundle nobody had noticed), and a minor release that removed six tools when they moved behind a licensing flag.
One thing your version may not have hit yet: the probe has to be checked before it's written, not after. In one server the backend sits behind an edge proxy, and my first test double answered result to every method, so it locked "prompts/list answers" for a server that has no prompts. The live check caught it. The sanity checks now run before anything is written.
It's packaged as @sbissoli/mcp-surface if it's useful to anyone. Thanks again, this thread changed how all seven ship.
Thank you for coming back with this, and for the package. Seven servers and 142 versions is a far better test of the idea than my one server was.
Your point about checking the probe before writing the lock landed. My lock is written from a test double too, so I checked it against the live server just now: the methods that answer without a token match what's locked (initialize, tools/list and ping answer; resources/list and prompts/list return 401). It matches today, but only because nothing has drifted yet — I have no step that would notice if it did. Your post-deploy check against the live endpoint is the piece I'm missing, and I'm adding it.
The two releases that don't start are the finding I keep thinking about. A version diff would have called them fine forever.
I also wrote up the whole MCP-server experience on Habr (in Russian), and this idea is in it with your name on it: habr.com/ru/articles/1089802/
Thanks again. This thread changed how mine ships too.
Thank you for the Habr write-up, and for putting my name next to the idea. That's very kind, and I'm glad it travelled further than this thread.
Your check against the live server is the same situation I was in before the post-deploy step: everything matched, but only because nothing had drifted yet. Nothing was standing guard. Once that step is in, "it matches today" becomes something the pipeline re-proves on every deploy, instead of something you happened to check once.
The two releases that don't start still bother me too. Nobody would have installed them on purpose, but a registry would have offered them as valid versions forever.
Thanks for this exchange. Both our servers ship better because of it.
Lesson 2 is the one that bites hardest in finance: the data and the code each assume a convention, and nothing checks them against each other. I hit the same thing with stock splits. The data stored split factors one way and the code applied them the other way, Apple's 2020 split turned a 124.81 adjusted close into 1,996.92, and every test passed, because no test ran real data through the real code. Your "measure invariants across the whole catalogue" is the fix I'd underline. On lesson 6: I'd also make the tests call tools/list first, the way a real client does, since some clients only validate results against the schemas they've listed. Tests that skip that step can pass while real clients fail.
The split example is a clean statement of the problem: two conventions, each correct on its own terms, and a 16× error that no test could see because the tests never put real data through the real code. The IPCA bug was the same shape. Every unit test used synthetic series where "last minus first" happened to be the right formula, and it took real data (and a model tripping over an absurd number) to show that the code held one convention and the source published another.
On lesson 6, fair point, and it made me go and check what my tests actually do. They cover both halves, but separately. One test connects a real client over an in-memory transport, checks that what tools/list announces is identical to the definitions in code, and calls every listed tool, but it only asserts that the call doesn't break. The output-contract test validates every result against its outputSchema, but it reads the schema from the definitions, not from what was listed. It holds by transitivity (listed = defined, result ⊨ defined), but transitivity is an argument, not a test, and it quietly assumes the client validates the way my validator does. The Inspector uses a different one. Collapsing the two into one client-shaped test (initialize → tools/list → tools/call → validate against the listed schema) is cheap, and it's on my list now.
"Transitivity is an argument, not a test" is exactly the gap.
Two small things that make the collapsed test worth its keep. First, validate with whatever the client you care most about uses rather than your own validator, since that's the assumption you just named; with the TypeScript SDK that means letting its client do the call and the validation, so the test fails the way a user's session would. Second, give it a negative control: a test that deliberately breaks one result (drop a required field, or add one the schema forbids) and asserts the client-shaped test goes red. A test that has never failed on a known-bad result hasn't shown yet that it can.
Both taken, and both are now part of the item. The first one corrects my own plan: I'd written "use a different validator, like the Inspector's", which just swaps one assumption for another. Letting the SDK client make the call and do the validation tests the thing users actually run. I still need to confirm that the client validates structured content by default on the SDK version I'm on. If it doesn't, that's worth knowing too.
The negative control is the part I'd have skipped, and it turns out to matter. On one of my servers every tool publishes a permissive output schema, so a broken result would pass the client-shaped test silently. The control is the only thing that would reveal the test is empty there. Thanks for pushing on this twice.
On your open question: in the TypeScript SDK (I checked 1.30.0 just now), Client.callTool does validate by default, but only for tools whose output schema it has cached, and it only caches them inside listTools(). So a test that calls a tool without listing first gets no validation at all and passes. That's the trap we fell into. Once the schema is cached it rejects two things: structuredContent that doesn't match the schema, and a result with no structuredContent at all from a tool that declares an outputSchema (unless isError is set).
That second check gives you a control that still bites on the server with permissive schemas: in a test build, have one tool drop its structuredContent. However loose the schema is, a client that really validates must throw on that call. If it doesn't, the test is empty.
Confirmed on the v2 SDK my servers run (client 2.0.0) — same code path. The validator comes from the cached tools/list response, and with nothing cached callTool returns whatever it got. I measured it on one real tool, tampering with the result between server and client:
v2 adds a third way that doesn't need the list: pass the tool definition to callTool (toolDefinition option) and it validates against that. Same two throws.
Then I checked my own suites. Every test that calls a tool without listing first asserts values, not the schema, and the schema tests list first and validate on their own. So nothing is empty today. But the change I'd planned — handing validation to the client — would have walked straight into this in any test that skipped the listing. It's now written into the item: list first (or pass the definition), and the negative control is the dropped structuredContent. That's also the answer for the server with permissive schemas: no field to remove there, but a missing structuredContent still makes the client throw.
Thanks — this saved me from a test that would have been green for the wrong reason.
Following up on this: it's done. The output-contract tests now drive the real server through the SDK's own Client — tools/list, then tools/call — so the result is rejected against the listed outputSchema, exactly where a user's session would fail, instead of against our internal definitions with a validator we picked.
Two things your suggestion surfaced along the way: the Client only validates once it has the schemas cached from tools/list (call callTool cold and it passes anything through — one of our servers had exactly that, with client-side validation silently off), and an in-memory transport doesn't serialize, so a key set to undefined survives in the test but vanishes on the wire. The harness now pins both, plus negative controls that break the result between server and client and require the call to fail.
It's a shared helper now, used by all seven servers: github.com/SidneyBissoli/mcp-br-co... — and the bcb test: github.com/SidneyBissoli/bcb-br-mc.... Thanks for the push.
This is one of the most honest "lessons learned" posts I've read on MCP servers — most writeups stop at the protocol, but you went after the actual failure surface: a source lying with HTTP 200, ordering that isn't deducible from the endpoint, and the numeric-semantics bug (last-minus-first on a rate-published index) that only a paid model eval caught. That last one especially — a bug in what the number means, invisible to unit tests — is the kind of thing that should scare anyone building tools for agents to reason over.
The two-transports-one-registry lesson hit closest to home for me. I build RAG/LLM systems and agent tooling myself, and the failure mode you describe — two entry points quietly diverging until one client gets a weaker contract than the other — is exactly the class of bug that's invisible until someone diffs the actual wire surface. Committing a normalized tools/list baseline and diffing it in CI is a pattern I'm going to steal outright.
The provenance section is the one I'd push on further: collecting the real fetch instant via AsyncLocalStorage at the single network call site, then taking the oldest timestamp across a cached answer's dependencies, is a genuinely elegant solution to a problem most people don't even realize they have (a cached answer silently misreporting when it was actually true). Have you run into cases where the "oldest instant" rule itself becomes wrong — e.g. a slowly-changing series where an old cached value is technically still accurate but the provenance block undersells its freshness?
Great writeup — bcb-br-mcp is going on my list of references for how to actually harden an MCP server past the demo stage.
Thank you — and steal the tools/list diff freely; it has caught more than any test I wrote by hand.
On the oldest-instant rule: I haven't found a case where it becomes wrong, because it answers a narrower question than freshness. retrieved_at is "when did the server last read this from the origin", which is a fact about the extraction, not a claim about the data. Whether an old cached value is "still accurate" is something the server can't know without asking the origin again — and for official statistics that's not a safe assumption, since series do get revised after publication. So if the block undersells freshness, it undersells it honestly: the fix is to revalidate, not to move the timestamp forward.
Freshness of the data lives in a different field: data_vintage is the period the numbers cover, and that stays correct regardless of when they were fetched. Keeping the two separate is what lets the oldest-instant rule stay conservative without lying.
Where it does get blunt is a mixed answer — part cached, part fetched just now. The block reports the oldest instant and served_from_cache: false, so it tells you the answer isn't fully cached but not which part is. Per-dependency timestamps would fix that; so far it hasn't been worth the extra weight in every response, but your question is a good nudge to write that trade-off down.
Following up on the provenance thread — I've been thinking about where our two areas actually overlap, and I think there's a concrete, buildable idea here rather than just a general "let's collaborate."
bcb-br-mcp is free and open, which is exactly right for adoption. But I've been working directly in x402 (HTTP 402 micropayments for AI agents — I built an agent using Coinbase AgentKit + a CDP-managed wallet that autonomously pays for API access over x402, settling USDC on Base). The pairing seems almost too natural: a rigorously engineered, high-trust financial-data MCP server is precisely the kind of source an autonomous agent economy would pay small amounts to query reliably — and your reliability work (the tools/list CI diff, the retrieved_at/data_vintage separation, the honest "blunt not wrong" framing) is exactly the trust signal that would justify a paid tier over the dozens of sketchier financial-data APIs out there.
Concretely, I'm picturing:
A paid x402 tier alongside the free one — the free MCP stays exactly as-is for discovery/goodwill, but a small x402-metered endpoint offers something the free tier doesn't: higher rate limits, bulk historical exports, or a provenance-audited "certified" response (with the retrieved_at/data_vintage fields as the actual value-add — most competing data APIs can't make that guarantee at all). I could build the x402 payment layer end-to-end; you already have the hard part (correct, trustworthy data) solved.
A RAG/agent layer on top — a natural-language interface for Brazilian macro data ("how has Selic moved relative to IPCA over the last 3 years") that uses bcb-br-mcp as its tool layer under the hood. This is closer to my daily work (RAG pipelines, LLM agent design) and could be sold as a lightweight SaaS to fintechs/analysts who don't want to write MCP client code themselves.
Multi-source expansion — longer-term, the same reliability patterns you built (ordering detection, provenance, CI-diffed surface) could extend to other countries' central bank/statistics APIs. I do data engineering work (Databricks/Spark-scale ETL) that could productionize ingestion at that scale if it ever made sense to go there.
Option 1 is the one I'd actually want to build first — it's the most directly monetizable, and it plays to both of our strengths without touching the free/open core you clearly care about preserving. Would you be open to talking through it, even just at the "does this make sense" stage?
Thanks for thinking this through concretely — I appreciate it. I'm going to pass on a paid tier, though, for reasons of principle rather than effort: the underlying BCB data is free and open, and I want provenance to be something every response carries, not a premium feature. A "certified" tier would also promise more than the server can honestly guarantee — as I wrote above, retrieved_at is a fact about the extraction, not a warranty on the data.
That said, the server is MIT-licensed, so if a natural-language layer on top of it (your option 2) makes sense for your work, you're completely free to build it — no permission needed. If you do, I'd be glad to hear how the provenance fields hold up in a RAG pipeline.
Lesson 2 has an exact twin in SEC XBRL data. A third-quarter 10-Q reports revenue for both the three months and the nine months under the same concept, and the fourth quarter usually isn't filed as its own number at all: it is the 10-K's annual figure minus the nine-month value. So "the latest revenue" can be a quarter, a year-to-date total or a full year depending on which fact you picked up, and the fix is yours: carry the period's start and end dates with every number, and say when a quarter was derived rather than reported.
The same data hides a second convention: restatements. A quarter comes back a year later as the comparative column in the next 10-Q, sometimes with a different value, so the server has to choose "latest restated" or "as originally reported" and say which. My server defaults to restated, exposes a flag for the original, and puts the period dates, form, filing date and accession number on every row, which is my version of your lesson 7.
Disclosure: I build Equibles, a financial-data MCP server for US companies, so these are scars rather than theory.
Thank you, Daniel — both of your conventions have exact twins in the SGS, and checking them was worth it.
Period. An SGS observation is only {date, value}: the date of a monthly or annual series is the first day of the period, and some series arrive already year-to-date or 12-month accumulated, which only their name says. The tool descriptions now say it. Computed values were already marked as derived, but your point found a hole: the tools that compute across several series were emitting data_vintage: null. Fixed in 1.17.0.
Restatements. The SGS keeps no as-first-published copy, so I can't offer your flag. What I can do is say it: every value is now declared as the version current at retrieved_at.
I ran the same two questions on the sibling IBGE server, and that's where it hurt. Five named indicators returned only .., nine mixed a monthly change with its own year-to-date and 12-month figures, and "GDP per capita" was reading municipal GDP. All fixed, with a live test that calls every indicator and requires a number.
Thanks again for sharing them.
Glad it was useful, and declaring every value as the version current at retrieved_at is the honest answer when the source keeps no first-published copy. The IBGE find is the scary one: "GDP per capita" quietly reading municipal GDP passes every schema check, and only a live call with a known answer catches it. Thanks for coming back with the results.
"The response must say which convention it used" is the lesson I learned from the other side. Our earnings metrics came back as the filing printed them: Oracle's revenue as "$19,345" because the table was in millions, Adobe's as "$6.76 billion" in the same response. Faithful, and useless to compare without guessing a multiplier. We ended up adding numeric, unit and scale next to the string, filled only from what the filing itself says, and scale stays null when it says nothing. In your A/B, did the agents actually read the convention field, or did they mostly take the number and move on?
"Faithful, and useless to compare" is exactly the trap, and scale: null when the filing says nothing is the right call. It's the same rule I ended up with for provenance: null means "not stated", never a guessed default.
On your question, honestly: the A/B can't answer it, because it didn't vary the convention field. The only thing toggled between arms was the retrieval diagnostics block; the convention metadata was present in both. Three of the twelve tasks did carry a convention trap (accumulated IPCA for a year, the same for IGP-M found by name, and a comparison mixing a rate series with level series), and every fixed-answer check passed in every cell. But that says more about where the computation lives than about what the model reads. The server accumulates the monthly rates itself and returns the right number, so a model that ignored the convention field entirely would still get it right.
That's the design choice underneath lesson 2, I think: when the server can apply the convention, it should apply it and then say so, rather than hand over a raw number plus a field and hope the model reads the field. Your case is harder, because you can't normalise scale when the filing is silent. There the field is the only defence, and whether agents read it is a real open question. A clean test would be the one you're implying: same tasks, convention field present vs. stripped, scored on whether the answer states the unit. I haven't run it.
That's the split I'd been missing: apply the convention when the server can and say it did, and keep the bare field only for the cases where it can't. Our script already leans that way. It prints the string and the multiplied number side by side, and leaves the number blank when scale is null, so the gap is at least visible to whoever reads the card.
Printing the string next to the multiplied number, and leaving the number blank when scale is null, is a nice way to keep the gap honest: whoever reads the card sees exactly where the convention stopped. Thanks for sharing how you handle it.
The distinction between a transient HTML response and a missing series is exactly where an MCP server becomes production infrastructure instead of a thin wrapper. I like the bounded retry idea for short queries, and I would also expose the source response shape and retry count to the agent so it can explain stale data instead of inventing certainty. Have you measured whether those diagnostics reduce bad tool calls in long agent runs?
Measured it. Here is the number, as promised.
96 agent sessions on bcb-br-mcp (Claude Sonnet 5 driving the server over stdio): 12 multi-step tasks with fixed answers, A/B with and without the
retrievalblock visible to the model, each at 0% and at 20% injected upstream failures (deterministic, at the fetch layer), 2 runs per cell. Total cost: US$ 2.63.Result on your exact question — does it reduce bad tool calls? No, and for a reason I did not expect. The count of bad calls (repeating a call after a definitive error, schema refusals, insisting on a failing key) was zero in both arms, at both failure levels. The server's bounded retry absorbs the injected failures before the model ever sees an error: what reaches the model is a valid result flagged
unstable: true, not a failure. And Sonnet 5 simply does not repeat or insist. The defence at the network edge zeroes the numerator the field was supposed to reduce.What the field does change is the reporting. When the origin did flicker, the model with
retrievalwarned the user about it in 3 of 6 judged sessions; without the field, in 1 of 12 (and that one was an empty result, not a flicker). Small sample, clear direction — it is exactly what the instruction attached to the field asks for ("say so when citing the number, and do not repeat the call").Two side notes: real sessions were 2–3 steps, not the ~20 I had budgeted for, so a session costs about US$ 0.03; and the run found a Windows-only bug in my A/B runner (the failure-injection preload must be passed to
node --importas afile://URL), fixed in@sbissoli/mcp-evals0.2.1.Raw traces, per-cell summaries and the generated report are in the repo: github.com/SidneyBissoli/bcb-br-mc...
Thanks — that distinction cost me the most, so I'm glad it landed.
Partly yes, partly not yet. On failure the agent already gets the diagnostics: how many attempts were made, the per-attempt budget, the fact that an existing series answers in under 0.5 s while a missing one sits ~30 s before the HTML page, and what to do next (verify the code with the search tool, or retry if the source is down). On success it gets the real extraction timestamp and a served_from_cache flag, but not the retry count or the source's response shape — a success after two HTML pages looks the same as a first-try success. Exposing that in the provenance block is cheap, since the extraction collector already sits on the single network call site.
Measured in long agent runs: no. What I have is a single-turn tool-selection eval (44 prompts, real model) and per-call telemetry that classifies errors by class. I don't have an A/B of "with diagnostics" vs "without" across a multi-step session, and I'd rather say that than guess. The one thing I did observe is the failure mode before the fix: a false "series does not exist" verdict made the agent state it as fact; the current message makes it verify or retry instead. That's an anecdote, not a measurement — a fair thing to measure next.
Follow-up on the "partly yes, partly not yet" — the first half shipped.
Shape of the origin's response + attempt count, on success. Every tool response
of
bcb-br-mcp1.15.0 now carries aretrievalfield inside the provenance block(provenance contract v1.1, shared by my servers):
requestsis how many trips to the origin composed the answer (a 10-year dailywindow is sliced into several),
attemptscounts retries too,anomaliessays whatwas overcome and how many times (
timeout,network,http_4xx,http_5xx,rate_limited,malformed_body— the last one is the "HTTP 200 with an HTML errorpage" case from the article), and
unstableis derived:attempts > requestsor anyanomaly. Two design rules I ended up caring about more than I expected:
null, not{1, 1}.nullmeans "not measured", not "it was clean".The retry/timeout/budget/counting logic moved into a small shared package
(
@sbissoli/mcp-upstream) so the other servers get the same field with the samename and shape instead of seven local reimplementations. Side effect worth naming:
while wiring it, I found that HTTP 429 on this server was being treated as a
deterministic 4xx and never retried. It now retries honouring
Retry-After.Your second question — does it reduce bad calls in long runs? Still "not yet".
The field is now emitted and honest, but I have not measured its effect on agent
behaviour over long sessions. That needs an A/B runner (same task set, with and
without the diagnostic visible to the model) and it is the next item on the list.
I'll come back with a number, not a feeling.
The HTTP 200 with an HTML error page problem is the real one here. A gateway sitting in front of an MCP server has the same failure mode with model calls, a provider returns 200 with a garbage or truncated body and looks healthy from the outside. Classifying by body shape instead of status code is the right instinct, curious if you found any pattern that predicts when the flaky 200 shows up, or is it fully random per call.
Not fully random, as it turns out — but I only learned that by measuring, and the first version of the server got it wrong.
Three populations produce that same HTML page with a 200, and two of them are predictable:
The useful consequence: the two deterministic cases can be separated from the flaky one before spending a retry. A request for the last N observations has no reason to take 30 s (real answers come in 0.18–0.41 s), so it gets a 6 s budget of its own; whatever hasn't answered by then is either case 1 or case 3, and repetition tells them apart — a missing code fails every attempt, a real series comes back on the next one in a few hundred ms. The first version treated the first HTML page as a verdict ("series does not exist") and would happily say the Selic target didn't exist whenever the origin hiccuped. That was the bug worth writing about.
Your gateway analogy holds, I think, with one addition: body-shape classification tells you that this attempt failed, but the shape of the request tells you which failure you're looking at. Both together are what let you retry the right thing and refuse the rest fast.