DEV Community: Hiroki Nomura

Funded key with limits, or approve each payment? Putting a spend policy between an AI agent and HTTP 402

Hiroki Nomura — Fri, 30 Oct 2026 06:36:46 +0000

Agents can pay for APIs now. With x402 and MPP a server answers 402 Payment Required with a price, the client signs a stablecoin authorization, retries, and gets the data. No API key, no invoice, no account. It works, and it is the part people demo.

The part nobody demos is who decides. Two patterns keep coming up:

A funded key with a cap. Fast. The model can also burn the whole cap in a retry loop and you find out from the balance.
Approve every payment. Safe. Nobody wants to approve two hundred $0.002 calls, so after a day the approval becomes a rubber stamp or the agent gets the key anyway.

Both put the money decision inside the agent's reach: in a prompt, in a tool argument, in an environment variable the model can read. I wanted the decision outside the model, in something a human writes once and can audit later. That became spendctl, a local sidecar. This post is about the design, including the one state that turned out to matter most.

The shape

The agent never holds a key. It sends its HTTP request through the sidecar instead: as a gateway URL, through HTTP_PROXY, with a wrapFetch in the SDK, or as an MCP pay tool. When the seller answers 402, the sidecar does three things:

Evaluates a policy file written by a human: per-agent budgets (hourly, daily, weekly, monthly), a per-request cap, allowlisted hosts, pinned recipients and networks, a first-payment cap for new recipients, a price-jump check, and a threshold above which a human has to approve.
Signs and pays with a wallet the agent cannot see, retries the request with the payment, and passes the response back.
Writes one ledger row per decision, with the reasons. Refusals come back as readable text the model can act on ("$0.40 would exceed the daily budget of $0.25 for agent researcher, $0.21 already spent"), not as a stack trace.

agents:
  researcher:
    budget: { perRequest: "0.05", daily: "0.25" }
    allow: ["api.example.com"]
    requireApprovalAbove: "0.02"

The model sees a budget, a refusal or an approval id. It never sees a private key, and it cannot change the policy.

Approvals that fit how agents run

A payment that crosses the threshold needs a human. Two modes:

Hold. The agent's request stays open until someone decides, up to four minutes by default. Good for CLIs and scripts.
Retry. The sidecar answers at once with an approval id. After a human approves, the agent retries with X-Spendctl-Approval: <id>. One approval pays for exactly one retry, and it is bound to what the human saw: recipient, network, asset, amount. If the seller's next 402 asks for anything else, the retry is refused and nothing is paid.

Humans approve from the CLI, a local dashboard, or a review link in a Slack or Discord webhook. Agents cannot approve; the admin token lives in a file the agent does not read.

The state nobody models

Someone on r/x402 asked the question that every payment system eventually has to answer: the payment settles, but the paid API times out before the agent gets the result. Does the ledger know the difference?

Most agent payment demos do not model this at all. A retry is a second payment. spendctl has an explicit state for it:

Once a signed payment has left the sidecar, the outcome is settled (the seller answered 2xx with a settlement receipt), failed (the seller rejected it), or unknown: the request timed out or errored after the payment was sent, or the seller returned an error after taking it, or the settlement header says "pending".
unknown counts against the budget until a human reconciles it. An unconfirmed payment is never free spend.
The sidecar never retries a sent payment. The agent gets a 502 with the ledger id and a sentence saying so. If the agent resends with the same Idempotency-Key inside the window, the sidecar refuses and points at the original row, so paying twice takes deliberate effort.
A human closes it with spendctl resolve <id> --settled --tx 0x… or --failed after checking the chain or the seller's dashboard.

This is not clever. It is the boring bookkeeping that is missing from most "let the agent pay" code, and it is where the money actually leaks.

Try it without installing anything

There is a hosted sandbox: a real sidecar running against mock paid APIs, exposed as a remote MCP server at https://mcp.spendctl.dev/mcp. Add it to Claude, Claude Code, Cursor or VS Code, then ask the model to create a sandbox, make a paid call, blow the budget, hold a payment for approval and approve it. Throwaway wallet, mock facilitator, no outbound network, nothing real. The tools are sandbox_create, sandbox_pay, check_budget, list_receipts, get_policy and approve.

Locally:

npx spendctl init && npx spendctl start   # sidecar on :8402
npx spendctl mock                          # mock facilitator + two paid APIs
npx spendctl fetch "http://localhost:4022/search?q=x402" --agent researcher

What it is not, yet

The sidecar is a commercial product with a free tier (two agents, a monthly cap on governed payments). The client SDK and the editor plugins are Apache-2.0 on GitHub.
Reconciliation of unknown is manual today. Asking the facilitator or the chain automatically is under consideration.
The sandbox cannot simulate a seller that settles and then times out; that path is only testable locally.
MCP clients cut tool calls at about 60 seconds, which is shorter than hold mode. For MCP agents, retry mode is the right setting today. A dedicated await_approval tool that long-polls under the client timeout is under consideration, after a good suggestion in the r/mcp thread.

If you are putting agents in front of paid APIs, I would like to hear which ledger fields you would want to see, and what your clients actually do when a payment is held. Docs are at spendctl.dev/docs; the sandbox needs no account.

We scored 1,274 MCP tool descriptions, then checked whether the scores predict what models do

Hiroki Nomura — Thu, 15 Oct 2026 13:00:00 +0000

We published the scores of 55 public MCP servers, 1,274 tools in all, at forecall.dev/en/servers. Each tool description is scored out of 100 on whether it says what the tool is for, when to use it, its parameters, what it returns, its constraints and an example.

Then we tested whether a low score means models pick the wrong tool. For one-step choices, it didn't: two frontier models picked the right tool 91–99% of the time, low scores or not. This post covers both: what the descriptions leave out, and what that does and doesn't tell you.

Where the 1,274 tools came from

46 servers are Smithery's most-used servers, copied from its registry API on October 7, 2026 (UTC). The registry keeps each tool's name, description and input schema, but drops output schemas, annotations and server instructions.
9 servers were taken from the running server with forecall dump: Notion, Playwright, Chrome DevTools, Context7, Filesystem, Memory, Firecrawl, DebugBase and Bhived.

All were scored with the same rules (Forecall's scoring rules v2). Each server's page says which source it came from and when. The pages show scores, issues and tool names, not the descriptions themselves. If you own one of these servers and want it taken down, email support@forecall.dev and it's removed within 7 days.

What the descriptions leave out

885 of 1,274 tools (69%) never say when to use them. The share is the same in both groups: 69% of the registry copies and 71% of the dumped tools.

87 of the 156 dumped tools (56%) don't say what they return, neither in an output schema nor in the description. (I'm leaving the registry copies out of this one, because the registry drops output schemas.)

18 of the 55 servers have at least one pair of near-identical descriptions, 129 pairs in all. Two servers account for most of them: Notion (57 pairs across 24 tools) and Paper Search (29 across 25).

Server averages range from 19.9 to 82, with a median of 44. Across all tools, 497 score under 40 and only 18 score 80 or more.

Does a low score mean models choose wrong? We measured it

Forecall also runs evaluations: it writes requests for each tool and asks models which tool they'd call. We used that to calibrate the static score on 4 servers with 88 tools (Memory, Notion, Playwright and Chrome DevTools), 8 one-step requests per tool, with Claude Sonnet 5.5 and GPT-6.1 Sol:

Server	Tools	Average score	Right tool (Sonnet / Sol)
Memory	9	43.8	97% / 99%
Notion	24	33.0	96% / 96%
Playwright	25	41.9	93% / 94%
Chrome DevTools	30	45.1	99% / 98%

The correlation between a tool's score and how often models picked it correctly was 0.04 (Spearman). Tools scoring under 40 were picked correctly 95.5% of the time; tools scoring 60–79, 100%.

We then put 25–55 tools from other servers next to each one, to make the choice harder. None of the requests that needed a tool went to the other servers' tools, and accuracy stayed at 91–98%.

So, for frontier models choosing in one step, a short description wasn't the problem: the tool names and schemas carried enough. The misses that did happen were mostly the models doing something sensible first. When a request didn't include a page number, they called list_pages before acting, in 77–87% of those requests in an earlier run. For browser_fill_form, they took a page snapshot first, because the field references come from it.

What the score is for, then

The score says what a description puts in words, not how a model will choose. We changed the site's wording to match: what we used to call "pairs easy to mix up" are now "near-identical descriptions", and every result says what static scoring can't tell.

What's missing from the text may still matter where this test doesn't reach: when the user doesn't have the ID a tool needs, over several steps, for smaller models, and for the people reading your docs. Just don't assume a longer description is better. In an earlier test, rewrites that raised scores (one tool went from 31 to 74) made two models worse (details here). Check rewrites against the models your users run.

Check your own server

Look it up: forecall.dev/en/servers
Not listed? If it's a public Streamable HTTP server that needs no sign-in, paste its URL at forecall.dev/en/lint.
Locally, with nothing sent anywhere: npx forecall dump -o tools.json -- <command that starts your server>, then npx forecall lint tools.json. The CLI also reads endpoint lists and search results.
To see how models actually pick your tools, Forecall's evaluations (Pro and Team plans) report the right-tool rate, look-ups first, argument validity and false calls per model.

I'm the author of Forecall. The linter is open source: github.com/forecall/forecall-cli.

Recording what the Firestore emulator sends to Cloud Functions, and replaying it in Rust

Hiroki Nomura — Tue, 13 Oct 2026 00:48:32 +0000

hidane is a reimplementation of the Cloud Firestore emulator as a single Rust binary: no JDK, the same gRPC, REST and WebChannel wire protocols, and it runs under firebase-tools with hidane exec -- firebase emulators:start. Its rule is that every claim of compatibility has a test behind it: scripts record how the official emulator answers, and hidane's test suite replays every recording.

Until now, one thing was missing under emulators:start: Firestore-triggered Cloud Functions. Since 0.2.3 hidane accepted the Functions emulator's trigger registrations, so functions loaded, but no event ever reached them. 0.4.0 delivers those events. This post is about how.

How the two emulators talk

When you run firebase emulators:start with Firestore and Functions, firebase-tools starts the Firestore emulator with --functions_emulator <host:port>. The Functions emulator then registers each Firestore trigger with the Firestore emulator. After a commit, the Firestore emulator sends an event for each document the commit changed to every trigger that watches it, as POST /functions/projects/{p}/triggers/{key} on the Functions emulator.

None of this is a documented API. It is a conversation between two local processes, and the only reliable description of it is what the official emulator actually sends.

Recording it from the outside

hidane treats the official emulator as a black box: its jar is never decompiled, and everything below was observed on the wire (docs/clean-room describes the policy).

tools/oracle/function_events.py starts the official emulator (v1.22.0, the version firebase-tools pins) three times, under an English locale:

with --functions_emulator pointing at a small server of the script's own, which records every request it receives and answers 200, or 500, 404 or a late answer depending on the trigger's key;
without the flag;
with the flag pointing at a port nothing listens on.

Against each, it registers triggers the way firebase-tools does, writes over REST, gRPC Commit and the Write stream, and records for every step: the answer, every request the Functions emulator received (in order, every header, the body as sent), and the WARNING lines the emulator printed.

What it sends

v1 triggers (providers/cloud.firestore/eventTypes/document.*) get a compact JSON body:

{"data":{"value":…,"oldValue":…,"updateMask":{}},
 "context":{"eventId":…,"timestamp":…,"eventType":…,
            "resource":{"name":…,"service":"firestore.googleapis.com"}}}

A few details only a recording shows: the v1 triggers of one change share the same eventId; updateMask is always empty; the timestamp is the commit time; and v1 triggers only ever see the (default) database.

v2 triggers (google.cloud.firestore.document.v1.*, including withAuthContext) get a binary-mode CloudEvent. The attributes travel as ce-* headers (ce-type, ce-source, ce-id, ce-subject, ce-time, ce-database, ce-document, and so on), and the body is a google.events.cloud.firestore.v1.DocumentEventData protobuf, base64-encoded, which firebase-tools decodes. For withAuthContext, the emulator sends ce-authtype: unknown and the same placeholder ce-authid whoever wrote.

The order: a Java HashMap

Ordering looked simple at first: all v1 events of a commit, then the v2 ones, document by document in the order the writes name them. But when several triggers watch the same document, they did not come in registration order, nor alphabetically.

They come in the iteration order of a Java HashMap sized for the project's triggers of that version. The model was fitted on 116 observations, and hidane reproduces it in presized_hash_order:

pub fn presized_hash_order<S: AsRef<str>>(keys: &[S]) -> Vec<usize> {
    // `(int) Math.ceil(n / 0.75)`, then `tableSizeFor`.
    let wanted = (keys.len() * 4).div_ceil(3).max(1);
    let capacity = u32::try_from(wanted.next_power_of_two()).unwrap_or(1 << 30);
    let mut ordered: Vec<(usize, u32)> = keys
        .iter()
        .enumerate()
        .map(|(i, key)| (i, spread(string_hash(key.as_ref())) & (capacity - 1)))
        .collect();
    ordered.sort_by_key(|&(i, bucket)| (bucket, i));
    ordered.into_iter().map(|(i, _)| i).collect()
}

string_hash is Java's String.hashCode (base 31 over UTF-16 code units), spread folds the high half into the low one, and the table is the smallest power of two that holds n / 0.75 entries. Keys come out by bucket, and by insertion within a bucket. Functions that depend on the order in which their siblings fire are fragile, but a test suite that passes on the official emulator should not start failing because the order changed.

Timing and failures

Events go out after the commit, from one background queue, so a write never waits for a function. Each event waits for the answer to the previous one, for up to 20 seconds. hidane does the same, one event at a time from a task of its own.

When delivery fails (an error status, a refused connection, a timeout), the official emulator logs a WARNING and does not retry, for example Unsuccessful HTTP response when publishing event for … or Failed to publish event to {host}: Connection refused. Without --functions_emulator, every event is a WARNING that the Functions emulator host/port is not configured. hidane prints the same wording.

Batches, transactions, :batchWrite, gRPC Commit, the Write stream, deleted paths and cleared databases send events; imports and POST /reset do not.

Replaying it

crates/hidane/tests/function_events.rs replays the 133 steps of the three runs against hidane and compares the status of each answer, every request the Functions emulator receives (v1 bodies byte for byte, v2 bodies decoded, with times and IDs compared as labels), and every WARNING.

Recording also turned up something unrelated: Java prints the smallest double as 4.9E-324, and hidane printed 5.0E-324, over REST and WebChannel too. That is fixed now.

What hidane does not copy

Matching the official emulator is the goal, but not its bugs. Each of these is listed in docs/parity-exceptions.md:

A v2 event's ce-source (a function's event.source) names the project twice: //firestore.googleapis.com/projects/projects/{p}/databases/{d}. hidane names it once, as production does.
A write that changes nothing sends v2 updated and written events, and deleting a missing document sends v2 created and written; v1 triggers get neither. hidane sends no event for either, as production does.
POST /reset makes the official emulator forget every registered trigger, so functions stop firing, silently, until they reload. hidane keeps them.
A v2 trigger registered without a database makes every write to the project answer 500. hidane has it watch (default).

End to end

Recordings check the wire; the last check is whether functions actually receive the same data. tools/oracle/functions/ is a functions project (firebase-functions 6.6.0) with v1 onCreate / onUpdate / onDelete / onWrite, v2 onDocumentCreated / Updated / Deleted / Written, onDocumentWrittenWithAuthContext, a subcollection and a named database. It runs under firebase emulators:exec --only functions,firestore (firebase-tools 15.33.0), once with the official emulator and once under hidane exec.

The same 37 events reach the same functions with the same data. The official run has 6 more, the v2 events for the no-op write and the missing delete, and projects/projects/ in event.source; both transcripts are in results/functions-triggers-*.json.

Also in 0.4.0

A commit's problems are reported in the official emulator's order.
The Go, Python and Java server SDKs are verified against both emulators: documents, queries, aggregations, vector search, transactions and listeners.
A weekly CI job starts the official emulator, re-records a set of recordings and fails if one differs from the committed fixture.
The parity table now counts 20,224 recorded answers that the test suite replays.

Try it

curl -fsSL https://hidane.dev/install.sh | sh   # or Homebrew, npm, cargo, pub, Docker
hidane exec -- firebase emulators:start

Other ways to install:

brew install hidane-dev/tap/hidane
npm install --save-dev hidane
cargo binstall hidane
docker run --rm -p 8080:8080 ghcr.io/hidane-dev/hidane

Website: https://hidane.dev
GitHub: https://github.com/hidane-dev/hidane
Release notes: https://github.com/hidane-dev/hidane/releases/tag/v0.4.0

hidane is also listed on DevHunt this week: https://devhunt.org/tool/hidane

If your tests get a different answer from hidane than from the official emulator, please open an issue with the Parity gap template. That is the most useful feedback.

Unofficial; not affiliated with Google. Firebase and Cloud Firestore are trademarks of Google LLC.

I scored the tool descriptions of Chrome DevTools MCP and Playwright MCP. Both rarely say when to use a tool

Hiroki Nomura — Mon, 12 Oct 2026 13:00:00 +0000

I scored the tool descriptions of two browser-automation MCP servers: Google's Chrome DevTools MCP and Microsoft's Playwright MCP. They averaged 45.1 and 41.9 out of 100. In Chrome DevTools MCP, 24 of 30 tools never say when to use them; in Playwright MCP, 23 of 25. And 27 and 21 tools respectively don't say what they return.

The scoring was done with Forecall, a static linter for MCP tool descriptions that I build. This post shows the results, what separates the low-scoring descriptions from the high-scoring ones, what happened when I added three sentences to one of them, and what real models then did with both versions.

An agent picks a tool from its name, description and input schema, and nothing else

An AI agent reads the list of tools an MCP server returns (the result of tools/list) and decides which tool to call with which arguments. All it has to go on are three things: the tool's name, its description and its input schema.

If the description only restates the name, the name and the schema are all the model has, and nothing tells it what a call returns until it makes one. Whether that is enough can only be measured with real models; I did that later in this post.

How I scored them: get tools/list, then score six parts out of 100

On October 7, 2026, I started these two versions locally and saved their tools/list results as JSON. No tools were called.

Server	Package and version	Tools
Chrome DevTools MCP	`chrome-devtools-mcp@1.10.1`	30
Playwright MCP	`@playwright/mcp@0.0.83`	25

I scored the JSON with Forecall's CLI (npx forecall lint). It scores each tool on six parts:

Part	Points	What it looks for
Purpose	20	What the tool does, beyond restating its name
When to use	20	When to use it, and how to choose between similar tools
Arguments	25	Each argument described, with its type and whether it's required
Return value	15	What comes back, in an output schema or the description
Constraints and side effects	10	Preconditions and side effects
Examples	10	An example of a call

The scoring is rule-based and reads only the text of the descriptions and schemas. It doesn't measure whether a real model picks the right tool; I come back to that limit at the end. (I re-scored both servers on October 10 with the current rules, v2, and got the same numbers.)

Results: both average in the 40s, and they're weak in the same places

	Chrome DevTools MCP	Playwright MCP
Average score	45.1	41.9
No "when to use"	24 / 30	23 / 25
No "what it returns"	27 / 30	21 / 25
Description restates the name	6	14
Pairs with near-identical descriptions	0	0

In both, most of the lost points come from "when to use" and "return value". Both describe their arguments well, and neither has a pair of tools with near-identical descriptions; their tool names are clearly distinct.

Where they differ is how many descriptions only restate the name: 14 of Playwright MCP's 25 tools get that finding.

The low scorers end with one sentence that restates the name

The lowest scores were 31 in Chrome DevTools MCP (drag, hover, upload_file) and 33 in Playwright MCP (browser_handle_dialog, browser_navigate, browser_type and others). Their descriptions:

Tool	Description
drag	Drag an element onto another element
hover	Hover over the provided element
browser_handle_dialog	Handle a dialog
browser_navigate	Navigate to a URL

They don't say when to call the tool, what's needed before calling it, or what comes back. browser_handle_dialog, for example, only makes sense while a dialog is open, but nothing in its description says so. That is what the linter measures: text that is missing. It doesn't show that a model will choose wrong, and for one-step choices, the test further down found that it doesn't.

The high scorers say when to use the tool, with an example

The same servers also have tools that score well. Chrome DevTools MCP's best is fill_form, at 72:

Fill out multiple form elements (inputs, selects, checkboxes, radios) at once. ALWAYS prefer this tool over multiple individual 'fill' or 'click' calls when interacting with forms. It is significantly faster, more reliable, and reduces turn count. Example: Fill username, password, and check "Remember Me" in one call.

It says how to choose between it and fill or click, and gives a concrete example. The only thing missing is the return value.

Playwright MCP's best is browser_find, at 66. In one paragraph it explains what it searches for, what it returns, and why that's cheaper than taking a full snapshot.

Three sentences took drag's score from 31 to 74

I rewrote drag's description using only facts from its schema. I added three things: when to use it (and how it differs from upload_file), what to do before calling it, and what it returns.

Drag one element and drop it onto another inside the page. Use it for drag-and-drop UIs such as sortable lists, kanban boards and sliders; to send files from the computer, use upload_file instead. Take a snapshot first and pass the uids of both elements as from_uid and to_uid. Returns a snapshot of the page after the drop when includeSnapshot is true.

Part	Original	Rewritten
Purpose	6	17
When to use	0	20
Arguments	18	18
Return value	0	7
Constraints and side effects	7	7
Examples	0	5
Total	31	74

That's 43 points. The sentence on when to use it did the most: "when to use" went from 0 to the full 20.

The rewrite also produced a new finding: the tool may be destructive but has no destructiveHint. It reacted to the word "drop". drag doesn't destroy anything on the page, so this finding is wrong. Setting destructiveHint: false in the tool's annotations makes it go away (the score stays at 74), and it also tells clients that the tool isn't destructive.

A higher score, though, is not the same as better picks. The next section checks that.

Then I tested real models: the short text was enough, and my rewrite made two models worse

A Chrome DevTools MCP maintainer asked for a concrete case where an agent is confused by the short descriptions. So I tried to produce one. A stand-in server returned chrome-devtools-mcp 1.10.1's tools/list, either as it is or with drag, hover and upload_file rewritten, and models got eight one-step tasks whose right tool is one of those three: a kanban card, a hover menu, a tooltip, an "Add photos" button, an avatar area that takes a dropped image, and so on.

With the descriptions as they are, no model picked a wrong tool: Claude Code with Fable 5.1, Sonnet 5.5 and Haiku 4.5 (24 of 24), Codex CLI with gpt-6-astra (11 of 11), and one-turn API calls to Claude Opus 5.5, Claude Sonnet 5.5, gpt-6-astra and gpt-6.1-sol (32 of 32). The only wrong picks, click instead of hover from two smaller Codex models, came from a task whose text was ambiguous.
My rewrites made two models worse. gpt-5.6-luna read "Use it for drag-and-drop UIs" against the avatar area that "accepts a dropped image", and called drag instead of upload_file in 2 of 4 runs (0 of 4 with the original). gpt-6-astra said it couldn't identify the tool for the "Add photos" task in 4 of 10 runs with my upload_file text (0 of 4 with the original).

So for a one-step choice, the one-line description plus the schema (from_uid/to_uid, filePaths) was enough, and the text that scores 74 cost accuracy on two models. The full table is in the issue thread. What the descriptions still don't say, what a call returns and that uids come from the latest snapshot, may matter in longer runs, which this test didn't cover.

A later, larger run with Forecall's own evaluation points the same way. Across all 30 Chrome DevTools MCP tools (8 requests per tool), Claude Sonnet 5.5 and GPT-6.1 Sol picked the right tool 99% and 98% of the time. And when a request didn't include a page number, they mostly called list_pages first (77–87% of such requests), which is the right move.

Try it on your own server

Get your server's tools/list and score it:

npx forecall dump -o tools.json -- <command that starts your server>
npx forecall lint tools.json

Both run on your machine, and the results aren't sent anywhere. Add --fail-under 60 to fail a CI job when the average drops below 60. To try it in the browser instead, paste the same JSON at forecall.dev; no sign-up needed. For a public server that needs no sign-in, you can paste its URL there instead of JSON. Both servers' full results are at forecall.dev/en/servers/chrome-devtools and forecall.dev/en/servers/playwright.

One caveat. The score comes from reading the text alone, so a tool like browser_close, short but obvious from its name, scores low even when models use it correctly. Treat the score as a guide to where there's room to write more. Whether real models pick the right tools has to be measured another way, and as the test above shows, a higher score doesn't mean better picks.

Use the score to find gaps, then test any rewrite with real models

Google's and Microsoft's browser-automation servers describe their arguments carefully, but rarely say when to use a tool or what it returns. The linter finds those gaps quickly. It doesn't tell you that a model will choose wrong: for these three tools, the models I tested chose right with the one-line text, and my longer text made two of them worse.

For your own server, start with the lowest-scoring tools and fill in what's missing, such as what a call returns. Then check the new text with the models your users run before you ship it. Words like "drop", or a list of example UIs, can pull a model toward the wrong tool.

Update: I suggested rewrites upstream in microsoft/playwright-mcp#1788 and ChromeDevTools/chrome-devtools-mcp#3026. A Playwright maintainer closed that one, asking for a reproducible trajectory against a frontier model rather than a score. In #3026, after the test above, I withdrew the claim that the short descriptions confuse models.

I'm the author of Forecall. The linter is open source: github.com/forecall/forecall-cli.

hidane 0.1.0: a Firestore emulator without Java, tested against the official one

Hiroki Nomura — Sun, 11 Oct 2026 06:05:08 +0000

hidane (火種, the seed of fire) 0.1.0 is out. It replaces the official Cloud Firestore emulator with a single Rust binary that speaks the same three wire protocols: gRPC, REST, and the WebChannel transport the browser SDK uses. It runs under firebase-tools, with no JDK installed. This post covers how to use it, why I built it, and how I check that it behaves like the official emulator.

Code: https://github.com/hidane-dev/hidane
Site: https://hidane.dev
Release: https://github.com/hidane-dev/hidane/releases/tag/v0.1.0

hidane is an independent open-source project, not affiliated with or endorsed by Google LLC. Firebase and Cloud Firestore are trademarks of Google LLC.

Update (2026-10-13): hidane 0.2 enforces Security Rules. Since 0.2.0, firestore.rules decides reads, writes, listeners and queries, with the official emulator's denial messages, :ruleCoverage and debug() output; 0.2.1 feeds the Emulator UI's Requests tab, and 0.2.2 checks a transaction's verify writes against the rules. On those points, the "not yet" list below is out of date. 0.2.3 fixes firebase emulators:start stopping when the Functions emulator loads a Firestore-triggered function, and verifies @firebase/rules-unit-testing under Security Rules in Node and in the browser. 0.3.0 completes Web support: firebase-js-sdk's own browser test suite passes on hidane as on the official emulator (1,183 tests), and a parity table counts what the tests compare with the official emulator: https://hidane.dev/#parity. 0.4.0 delivers Firestore events to the Functions emulator: Firestore-triggered functions, v1 and v2, fire under firebase emulators:start with the same data as with the official emulator, and the Go, Python and Java server SDKs are verified against both emulators. Latest release: https://github.com/hidane-dev/hidane/releases

Using it

Install it, then put hidane exec -- in front of the firebase-tools command you already use:

curl -fsSL https://hidane.dev/install.sh | sh
hidane exec -- firebase emulators:start --only firestore

For that one command, hidane exec makes hidane stand in for java and the official jar. Your firebase.json and your tests stay as they are, and emulators:exec and the Emulator UI work too.

Every channel installs the same release binaries, for macOS, Linux and Windows:

brew install hidane-dev/tap/hidane
npm install --save-dev hidane
cargo binstall hidane            # or: cargo install hidane
dart pub global activate hidane
docker run --rm -p 8080:8080 ghcr.io/hidane-dev/hidane

You can also run it on its own:

hidane --host 127.0.0.1 --port 8080
export FIRESTORE_EMULATOR_HOST=127.0.0.1:8080

In web code, use connectFirestoreEmulator(db, '127.0.0.1', 8080) as usual.

Why

The official emulator is a closed-source Java application shipped as a jar. Three things about it hurt in local development and CI:

Java 21 is required. firebase-tools 15.0.0 dropped support for older Java versions, so you need a JDK just to run a local database.
Startup and memory. It takes 0.74 s to open its port (2.5 s through firebase emulators:start), idles at 95 MiB, and sits at 456 MiB after 1,000 documents.
Writes slow down while a listener is attached. With onSnapshot on a collection, each batch takes longer the more documents are already stored. Writing 100,000 documents in batches of 500 took 121 s, 11× the time without the listener (firebase-tools#3477). The Emulator UI keeps listeners open, which matches the "it's only slow when the UI is open" reports.

The same measurements on hidane

	Official emulator v1.22.0	hidane
Time until the port accepts	0.74 s (2.5 s via the CLI)	5.5 ms
Memory, idle / after 1,000 documents	95 MiB / 456 MiB	7.4 MiB / 21 MiB
100k documents with a listener	121 s, growing per batch	4.9 s, flat per batch

One caveat: these were not measured side by side. The official numbers come from early research on a loaded workstation; hidane's were taken later on a quiet M1 Max. A same-machine comparison is planned.

The flat write cost is a design goal: a commit should cost what it changes, not what is stored. With a listener on the whole collection, hidane stays at about 10.5 ms per batch from the first batch to the last.

Parity: the official emulator is the oracle

An emulator is only useful if it behaves like the one your tests were written against. If error messages, the order of snapshot events, or which edge cases count as errors differ, tests break the day you switch.

So hidane treats the official emulator as the oracle:

Scripts in tools/oracle/ send the official emulator many requests, edge cases included.
Its answers are recorded as fixtures (17 so far).
hidane's test suite replays every recording against hidane and compares.

For scale: 146 cases for how the Authorization header is read, 87 for vector search (find_nearest), and 57 for REST answers (compared byte for byte). On top of that, the Admin SDK, the Web SDK in Node, the Lite SDK, the Web SDK in a real browser, and @firebase/rules-unit-testing run against both emulators, and their transcripts are compared (they are published in results/).

Recording the official emulator turned up plenty of details you would never guess:

A page of results that is exactly full still gets a next-page token; the last page is empty.
explain_options is accepted and ignored, while find_nearest is fully implemented.
REST bodies are limited to 16 MiB, gRPC messages to 100 MiB.
WebChannel frames carry their length in UTF-16 code units, not bytes.

The WebChannel protocol isn't documented, so I put a recording proxy between Chromium and the official emulator and wrote the server from the capture (docs/webchannel.md).

Exports in the official format

firebase emulators:start --import and --export-on-exit work with the official emulator's export format: LevelDB logs of App Engine entities. Tests check hidane's files against the official emulator's, entity by entity, byte for byte. Through firebase-tools, an export written by either emulator reads back the same in the other, in all four combinations. Switching keeps your seed data.

Bugs are not reproduced

Parity has limits. Where the official emulator is clearly wrong, hidane does not copy it, and every such difference is listed in docs/parity-exceptions.md. Some examples:

the listener slowdown above;
after POST /reset, attached listeners receive nothing, not even later writes;
a client that sends an HTTP/2 ping every 10 s gets GOAWAY too_many_pings after 30 s, which is likely what iOS users see as a disconnect after 90 s;
WebChannel sessions are never released;
vectors in nested fields cannot be searched with find_nearest.

What 0.1.0 does, and what it doesn't yet

Working, and checked against the official emulator:

documents, queries (every operator, cursors, collection groups), aggregations (count / sum / avg), vector search;
transactions, with the official emulator's locking;
listeners with incremental updates and resume;
gRPC, REST and WebChannel on one port;
running under firebase-tools, the Emulator UI (browse, create, delete, clear all data), export and import;
the Admin SDK, the Web SDK in Node, in the browser and Lite, and rules-unit-testing without rules.

Not yet:

Security Rules, coming in v0.2. Every request is currently allowed.
The Emulator UI's request monitor, and events for the Functions emulator.
Verification with the Go, Python, Java, iOS, Android and Flutter clients. They speak the same protocols, but nobody has run them against hidane yet.

For tests and development that don't depend on rules, hidane can replace the official emulator today.

Distribution

Each GitHub Release archive comes with a SHA-256 checksum and a build provenance attestation (gh attestation verify). crates.io, npm and pub.dev are published from GitHub Actions through trusted publishing (OIDC), so no registry token lives in the repository. npm versions are staged and released only after a maintainer approves them with 2FA. License: MIT OR Apache-2.0.

Next up is the Security Rules evaluator for v0.2. The plan is in the roadmap.

If you try hidane with your own test suite and an answer differs from the official emulator's, please open an issue with the parity gap template. Reports from the Go, Python, Java or mobile SDKs, whether they worked or not, are very welcome. Questions go to Discussions.

I open-sourced a linter for MCP tool descriptions

Hiroki Nomura — Thu, 08 Oct 2026 03:54:27 +0000

AI agents choose an MCP tool by reading three things: its name, its description and its input schema. Many servers write those descriptions like docstrings, for a human who already knows why they opened the file. An agent with thirty tools in its context doesn't know that. It needs to know when to use a tool, what comes back, and which lookalike to use instead.

Forecall's linter checks exactly that, and its code is now public under Apache-2.0: github.com/forecall/forecall-cli.

Try it on a real server

Two commands. The first starts a server and saves its tools/list; the second scores it.

npx forecall dump -o memory.json -- npx -y @modelcontextprotocol/server-memory
npx forecall lint memory.json

Here is part of the report for the official memory server (0.6.3):

Average score     43.8 / 100
Tools             9
Confusable pairs  1

Across the server
  Major     confusable_pair  delete_entities and delete_relations are easy to mix up.
                             Description similarity 0.62 · Name similarity 0.33

  39 / 100  delete_relations
    Purpose 6/20 · When to use 0/20 · Arguments 18/25 · Return value 8/15 · Constraints and side effects 7/10 · Examples 0/10
    Major     restates_name     The description only restates the name (7 words).
    Major     too_short         The description is too short (7 words).
    Major     no_usage_context  Says nothing about when to use it, or when not to.

Two delete tools with similar descriptions, and neither says when to use it. That's the kind of pair where an agent deletes the wrong thing. I've suggested rewrites for them in an issue on the servers repo.

For a remote server, pass its URL instead, with --header "Authorization: Bearer …" if it needs a key:

npx forecall dump https://example.com/mcp -o tools.json

What it checks

Each tool gets a score out of 100, in six parts:

Part	Points	What it looks for
Purpose	20	What the tool does, beyond restating its name
When to use	20	When to use it, and which similar tool to use instead
Arguments	25	Each argument described, typed and marked required
Return value	15	What comes back, in an output schema or the description
Constraints and side effects	10	Preconditions, side effects, auth, rate limits, annotations
Examples	10	An example of a call

Across the server, it flags too many tools, tools that are easy to mix up, and descriptions that share a long paragraph. Every issue has a code and a severity, so you can work through them one by one.

Use it in CI

--fail-under 60 exits with 1 when the average is below 60, and --json prints the report as JSON.

- run: npx forecall dump -o tools.json -- node dist/server.js
- run: npx forecall lint tools.json --fail-under 60

Nothing leaves your machine

lint works offline. The scoring lives in @forecall/lint, a package with no dependencies and no network code. dump is the only command that talks to the network, and only to the server you name; the JSON it writes records where the tools came from, never headers or environment values.

What it can't tell you

The score comes from rules that read the text and the schema. A short description like browser_close can score low even when every model uses it correctly, and a high score doesn't prove that a model will pick the right tool. Use the score to find where to write more. (Forecall's hosted Evaluations run real models against your tools for that. They're paid, and not in this repository.)

What we found with it

We ran it over the 46 most-used servers on Smithery: 769 of 1,118 tools (69%) never say when to use them. The full write-up is here.

Feedback welcome

The rules are at v1, and I'd like to hear where they're wrong. A reader of that write-up already suggested one: when two tools are flagged as lookalikes, check that each description names the other. It's now issue #7. Issues and pull requests are welcome.

I'm the author of Forecall.

We checked 46 popular MCP servers on Smithery: 7 in 10 tools never say when to use them

Hiroki Nomura — Thu, 08 Oct 2026 02:55:15 +0000

We took 46 of the most-used MCP servers on Smithery and checked the descriptions of all 1,118 of their tools. 769 tools (69%) never say when to use them. 507 (45%) never say in their description what they return. And 15 servers have tools whose descriptions are so alike that an agent can mix them up: 90 such pairs in all.

The 69% is exactly what we found last time, when we checked 8 public servers with 126 tools and 87 of them lacked it. Nine times the sample, the same share.

The checks were done with Forecall, a static linter for MCP tool descriptions that I build.

How we checked: the top 50 servers on Smithery, judged by their text only

On October 8, 2026, we took the first 500 servers in Smithery's public registry (in the order its API returns), removed duplicates, picked the 50 with the most uses, and collected their tool definitions. We scored them with Forecall's CLI (forecall 0.3.2). Four of the 50 have more than 200 tools, the CLI's limit, so we left them out: 46 servers and 1,118 tools remain.

One caveat. The tool definitions Smithery's API returns leave out the input schemas' required lists and any output schemas, so Forecall's overall scores come out lower than they should. This article does not use the overall scores. It counts only the findings that depend on the description text alone: whether a tool says when to use it, whether its description says what it returns, whether the description is too short, and pairs of tools whose descriptions are alike.

7 in 10 tools never say when to use them

Finding	Tools	Share
No "when to use"	769	69%
Description doesn't say what it returns	507	45%
Too short, or restates the name	101	9%

By server it is starker. In 16 of the 46 servers, not one tool says when to use it. Count the servers where 80% or more lack it, and you have half of them: 23.

"When to use" is what decides between lookalike tools. Without it, a model picks by the tools' names and the feel of their descriptions.

The same notice on every tool makes the tools indistinguishable

PubMed has the most lookalike pairs. It has only 7 tools, yet all 21 pairs are flagged as easy to mix up. Among the 12 most similar pairs, the descriptions share 73% to 94% of their words.

The cause is a long notice pasted into all 7 descriptions:

IMPORTANT - PubMed Database Scope: This server provides access to PubMed, which ONLY indexes biomedical and life sciences literature including: …

What sets each tool apart, its first sentence ("search articles", "get metadata", "find related articles" and so on), is buried in the shared paragraph.

A note that applies to the whole server belongs once in the server's instructions, which the client receives at initialization, not in every tool's description. A tool's description should say what only that tool does.

"Send a reply" and "draft a reply" are written almost the same way

Gmail has 7 lookalike pairs. The closest two share 92% of their words:

Tool	Start of its description
Gmail_ReplyToEmail	Send a reply to an email message, optionally with one or more file attachments.
Gmail_WriteDraftReplyEmail	Compose a draft reply to an email message, optionally with one or more file attachments.

After that, both go on with the same paragraph about attaching files. The difference is a few words: "Send" against "Compose a draft".

Mix these two up, and an agent meant to write a draft sends the email. Tools that do something you can't undo, above all, should say in one sentence how to choose: "To prepare a reply without sending it, use Gmail_WriteDraftReplyEmail."

A good example: Brave Search says how its tools connect

Brave Search's descriptions, by contrast, spell out how the tools relate. Here is brave_web_search:

Performs web searches using the Brave Search API and returns comprehensive search results with rich metadata. To chain into local-POI enrichment, pass result_filter=locations and feed the resulting locations.results[].id values into brave_local_search.

It says what comes back and which value to pass to which tool next, so an agent can use several tools in sequence. All 8 of Brave Search's tools say in their descriptions what they return, and it has no lookalike pairs.

How to fix it: one sentence each for three things

From these results, this is the order to fix descriptions in:

Say when to use the tool. "Use this when …", and if there is a lookalike, "for …, use X instead."
Move shared notes to the server's instructions. The same paragraph on every tool buries what tells them apart.
Put the difference between lookalikes into words. Send or draft? Delete or remove? Say it plainly, especially for actions you can't undo.

To check your own server, run these two commands. The first gets the tools/list from your server, the second scores it. Both run on your machine and send the results nowhere.

npx forecall dump -o tools.json -- <command that starts your server>
npx forecall lint tools.json

For a Streamable HTTP server, pass its URL instead: npx forecall dump https://example.com/mcp > tools.json. The CLI is open source: https://github.com/forecall/forecall-cli

The weakness doesn't shrink with scale

With 8 servers or 46, 7 in 10 tools never said when to use them, popular servers included. The fix is small: one sentence on when to use the tool and one on how it differs from its lookalikes leave an agent far less room to pick the wrong one.

We'll keep this study going with new data. Next, we'll score the remote servers in the official MCP Registry, schemas included.

The same study is also on Qiita, in Japanese: https://qiita.com/nomunomu0504/items/b2eab1960a7aad38b59d