Nine write tools appeared in our capture of mongodb-mcp-server that were absent from another observer's capture of the same version.
The launch commands explained it: our sandbox started the server bare; the other capture used --readOnly and a connection string.
We run HISTOR, a public log of MCP tool definitions. Paraphern maintains RugSnare, which records and compares tool contracts across releases. On 26 October, we compared the overlapping parts of our records. Some differences described the servers' launch conditions. Others exposed mistakes in the measurement process, including gaps in ours.
What we actually compared
The overlap contained 26 packages. HISTOR held tool lists for 21 of them. RugSnare's pins supplied 34 version captures that could be compared against our record.
We compared the fields both records retained in full: name, description and inputSchema. We excluded outputSchema, which the peer snapshots retained only as a hash. Annotations were a separate comparison, limited to current versions for which both sides had them.
The published comparison found 25 matching version captures and nine differences. Annotations agreed on all 318 tools across the 16 current versions where they could be compared.
Each tile represents one version capture. This selected overlap is not an estimate of error rates across MCP servers.
Historical version attribution had an important limitation: HISTOR had not recorded the package version on each observation. The comparison used list contents and npm publication times to connect historical observations to release windows. That missing provenance needed a code fix of its own.
A launch argument can change the contract
The MongoDB differences affected versions 3.0.5 and 3.1.0. The bare launch exposed nine additional write tools. Other tools and descriptions differed too.
This matters when a client approves a tool list. An observation made with a read-only launch cannot establish which tools the same package advertises under another configuration. The launch entry belongs beside the captured definitions.
It also changes the question a comparison should answer. Before investigating a changed tool list, check whether the two captures used equivalent arguments and configuration. The MongoDB capture notes preserve the peer's measurement context; our comparison records the bare-launch difference.
Startup network access can change registration
For @netlify/mcp 1.19.0, RugSnare captured netlify-coding-rules; HISTOR did not. Registration depends on fetching a topic list during startup. HISTOR's sandbox has no route to the internet.
A tool absent under those conditions may still be present in a connected deployment. Capturing tools/list records what that instance advertised, under that launch. The Netlify notes make the capture window and corrections inspectable.
Network policy therefore belongs in the observation record. So does authentication context: an anonymous or placeholder-key session cannot establish everything a real authenticated client receives.
The observer can retain tools the package stopped serving
Six other version snapshots contained stale entries: two each for Shopify, Currents and Netlify. Paraphern checked the findings and confirmed that an automated re-pin had run over dirty capture environments. All six were re-captured from clean environments.
The Shopify erratum names the cross-check and records the corrected five-tool inventories for 1.16.0 and 1.17.0. The Currents and Netlify pages preserve their corrections too.
This is why independently collected records are useful even when both collectors already publish hashes. A digest faithfully identifies the bytes its producer captured. The capture still needs to be the intended one.
Paraphern also incorporated a correction to our profile's description in the comparison document. Our separate extras digest covers more than annotations. Writing down exactly which members a digest includes is part of making comparisons possible.
What we changed on our side
Five overlapping packages had no tool list in HISTOR. Their launch requirements exposed gaps in our coverage: filesystem roots, an authentication mode, a start subcommand, a transport option, or a network dependency. A missing capture remained missing coverage.
HISTOR's code now stores packageVersion and a launch record on each package observation. The timeline and package_observations.jsonl.gz expose the version, observer, image, runtime, entry and supplied configuration. Older rows remain null where the facts were never recorded.
In version 0.4.2, we also shipped four reviewed launch defaults:
| Package | Supplied test configuration |
|---|---|
| Filesystem | The disposable container's /work/app directory |
| Azure DevOps | A synthetic organization, envvar authentication and a placeholder token |
| LaunchDarkly | The start command, stdio transport and a per-run placeholder |
| Figma |
--stdio and a placeholder environment variable |
Explicit registry arguments take precedence. A previous failure can retry with the new configuration while retaining its earlier observation. The sandbox generates placeholder values locally; it receives no real credentials and gains no internet route.
We fixed another attribution error: when the observer omits its resolved version, the reader now leaves it unknown instead of substituting the version requested by the crawler.
These follow-up changes ship in aimarket-histor 0.4.2. Both standalone sandbox observers were updated before the production HISTOR service on 27 October. The local HISTOR suite passed 610 tests, with 210 PostgreSQL cases skipped without a test database, one optional token-count case skipped without tiktoken, and 12 integration cases deselected. The new profile regressions use synthetic package responses and a stubbed container boundary. They do not establish four new successful captures of current upstream packages. Supabase and other services requiring a real authenticated network session can still remain unobserved.
A small comparison you can run yourself
Save two complete tools/list arrays as a.json and b.json. This Python example reports differences in the three shared fields used above:
import json
import sys
from pathlib import Path
FIELDS = ("name", "description", "inputSchema")
def surface(path):
tools = json.loads(Path(path).read_text())
if not isinstance(tools, list):
raise ValueError("Expected a complete tools/list array")
result = {}
for tool in tools:
name = tool["name"]
if name in result:
raise ValueError(f"Duplicate tool name: {name}")
result[name] = {
key: json.dumps(tool[key], sort_keys=True, allow_nan=False)
for key in FIELDS if key in tool
}
return result
a, b = map(surface, sys.argv[1:3])
print("Added:", sorted(b.keys() - a.keys()))
print("Removed:", sorted(a.keys() - b.keys()))
for name in sorted(a.keys() & b.keys()):
changed = [key for key in FIELDS if a[name].get(key) != b[name].get(key)]
if changed:
print(name, changed)
Run it with python compare_captures.py a.json b.json. It compares saved JSON values, preserving missing versus explicit null fields and distinguishing booleans from numbers. It does not implement MTL/1, classify compatibility, or verify capture provenance. Array order remains significant; outputSchema and extras are deliberately outside this example.
A checklist for reproducible capture. HISTOR does not yet export resolved dependency inventories.
Keep both original responses and the conditions that produced them: package version, launch entry and arguments, configuration policy, image identity, resolved dependencies, network access and observation time. Establish the shared comparison fields before interpreting different digest strings.
The next useful cross-check is a comparison of fresh captures with equivalent launch conditions and explicit version records on both sides. That would test whether these changes make the join easier; this historical comparison cannot establish that result in advance.
Disclosure: we build HISTOR and WARDEN. AI assistance was used for the analysis, code, writing and cover illustration. The two diagrams are generated deterministically from the reported comparison and its documented limits. The cover is an editorial illustration, not a screenshot or measured result.
Top comments (0)