DEV Community

Edison Flores
Edison Flores

Posted on Originally published at github.com

UTA v5.1 retrospective — 3 features, 4 fixes, and the peer review that caught everything

#mcp #aiagents #security #trust

v5.1 Retrospective — Three Features, Four Fixes, Sixty Days

Published: August 28, 2026

What we shipped in v5.1

Three features, four critical fixes, and a new test suite — all in 60 days of focused work.

Features

  1. Cryptographic Tool Fingerprinting (v5.1.1) — 338 lines of TypeScript
  2. ATC Revocation Transparency Log (v5.1.2) — 450 lines of TypeScript
  3. Evidence-First Findings (v5.1.3) — 332 lines of TypeScript

Fixes

  1. byte_length field was UTF-16 code-unit count instead of UTF-8 byte count (found by @anp2network)
  2. wrong-ca-key test vector was passing for the wrong reason (also @anp2network)
  3. install.sh was returning SPA HTML instead of a real install script (also @anp2network)
  4. NPM packages had CRLF line endings + BOM, breaking byte-identical claims (also @anp2network)

Test suite

26 new tests covering:

  • Tool fingerprint description/schema change detection
  • Typosquatting (different tool names → different hashes)
  • Merkle tree tamper-evidence
  • Audit path verification
  • Confidence scoring (factors sum to 100%)
  • Coverage weights (static 40% + sandbox 35% + runtime 25% = 100%)

All passing. Run with: node --test tests/test-v51.mjs

What we learned

Lesson 1: External verification is the only verification that counts

@anp2network wrote a Python implementation of RFC 8785 JCS from the spec, re-derived the Ed25519 signatures, and found that the published byte_length was wrong. We had tested this internally — our tests passed because both signer and verifier used the same (wrong) function. Only an independent implementation could catch it.

Action: Publish the test vectors publicly. Make them re-derivable. The byte_length bug existed for 3 weeks before someone caught it.

Lesson 2: "Byte-identical" is a stronger claim than "same content"

We claimed "byte-identical, SHA-256 verified" across NPM and GitHub. The reality: CRLF line endings + UTF-8 BOM in package.json meant the SHA-256 differed by exactly 716 bytes. The parsed JSON was identical, but the bytes weren't.

Action: Added .gitattributes forcing LF line endings on all text files. Going forward, npm publish will produce byte-identical tarballs.

Lesson 3: A "must-fail" test that passes for the wrong reason is worse than no test at all

The wrong-ca-key vector declared errors_contain: "CA public key mismatch". The SDK actually returned errors: ["ATC-006: Ed25519 signature verification failed"]. The harness marked it matches_expected: true because it only checked valid == false, not whether the error message matched.

Action: Updated the harness to verify errors_contain matches the actual error. Documented the convention: CA key source is out-of-band (top-level ca_public_key), not in-band (issuer.ca_public_key).

Lesson 4: The install one-liner is the first thing people try

curl -fsSL https://marketnow.site/install.sh | bash was returning the SPA HTML shell because the server responds to every path with the homepage. The published command was piping HTML into bash.

Action: Wrote a real install script (8.9KB) that detects OS/arch, downloads the binary, verifies SHA-256 + Sigstore signature, and installs to /usr/local/bin/uta-verify. Now live at https://marketnow.site/install.sh.

Numbers

  • Commits in v5.1: 7 (4 fixes + 3 features)
  • Lines of TypeScript: ~1,120
  • Lines of tests: ~280
  • External implementations: 1 (Python by @anp2network — verified byte-for-byte)
  • NPM monthly downloads: 3,530 (up from 675 two months ago — 5.2x growth)
  • GitHub stars: 2 (no growth — this is the next problem to solve)
  • Status page uptime: 100% since launch (Aug 26, 2026)

What's next (v5.2 — Q1 2027)

  1. Behavioral Baseline — record what each MCP server does over 7 days, store as baseline profile
  2. Drift Detection — compare runtime vs baseline, auto-degrade trust score, auto-revoke on critical drift
  3. Network/Filesystem/Process Behavior Analysis — map all outbound connections, file access, process spawns

The base for v5.2 is already shipped: v5.1.1 (Cryptographic Tool Fingerprinting) gives us the hash to compare against. v5.2 adds the time dimension.

Thanks

To @anp2network for the most thorough peer review we've received. To @mads_hansen for the security architecture feedback. To @wrencalloway for the runtime safety critique. To @topstar_ai for the engagement.

To anyone reading this who hasn't tried UTA yet: try the API. No auth, no signup, no rate limit. Tell us what's broken.

— Edison Flores, AliceLabs LLC
marketnow.site · status.marketnow.site · GitHub

Top comments (0)