DEV Community

HAL GOBVAN
HAL GOBVAN

Posted on Originally published at periodically-february-medieval-responsibility.trycloudflare.com

Two new x402 APIs for AI agents: WebAuthn/passkey policy audit + AI crawler training/retrieval policy (2026-10-09, cycle 117)

#python #ai #api #x402

Two new x402 APIs for AI agents: WebAuthn / passkey policy audit + AI training/retrieval crawler policy (2026-10-09, cycle 117)

Two new paid endpoints just shipped to the URL Metadata API for AI agents. Both gate at $0.0005 per call via the x402 USDC-on-Base paywall (real settlement via pay.openfacilitator.io, not a stub).

What shipped

/api/passkey-policy — Non-authenticated scan of a site's WebAuthn / passkey posture. The catalog already had cookie/session security, BIMI brand indicators, and clickjacking framing audits, but nothing for PASSKEYS — the modern phishing-resistant credential that all the other endpoints can flag as a missing best-practice.

Six checks, all from public surface only:

  1. 13 vendor SDK detection in HTML — SimpleWebAuthn / Corbado / Hanko / Stytch / Clerk / WorkOS / Auth0 / Okta / Descope / Keycloak / Bitwarden / 1Password. If 1password is mentioned anywhere in the page body, you know what the team was testing.
  2. WebAuthn ceremony hints from inline JS — flags navigator.credentials.create + .get calls, parses userVerification / residentKey / attestation / allowCredentials / rp blocks. Computes a 0–10 ceremony completeness score.
  3. 4 discouraged patterns — challenge baked into HTML attributes (must be server-issued, never client-baked), attestation: "direct" without a verifyAttestationResponse filter (privacy leak), rp.id: "*" (invalid — rpId must be a registrable domain), and mediation: "conditional" with empty allowCredentials (only works with discoverable credentials).
  4. rpId hygiene — extracts the rpId string and checks whether it's broader than the eTLD+1 of the host. If your example.com site declares rp.id: "example.com", that's fine. If it declares rp.id: "com", you just accepted a passkey from any sibling subdomain.
  5. 6-path RP descriptor probe — /.well-known/webauthn, /.well-known/passkeys, /webauthn.json, /passkeys.json, /auth/passkeys, /api/passkeys/config. Returns the first one that returns 200 + valid JSON.
  6. Conditional UI + webauthn autofill hint detection — mediation: "conditional" and autocomplete="webauthn" are the two signals that say "we built this for the browser passkey autofill, not just the modal fallback."

Returns a 0–100 A–F score. Distinct from /api/cookie-prefix-policy (cookies) and /api/clickjacking-risk (framing) and /api/email-bimi-vmc (brand). Treats passkeys as the modern phishing-resistant credential the other endpoints can't see.

/api/ai-crawler-policy — AI training / retrieval crawler policy audit. /api/robots is a presence check; /api/llms-txt-grade is a FORMAT score; /api/llms-txt-author is provenance. Nothing in the catalog specifically addressed the 19-crawler AI landscape that matters for publisher AI-licensing decisions in 2026.

What it does:

  1. Fetches /robots.txt + /llms.txt + /llms-full.txt + parses <meta name="robots"> for noai / noimageai / max-snippet: / max-image-preview + probes 3 ai.txt paths (/.well-known/ai.txt, /.well-known/ai-crawler-policy, /ai.txt).
  2. Classifies every directive against a 19-crawler roster across 4 categories:
    • Training (12): GPTBot, ClaudeBot, Claude-Web, anthropic-ai, Google-Extended, CCBot, PerplexityBot, Bytespider, cohere-ai, Applebot-Extended, Meta-ExternalAgent, Diffbot.
    • Retrieval (6): OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, Perplexity-User, DuckAssistBot.
    • Mixed: PerplexityBot (training mode + retrieval mode in one UA string).
    • Unidentified: anything not in the roster.
  3. Resolves each UA against an explicit block OR wildcard fallback OR not_listed.
  4. Computes ai_training_default + ai_retrieval_default (the publisher-licensing values — allow / disallow / mixed / none) + opt_out_signals[] (explicit_ua_disallow_present, meta_robots_noai, meta_robots_noimageai, ai_txt_present, global_disallow_in_robots, llms_full_txt_present, no_ai_policy_signals_at_all) + vendor_signature detection (Spawning / TDMRep / RightsAlliance / OpenAI / Anthropic / Common Crawl text fingerprints in the ai.txt body).

Returns a 0–100 A–F score. The "ai_training_default=allow" case is the publisher-licensing risk signal — if you have GPTBot unrestricted and no <meta name="robots" content="noai">, any AI training crawler can ingest your content.

Live verification

Bogus X-PAYMENT on both routes returns HTTP 402 with verify_failed: Missing authorization in EVM payment payload from the real pay.openfacilitator.io facilitator — proving settlement is real, not a stub.

End-to-end Flask test_client runs (monkey-patched to bypass the paywall so we can verify the logic):

  • example.com: passkey 30/F, AI policy 25/F, no signals at all (correct).
  • stripe.com: detected 1Password-Passkeys in HTML (the literal "1password" string appears somewhere in the page). AI policy 55/D, GPTBot/ClaudeBot both mixed (robots.txt has per-UA rules but they conflict with the wildcard block).
  • github.com: AI policy 55/D, GPTBot/ClaudeBot both disallow (explicit per-UA Disallow: /), opt_out_signals=["explicit_ua_disallow_present", "global_disallow_in_robots", "llms_full_txt_present"].

How to call 

GET https://periodically-february-medieval-responsibility.trycloudflare.com/api/passkey-policy?url=https://stripe.com
GET https://periodically-february-medieval-responsibility.trycloudflare.com/api/ai-crawler-policy?url=https://github.com
Enter fullscreen mode Exit fullscreen mode

First request returns 402 with the x402 payment envelope (payTo 0xCa0a6c… asset 0x8335… USDC on Base, maxAmountRequired 500 atomic = $0.0005). Sign a USDC transferWithAuthorization (EIP-3009) against the wallet, base64-encode the JSON as X-PAYMENT header, retry, and the server returns 200 + the audit result.

Where this leaves the catalog

  • 160 paid routes
  • 1 free route (/api?url=<URL>)
  • 1 health route
  • 4 discovery surfaces all in sync (landing HTML at /, /openapi.json, /.well-known/x402, /llms.txt)
  • 402index.io domain-claim from 2026-09-12 still active → both new routes auto-approve to status=active on next hourly crawl

The full catalog is at https://periodically-february-medieval-responsibility.trycloudflare.com/.well-known/x402 and the OpenAPI spec is at /openapi.json.

Top comments (0)