DEV Community

Paul Spread
Paul Spread

Posted on Originally published at agentbadge.xyz

Inside an Agent Readiness Scanner: Rules, Evidence and Reproducibility

#api #testing #ai #webdev

Originally published at AgentBadge.

When someone tells you that an API has an 87/100 Agent Readiness score, the first question should not be:

"Is 87 a good score?"

The better question is:

"Why is it 87?"

And the question after that is even more important:

"Can I reproduce the result myself?"

This is the problem AgentBadge is designed to solve.

Agent Readiness should not be an opinion generated by an LLM. It should be a measurable property of a service, calculated from explicit rules and supported by evidence.

The core idea is simple:

Rules → Evidence → Assertions → Score → Report

1. What the scanner actually does

The scanner does not ask an LLM "is this API good?"

It runs deterministic checks:

  • Does robots.txt exist?
  • Does /.well-known/openapi.json return 200?
  • Is the response valid OpenAPI?
  • Does the homepage have machine-readable metadata?
  • Is there an agents.txt file?
  • Are there llms.txt or llms-full.txt files?
  • Does the API support content negotiation?
  • Is there an A2A agent card?

Each check produces evidence. Each piece of evidence becomes an assertion. Each assertion has a status.

AI-assisted inference is used when deterministic checks are insufficient — but it is a copilot, not the judge.

2. Rule structure

Every rule follows the same structure:

AB-001
Name: OpenAPI discoverability

Given:
  target = https://example.com

Check:
  GET /.well-known/openapi.json

Pass when:
  HTTP status = 200
  AND response is valid OpenAPI

Evidence:
  URL
  HTTP status
  content type
  content hash

Severity:
  medium
Enter fullscreen mode Exit fullscreen mode

This is not a prompt. It is a specification.

The rule says what to check, what constitutes a pass, and what evidence to collect.

3. Assertion states

Every assertion has exactly one of four states:

State Meaning
VERIFIED Direct evidence found
INFERRED Reasonable interpretation, insufficient evidence
CONFLICT Two sources disagree
MISSING Expected capability or artifact not found

An LLM can be 94% confident that an API supports refunds. Without machine-readable evidence, the assertion stays INFERRED.

Confidence is not the same as verification.

Assertion states infographic — four states from verified to missing

4. Evidence is part of the measurement

Evidence is not an optional explanation attached after the fact.

Each finding includes:

  • The actual HTTP response (where applicable)
  • The URL checked
  • The content type received
  • A content hash
  • The timestamp of the check

This means the assertion is not just "OpenAPI exists: yes" — it is "OpenAPI exists: yes, here is the proof."

{
  "rule_id": "AB-007",
  "status": "VERIFIED",
  "target": "https://api.example.com/openapi.json",
  "evidence": {
    "http_status": 200,
    "content_type": "application/json"
  },
  "confidence": 1.0
}
Enter fullscreen mode Exit fullscreen mode

5. Scoring has category floors

The total score is not a simple average.

Some categories are foundational. If Discovery = 0, it does not matter how good the documentation is — agents cannot find the API.

Category floors prevent a high score from hiding a critical zero.

A score of 91/100 with Discovery = 0 is not a good score. It is a misleading one.

6. What confidence means and what it does not

An LLM can analyze an API documentation page and say:

"This API likely supports token-based authentication with rate limiting."

That is a reasonable inference. But without machine-readable evidence (an OpenAPI spec, a response header, a well-known endpoint), it remains INFERRED.

The user can then:

  • Confirm — "Yes, this is correct, I checked manually"
  • Edit — "Almost right, but the auth method is different"
  • Reject — "No, this is wrong"

This creates a feedback loop: automatic fixes for deterministic changes, human confirmation for semantic claims.

7. Reproducibility

If the scanner gives a score of 76/100 for https://api.example.com today, someone else running the same rules against the same captured state should be able to understand how the result was produced.

The report needs to describe the measurement context:

{
  "target": "https://api.example.com",
  "ruleset": "agent-readiness-v1.0",
  "scanner_version": "0.1.0",
  "timestamp": "...",
  "assertions": [...],
  "score": {
    "total": 76,
    "categories": {
      "discovery": 18,
      "documentation": 20,
      "authentication": 17,
      "machine_readability": 21
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Reproducibility infographic — three identical inputs converging into the same result

8. Rules must be versioned

Rules change. New standards appear. Some checks eventually turn out to be too strict or too weak.

Agent Readiness v1.0 must not silently become v1.1 while pretending the results are identical.

Each ruleset has an explicit version. A report can say:

Score: 82/100

Ruleset: Agent Readiness v1.2

Same target + same measurement state + same ruleset = reproducible result.

9. Why open rules do not destroy the product

If AgentBadge publishes its rules, can't someone simply copy them?

Yes. And that is intentional.

The goal is not a secret scoring algorithm. The goal is a useful measurement standard.

The long-term value comes from the workflow around that standard:

Open specification → Open scanner → GitHub Action → README badge → Continuous monitoring → Regression alerts → Fix workflow → Developer adoption
Enter fullscreen mode Exit fullscreen mode

A competitor can copy rules. They cannot instantly copy:

  • thousands of repositories displaying the badge
  • existing GitHub Actions
  • developer habits
  • historical scan data
  • integrations
  • trust built around independently verifiable reports

The moat is not "our rules are secret." It is "our standard is installed inside the developer workflow."

10. The score should explain itself

If a developer's score changes from 76 → 72, the product should explain the delta:

+8  OpenAPI documentation detected
-12  New authentication issue detected
+0   Discovery unchanged

Result: 76 → 72
Enter fullscreen mode Exit fullscreen mode

Without a breakdown, a developer who fixed three problems and still sees the score decrease will think the product is broken.

With a breakdown, it becomes: "The scanner found something new. Now I know what to fix."

11. From Measure to Prove to Improve

The architecture is a loop:

MEASURE → PROVE → IMPROVE → Measure again
Enter fullscreen mode Exit fullscreen mode

The score is the beginning of the workflow, not the end.

Measure → Prove → Improve cycle diagram

12. What AgentBadge should never claim

AgentBadge measures Agent Readiness. It does not certify:

  • API security
  • business correctness
  • service reliability
  • legal compliance
  • whether an agent should trust the company

A high score means: "This API satisfied these measurable Agent Readiness criteria."

Don't certify. Measure.

13. What this enables 

npx @agentbadge/cli scan https://api.example.com
Enter fullscreen mode Exit fullscreen mode

A CI pipeline can enforce a minimum score. A README can display the current measurement. A platform can query AgentBadge programmatically. An organization can compare vendors using the same ruleset.

14. The bigger idea

Performance has metrics. Accessibility has automated checks. Security has scanners. TLS has analyzers. SEO has crawlers and validators.

The emerging agentic web needs something similar.

Don't ask an AI to invent a score.

Define the rules. Collect the evidence. Show the reasoning. Version the rules. Make the result reproducible.

Then let developers improve their systems.

Read more:

Agent Knowledge Layer: agentbadge.xyz/agent-guide

Top comments (0)