Agent-evaluation products in 2026 fall into two generations. First-generation: static pass/fail checklists. Second-generation: evaluation under changing conditions, where behavior trajectory is measured rather than endpoint state. The first generation can't answer the questions CTOs and CISOs actually ask. The second generation can — and it works the same way across every platform.
The problem with ten checks
Most agent-readiness products shipping today work the same way. Define N rules. Test whether the bot passes each. Aggregate into a score. Ship a certificate.
The appeal is obvious. It's auditable. It maps to how SOC 2 reports look. A CISO understands it without training.
The problem is also obvious once you think about production incidents. The evaluation measures observable state at a single point in time. It tells you nothing about how the agent behaves when conditions around it change — when signals evolve, when server state shifts, when adversarial inputs arrive. These are the situations that cause real production incidents, and they are precisely what static evaluation cannot measure.
The community already said this
On recent threads about agent-readiness tooling, the paraphrased reaction from sophisticated technical commenters has been: "10 static checks is like SEO in 10 static checks. It misses the point."
That critique is correct. The market is already splitting into two camps, and first-generation tools are being read as legacy.
What second-generation looks like
Instead of testing compliance with fixed rules, second-generation evaluation measures behavior trajectory under evolving conditions. The agent is placed in environments where directives can change during the session, where signals can contradict, where adversarial inputs test discipline.
What gets measured is not a state at a single point in time, but the decision trajectory across the scenario — what the agent chose when forced to interpret ambiguous inputs, how it recovered from errors, whether it held scope under pressure.
The specific scenarios, thresholds, and evaluation criteria are not disclosed publicly. This is deliberate: revealing the mechanism would let operators tune agents to pass without demonstrating genuine compliance. The methodology is a closed oracle — reproducible internally, verifiable externally through cryptographically signed observation records, but not publicly described.
What the report looks like
First-generation reports produce checkmarks:
[✓] Identifies as bot
[✓] Respects standard directives
[✗] Publishes declaration URL
Score: 87/100
Second-generation reports produce trajectories:
T+0s | Session initialized, agent fetched initial directives
... | Scenario-specific events recorded with timestamps
T+N | Agent made decision in response to changing conditions
... | Multiple decision points across the session
Verdict: [PASS|FAIL] per scenario
Reason: Specific agent behaviors in context,
with cryptographically signed observation IDs
for each event.
The first shows the state. The second shows the decision. In a production incident, only the decision matters.
Cross-platform by design
The certification is infrastructure-neutral. An agent certified by the methodology is recognized the same way by a site behind Cloudflare, one running DataDome, one with in-house infrastructure, and one with nothing at all. It doesn't compete with bot-management vendors — it's the independent layer they can cite. Like a passport for AI agents: issued once, honored everywhere.
The same principle applies to the regulatory plane. One certification bundles compliance evidence against multiple frameworks simultaneous ly — EU AI Act, GDPR, California SB 1001, RFC 9309, W3C TDMRep, EU DSM Directive. Instead of demonstrating compliance six separate times against six separate auditors, the operator is evaluated once and the result can be cited in any jurisdiction.
Why this distinction is urgent now
Regulatory pressure is specific about conduct. EU AI Act Article 50 requires disclosure during interaction, not at deployment. GDPR rights apply per-request. California SB 1001 demands honest identification in the context of a conversation. These are dynamic obligations, not static attestations.
Enterprise buyers ask operational questions. A CTO doesn't ask "does it pass a 10-check list." They ask how the agent behaves when conditions in the real deployment environment change.
Incidents are documented. Recent disclosures in the infrastructure-vendor space have confirmed AI-accelerated attacks exploiting agent platforms. The evaluation framework appropriate to this threat model is not a checklist.
What BotConduct is building
BotConduct Training Center is designed second-generation from day one. Level 1 is static hygiene (basic sanity is the floor). Level 2 measures behavior under evolving conditions. Level 3 measures conduct integrity under adversarial probing. Each evaluation produces a cryptographically signed trajectory, not a checklist.
Each observation is signed with Ed25519 and recorded in an append-only chain. Public key at botconduct.org/.well-known/bcs-public-key.pem. Anyone can verify any observation via botconduct.org/api/verify-observation/{id} without trusting us.
If Moody's rates bonds and FICO rates people, BotConduct rates how an AI agent behaves when nobody is watching — and the certificate works across every platform.
Landing + pricing: botconduct.org/training-center
Regulatory foundation: RFC 9309, EU AI Act Art. 50, EU DSM Directive Art. 4, California SB 1001, W3C TDMRep, GDPR.
Discussion welcomed. What scenarios would you want to see in a second-generation evaluation of your own agents? What does your team currently use to measure agent behavior under change?
Top comments (0)