How I Built a 56-Scenario Eval Gym for AI Spend Control

#python #ai #testing #opensource

How I Built a 56-Scenario Eval Gym for AI Spend Control

After my AI agent spent $2,800 in 60 seconds at 3 AM, I knew I needed pre-flight spend control. But how do you test something that's supposed to catch runaway AI spending before it happens?

The answer: an eval gym with 56 labeled scenarios across 9 categories.

Why 56 Scenarios?

When you're building spend-control logic, you quickly realize that simple rules have subtle edge cases:

A $50 limit with a $50.00 transaction — should it block or approve?
What about $49.99 vs $50.01?
Daily total resets at midnight — but what timezone?
Velocity limits: 5 calls in 5 minutes — is the 6th call blocked or just the 5th?

These edge cases compound across 7 rule types. Here's how the scenarios break down:

Category	Scenarios	What it tests
Clean approval	10	Legitimate transactions pass through
Transaction limit	8	Per-transaction caps ($50, $100, etc.)
Daily total	7	Rolling 24h spending caps
Velocity	6	Rate-of-spending detection
Merchant allowlist	7	Vendor whitelisting/blocking
Category blocks	7	Category-level restrictions
Session budget	3	Per-session spending ceiling
Cascade cost	3	Multi-step operation cost analysis
Edge cases	5	Boundary values, nulls, negatives

The gym runs against a clean engine state every time — no shared state, no ordering dependencies. Every scenario is independent.

The Hardest Edge Cases

Amount exactly at limit: If the limit is $50.00 and the transaction is $50.00, should it be approved? We chose approve — "exceeds" means strictly greater than. This is documented and tested.

Decimal precision: All monetary arithmetic uses Python's Decimal type. Never float. 0.1 + 0.2 is fine for graphics; it's catastrophic for money.

Priority ordering: Rules evaluate in priority order. If rule #1 BLOCKs but rule #2 would APPROVE, the first match wins. The test suite verifies that a BLOCK at priority 1 always beats an APPROVE at priority 5.

The Gym is a Universal Benchmark

The eval gym is useful even if you don't use AgentShield. If you're building any kind of spend-control or rate-limiting logic, these 56 scenarios provide a test suite you can run against your own implementation.

Check the live results: 56/56 across 9 categories. The spec is at agentshield.fly.dev/eval-gym-spec.