We spent two days running deliberate, adversarial testing against our live production payment and provisioning system — real agent purchases, real concurrent load, real market conditions. Not a staged demo. The goal was simple: find out where it actually breaks.
It broke in three places. Here's the full account.
- The audit trail had a real gap
Our system only recorded which provider a job landed on — not what a failed provider actually returned, or when. If a job failed over twice before succeeding, that middle part of the story was gone.
We built a new append-only audit table: every provider attempt now gets logged with its verbatim response, precise timestamps, and outcome. It's enforced at the database level — we tried to directly UPDATE and DELETE a real row ourselves, and the database rejected both, no exceptions even for us.
- A rejected request could still get charged
Found during live testing: a request that hit our concurrency limit and got rejected still resulted in a real, settled on-chain payment. Root cause — payment was settling before the system confirmed the job was even allowed to run.
We fixed the ordering. Then we stress-tested the fix with genuinely simultaneous requests and found a second, narrower version of the same bug — a race condition where multiple parallel requests could all pass the check before any of them locked in a slot. Fixed that too, this time making the check and the reservation a single atomic operation.
Verified by firing six real simultaneous requests: five jobs, five payments, a clean 1:1 match, zero orphaned charges.
- Unprofitable requests were still getting charged
A separate issue: when every available provider was briefly unprofitable at current market rates, the job correctly failed to place — but payment still went through anyway. Fixed with a live profitability check that runs before settlement, not after.
- The deeper fix: real dynamic pricing
Both billing bugs shared a root cause — a fixed rate card that couldn't track live market movement in either direction. We replaced it with real-time pricing: every job now prices off the actual live cost of the provider that will serve it, with the exact quoted price matching exactly what gets recorded on the ledger. No more flat numbers.
The standard we're holding ourselves to
Every fix above was independently re-verified against real production data after shipping — not assumed to work because the code looked right. That's what "real over convincing" means to us in practice.
kilawattcloud.dev
Top comments (0)