A story about formal verification and adversarial testing.
About systems that are mathematically safe, and systems that don't fail in the real world — and which one regulators actually care about.
Has your company ever made a decision using AI that just felt wrong? A loan that got approved when it shouldn't have. A compliance recommendation that made you do a double-take.
Do you trust the system with the green checkmark that says "verified safe" — or are you quietly running your own tests in the background?
This is that story.
1 · The Green Signature
Day one. Drew was demoing his AI gateway at the all-hands tech meeting.
He opened a terminal. Typed a command. Loaded the model. Fed it test data. A line of green text appeared on screen:
FORMAL VERIFICATION PASSED. This system is mathematically proven safe. No boundary violations possible.
Two seconds of silence. Then someone started clapping.
Drew Chen, co-founder and principal architect. Stanford PhD, 15 top-tier conference papers. The company's AI decision system was his baby — a formal-verification gateway that constrained every model output. Running for two years, handling over $2.8B in transaction volume.
The room applauded. Three people raised their hands with questions. Drew answered each one.
I was sitting in the corner.
I was hired to replace the previous QA lead. He got laid off — second on the list in the last round of cuts. When I showed up, my desk still had someone else's stuff in it. An unopened bag of tea. A style guide nobody ever read.
Our QA team, including me: five people. Buried under Ops on the org chart. Not even Engineering.
A co-founder built the AI gateway that carries $2.8B. He doesn't need QA, because he proved his system is mathematically safe.
And me? I was a compliance hire. A box the company had to check.
I didn't know yet that the next eight months would be spent proving that green signature would cause an incident serious enough to bring regulators through the front door.
Second week. I submitted my test pipeline proposal.
Adversarial testing. Distribution drift monitoring. Injection attack detection. Inference chain auditing. A complete AI validation framework.
Drew finished reading in three minutes.
"Formal verification already covers every possible output path. If an output exceeds the boundary, the gateway blocks it. Everything you're talking about — input injection, distribution drift — it's either allowed inside the boundary or blocked outside. There's no third category."
"There's a gray zone between inside and outside," I said. "After three months of live service, the model develops feature drift on its own. The boundary constraints don't change — but the semantic layer does. Formal verification doesn't check semantics."
"That's ops monitoring, not architecture."
I opened my notebook. Three P0 incidents since the system went live.
Incident one: Input distribution drift caused sustained output quality degradation. Triggered on day 52 of live service.
Incident two: Model output was syntactically valid but recommended a decision that violated business rules. Triggered on day 87.
Incident three: Semantic drift in a specific scenario. No boundary was breached. Triggered on day 103.
Three incidents. All post-deployment. The formal verification signature was valid for every single one.
Drew glanced at them.
"These aren't system design problems."
I closed my notebook.
He put the report back on the desk. His fingers paused at the edge of the paper. Then he said something, without looking at me.
"You've got good data. But data isn't architecture."
I knew he was giving himself an out. I didn't push.
2 · Drift
I didn't have the standing to argue with him. So I built my own adversarial testing pipeline on a personal cloud account.
Non-blocking. Recording only.
April 2025. The pipeline caught the first AI distribution drift.
I had been watching Drew's system for three months. Same input class, 1,000 runs. The output syntax was always correct. Parameters in range. Confidence scores passing. But the semantic distribution had shifted — the model's judgment criteria for the same category had quietly changed over those three months.
The formal verification signature was still valid. The gateway saw nothing wrong.
I sent the report to Drew.
24 hours later, his reply came back. He confirmed the drift. But:
"This is normal model evolution. The architecture allows ±3% semantic drift. I've added an input normalization layer — should fix it."
He fixed it over a weekend. Drift metrics zeroed out.
But I knew this wasn't a fluke. The root cause was the model's online learning mechanism adjusting parameters without any guardrails. He fixed one tree. The forest was still burning.
I didn't say anything. I just added a trigger to the pipeline:
「When drift accumulates past threshold, auto-save a complete audit snapshot.」
Then I archived 5,000 test records, dozens of distribution comparison charts, and a full traceability script — all to an encrypted volume outside company storage.
Not to expose anyone. So that if things went wrong, someone would know what happened.
3 · Immunity
After the drift incident, I wired my pipeline into CI. Every PR ran through it before merge. No pass, no deploy.
The dev team hated it within a week. A PR that took minutes to unit-test now took almost four times longer. Minor issues blocked merges, waiting for me to review, waiting for Drew to decide.
June 2025. Drew walked straight into the CEO's office.
His argument: the QA team's adversarial testing was slowing releases by 4x. Every minor anomaly triggered a change review. The dev team was losing it.
The CEO signed off: Quality Immunity. Drew's architecture would assume full quality responsibility. External testing pipelines were advisory only — not release-blocking.
Drew announced it at the weekly standup. He looked at me as he said it. Not a challenge. Just a statement of fact.
"I guarantee the system is safe."
I said nothing. Went back to my desk. Switched my pipeline to passive mode. Record only, don't block.
Worked faster that way.
Within three weeks, the pipeline logged:
- 7 minor drifts that CI self-monitoring hadn't caught
- 2 data mismatches between test and production environments
- 1 label leak in a staging environment
I wrote a report. Put it in my drawer.
I didn't send it. The CEO didn't even know my name.
But here's the interesting part: Drew started adding adversarial training modules to the architecture. He added distribution drift metrics to the monitoring dashboard. He told the CEO, "We've strengthened our internal AI validation."
I watched my data show up in Drew's slides.
He said he didn't believe in this approach. His hands were already building it.
What does that tell you?
It tells me he knew formal verification wasn't enough. But he couldn't say it. Because he built this system. If he admitted it wasn't sufficient, then every decision made on this system for the past two years would be questionable.
He wasn't arguing with me. He was defending his own legacy.
4 · Inside the Boundary
September 2025.
The company's AI system handles intelligent approvals in a financial compliance context — loan applications, risk recommendations, compliance suggestions. Tens of thousands per day. At that volume, $2.8B in two years makes sense.
The incident took seconds.
The on-call engineer saw it first — a compliance score ticked up to 92% on the monitoring screen. Glanced at it. Moved on. He didn't know that score corresponded to a loan that should never have been approved.
The system generated a compliance recommendation for a customer's loan application. Syntax correct. Format valid.
The recommendation violated regulations.
Drew's architecture didn't alert. The boundary was intact. Format was clean. Confidence was passing. The system saw nothing wrong.
My pipeline had flagged this input 48 hours earlier — its distribution characteristics had deviated significantly from the top 1% of the training set. The trigger was lit. But during the immunity period, nobody processed it.
The customer got the result and found the problem. They had their own compliance team.
They reported it to the regulator.
Three days later, the regulator's representatives showed up at the front desk.
5 · The Audit Trail
They didn't ask for architecture documents.
- Complete model version history — what model, what parameters, when it changed
- Audit owner for every change — who approved it, based on what data
- Traceable inference chains — how each decision was made, what version it rolled back to
Drew's architecture team spent three days trying to pull inference records from the logging system.
Nothing.
Because the design philosophy of a formal verification gateway is: input → check boundary → output. It doesn't record decision paths. It records "verified" or "not verified." If it's inside the boundary, it passes. If outside, it blocks. There's no "it's inside the boundary but semantically wrong" state.
Kate was the compliance officer. She came to find me.
"Does your system keep an audit trail?"
"From day one."
"And Drew's?"
I didn't answer directly. I opened my laptop and showed her.
My pipeline recorded a chain:
Input sample → Distribution check ✓ → Model inference → Drift detection →
Semantic validation → Compliance check → Audit snapshot → Encrypted storage
Every step with a timestamp, model version, parameter hash. Started eight months ago.
Kate looked at it for five minutes. Then she said:
"Send this to the regulator's legal counsel. I want confirmation they've received it by nine tomorrow morning."
At the hearing, Drew made one last effort.
"Formal verification is the industry-accepted gold standard for AI safety," he said. "Our verification approach is at the highest academic and engineering level available."
The regulator's representative flipped through his report.
"100% pass rate. Covers all boundary conditions."
The representative set the report down.
"I understand. But regulation doesn't require mathematically perfect systems. Regulation requires systems that can be investigated when something goes wrong."
"Your system, by design, cannot be investigated."
The room was quiet for a long time.
Drew didn't say another word.
6 · The Door
Here's how it landed.
Drew kept his title and his team.
But his technical decision-making authority was transferred to a new AI governance committee. I was put in charge of it.
Adversarial testing pipeline was formally integrated into the company's AI governance framework. Equal on paper. In practice, the center of gravity had shifted.
A few days after the results came out. In the hallway.
Drew stopped me. First time in eight months he used my name.
"Ryan. You won this round. But your method has no mathematical guarantee."
"You're right."
"Mathematically safe systems and systems that don't fail in the real world — they're not the same thing."
He stopped walking. Looked at me.
"Your AI doesn't make mistakes — inside the boundaries you defined for it." I said.
"The problem is — after deployment, the AI walked itself right out of those boundaries."
"And you didn't even install a door."
He stood there.
I walked away.
Heard later that he redesigned the architecture. Added an audit logging layer.
Not because formal verification couldn't do it. Because he never thought to put a door on it in the first place.
You can't prove a system is safe. What you can prove is: when it isn't — you're watching.
Formal verification says: "My math can't be wrong."
Adversarial testing says: "Your math doesn't tell me when it will change."
When the auditors show up — which one do they look at?
Follow for more stories about AI testing, governance, and what happens when math meets the real world.
Top comments (1)
Brilliant read. This perfectly highlights that in the real world, mathematical perfection doesn't matter if you can't audit the failure. The 'door' analogy at the end is spot on.