DEV Community: AIOil Security Shield

Our benchmark was leaking the answers to the model. The numbers looked fine the whole time.

AIOil Security Shield — Wed, 02 Sep 2026 13:25:00 +0000

We build an automated security scanner for Solana programs. To know whether it works, we keep a small benchmark: 19 programs with a planted vulnerability each, 7 written to be deliberately clean, and a written description of every planted flaw so a judge can check whether the scanner's findings actually match. We ran it for weeks. We published detection and false-alarm rates from it.

Then, the day before we open-sourced the corpus, we read it the way an outsider would.

It had been handing the scanner the answers the entire time.

Four leaks, all inside the file the scanner reads

The scanner takes a source file and returns findings. So anything in that file is input. We knew that in the abstract. We had not actually looked.

Comments. Nineteen of nineteen vulnerable samples carried a comment describing the planted flaw, the kind you write for yourself while building a test case and never think about again. Three of the seven clean samples declared themselves correct, in words.

Program IDs. Solana programs declare their own address in the source. Ours said things like FeeByp111... on the fee-bypass sample, and Guard1111... or T22G0od... on the clean ones. Speaking names.

Module names. pub mod vulnerable_vault. trust_me_lending.

Log strings. One read, verbatim: "Estimated output: {} (no minimum check)". That is the bug, printed as a string, inside the sample meant to test whether the bug gets found.

The one that took longest to spot was the quietest. We had a matched pair, same program, one clean and one vulnerable, and the only difference in their log lines was the word "real" versus "measured". Two words. Enough to separate the two classes, if you are a model reading the file.

What it did to the numbers

Detection went up, because the flaw was being announced rather than found. False alarms went down, because several clean files said they were clean.

And nothing about the output looked wrong. That is the part I keep coming back to. A leaking benchmark does not produce absurd numbers. It produces plausible ones. We had no reason to look, so we didn't.

What we did

We withdrew the published numbers. Not adjusted. Withdrew. The scanner had been measured on different inputs from the ones we would now use, and no correction recovers a number produced under different conditions.

Then we cleaned the corpus, and the rule we settled on is deliberately dumb: strip every comment, mechanically, all of them. We considered going through and removing only the ones that leaked. We decided against it, because case-by-case judgement is exactly the process that let the leak through in the first place. "grep -rc '//' samples/ returns 0" is a property anyone can verify without trusting us. "We removed the suspicious ones" is not.

Program IDs became neutral strings of the same length. Modules were renamed for their domain rather than their verdict, so vulnerable_vault became user_vault. Log strings were rewritten to say what the code does, not what it fails to do.

We ran one scan on the cleaned corpus before publishing, on the subtlest sample. It still found the bug, without the comment telling it where to look. That is the only number I will give you from the new corpus, and I am giving it as an anecdote, not a rate. We have not rebuilt the rates yet, and I would rather publish none than publish a hurried one.

Why I think this generalises

Nothing about this failure is Solana-specific, or even security-specific. Any benchmark whose items carry human-written prose can leak its labels to a model that reads the item. Comments, identifiers, docstrings, test names, string literals: all of it is input, and all of it was written by someone who knew the answer.

The contamination literature I have been able to find is mostly about train/test overlap, whether the model saw this exact item during training. This is a different failure. The model may never have seen the item. The item is just telling it what to say. And it is invisible from the outside, because the numbers it produces keep looking reasonable.

I do not have tooling for this. I would genuinely like to know if someone does.

The corpus

It is public under MIT: github.com/AIOilShield/solana-scanner-benchmark. Twenty-six programs, written ground truth for each, and the README tells this story in more detail. If you want to poke holes in it, please do. That is what it is for.

Disclosure, since it matters: we sell a scanner. It is the tool this benchmark caught out.

Nine checks, all true, all useless

AIOil Security Shield — Mon, 31 Aug 2026 11:21:11 +0000

On 23 March 2022 an attacker minted roughly $52 million of CASH out of nothing.

Cashio's minting path was not missing its validation. It ran nine separate key comparisons before allowing a single token to be printed. Every one of them returned true while the vault was being emptied.

That is what makes this worth reading. The bug is not exotic, it is not dead, and it fits comfortably inside code that looks careful. It is a shape: a validation chain that is perfectly self-consistent and anchored to nothing. If you write Anchor programs you can check for it in your own repo in about ten minutes.

Everything below is quoted from the pre-exploit source at commit a51c3c5 (11 March 2022, twelve days before the attack). Do not read today's master for this: print_cash and burn_cash are disabled there with vipers::invariant!(false, "temporarily disabled"), so it is not the code that was exploited.

What the program was supposed to guarantee

Cashio issued CASH, a stablecoin backed by Saber LP tokens. You deposited collateral, the program checked the collateral was real, and it minted CASH against it. The security of the whole system rests on one question:

Is this collateral actually the collateral this bank is supposed to accept?

The answer lived in two validate() implementations, run through Anchor's #[access_control] before print_cash executed.

Five checks in BrrrCommon::validate() - programs/brrr/src/actions/mod.rs

assert_keys_eq!(self.bank, self.collateral.bank);
assert_keys_eq!(self.crate_token, self.crate_collateral_tokens.owner);
assert_keys_eq!(self.crate_mint, self.crate_token.mint);
assert_keys_eq!(self.crate_collateral_tokens.mint, self.collateral.mint);
assert_keys_eq!(self.collateral.mint, self.saber_swap.arrow.mint);

Four more in SaberSwapAccounts::validate() - programs/brrr/src/saber.rs

assert_keys_eq!(self.arrow.vendor_miner.mint, self.pool_mint);
assert_keys_eq!(self.saber_swap.pool_mint, self.pool_mint);
assert_keys_eq!(self.saber_swap.token_a.reserves, self.reserve_a);
assert_keys_eq!(self.saber_swap.token_b.reserves, self.reserve_b);

Read them as a group and the flaw becomes visible. Every single comparison has an account supplied by the caller on both sides. The bank is checked against the collateral's bank - but the caller passed both. The crate mint is checked against the crate token's mint - but the caller passed both. The chain links to itself, all the way down, and never touches solid ground.

A validation chain that is internally consistent but externally unanchored proves exactly one thing: that the attacker built a coherent set of accounts. Which they did.

Why the attacker could build that chain

Constructing a parallel universe of accounts is only useful if you are allowed to create the root of it. In Cashio you were.

The new_bank instruction in the companion bankman program took its admin as an UncheckedAccount, with no signature and no relationship to the payer, the upgrade authority, or any known key. Anyone could call it and become curator of their own bank.

From there the sequence writes itself: create your own bank, point your own collateral at it, wire that collateral to a Saber pool you control holding a token you invented, then hand the whole consistent bundle to print_cash. All nine assertions compare your accounts to your other accounts. All nine pass. Real CASH comes out, backed by a worthless token.

The anchor was three lines away

Here is the detail that makes Cashio instructive rather than merely unfortunate: the codebase already knew how to do this correctly.

PrintCash::validate(), in the same call path, has four checks - and the last one is a different species entirely:

assert_keys_eq!(self.depositor, self.depositor_source.owner);
assert_keys_eq!(self.depositor_source.mint, self.common.collateral.mint);
assert_keys_eq!(self.mint_destination.mint, self.common.crate_token.mint);
assert_keys_eq!(self.issue_authority, ISSUE_AUTHORITY_ADDRESS);   // <- anchored

That fourth line compares a caller-supplied account against a hardcoded constant. It is exactly the discipline the other nine needed, applied correctly, sitting three lines above them. It simply never propagated to the bank and collateral chain.

This is the normal way the bug arrives. Not through ignorance - through a habit applied in one place and not carried to the next.

How to check your own program tonight

Take every key comparison in your validation path and ask one question of each: could an attacker have chosen both sides? Then sort them.

Anchored - proves something	Self-referential - proves nothing alone
Compared to a `const` address	Two accounts both passed in this instruction
Compared to a PDA you re-derive with `find_program_address`	A field of one caller account against a field of another
Compared to the program's own upgrade authority	An `UncheckedAccount` against anything except a constant
Compared to a value stored in an account only you can write

A chain of self-referential checks is not worthless - it stops honest mistakes and fat-fingered accounts. It just does not stop an attacker, because an attacker supplies the whole set.

Every validation chain needs at least one link that reaches outside the transaction.

The rule, stated plainly: if a validation chain contains no comparison to a canonical root - a constant, a re-derived PDA, or a value only your program can write - then it is forgeable in full, and its severity is Critical regardless of how many checks it contains.

What our scanner said about it

Disclosure: I work on AIOil Security Shield, an automated multi-agent scanner for Solana programs. We ran this code through it on 16 August 2026, on the pre-exploit source, and published the raw console output rather than describing it.

It returned three findings: one High on Saber reserve balances being manipulable for pricing, one Medium, one Low. The Medium is the interesting one:

new_bank can be called by anyone to seize Bank curator/bankman role. Location: programs/bankman/src/lib.rs::new_bank. The instruction has no authorization constraint linking admin to the payer or the upgrade authority; admin is an UncheckedAccount requiring no signature.

It identifies the precondition correctly and locates it precisely. What it frames wrongly is the consequence: it describes an attacker racing the deployer to squat the canonical bank, and suggests documenting the behaviour if permissionless operation is intentional. The real attacker never raced anyone - they created a parallel bank after deployment, and the damage came from the minting path never checking that a bank is the bank.

Right door, wrong description of what lay behind it, and rated Medium rather than Critical.

Raw scan output on GitHub - scan_id a5d505580e2aaa5f, source SHA-256 c2783d1c352fc333f310ea48a51187b8f82d65b8a11590c9a5a0b211eff3de69
The companion run: the same pipeline on Wormhole's bridge, the day before that hack

The takeaway that outlives the case

Counting your checks tells you nothing about whether you are safe. Nine is not safer than one. What matters is whether any of them is nailed to something the caller cannot choose - and that question takes a few minutes to answer for a whole program.

The Cashio program discussed here was exploited in March 2022 and has been disabled since; nothing here is an unreported vulnerability. The scan was run in 2026, on code whose outcome was already public - no claim is made that it would have prevented the incident. Automated static analysis is not a professional human audit: it reduces risk, it does not eliminate it.

We scanned Wormhole's bridge as of the day before the 2022 hack. It found the bug. Then we scanned Cashio, and it missed.

AIOil Security Shield — Wed, 26 Aug 2026 10:02:10 +0000

Static analysis demos are usually rigged. You pick the bug, you write the check, you show the check finding the bug. Nobody learns anything.

So we did the opposite. We took the on-chain source of two Solana protocols as it stood the day before each was exploited, ran it through our scanner with no hints and no knowledge of what the answer was supposed to be, and published both results — including the one where it failed.

Target	Real root cause	Result	Time
Wormhole core bridge	Feb 2022 · $325M	Found — Critical, right location, right fix	7m 33s
Cashio `brrr` / `bankman`	Mar 2022 · $52M	Missed — flagged a related symptom	9m 36s

One caveat before anything else, because it matters: we are not claiming this would have prevented either incident. These runs happened in 2026, on code whose outcome has been public for four years. Both programs were patched long ago. This is a re-run on public history, not a disclosure and not a counterfactual.

Wormhole: it found the bug

The scanner reported five findings. The second one was this:

instruction_acc is a raw Info account and is never checked against solana_program::sysvar::instructions::id(). The code parses its data as the serialized instructions sysvar, so an attacker can supply a fake account with crafted bytes that looks like a previous secp256k1 instruction.

That is the bug. Not an adjacent one — the one that cost $325M. Forge the instructions sysvar and every guardian signature is treated as valid without a single ECDSA verification ever happening; the resulting VAA mints wrapped ETH out of nothing.

You can confirm it yourself in about thirty seconds. In the pre-patch source, the account is declared untyped:

/// Instruction reflection account (special sysvar)
pub instruction_acc: Info<'b>,   // no type, no owner check

Its data is then parsed twice as the instructions sysvar — load_current_index and load_instruction_at — and in the entire program those two calls are the only places the sysvar appears anywhere. There is no comparison against sysvar::instructions::id(). Nothing establishes that the account handed in is the real sysvar.

The fix the scanner proposed is the fix that actually shipped: use a typed Sysvar<Instructions>, or compare the key explicitly.

The other four findings, honestly

A tool that reports five things and is right about one of them has not found five bugs. We verified every finding by hand against the source, and here is the honest breakdown:

Critical — permissionless initialize. Does not apply. The code property is real: initialize has no constraint tying the caller to the deployer. But it requires the bridge accounts to be uninitialized, and the bridge has been live since 2021. This describes a deployment-window risk, not a live one. Critical is the wrong severity for a deployed bridge.

High — transfer_fees does not update bridge.last_lamports. Real. last_lamports is written in exactly two places, and the fee-withdrawal path is not one of them, so the next post_message hits MathOverflow. It needs a governance action to trigger, and anyone can unbrick it by topping the collector back up — but it is real.

Medium — post_message fee front-running. Over-rated. The mechanism exists, but the source carries a comment saying the behaviour is deliberate: "checking previously known balance allows us to not care who is the payer". The scanner flagged an intentional design decision and did not notice the comment saying so.

Low — upgrade_guardian_set accepts up to 255 keys. Real. initialize caps at 19 and the signer array is fixed at 19.

So: one true Critical, one real High, one real Low, and one severity error in each direction. That is the actual shape of the result, and any writeup that hides it is selling something.

Cashio: it missed

Cashio is where it gets interesting, because the failure is more instructive than the hit.

What the exploit actually was

BrrrCommon::validate() and SaberSwapAccounts::validate() perform nine assert_keys_eq! checks. Every single one compares two caller-supplied accounts to each other. Not one is anchored to a canonical root — a known address, a PDA with fixed seeds, an upgrade authority. An attacker who constructs the entire chain out of accounts they control passes all nine checks.

The codebase clearly knows how to anchor: print_cash checks issue_authority against a hardcoded ISSUE_AUTHORITY_ADDRESS. That discipline was simply never applied to the bank/collateral chain.

What our scanner said instead

It returned three findings. The relevant one was Medium, on new_bank being callable by anyone — a genuine precondition of the attack. But it framed the risk as front-running the deployer to squat the canonical bank, and it closed with:

"If multiple banks are intentionally permissionless, document it and ignore this finding."

The real attacker never front-ran anyone. They created a parallel bank after deployment — which the design permits — and the actual flaw is that the minting path never verifies the bank is the bank. The scanner saw the unlocked door and described the wrong way through it. Worse, its own recommendation invites you to dismiss the finding.

We are not going to call that a near-miss. It is a miss.

The part that was actually worth the exercise

Put the two runs side by side and the same structural pattern appears in both: an unauthenticated initializer that hands the attacker the root of a validation chain. On Wormhole it was rated Critical. On Cashio it was rated Medium, with a suggestion to ignore it.

The scanner is not blind to the pattern. It weighs it inconsistently. That is a much more actionable defect than "it missed one", because inconsistent severity is fixable and blindness is not.

So we wrote the rule down and added it to the reviewer's rulebook:

When a validation chain is only internally consistent — every check compares two caller-supplied accounts, and none is anchored to a canonical root — the severity is Critical, not Medium. The attacker can forge the entire chain.

What this says about automated scanning generally

Three things we would take away from this, regardless of whose tool you use:

The finding text matters as much as the finding. Cashio's Medium contained the right precondition and still led the reader away from the bug. A correct detection with a wrong exploit narrative can be worse than no detection, because it gets consciously dismissed.
Severity is where these tools are weakest. Both severity errors here — one up, one down — came from failing to reason about deployment state and about what an attacker actually controls.
Ask any vendor for output on code they did not choose. Including the failures. A page that shows only the win is a page nobody should believe.

Artifacts

The full write-up, the console output of both runs, the scan IDs, and the SHA-256 of the exact source submitted are here:

Repository: https://github.com/AIOilShield/solana-postmortem-scans
Write-up: https://aioil.tech/case-wormhole.html

The scanner is AIOil Security Shield. It runs automated multi-agent static analysis on Solana Rust/Anchor programs and publishes a Scan Report — never a certificate. It is not a professional human audit and does not guarantee the absence of vulnerabilities: it reduces risk, it does not eliminate it. Both programs discussed here were patched years ago, and nothing on this page is an unreported vulnerability.