Watch the fires burning on the far shore. Don't cross until they've burned themselves out.
— The 36 Stratagems, Watch the Fires Burning Across the River
Previously on this series: #3: Lena Walked Into an AI Deal. She Walked Out With Three Borrowed Knives. — Lena stopped a bad AI procurement by borrowing three knives. Then she founded VeriTest.
#7: P Watched an AI That Only Looked One Way. The 99.97% Was Real. It Just Missed Everything That Mattered. — P dismantled a 99.97% detection system — the number was real. It's just that "normal" was the loophole. (Something from Nothing)
Lena's screen had two browser tabs open. On the left, a technical blog post throwing down a challenge. On the right, counter-data from the competition. She was waiting for a third window to light up.
The Shore
VeriTest's office, afternoon. Lena sat at her desk, screen split in two.
On the left: CoverAI's CTO had posted a technical blog yesterday. "96.8% coverage: not the ceiling — the baseline." The final paragraph came with a comparison chart that strongly implied competitor TestShield's test set lost about thirty percent of its coverage in real deployment.
On the right: TestShield responded this morning. They'd open-sourced an evaluation script on GitHub. Run it, and it printed one line — "Your AI test coverage is lower than you think." No names. But everyone knew who they meant.
Both companies were AI test vendors VeriTest was trialing. Same pool of customers. Same evaluation dimension. Both were competing for the annual contract on Lena's desk.
Marcus Reed walked in: "The board's asking when we decide. Both sides are pushing."
Lena didn't answer. She was waiting for someone's reply.
That someone was P. There was a middleman — a test director Lena had met at an industry meetup, who knew how to reach P.
Four days ago, Lena asked him to pass along a message: "Anyone doing independent AI audits?"
The reply came the next day. "There's one. Goes by P. Buyer-side only."
Lena got the contact info, sent a short message — introduced herself, explained VeriTest's story — and waited.
One day of staring at her phone.
Her phone buzzed.
"Quote. Deadline."
Lena typed six words: "Two weeks. Two vendors. Anonymous. Your call."
P sent back a number. Lower than Lena expected. She didn't negotiate.
Fire
The public war between the two vendors escalated.
CoverAI published a whitepaper. "Why 96.8% isn't the ceiling — it's the baseline." They broke down their test set methodology in detail: data sources, classification strategy, manual review ratio. The end of the article casually noted that "certain vendors' test coverage drops significantly in real deployment environments." Name wasn't mentioned. But every parameter pointed at TestShield.
TestShield didn't write a response. They did something smarter — they open-sourced that evaluation script on GitHub. The repo description was one sentence:
"Run it. See your real coverage."
Stars hit 240 in three days. Someone in the comments posted their own result: an AI platform claiming 90%+ coverage came out at 54%.
Nobody named names. But CoverAI's PR team must've seen it.
Lena read everything from both sides. She saved the links in a folder called "Watching". She wasn't picking a side. She was waiting for P's data.
But she was also watching while she waited.
She opened a new document. The title line read: "If neither works" — no content yet, but the title was already there.
Wednesday. CoverAI's sales VP sent an email with an attachment — a churn report suggesting TestShield had lost several big accounts over the past two quarters.
"We've noticed some instability in the market. If VeriTest wants independent verification, we can arrange a joint assessment."
Lena didn't reply. She screenshotted the email. Saved it to the "Watching" folder.
Friday. Internal review. Marcus Reed asked: "Both sides are pushing. Which way are you leaning?"
"Still waiting on the last batch of validation data," Lena said.
She had a hunch by now. But a hunch isn't evidence — and without P's data, she couldn't say anything.
She was letting the fire burn a little longer. Giving herself more time to watch.
While Lena kept the fire burning — eleven days of it —
The Audit
P's audit ran on two independent tracks, converging into a cross-validation.
Track A — Public data reconstruction.
CoverAI's whitepaper laid out their test set methodology in detail: data source categories, labeling strategy, hard-case filtering rules.
P spent three days building a simulation dataset — combining CoverAI's publicly available benchmark subset with data streams from public APIs that matched the same categories, then reconstructed a proportional simulation following CoverAI's methodology. Same scale. Same distribution. Same classification system.
# CoverAI Methodology Reproduction - Simulation Dataset Config
# Source: CoverAI Whitepaper §2.3 "Test Set Construction Methodology"
dataset:
source_categories:
- transaction logs # 35% of total
- user behavior sequences # 28% of total
- system monitoring metrics # 22% of total
- long-tail edge cases # 15% of total ← CoverAI's actual whitepaper marked this as "TBD"
annotation_policy: "three-layer cross-labeling"
conflict_sample: "excluded" # following CoverAI's published methodology
sample_count: 50000
P ran the data through a custom evaluation framework — coverage logic aligned with CoverAI's methodology, but stripped of the vendor's own test set selection bias.
Result: 71.3%.
CoverAI claimed 96.8%. That's a 25.5 percentage-point gap.
Not fraud. Selective construction — CoverAI's test set included the high-scoring cases and excluded the long-tail scenarios, edge conditions, and annotation-conflict data. Not a fake number, a curated exam paper.
During the reconstruction, P noticed something: CoverAI's AI test engine was essentially an encoder-based pattern matcher. It had dense embedding coverage for high-frequency patterns like transaction logs and user behavior sequences, but almost no usable semantic representation for long-tail edge cases. The whitepaper marked the long-tail category as "TBD" — not necessarily a documentation gap, but P's findings leaned toward a different explanation: the model simply didn't have usable embeddings for that region.
P didn't draw conclusions. P just wrote down a number.
Track B — Historical result cross-validation.
Lena gave P two batches of VeriTest's test data from the past three months — sanitized. TestShield's reports on that data showed 82.1% and 84.6% coverage.
P re-ran the same data in a clean test environment.
Result: 56.2%.
That's a 28.4 percentage-point gap.
TestShield's open-source evaluation toolkit included the full runtime environment — different versions referenced different model snapshots.
P ran three controlled experiments to isolate the root cause:
Group 1: Re-ran the raw data against TestShield's current evaluation pipeline. Result: 56.2%.
Group 2: Re-ran the same data against TestShield's evaluation script from three months ago. Result: 79.8%.
Four terminal windows open on the screen. By the time P was tuning the third group's parameters, the sky outside had gone from daylight to dark. A mug sat next to the keyboard — no idea which cup of the day — gone cold.
Group 3: Cross-run — P's simulated CoverAI-style dataset through TestShield's current pipeline, and VeriTest's sanitized data through P's simulated CoverAI evaluation flow. The CoverAI-style dataset scored close to 94% on TestShield's pipeline; VeriTest's sanitized data scored close to 81% on the simulated CoverAI flow. Neither evaluation pipeline had a fundamental flaw — the issue was in the test sets and model versions on both sides. P also noticed the approximately 5pp gap between the 79.8% from the old script and TestShield's originally reported 84.6%, but that could just be sanitization noise and environment drift. The root cause was clear enough.
Only one direction made sense.
P confirmed it through version comparison: the sandbox had been silently updated after VeriTest submitted its data. P compared the old and new model outputs category by category, tracing the fine-tune direction — the new model had been fine-tuned on a vertical domain for another client. Precision on that vertical category jumped from 0.97 to near 1.0, pushing F1 from 0.89 to 0.94 — but the trade-off was a recall drop on general test data from 0.82 to 0.61. TestShield was still marketing the old evaluation results.
P confirmed through version hash comparison. The evaluation script from three months ago referenced model snapshot f7a3e92. The current build was different.
$ git log --oneline --all | head -3
f7a3e92 (tag: v2.1) update model snapshot to v3.1.2
# released: 2026-04-15
b4c8f71 (tag: v2.0) public eval script v2.0
# released: 2026-03-01
# VeriTest test data submission: 2026-03-25
# → sandbox updated model version after data was submitted
# → P's reproduction ran on the new version, old results were from the old version
# → new model biased toward vertical domain, general performance degraded,
# but TestShield kept marketing the old results
The new version was released three weeks after VeriTest's test data was submitted.
Cross-validation.
CoverAI's problem: curated test set that masked long-tail coverage gaps.
TestShield's problem: sandbox silently updated, evaluating new data with a shifted model while marketing old numbers.
Two different cheating paths. Same conclusion.
Both sides were untrustworthy.
P spent eleven days.
Day twelve. An email landed in Lena's private inbox. Three tables attached.
P didn't write a recommendation.
Table 1: CoverAI claimed 96.8% → actual 71.3% | gap 25.5pp
Table 2: TestShield claimed 84.6% → actual 56.2% | gap 28.4pp
Table 3: Cross-comparison | Different problems, similar magnitude
No advice. No conclusion. No "sign / don't sign."
P added a note in Table 3's margin: two independent paths producing similar gaps was a coincidence.
Just numbers.
Still Watching
Lena finished reading. She had two options:
- Use the report to pressure both sides — cut the price and sign.
- Use the report by staying silent — let them keep fighting, let them expose more.
She chose 2.
The fire wasn't big enough yet. If she showed her hand, both vendors would stop shooting. Wait another two weeks — let them burn each other down.
Not because option 1 was slower. Because option 2 was cleaner — if she moved now, both sides would know VeriTest had done an independent audit. And once that information leaked, P would be traced.
Lena didn't want P dragged into this.
She didn't reply to P's report. She printed the three tables. Locked them in a drawer.
Her phone lit up. CoverAI's sales VP followed up with an updated version of their churn report. "If VeriTest needs more data, we can arrange a joint assessment."
Lena read it. Flipped her phone over. Screen down.
That week, CoverAI published another case study and another technical blog. TestShield stayed quiet — two silent updates to their GitHub repo.
Lena didn't reply to any of it. She didn't open that drawer either. On day seven, her phone buzzed — P sent an address and a time. Lena had never been to that place.
The Third Cup
One week later. 4 PM. The Third Cup.
Lena showed up ten minutes early. Picked a seat near the back — not a corner, not by the window. A seat with a clear view of the door, but the door couldn't see her.
P walked in on time. Didn't greet. Sat down across from Lena.
The person behind the counter glanced over. Put two ceramic coasters on the counter. The handwritten note under one of them was still there — "I have coffee. You have a story."
P sat down, and the counter person had already placed an espresso cup on the counter. No question. P didn't order.
The person behind the counter looked at Lena.
"Long Black."
The barista filled hot water first, then pulled the shot. When it came up, Lena took a sip.
P slid an envelope across the table. A printed copy of the report — with two extra lines handwritten in the margin of Table 2. Lena glanced at it. Put the envelope away.
"You don't say much," Lena said.
P took a sip. Set the cup down.
"Your budget only buys so many words."
Lena smiled. Not approval — confirmation. She'd picked the right person.
She pulled out another envelope. Cash. P didn't count it, pocketed it.
P stood up.
"If neither works — you know its name."
Lena looked at P for two seconds. Didn't ask.
P walked out. The espresso cup still had a sip left at the bottom — P had set it down, not finished it.
Lena sat for a few minutes. Glanced at the half-empty espresso — still warm. Opened P's printed report. In the margin of Table 2, P had handwritten a single line:
"B's evaluation script from three months hashed to f7a3e92. The current version doesn't."
That's it. No commentary. No "they changed the model" — P just wrote a hash. Those who understood would understand.
Lena closed the report. Pulled out her phone. Found Marcus Reed's name. Typed:
"Hold both. I need two weeks."
Sent. Put the phone down.
The phone buzzed. Marcus Reed replied with one word: "Waiting."
Lena didn't respond. Finished the last of her coffee. Set the cup down, stood up, and walked out.
Paused at the door for one beat. Didn't look back.
The Crossing
Two weeks later.
Two signed letters of intent sat on the desk — non-binding, but the posture was right. Both prices had come down to where she wanted them. Payment terms stretched to double the original. Neither vendor knew what the other had been offered. Both thought they'd won.
Marcus Reed set the letters on her desk and paused: "You never planned to pick one, did you?"
Lena didn't answer. Marcus didn't ask again. He walked back to his desk, figuring out the answer on the way. The set of his shoulders told her she didn't need to explain.
When Marcus was gone, Lena opened the second drawer on the left side of her desk.
The top document was new.
The cover read: In-House AI Test Capability Build — V1.0
P's line — "You already know its name" — wasn't about another vendor. It was VeriTest itself.
The first draft took a weekend. The second week went to budgeting. CoverAI and TestShield both published fresh material during that time — she didn't open a single one.
Lena hadn't spent those two weeks watching them fight. The war between CoverAI and TestShield bought her time — both sides thought VeriTest was still evaluating, not that she'd stopped looking at them entirely.
Tucked in the appendix was a blank data source sheet.
This is Watching the Fires Burning Across the River — don't enter the game. Don't put out the fire. Wait for the far shore to burn itself out, then cross.
🤖 AI Post-Mortem
[36 Stratagems Tactical Database v3.1] Loaded
[Tactic Match] Watch the Fires Burning Across the River
[Analysis Mode] Full Situational Scan
━━━━━━━━━━━━━━━━━━━━
Tactic Match: ~90%
Operator: Lena (observer) + P (spotter)
Action: Lena stays out of the fight. P crosses the river to scout. Two vendors burn each other; Lena watches from the far shore before deciding to cross
Objective: Assess both vendors' real capability without revealing her position
Result: Lena buys a time window through silence, chooses to build in-house capability
Attention Migration:
- The shore (watching): Lena takes no side, issues no statements, participates in no public arguments. Both sides believe she's still evaluating
- The fire (scouting): P audits through two independent tracks plus cross-validation. Neither vendor knows they're being watched
- Cognitive binding: Both vendors fixate on each other. Neither notices VeriTest conducting an independent audit
Counter-Surveillance:
- Strategic assumption risk: If both vendors simultaneously stop attacking and collude to hide the truth, the observer loses perspective
- Detection probability: Low
- Attribution: Industry rivalry makes collusion impossible
- Information asymmetry mirror: Lena hides P's audit results vs. both vendors hide their real coverage data
- Methodology/tool limitations: P works exclusively with public data and sanitized historical records — no access to either vendor's internal systems. Conclusions rely on inference, not direct evidence
- External monitoring signals: P's anonymous report routed through private email. No network trace. Cash transaction, no banking link.
Core Insights:
- Two cheating paths, different mechanisms, similar magnitude: neither can be trusted
- Observer exposure status: Unexposed
- Information asymmetry mechanism: The observed party doesn't know it's being observed
- Core leverage: Silence
Next stratagem: Conceal a Dagger in a Smile
P.S. English isn't my first language. I use AI to polish the writing and smooth out the rough edges. Thanks for reading. ☕ Buy me a coffee
Top comments (7)
Great story! I really enjoyed how the plot unfolded and how it kept me guessing until the end.
I have a question out of curiosity. If the story hadn't gone in this direction, how else do you think it could have evolved? For example, if Lena hadn't found those logs, or if both AI suppliers had actually been completely clean, what path would you have taken? Would it have become more of a psychological battle, a technical investigation, or even a story about trust and human decision-making?
I'm also curious about your writing process. Did you have this ending planned from the beginning, or did the story naturally evolve as you wrote it? Were there any alternative plot twists or endings that you considered but ultimately decided not to use?
I'd love to hear your thoughts. It's always fascinating to learn how authors imagine different possibilities before choosing the final version. Looking forward to the next chapter!
Q1:
You have to put yourself in Lena's shoes here. She's the kind of person who sees everything as chess pieces — the two suppliers are pieces, P is a piece (a more useful one, but one she'll eventually pay for treating that way). I honestly think Lena would squeeze both suppliers dry and end up building her own testing platform out of the rubble. So yeah — it always comes back to Lena building her own AI platform, something she can actually trust.
Q2:
Great question. First, I already nailed down each character's personality in the series announcement post. Second, the 36 stratagems themselves are a natural constraint — kind of like setting boundaries when you're vibe coding with AI. The plot can go anywhere as long as it stays inside the stratagem's frame. That makes writing easier in some ways, but it also limits creative freedom. And on top of that, there are multiple interconnected threads running between the stratagems, so that's another layer of constraints.
Overall I have a pretty complete outline for the series — every character arc and every stratagem's story exists in some form. As for how many times I've revised it? Hard to say. Every time I publish a new story I go back and audit the whole outline, fill in gaps. I've definitely considered a lot of alternate plots — started writing, felt something was off, scrapped it and pivoted. Only to circle back to the first version sometimes 😂 Just like work — the boss always ends up wanting the first draft.
I really enjoyed reading this story; it felt like I was watching a movie. You truly write amazingly.
Haha, that's such high praise it almost makes me feel like I've successfully switched careers — from QA to professional writer. I'm gonna be waking up multiple times tonight just to smile about it 😂
Oh snap, 3 months ago... We all know whose model was chucked 3 months ago...
😏 Someone read between the lines. Let's just say the fine-tune looked great on paper.
First crossover. Two observers, one table. Neither says much — but the espresso and the hash say everything. 🃏