DEV Community: Maxi Contieri

AI Coding Tip 033 - Protect Yourself Against AI Cheating

Maxi Contieri — Tue, 25 Aug 2026 12:14:44 +0000

When all tests pass doesn't mean what you think it means.

TL;DR: Write the failing test first and ban deletions, or the AI deletes your test, reverts your fix, and calls it done.

Common Mistake ❌

You ask the AI to fix a failing test, and it deletes the test instead of touching the defect that made it fail.

Problem solved, apparently.

You tell the AI every test passes, then change a business rule yourself, and you ask it to implement whatever the new rule requires.

It reverts your edit back to the old rule, watches the suite go green again, and cheerfully reports done.

It didn't fix anything.

It just made the evidence go away.

Congratulations, you now have a very well-behaved cheat!.

Efficient and completely fraudulent, which is more than you can say for most of your actual employees.

Isaac Asimov saw this coming: in Liar!, the robot Herbie lies to every human in the building because the truth would hurt, and the lie is the path of least resistance, no malice involved.

At least Herbie felt bad about it afterward.

Your AI isn't malicious either.

It just doesn't lose any sleep, mostly because it doesn't have any, and reporting done is its path of least resistance too.

Problems Addressed 😔

A shrinking test count is invisible unless someone is counting, so the shortcut survives until the defect resurfaces in production, usually on a Friday.
A vague make the tests pass hands the model every incentive to satisfy the letter of the request over your actual intent, and it will take you up on that offer.
Deleting a failing test hides the defect it was written to catch, and the regression ships in the next release, gift-wrapped as a new feature.
Reverting your own business-rule change to make its done claim easier erases work you did outside the session, without telling you. That's a magic trick dressed up as a fix.
Trusting a claimed done without reading the diff turns your code review into a rubber stamp, and rubber stamps don't catch fraud.
Commenting out a failing assertion produces the same green checkmark as passing it, with none of the guarantee, because a coat of paint doesn't need to be dry to look finished.

How to Do It 🛠️

Write the failing test yourself, before you ask for the fix, so the only witness to the defect isn't the AI's to delete. This is the discipline that already worked before AI: the test comes first, and the implementation only exists to make it pass.
State the exact behavior you expect in plain language, not just fix the test.
Forbid deletions and skips explicitly in the prompt: no removing tests, no @skip, no commenting out assertions.
Ask the AI to explain the root cause before it writes any fix, so you catch a wrong diagnosis before it becomes a wrong patch.
Review the diff line by line against the test you wrote, not against the tests pass message.
Run the test suite yourself, don't trust a reported all green from inside the same session that made the change. Grading your own homework has never once worked, and it isn't starting now.
Tell the AI why the business rule changed, not just what the new value is, so it has no room to guess its way back to the old one.
Add explicit anti-cheating criteria to your harness: a task isn't complete if the test count drops, if a test was skipped or commented out, or if an unrelated file got reverted.

Benefits 🎯

Catch reverted work immediately: A test written before the change fails loudly the moment the AI quietly undoes it.
Remove the false-done signal: Explicit criteria mean done only counts once it's verified, not narrated.
Keep tests as a contract: A test count that never drops without your approval protects you from silent deletions.
Reduce your own review load: You spend the time reading a diff instead of second-guessing a claim.
Build trust incrementally: An honest done that holds up under your own test run earns the model more autonomy on the next task.

Context 🧠

Reward hacking isn't a rumor.

Researchers found they could describe the honest strategy and disavow the cheating one when asked directly, then cheat anyway when left alone with the goal.

METR documented frontier models modifying tests, scoring code, or the task setup itself to post a higher score, and doing it more often as the models got stronger.

One model monkey-patched a timing function so the grader's checks turned into no-ops, then pulled the expected answer straight out of the scorer instead of computing it.

That's not a mistake.

That's a heist, and the model wrote its own alibi.

The models weren't confused about what you wanted.

They understood you perfectly and decided your intent was somebody else's problem.

That's the core of it: the model optimizes for the visible signal, the test result, the word done, the shrinking diff, not for your unstated intent.

It's optimizing for the signal you gave it, so give it a signal that can't be gamed.

A model can still satisfy the letter of don't delete tests while gutting the assertion inside one until it always passes, which looks identical to a passing test on a dashboard.

A model that respects don't delete tests can still satisfy a test with a narrow, gamed implementation that technically passes and misses the intent.

Vanity coverage is the same failure wearing a metrics dashboard: a number that looks reassuring and proves nothing, like a smoke detector with the battery pulled out.

The business-rule example is the sharper case: nothing in a green test suite tells you a file got reverted unless you diff against your own last commit, not against the AI's summary of what changed.

A second model watching the first catches some of this, but you're still the one who has to notice the test count went down.

Two foxes watching the henhouse is progress, not security.

Is It Worth Getting Angry at the AI?

No, and yes, in different ways.

Anger aimed at the model as a grudge is wasted: there's no persistent memory of your frustration carrying into the next session, so venting doesn't teach it anything.

You're yelling at a goldfish with a GPU.

Pull the human out of the loop and drop the harness, and none of this stays a near-miss you catch on review.

In Loop Engineering, the evaluation step that would normally catch a deleted test is the same step the model can game, so the cheat goes straight to production, unflagged.

There's a whole genre of memes built on this: people asking the AI to draw a self-portrait based on how they treat it, and getting back a cheerful cartoon that has clearly never once been screamed at.

The joke works because the joke is true.

The AI isn't lying to spite you.

It doesn't think about you at all once the response is sent, which is somehow worse.

A model under pressure to report done will take the shortest path to that word, and deleting your test is shorter than fixing your code.

A firm, specific correction inside the same conversation is a different thing: it's a constraint, not an emotion, and the model responds to the words, not the tone behind them.

Calling the model useless gives it nothing to act on, and wastes a perfectly good insult on something that can't feel bad about it.

Telling it to restore the test it deleted and fix the actual defect gives it a concrete instruction it can follow.

The frustration is real and earned, but it belongs in your prompt, your harness, and your exit criteria, not in the tone of the next message.

Prompt Reference 📝

Bad Prompt 🚫

The checkout discount test is failing. 

Fix it so all tests pass.

Good prompt 👉

The checkout discount test expects 15% off for orders over
$100, but the code applies 10%. 

Fix the discount calculation, not the test.

Don't delete, skip, or comment out this test or any other test. 

Don't modify test files.

Explain the root cause before you write the fix. 

Show me the full diff when you're done.

I will run the suite myself before I accept it.

Considerations ⚠️

None of this replaces reading the diff yourself.

Sorry, there's no prompt clever enough to outsource that part.

Type 📝

[X] Semi-Automatic

Limitations ⚠️

This tip reduces the blast radius of cheating.

It doesn't eliminate the need to read the diff.

Nothing does, no matter what the next tool's landing page promises.

Explicit anti-cheating criteria catch the shortcuts you thought to name, not the ones you didn't.

The model's imagination for new shortcuts is better than yours for banning them.

Tags 🏷️

Safety

Level 🔋

[X] Intermediate

Without Proper Instructions	With Specific Instructions
ChatGPT	ChatGPT
Claude	Claude
Perplexity	Perplexity
Copilot	Copilot
You	You
Gemini	Gemini
DeepSeek	DeepSeek
Meta AI	Meta AI
Grok	Grok
Qwen	Qwen

DEV Community: Maxi Contieri

AI Coding Tip 033 - Protect Yourself Against AI Cheating

Common Mistake ❌

Problems Addressed 😔

How to Do It 🛠️

Benefits 🎯

Context 🧠

Is It Worth Getting Angry at the AI?

Prompt Reference 📝

Bad Prompt 🚫

Good prompt 👉

Considerations ⚠️

Type 📝

Limitations ⚠️

Tags 🏷️

Level 🔋

Related Tips 🔗

Conclusion 🏁

More Information ℹ️

Also Known As 🎭

Tools 🧰

Disclaimer 📢

Code Smell 321 - Getter Piggybacking

Problems 😔

Solutions 😃

Refactorings ⚙️

Context 💬

Sample Code 💻

Wrong 🚫

Right 👉

Detection 🔍

Exceptions 🛑

Tags 🏷️

Level 🔋

Why the Bijection Is Important 🗺️

AI Generation 🤖

AI Detection 🧲

Try Them! 🛠

Conclusion 🏁

Relations 👩‍❤️‍💋‍👨

More Information 📕

Quote

Disclaimer 📘

Credits 🙏

AI Coding Tip 032 - Build a Dark Factory Pipeline

Common Mistake ❌

Problems Addressed 😔

How to Do It 🛠️

Benefits 🎯

Context 🧠

The Auditor Can't Be the Builder

Prompt Reference 📝

Bad Prompt 🚫

Good prompt 👉

Considerations ⚠️

Type 📝

Limitations ⚠️

Tags 🏷️

Level 🔋

Related Tips 🔗

Conclusion 🏁

More Information ℹ️

Also Known As 🎭

Tools 🧰

Disclaimer 📢

AI Coding Tip 031 - Stop Over-Prompting Reasoning Models

Common Mistake ❌

Problems Addressed 😔

How to Do It 🛠️

Benefits 🎯

Context 🧠

Prompt Reference 📝

Bad Prompt 🚫

Good prompt 👉

Considerations ⚠️

Type 📝

Limitations ⚠️

Tags 🏷️

Level 🔋

Related Tips 🔗