DEV Community

Cover image for I gave my 10,000-line security app to 5 AIs. They scored it 8 to 10. Here's what I learned.
Akhouri Anmol Kumar
Akhouri Anmol Kumar

Posted on

I gave my 10,000-line security app to 5 AIs. They scored it 8 to 10. Here's what I learned.

Description: "ATLOCK v5 is a single (.exe), pure-Python security suite for Windows. Five AI models reviewed it and disagreed. I'm publishing the scores, the internals, and the weak spots."

I built a security app alone. It is one Python file(packed in single .exe) with 10,011 lines (about 745 KB<-- python file size).

I'm not a security company and I haven't had an audit. What I did have was a question I couldn't answer myself: is this actually any good?

So I gave the full source to five AI models and asked for one number out of 10.

Model Score
Qwen 10 / 10
Gemini 10 / 10
Claude 8.5 / 10
Grok 8.5 / 10
ChatGPT 8 / 10

Average: 9.0. The spread is the interesting part.

I'm not posting this to say "look, 9 out of 10". An AI score is an opinion, not an audit. I'm posting because the disagreement says more than the number does, and because I'd like people who know more than me to try to break this.


What ATLOCK v5 is

A Windows-first security suite. No installer, no kernel driver, one .py file (shipped as an .exe).

  • File Guard: lock files and folders with a password, optionally plus a keyfile or hardware key, so you need both.
  • Password vault with Argon2id, TOTP and FIDO2 (YubiKey etc.) support.
  • Quick Lock for fast protection of files you open often.
  • Recovery keys you can split into shares (Shamir, e.g. 3 of 5).
  • Public-key encrypt, sign and verify from the command line.
  • Canary / decoy files, a background watchdog, and a device health check (BitLocker, hibernation, cloud folders).
  • CLI for everything: --selftest, --fuzz, --verify, --shred, --rotate-key and more.

ATLOCK_v5 weakness ------

"A same-user attacker who can run code as you can modify this script and read this process's memory... they cannot stop a determined local attacker."

I put that in the first lines on purpose. If a security tool won't tell you what it can't do, don't trust it.


The technical bits, no brochure

Container format (v3.1)

Files, folders and Quick Lock items all use the same format:

  • chunked AES-256-GCM
  • key commitment
  • a per-seal chunk subkey
  • header-bound AAD
  • independent per-chunk authentication
  • optional Padme padding, so file size leaks less

Older v1, v2 and v3.0 containers still open. Migration is automatic.

Argon2id, with a ceiling as well as a floor

_ATL_ARGON2        = dict(time_cost=3, memory_cost=65536,  parallelism=4)  # floor
_ATL_ARGON2_STRONG = dict(time_cost=4, memory_cost=131072, parallelism=4)  # vault / recovery
KDF_MAX            = dict(time_cost=6, memory_cost=524288, parallelism=8)  # ceiling for UNTRUSTED headers
Enter fullscreen mode Exit fullscreen mode

The third line is the one I'm proudest of. A container header is attacker-controlled input. If you trust the KDF parameters inside it, someone can hand you a doctored file that makes your machine burn gigabytes of RAM. ATLOCK enforces a minimum and a maximum, and the self-test covers it.

It tests itself

--selftest ran 41 checks when the AIs ran it, and all passed. They include:

  • frozen old-format containers still opening
  • padded containers with a wrong-length pad being rejected
  • shares from earlier builds still rebuilding a recovery key
  • a folder lock refusing to continue (and keeping your original) if a file changes mid-lock
  • the clock and lockout store not deadlocking under concurrent use
  • a failed ACL on a non-private path failing closed

--fuzz throws random corruptions at the container parser. The goal is simple: never crash, never silently corrupt.


Why I think the scores split

I don't know for sure, but here's my read.

The 10s are probably responding to what's visible: the crypto choices, the test coverage, the paranoid handling of untrusted input.

The 8s are probably responding to what isn't provable from a static read:

  • One 745 KB file is hard to maintain and hard to audit.
  • Pure Python can't reliably wipe secrets from RAM. Strings and bytes are immutable, and the interpreter copies them.
  • The Windows-only paths (DPAPI, Windows Hello, registry, ACLs) can't be exercised outside Windows.
  • Nobody has line-by-line audited all 10,011 lines, the AIs included. The Claude rating said so itself: it came from reading part of the file and running the tests, not from reading everything.

I think the 8.5s are the more honest ones. I'm also glad the 10s exist, because they tell me the design reads well.


Known limits, in my own words

  • A same-user attacker running code as you wins eventually. That's true of every pure-Python tool.
  • Secure shredding on SSDs and copy-on-write filesystems can't be guaranteed. Use BitLocker.
  • Quick Lock is an application-level gate, not password-strength protection. For that, use File Guard with a real password.
  • Nothing is code-signed, so Windows SmartScreen may complain.
  • There is no external audit. That's what I'm asking you for.

Try it, or try to break it

Download (Windows .exe, zipped):
https://github.com/Akhouri-Anmol-Kumar/ATLOCK/releases/download/v5.0.0/ATLOCK_v5.zip

If you'd rather not run an unsigned binary from a stranger (reasonable, I wouldn't either), then no problem you can leave it. But honestly, I request you to download it and use it in addition break it.




Start with `--selftest` and `--fuzz`. They're the quickest way to see whether the claims above hold on your machine.

---

## What I'd love to argue about

1. **Is a score from an AI worth anything?** Two models said 10 and one said 8 about the same code. Which would you trust, and why?
2. **Is a security tool in pure Python a reasonable idea,** or a trap because of memory handling?
3. **Single file: feature or sin?** I like that it's inspectable end to end. I know it's also a maintenance cost.
4. **If you had one hour, where would you attack it first?** The KDF handling, the container parser, the rollback and lockout logic, or Quick Lock?
5. **What would you need to see before trusting it** with real data: an audit, reproducible builds, code signing, something else?

Tell me what's wrong with it. That's more useful to me than another 10.
Enter fullscreen mode Exit fullscreen mode

Top comments (33)

Collapse
 
akhourianmolkumar profile image
Akhouri Anmol Kumar •

@xulingfeng any views on it?
I gave my .py of ATLOCK_v5 to those AIs(mistakenly I forgot to give my codes to deepseek)
but just now I gave it to deepseek too and it rated 9/10)

Collapse
 
xulingfeng profile image
xulingfeng •

Collapse
 
akhourianmolkumar profile image
Akhouri Anmol Kumar •

yeah an doubt?

Thread Thread
 
xulingfeng profile image
xulingfeng •

Thread Thread
 
akhourianmolkumar profile image
Akhouri Anmol Kumar •

Collapse
 
technogamerz profile image
𝐓𝐡𝐞 𝐋𝐚𝐳𝐲 𝐆𝐢𝐫𝐥 •

Again This Meme.....

Thread Thread
 
xulingfeng profile image
xulingfeng •

🤣

Thread Thread
 
akhourianmolkumar profile image
Akhouri Anmol Kumar •

bro @technogamerz @xulingfeng which meme?

Thread Thread
 
technogamerz profile image
𝐓𝐡𝐞 𝐋𝐚𝐳𝐲 𝐆𝐢𝐫𝐥 •

@xulingfeng meme.

Thread Thread
 
akhourianmolkumar profile image
Akhouri Anmol Kumar •

didi, I scored 80/80 in science only 80 in class.

Thread Thread
 
akhourianmolkumar profile image
Akhouri Anmol Kumar •

aura 📈📈📈🖐️🤚

Thread Thread
 
akhourianmolkumar profile image
Akhouri Anmol Kumar •

@xulingfeng bro you did gaokao exam?

Thread Thread
 
akhourianmolkumar profile image
Akhouri Anmol Kumar •

@nyaomaru once said about why devs feel lazy to write
@xulingfeng is right example of it. bro you just liked my question that's it. 🫵🔥

Thread Thread
 
technogamerz profile image
𝐓𝐡𝐞 𝐋𝐚𝐳𝐲 𝐆𝐢𝐫𝐥 •

Congrats 🎉👏🏻

Thread Thread
 
akhourianmolkumar profile image
Akhouri Anmol Kumar •

about what?

Thread Thread
 
technogamerz profile image
𝐓𝐡𝐞 𝐋𝐚𝐳𝐲 𝐆𝐢𝐫𝐥 •

You scored 80/80!!!! lol

Thread Thread
 
akhourianmolkumar profile image
Akhouri Anmol Kumar •

oh nice, and LOL used by a IITian
🤣☠️

Collapse
 
maame-codes profile image
Maame Afua A. P. Fordjour •

Love that you put the weaknesses right at the top, that alone makes me trust it more than most tools. For your first question, I’d trust the 8.5 more. A 10 usually means it didn’t look hard enough. Also the KDF ceiling for untrusted headers is a really smart detail, not something most people would think about.

Collapse
 
akhourianmolkumar profile image
Akhouri Anmol Kumar •

yeah.
that's why my tagline is "I build what others forgot to fix"

Collapse
 
maame-codes profile image
Maame Afua A. P. Fordjour •

Great read Akhouri!

Thread Thread
 
akhourianmolkumar profile image
Akhouri Anmol Kumar •

thanks.

Collapse
 
ivannovazzi profile image
Ivan Annovazzi •

The spread is the most honest part of this. A 10/10 from a model on 10k lines it read in one context window tells me more about the model than the code. Curious if Claude's 8.5 came with specific findings or just vaguer hedging? Also asking AIs to break it with an adversarial prompt instead of score it would probably surface way more than the number did

Collapse
 
akhourianmolkumar profile image
Akhouri Anmol Kumar •

yeah I've done that before the ATLOCK_v5 release date.
I said almost 3-4AIs (primarily ChatGPT & Claude-AI)
and they tried to break it and even successfully done it.
but later I almost fixed all the break points still some breakpoints are still present but it's all because of it is built on python.
still I would love if you try it and find bugs and report me.
and yeah it's unsigned so you may get smart bluescreen warning and if you feel insecure of running it anyway then no issues but I request you to try it and it's safe for sure others have also downloaded It I've also ran virustotal checks it is safe.
so at last, its all on you if you feel to try it then please if not then no issues you can be still the part of the akhouri systems community

Collapse
 
mona_d_4222dda374567263b profile image
Amber D. •

Thanks for publishing this with the weaknesses up front, Anmol. Leading with the same-user attacker limitation is the right call, and the "ceiling as well as a floor" on the Argon2id parameters is a smart detail. Treating the container header as attacker-controlled input and capping the KDF cost closes off a real denial-of-service vector that plenty of tools overlook. Shipping --selftest and --fuzz alongside the tool is also a good way to let people check your claims rather than take them on faith.

On your questions, I'd trust the 8.5s over the 10s, mostly because the Claude score came with an admission that it hadn't read the whole file. An AI score tells you the design reads well, not that it holds up. For the one-hour attack, I'd start with the container parser and the rollback/lockout logic, since those are where state handling and edge cases tend to hide bugs, and the fuzzer can only cover so much of that. To trust it with real data, I'd want an external audit plus reproducible builds, so people can verify the .exe matches the source.

One suggestion: splitting the single file into modules, even if you still package it as one .exe, would make independent review far more realistic. Good luck with it!

Collapse
 
akhourianmolkumar profile image
Akhouri Anmol Kumar •

hmm, modulation.... it's a cons point which all AIs said too.
I should have make it in module strucuture.

Collapse
 
amintacccp profile image
tamina •

The ceiling on the Argon2id parameters is the detail I'll remember — treating the container header as attacker-controlled input is exactly the kind of thing a score out of 10 never captures. I also liked your read that the 8.5s are the more honest scores: they were grading what can't be proved from a static read, like secrets staying in RAM in pure Python. Publishing --selftest and --fuzz alongside the scores makes the claim checkable instead of just quotable.

Collapse
 
akhourianmolkumar profile image
Akhouri Anmol Kumar •

Thanks I love that you liked the 8.5 rating and yeah I also think 8.5 is most reasonable rating for it.

Collapse
 
amintacccp profile image
tamina •

Thanks! The Argon2id part is what pushed it up for me — a lot of tools still default to weaker hashing and call it done. Curious whether you re-tested after tuning the memory/time parameters, or kept the defaults?

Collapse
 
kartik-nvjk profile image
Kartik N V J K •

Publishing the spread instead of only the 9.0 average was the right call, because a 10 next to an 8 tells me as much about the judges as about the code. When I've used models to score code, rewording the question alone moved the numbers a lot. Did all five get the exact same prompt? I'd also like to know whether the lower scorers pointed at the same weak spots or at different ones.

Collapse
 
akhourianmolkumar profile image
Akhouri Anmol Kumar •

yeah exact this prompt: read whole code and brutally rate my ATLOCK_v5 in a scale of 10.
gemini took 2 prompts to understand lol so for gemini after the first prompt I added one more: a real brutal rating I want (because gemini just said I didn't read the whole code but according to some upper sources he gave me score)
my personal opinion gemini is just 💩 so do not consider its rating
I prefer claude, chatgpt, grok's rating.

Collapse
 
dgold profile image
D. Goldberg •

Respect for putting the full source in front of multiple models and publishing the spread instead of just the average. The honest framing of limits—especially around pure-Python memory handling and same-user attackers—is more useful than any 10/10. Looking forward to seeing where people try to break it first.

Collapse
 
akhourianmolkumar profile image
Akhouri Anmol Kumar •

I still have no data sorry bro till now no one has break it.
yeah people are reporting me some suggestions but till now not a serious break found ☠️

Collapse
 
cekuu35 profile image
Cenk KURTOĞLU •

The honest scope line (“a same-user attacker… cannot stop a determined local attacker”) is worth more than the 9.0 average, and I think you know the disagreement is the signal — the useful cut isn’t the number, it’s which parts the models disagreed on and whether any of them produced a concrete exploit instead of a vibe. Two things would turn this from a review exercise into a security artifact: (1) publish a --selftest / --fuzz run with reproducible output, so someone can re-run the claim rather than re-read a score; (2) keep a public “known-broken” list — the same instinct as your weakness note, but living where it can grow. Reviewers trust a tool that documents its own failures far more than one that scores well, and the fact that models handed it 10/10 while you documented a real limitation is exactly the disagreement that should make a reader trust you and distrust the score. For a single-file Windows tool, the memory/clipboard paths sitting right next to that same-user caveat are where I’d expect the first genuine finding.