DEV Community

Cover image for I Let an AI Audit My Password Vault. It Lied to Me With Total Confidence. 🔐
Akhouri Anmol Kumar
Akhouri Anmol Kumar

Posted on

I Let an AI Audit My Password Vault. It Lied to Me With Total Confidence. 🔐

#debugging #security #opensource #python

🚨 "Brother, I read the whole code carefully. These bugs are real."

That's how it started.

I'd been building ATLOCK, a Windows security suite with an encrypted password vault, file guard, 2FA, the works. v5 was nearly done. So I did what every developer does at some point: I pasted 8,500 lines of Python into an AI and asked,

"Any bugs?"

It came back with six. Emojis, severity colors, function names, even a smug little "I didn't invent these, you can verify them yourself."

Reader, I took that offer literally. 😏

🧪 The Rules of the Game

For every claim I did one thing: open the actual file and check.
No vibes. No trust. Just grep and line numbers.

Final score:

# Claim Verdict
1 Vault recovery key crashes 🔴 Real
2 Watchdog silently skips services 🤥 Fiction
3 Enabling 2FA locks you out 🟠 Real
4 HIBP request on every keystroke 🟡 Real
5 Wrong key derivation for v2 vaults 😇 Real, but harmless
6 Factory reset race condition 🟡 Real

Four real bugs, one fake, one harmless. Let's go through them. 🍿

💥 Bug #1: The Feature That Could Never Have Worked

Creating a Vault Recovery Key called this:

enc_b64 = base64.b64encode(bytes(self._sess.view())).decode("ascii")
Enter fullscreen mode Exit fullscreen mode

And SecureBuffer looks like this:

class SecureBuffer:
    def get_bytes(self) -> bytes: ...
    def wipe(self): ...
    # view()? Never heard of her.
Enter fullscreen mode Exit fullscreen mode

Click "Create Vault Recovery Key" → AttributeError. 💀
It's the one feature meant to save you when you forget your password, and it crashed on creation.

Fix: one line.

enc_b64 = base64.b64encode(self._sess.get_bytes()).decode("ascii")
Enter fullscreen mode Exit fullscreen mode

Lesson: a recovery feature nobody tests is a recovery feature that doesn't exist.

🔒 Bug #3: Enabling 2FA Locked Me Out of My Own Vault

This one is my favorite, because it's sneaky.

When you enable TOTP in Settings, the app creates a fresh PasswordVault(), writes the 2FA config to disk, and shows you a lovely QR code. Everything looks perfect.

But the main app is holding a different vault object, loaded once at startup, with a memory that has no idea 2FA now exists.

So you lock the vault, try to unlock it, and your own app says:

"TOTP is required by your security policy but not set up. Access denied." 🙃

Yes. The policy says "2FA required", the stale in-memory copy says "2FA doesn't exist", and the vault, being a good fail-closed security tool, slams the door on you.

Funny part: it's the safe kind of bug. Annoying, but it never leaked anything. A password manager that locks you out beats one that lets everyone in. 😅

Fix: re-read the 2FA config from disk right before checking the second factor.

def _sync_mfa_from_disk(self):
    with contextlib.suppress(Exception):
        disk = atlock_read_json(self._path, None)
        if isinstance(disk, dict):
            if isinstance(disk.get("mfa"), dict): self._db["mfa"] = disk["mfa"]
            else: self._db.pop("mfa", None)

# in unlock(), right before the 2FA check:
self._sync_mfa_from_disk()
ok2, msg = atl_second_factor(self._db, master, "Vault unlock", ask_code=self.ask_code)
Enter fullscreen mode Exit fullscreen mode

Bonus: this also covers someone enabling 2FA from the CLI while the app is open.

🌐 Bug #4: Typing a Password Sent a Small Army of Network Requests

When you add a vault entry, ATLOCK checks your password against Have I Been Pwned. Great feature. Terrible implementation:

self._pw_e.bind("<KeyRelease>", self._on_password_change)  # every. single. key.
Enter fullscreen mode Exit fullscreen mode

Type a 12-character password and the app spawns about 7 threads and 7 API calls. Rate limits and UI lag, free of charge.

Fix: a classic debounce. Wait until the user stops typing for 700 ms, then check once.

self._hibp_after = self.after(700, lambda: self._run_hibp(pw))
Enter fullscreen mode Exit fullscreen mode

If the password changed while the request was in flight, the stale result is thrown away. No more flickering "✓ Safe" labels for passwords you've already deleted.

🗑️ Bug #6: Factory Reset vs. The Process That Wouldn't Leave

Factory Reset worked like this:

  1. Spawn a new process that deletes everything
  2. Close the current app

The problem: step 1 happens while the old app still holds its file handles. On Windows, deleting a file another process still has open gives you PermissionError, and you get a half-erased reset. Spooky. 👻

Fix: the reset process now receives the parent's PID and politely waits for it to exit.

subprocess.Popen([sys.executable, sys.argv[0], "--reset",
                  f"--reset-wait-pid={os.getpid()}"])
Enter fullscreen mode Exit fullscreen mode

On Windows it waits on the process handle (up to 10 seconds) before wiping anything. Manners matter, even in a destructive operation.

😇 Bug #5: Real, But Nobody Is Ever Affected

The claim: Quick Lock always derives the master key with the legacy v1 function, even for v2 vaults.

True in the code. But I traced who can ever reach it: TOTP and FIDO2 setup require a v3 vault. A legacy v2 vault can never have a second factor, so there's nothing to unwrap with the wrong key.

So: technically inconsistent, practically unreachable. I left it alone. Knowing what not to fix is also engineering. 🧘

🤥 Bug #2: The One That Never Existed

The most dramatic claim:

"Watchdog calls DefenderGuard.run_ps, but the method is _run_ps! It silently skips services and scheduled tasks! A big hole in persistence detection!"

It even offered proof: "the WMI function spells it correctly, so the other two are obviously typos."

I searched the entire file:

grep -n "DefenderGuard\.run_ps" ATLOCK_V5.py
# (nothing)
Enter fullscreen mode Exit fullscreen mode

Zero results. Every call was already _run_ps. The bug was invented, complete with a convincing explanation of how it happened and why it matters.

And the best part? I fed the fixed file back to the AI, and it confidently re-confirmed all the old bugs, including the fake one, even the ones I'd already fixed. It even reproduced the typo with a stray space (DefenderGuard. run_ps) that exists nowhere in my code. 🤡

🧠 What I Actually Learned

1. AI code review is a great scout, and a terrible judge.
It found four real bugs I'd missed. That's genuinely valuable. It also invented one with the same confident voice. You can't tell them apart without checking.

2. Demand line numbers.
A real bug can be pointed at. If a claim has no line number, it's a rumor.

3. grep is a lie detector.
Every claim here took about ten seconds to verify. Ten seconds beats a week of "fixing" a bug that doesn't exist.

4. Trace before you fix.
Bug #5 was real and still not worth a patch. Reachability matters more than correctness-in-the-abstract.

5. Fail-closed bugs are the good kind of bad.
When your security tool breaks, it should break shut.

🛡️ So... What's ATLOCK?

ATLOCK is a Windows security suite by Akhouri Systems: an Argon2id-based encrypted password vault, file guard, TOTP and FIDO2 2FA, panic/decoy vault, breach checks, startup watchdog, and more.

  • 📦 v4 is out now: Download ATLOCK v4
  • 🔥 v5 (everything you just read about) is coming soon, and I'm releasing it when Akhouri Systems crosses 1,000 downloads.
  • 📊 Current progress: 967 / 1000

That's 33 downloads away. If you've ever wanted to be the reason a security tool ships its next version, you know what to do. 😄

⭐ Repo: github.com/Akhouri-Anmol-Kumar/ATLOCK

💬 Your Turn

Has an AI ever confidently "found" a bug that didn't exist in your code? Drop your best hallucination story below. 👇 I'll start a collection.

Built by Akhouri Anmol Kumar, one very suspicious grep at a time. 🔍

Top comments (1)

Collapse
 
akhourianmolkumar profile image
Akhouri Anmol Kumar •

mustafa bro I'm preparing for battle