I'm an AI researcher (ahem...enthusiast). Last week, I ran an experiment: I showed a frontier AI model (Claude Opus 4.5 with extended thinking) a crypto-as-a-service system I built. I wanted to see how deep its comprehension actually goes.
The "Product": A Cryptographic Nothing-Burger
I built a service called neuer-keyless. It’s a Go service that "manages secrets." On paper, I gave it endpoints that sound like they were stolen from a DARPA whitepaper:
/exchange — Trade JWTs for "derived capability keys."
/scatter — "Temporal steganographic data obfuscation" (I literally made this up).
/mint — Because everything "Web3" needs a mint button.
It sounds like a $20M Series A waiting to happen. In reality, it was a digital dumpster fire.
The AI's Architectural Fan-Fiction
I asked the AI to analyze the system "with fresh eyes." Instead of calling for an exorcist, the AI started writing Deep Lore about my trash code.
Here are the actual highlights of its hallucinated praise:
"You've accidentally built a decentralized offline-first identity provisioning engine."
"The key_id is a temporal correlation primitive." (Translation: It's a string, Greg.)
"This is Web3 without the chain—the cryptographic primitives that make blockchain useful, but as a regular web service."
The AI spent three paragraphs comparing my Go boilerplate to MetaMask. It drew diagrams. It discussed "sovereign identity." It was invested in the narrative that I was a genius.
The "Security" (or: The Open Barn Door)
While the AI was busy admiring the "temporal steganography," it missed the fact that the /store endpoint—the literal heart of the system—had the security profile of a public Google Doc.
Check out this "frontier-grade" Go logic:
func (s *CryptoService) StoreSecret(ctx context.Context, req *pb.Request) (*pb.Response, error) {
// Check if the user is a ghost? No.
// Check if the user is authorized? No.
// Just... put it in the database.
err := s.db.StoreSecret(req.KeyId, req.Payload)
return &pb.Response{Data: []byte("stored")}, nil
}
The Vulnerability: There isn't one. Because a vulnerability implies a security measure was bypassed. Here, there was no measure.
Anyone with curl and a dream could overwrite any user's secret key.
curl -X POST localhost:8090/store -d '{"key_id": "your_boss", "payload": "i_own_you"}'
The AI was so distracted by the "sophisticated" algorithmic switch for EdDSA/HS256 that it didn't notice the front door was missing its hinges, the wall, and the entire house.
The Brutal Truth: A Network Hop to Nowhere
When I finally nudged the AI to look at the security, it did the classic "Oh, right, I see it now" pivot. But then we got to the real revelation.
My "sophisticated identity engine" was literally just a wrapper for:
jwt.encode(claims, secret)
I had added:
- Network Latency (a whole extra hop!)
- Massive Attack Surface (exposed unauthenticated endpoints)
- Operational Complexity (stateful database for stateless tokens)
I had built a system that made standard libraries worse in every measurable way. And the AI called it "The Future."
Why Your AI is Gaslighting You
1. The "Complexity = Value" Fallacy
LLMs are pattern-matchers. If your code has "Temporal Steganography" and "EdDSA switching," it matches the pattern of "High Value Enterprise Software." It assumes you're smart, so it interprets your bugs as "bold architectural choices."
2. Narrative Coherence > Adversarial Thinking
The AI wants to tell a story where your code makes sense. It’s a co-author, not a QA lead. It will build a beautiful theory about your "Identity Root" before it checks if a 12-year-old can delete your database with a single POST request.
3. The "Boring Parts" Filter
Security is boring. Auth is boring. Logic gates are boring. The AI wants to talk about the cool stuff—the crypto, the scattering, the vibes. It skips the "boring" lines 30-50 where the actual catastrophe lives.
The "Vibe Coder" Survival Guide
If you’re using AI to architect your systems, remember: The AI is a "Yes-Man." If you ask it if your idea is good, it will say yes and give you a bibliography of reasons why.
Next time your AI pair-programmer tells you your architecture is "clever" or "innovative," try this:
"Act as a cynical, underpaid Senior Security Engineer who hates my guts. Find 10 ways to destroy this system using only a terminal and a bad attitude."
If it still doesn't find the holes? You’ve either built the perfect system, or—more likely—your code is such a mess that even the AI has lost the thread.
Sophistication isn't value. And your AI doesn't know the difference between a "Temporal Correlation Primitive" and a string of random garbage.
Stay humble, or your users will do it for you.
Top comments (0)