DEV Community

Cover image for "AI thinks" is the most dangerous phrase in software engineering right now
Lucas
Lucas

Posted on

"AI thinks" is the most dangerous phrase in software engineering right now

Let me give you the one sentence that I want tattooed on every IDE and pinned to every engineering team's wall:

Artificial intelligence, as it exists today, is a system for recognising patterns in data and using those patterns to make predictions.

That's it. Not reasoning. Not understanding. Not consciousness. Pattern recognition. Prediction. At extraordinary scale, but prediction nonetheless.

Why does this matter practically? Because the failure modes of prediction are categorically different from the failure modes of reasoning.

A reasoning system that gets something wrong fails in traceable ways: a faulty premise, a logical gap, a missing piece of evidence. You can find the error and fix it.

A prediction system that gets something wrong produces an output that is statistically consistent with correct outputs. The wrong answer looks exactly like the right answer. It's formatted the same. It's expressed with the same confidence. The code compiles. The explanation is coherent.

The bug is subtle. The hallucinated library name sounds real. The fabricated API signature matches the pattern of legitimate API signatures so closely that a developer under time pressure won't catch it.

This is why 45% of all deployments linked to AI-generated code led to problems in 2025, according to Harness's State of Software Engineering report. Not because the code obviously fails. Because it plausibly succeeds, right up until it doesn't.

Here's the hierarchy you need to hold clearly:

  • AI = any machine simulating intelligent behaviour (umbrella term)
    Machine Learning = learns from data instead of explicit rules

  • Deep Learning = multi-layer neural network architecture

  • LLMs = deep learning on text, trained to predict the next token

Claude, GPT-5, Gemini, they are all doing one thing, billions of times per second:

Predicting what token comes next.

That's the mechanism behind every function it writes for you. Every architecture suggestion. Every code review. Sophisticated? Yes. Useful? Enormously. A substitute for reasoning? No.

The engineer who holds this clearly catches the confident wrong answer before it ships.

The engineer who thinks the model "knows" things approves the PR at 11pm on a Friday.

Tomorrow: how we got here; a 70-year story that makes right now make sense.

Top comments (2)

Collapse
 
mehmetcanfarsak profile image
Mehmet Can Farsak • Edited

This is such an important article. The prediction vs reasoning distinction is spot-on — especially when working with AI coding agents.

I've seen the "confident wrong answer" problem firsthand with Claude Code. It'll write code that compiles and looks correct but has subtle architectural flaws. The wrong answer looks exactly like the right answer, just like you describe.

That's actually what led me to put together a small plugin (Brainstorm-Mode on GitHub under mehmetcanfarsak). It forces a separation between the thinking phase (where you actually reason about architecture and trade-offs) and the execution phase (where the agent generates code). During brainstorming, write/edit/bash tools are blocked so you can have a real discussion about the problem instead of watching the agent race ahead with plausible-but-wrong implementations.

It's pretty lightweight — just a settings.json config that plugs into the Claude Code hooks system. Not a complete solution, but it enforces the "reason before you predict" discipline that this article is talking about.

Collapse
 
mehmetcanfarsak profile image
Mehmet Can Farsak • Edited

Another great point — the prediction-vs-reasoning distinction really resonates.

This connects to something I've seen in practice: during brainstorming, AI agents treat their predictions as actionable plans and start executing before you've even reviewed them. It's like the agent is "approving the PR at 11pm on a Friday" but in real-time.

I put together a small plugin for that exact problem (Brainstorm-Mode on GitHub under mehmetcanfarsak). It blocks tool execution during the thinking phase — so the agent can still explore ideas and predict what to do, but can't actually make changes until you say go. It's not a replacement for reasoning, but it does enforce that "reason before you act" discipline you're talking about.

Pretty lightweight setup (Python/TypeScript, works with Claude Code and Codex). Worth checking out if you've ever had an agent "helpfully" break something while you were still thinking through the approach.