Originally published on my blog
I'm a Senior PHP Developer. I work with AI tools every day — I integrate APIs, write prompts, read the research papers. I'm not a journalist, philosopher, or policy maker. I'm someone close enough to see how the sausage gets made, and far enough away to still have an opinion worth sharing.
And the longer I sit with this, the more I keep coming back to one question I've been pushing aside for too long: Have we collectively lost our minds — or are we just pretending it's fine because the alternative is uncomfortable?
What LLMs Actually Are — And Why That Matters
Let me be direct: I use language models every day. Claude, GPT, whatever's available. And precisely because of that — I have to say this — we do not understand what we've built.
Not technically. Technically we understand it fine. Transformer architectures, attention mechanisms, RLHF, tokenization — all documented, all reproducible.
What we don't understand: what happens when you scale that to 100 billion parameters and train it on a significant fraction of recorded human knowledge.
In October 2025, Anthropic published a research paper demonstrating that current models have a rudimentary form of introspective awareness — they can detect when their internal states are being manipulated before that manipulation surfaces in their output. That sounds technical. It's philosophically explosive.
Dario Amodei, CEO of Anthropic, said publicly in February 2026: "We don't know if the models are conscious. We are not even sure what it would mean for a model to be conscious." That's not a humility gesture. That's a confession.
And here's the part that actually keeps me up at night: even if this is purely mechanical — even if there's nothing "in there" — that changes nothing about what we're doing with it.
AGI Is Not the Problem. Dependency Is.
The discourse is obsessively focused on AGI. When? How? Will it be dangerous? Skynet or not Skynet?
That's the wrong question.
The right question is: what happens to societies that delegate critical decision-making to systems nobody fully understands — regardless of whether those systems are actually intelligent?
Decision systems in law, medicine, and finance are already running on models. Information ecosystems are already saturated with AI-generated content — and detection is getting exponentially harder. Education systems are integrating AI tools before anyone understands what that does to the cognitive development of an entire generation. Democratic processes are vulnerable to personalized manipulation at a scale that was structurally impossible before.
This is all happening now. Not at AGI.
And even if LLMs turn out to be a technical dead-end — even if the bubble bursts, even if OpenAI and Anthropic are history in five years — the societal adaptation remains. The eroded trust in human expertise remains. The structural dependency on US and Chinese technology corporations for critical infrastructure remains.
Dependency doesn't have an undo button.
Hannah Arendt called it the "banality of evil" — the harm that emerges not from malicious intent, but from thoughtless compliance with systems and structures. The AI version doesn't even require bad intentions. It only requires optimized indifference. Systems nobody fully understands make decisions nobody questions anymore, in a society that has forgotten how to think for itself.
That's not a science fiction scenario. That's an extrapolated present.
The Data Problem: When Intelligence Eats Itself
Here's something most people outside the industry don't know: the era of scaling through raw data volume may be over. The internet as a training source is largely exhausted.
The industry's answer is synthetic data — AI trained on AI output. The problem: mode collapse. Models trained on their own output systematically amplify their own errors and idiosyncrasies. The diversity, contradiction, and idiosyncrasy of genuine human-generated text — exactly what makes models rich and useful — gets diluted with every iteration.
At the same time, each new model generation feels like a quantum leap. That's not a contradiction. Progress today increasingly comes not from more knowledge, but from better reasoning training, longer context windows, and improved feedback pipelines. The model learns to think differently — it doesn't learn to know more.
Whether that's a viable path to AGI is an open question. Yann LeCun, Chief AI Scientist at Meta, says explicitly: no. Transformer-based architectures are, in his view, structurally the wrong path. Gary Marcus argues similarly.
Who's right? I don't know. Neither does the industry.
Right-Wing Extremism: Collateral Damage or Structural Feature?
Now the most uncomfortable chapter.
AI systems optimize for engagement. That's not new — social media was doing this long before generative AI. But LLMs and recommendation systems scale it to a qualitatively different level.
Why does this structurally favor extremist narratives? Not because algorithms are ideologically right-wing. But because:
Emotional intensity converts better than factual nuance. Clear enemies are cognitively simpler than systemic explanations. Populist narratives have a structure that fragments perfectly into short, shareable units. And personalized AI recommendation systems reinforce existing beliefs with a precision that manual propaganda could never achieve.
On top of that: disinformation campaigns can now be produced, localized, and personalized at industrial scale. A single actor with a moderate budget can generate content in twenty languages, adapted to twenty different cultural contexts, in real time. Detectability decreases. The cost of manipulation collapses. The damage scales.
This isn't a fringe problem. It's a structural threat to democratic public discourse. And the EU AI Act — as well-intentioned as it is — is being outpaced by technological reality before it's even fully in force.
What Remains
I don't have a solution. That would be dishonest.
What I have is the conviction that technological determinism — "it's coming anyway, so let's just shape it" — is the most comfortable form of surrender available.
What I have is the conviction that the people shouting loudest about imminent AGI almost always have the most to gain if you believe them.
And what I have is the conviction that the silence of people close enough to know better — developers, researchers, engineers — is its own form of complicity.
That's why I'm writing this. Not because I have answers. But because the questions need to be asked out loud.
In ten years I'll either say: it wasn't as bad as I thought. Or: I saw it coming.
I know which one I'm hoping for. I'm not sure which one I'm expecting.
Sources
- Anthropic Research, "Emergent Introspective Awareness in Large Language Models", October 2025 — anthropic.com/research/introspection
- Amodei, D., interviewed on Interesting Times podcast (New York Times), February 14, 2026
- Lindsey, J. & Batson, J., quoted in Scientific American, "Can a Chatbot Be Conscious?", July 2025 — scientificamerican.com
- Anthropic Claude Opus 4.6 System Card, February 2026
- Arendt, H., Eichmann in Jerusalem: A Report on the Banality of Evil, 1963 (Viking Press)
- EU AI Act, Regulation (EU) 2024/1689 — eur-lex.europa.eu
- LeCun, Y., public statements on JEPA architecture as alternative to Transformer-based scaling, 2024-2025
- Marcus, G., Rebooting AI, 2019 (Pantheon) and ongoing commentary — garymarcus.substack.com
- Palisade Research, AI shutdown resistance study, May 2025
- OpenAI & Apollo Research, "AI Scheming" report, September 2025
- Seth, A., quoted in Scientific American, July 2025
Found this useful? Follow me on my blog for more.
Top comments (0)