The Man Who Summoned Ghosts | Chapter 3: Stepping Away, Again

#ai

Leaving OpenAI again, building Eureka Labs, and turning education into a product.

Originally published on Lei Hua's Substack.

Anchors:
2024-02-20 · Let's build the GPT Tokenizer
2024-03-20 · Sequoia AI Ascent · Making AI Accessible · https://www.youtube.com/watch?v=c3b-JASoPi0
2024-06-23 · UC Berkeley AI Hackathon Keynote · https://www.youtube.com/watch?v=tsTeEkzO9xc
2024-09 · No Priors Ep. 80 · https://www.youtube.com/watch?v=hM_h0UA7upI

Epigraph

"I love startups and I love companies, and I want there to be a vibrant ecosystem of them. ... I would say a bit more hesitant about kind of, like, five mega corps kind of like taking over."
— Andrej Karpathy, Sequoia AI Ascent · 2024-03

I. Leaving Again

On February 13, 2024, Karpathy announced he was leaving OpenAI for the second time. The stated reason was simple: "I want to spend more time on my own personal projects." No drama, no conflict statement, no flurry of farewell tweets from colleagues. Quiet, clean — like the first time.

But this time, he was no longer just an engineer letting go of the wheel. When he left, the LLM-OS metaphor from his Intro to LLMs had already entered industry vocabulary. When he left, his own YouTube channel had hundreds of thousands of subscribers. When he left, OpenAI was already on the edge of internal fracture — three months later Ilya Sutskever would leave, five months later Jan Leike would leave, the superalignment team would collapse. Karpathy had walked out one step ahead of all of it.

II. The First Words After Leaving — No Longer Polite to the Tokenizer

A week after leaving, he released his first public work as a free agent: Let's build the GPT Tokenizer. On the surface, it's a 2-hour-13-minute tutorial paired with his GitHub repo minbpe. But its real significance is in the rant at the end.

He systematically listed the bugs that tokenization creates inside LLMs: the model can't spell, can't do basic arithmetic, struggles more with JSON than with YAML, has uneven performance across languages, fails on character-level tasks ("how many r's in strawberry") — each one rooted in this one seemingly innocuous preprocessing step. His conclusion was cold: tokenization is legacy technology and we should try to escape it.

This is a subtle but real turn. He would not have spoken this way from inside OpenAI. Not because OpenAI forbids it, but because when you are a researcher at a frontier lab, you carry an unspoken duty to speak with respect for the core technologies your lab is built on. The moment you step outside, that duty falls away. The price of speaking freely becomes much lower.

What we will see later is that this freer voice rises to a peak in October 2025 — but the seed of that sharp soberness was planted at the end of this tokenizer video in February 2024.

III. Sequoia Ascent — A First Draft of the Founder Identity

A month later, in March 2024, he appeared at Sequoia's AI Ascent conference for a fireside chat with Stephanie Zhan. In her introduction, she described him as "an incredible, fascinating futurist thinker; a relentless optimist; and a very practical builder." That is the 2024 Karpathy — still firmly in the optimist column.

In that conversation, he laid out his LLM-OS vision systematically for the first time to a room of VCs and founders. But what deserves remembering more is the long, almost uninterrupted monologue he gave when asked what he had learned from working with Elon.

"He likes very small, strong, highly technical teams. Companies, by default, the teams grow and they get large. Elon was always like a force against growth."
"He doesn't like large meetings. If you're not contributing, and you're not learning, just walk out. And this is fully encouraged."
"Usually, a CEO of a company is like a remote person five layers up. It's not how he runs companies. ... If the team is small and strong, then engineers and the code are the source of truth."
"I like to say that he runs the biggest startups."

When you listen to that passage, there is a quiet sense: he is not describing Elon. He is describing what he is about to do. Small teams, deep technical work, removing bottlenecks, accountable to engineers and not middle managers — all of this would surface in Eureka Labs. He had not yet announced Eureka Labs in that Sequoia conversation; but he was already publicly rehearsing the working style he would adopt.

The interview ended on a soft but firm note. Asked what gave him the most meaning going forward, he said: "I want the ecosystem to be like a coral reef — a vibrant ecosystem of all kinds of cool, exciting startups in all the nooks and crannies of the economy."

Stephanie Zhan teased him on stage: "Genuinely, Andrej dreams about coral reefs." The room laughed.

IV. Eureka Labs and the Berkeley Pep Talk

June 23, 2024. UC Berkeley AI Hackathon. 1,200 student hackers filled the SkyDeck awards ceremony. Karpathy walked on stage and gave a talk unlike anything he had ever given publicly — the most motivational, the most educator-flavored, the least researcher-flavored appearance of his life.

He used as his example a weekend project he had built — awesomemovies.life. He confessed: this wasn't his first time building this kind of thing. It was his twentieth. Each iteration took one weekend. Each one was imperfect. But each one taught him something new. The point wasn't the site itself; it was the accumulation — a Malcolm Gladwell "10,000 hours." What looks like "vibe" or "talent," he was saying, is mostly the muscle memory left by a lot of patient practice.

The whole talk's register is encouraging — across all his public appearances, this is the moment when he most clearly sounds like a teacher. The audience was young people about to enter AI; he wasn't a researcher among peers; he was a teacher telling students: you can do this, but you must repeat.

Less than a month later, on July 16, 2024, he announced Eureka Labs on X. "This is something I've been doing my whole life," he wrote, "but now it's finally my full-time job."

The Berkeley talk and the Eureka Labs announcement were less than 30 days apart. The talk wasn't a coincidence — it was Eureka Labs' spiritual manifesto. He rehearsed the identity in front of 1,200 students before formally registering it as a company.

V. No Priors — An Undertow of Doubt

By the fall of 2024, Eureka Labs had been alive for two months. He appeared on Sarah Guo and Elad Gil's podcast No Priors. The conversation ranged widely — Tesla vs. Waymo's self-driving paths, the shared neural networks between Optimus and the Tesla fleet, his views on the future of education.

But in that conversation, he first uttered, in public, the seed of what would later become "AGI is still a decade away." He proposed a conjecture — he called it the cognitive core: the part of the model that reasons, plans, and thinks might need to be very small, perhaps only ~1B parameters. Current frontier models are large, he suggested, because they carry too much knowledge. But knowledge and intelligence are not the same thing; a bigger model is not, automatically, a smarter one.

It was a researcher's detail, the kind of thing only insiders care about. But it is the true origin of the sharpest lines from the Dwarkesh interview thirteen months later — "the model is lossy compression, not a knowledge source," "AGI is still a decade away," "it's slop." All of it grew from the judgment of "knowledge ≠ intelligence" that he first stated in September 2024.

By the end of 2024, what outsiders saw was a vibrant founder — newly launched Eureka Labs, sharing the LLM-OS vision in podcasts and on stages, holding a coral-reef optimism for the ecosystem. But his own internal pessimist thread had quietly begun, in the cognitive-core conjecture.

One Line for This Chapter

In chapter three, he has become a founder on the outside, but he is still a researcher inside — and this time, the subject he is researching is exactly where AI still falls short. His optimism and his pessimism aren't a contradiction. They are two outward-facing directions of the same engineer's heart.