DEV Community

Cover image for I Built Production AI Systems Through Karpathy's 'Magnitude 9 Earthquake.' Here's What Actually Works.
Daniel Nwaneri
Daniel Nwaneri

Posted on

I Built Production AI Systems Through Karpathy's 'Magnitude 9 Earthquake.' Here's What Actually Works.

#webdev #programming #career #ai

After my $5/month RAG article reached 1,800+ developers in one week, many asked: "What skills actually matter in the AI era?" This is my answer - based on building production systems through what Karpathy calls a *'magnitude 9 earthquake.'**

"I've never felt this much behind as a programmer."

That's not a junior developer speaking. That's Andrej Karpathy - former Tesla AI Director, OpenAI founding member, one of the most respected AI researchers on the planet.

His recent tweet sent shockwaves through the developer community, racking up 14.8M views and sparking a collective existential crisis among programmers worldwide.

But here's what caught my attention: while Karpathy and others were theorizing about feeling "behind," I was actually building production AI systems. Systems handling 500K+ API calls daily. Systems that taught me something crucial - there's a massive gap between the conversation happening on Twitter and the reality of building with these tools.

While think pieces like "The Fog of Code" are debating what this means philosophically, I wanted to share what I've learned building production AI systems. Not theory. Not hot takes. Just patterns that work (and fail) at scale.

Let me show you what's actually happening in the trenches.

The Journey Nobody Predicted

In January 2023, Karpathy tweeted: "The hottest new programming language is English"

It seemed visionary at the time. Learn to prompt well, and you'd unlock massive productivity gains. The community rallied around "prompt engineering" as the new must-have skill.

Fast forward to December 2025, and that same Karpathy tweets:

"There's a new programmable layer of abstraction to master (in addition to the usual layers below) involving agents, subagents, their prompts, contexts, memory, modes, permissions, tools, plugins, skills, hooks, MCP, LSP, slash commands, workflows, IDE integrations..."

Notice what happened? "English is programming" wasn't wrong - it was incomplete.

Prompting turned out to be just the entry point. The real revolution is an entire orchestration layer that sits between your prompts and production systems. And almost nobody is teaching this.

What the New Stack Actually Looks Like

A quick note on perspective: Most articles about AI orchestration come from Silicon Valley engineers with unlimited compute budgets. I'm building from Port Harcourt, Nigeria, on Cloudflare Workers, where cost and infrastructure constraints are real. This shapes everything you'll see below.

Let me break down what Karpathy's describing, not from theory, but from building real systems.

The Architecture 

Traditional Stack (2023):
├── Frontend (React/Vue/etc)
├── Backend (Node/Python/etc)
├── Database (Postgres/MongoDB)
└── Infrastructure (AWS/Vercel/etc)

New Stack (2025+):
├── Traditional Stack (still needed!)
└── AI Orchestration Layer
    ├── Agent Framework
    ├── Context Management (MCP)
    ├── Tool Integration
    ├── Vector Databases
    ├── Prompt Engineering
    ├── Memory Systems
    └── Reliability Patterns
Enter fullscreen mode Exit fullscreen mode

The traditional stack didn't disappear - it became the foundation for something bigger.

Real Example: Building a Production MCP Server

I recently built an MCP (Model Context Protocol) server for a client's ChatGPT app. Here's what that actually involves:

The "Simple" Description:
"Build an AI-powered affiliate tracking system with semantic search."

The Production Reality:

  1. MCP Server Architecture 
// This isn't just "prompting Claude"
// It's building infrastructure for AI to interact with

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const mcp = new MCPServer({
      name: "vetster-affiliate",
      version: "1.0.0"
    });

    // Register tools the AI can use
    mcp.registerTool({
      name: "search_veterinarians",
      description: "Search for veterinarians using semantic search",
      parameters: {
        query: { type: "string" },
        location: { type: "string" }
      },
      handler: async (params) => {
        // This is where traditional engineering meets AI
        const embedding = await generateEmbedding(params.query);
        const results = await vectorSearch(embedding);
        const affiliateLinks = enrichWithTracking(results);
        return affiliateLinks;
      }
    });

    return mcp.handleRequest(request);
  }
};
Enter fullscreen mode Exit fullscreen mode
  1. Vector Search with Workers AI 
// Cost-optimized semantic search on the edge
async function vectorSearch(embedding: number[]) {
  const results = await env.VECTORIZE.query(embedding, {
    topK: 10,
    returnMetadata: true
  });

  // Reliability pattern: fallback to keyword search
  if (results.matches.length === 0) {
    return await fallbackKeywordSearch();
  }

  return results;
}
Enter fullscreen mode Exit fullscreen mode
  1. Observability & Analytics 
// Because AI systems fail in non-obvious ways
await env.ANALYTICS.writeDataPoint({
  blobs: [userId, searchQuery],
  doubles: [responseTime, confidence],
  indexes: [timestamp]
});
Enter fullscreen mode Exit fullscreen mode

The Point: This isn't "writing prompts." This is systems engineering with AI components.

The Skills That Actually Matter

What Karpathy Got Right

He's absolutely correct about the "magnitude 9 earthquake." The profession is being refactored at a fundamental level.

But here's what I learned from building at production scale:

1. Orchestration Over Prompting

Everyone talks about prompt engineering. Almost nobody talks about:

  • How do you handle AI failures gracefully?
  • How do you manage costs at scale?
  • How do you make stochastic systems feel reliable?
  • How do you debug when the AI does something unexpected?

Production Example:

My FPL Hub (Fantasy Premier League analytics platform) serves 2,000+ users with 500K+ API calls daily. When I added AI-powered insights, here's what I learned:

// Naive approach (doesn't work in production)
const analysis = await openai.chat.completions.create({
  model: "gpt-4",
  messages: [{ role: "user", content: prompt }]
});

// Production approach
async function generateAnalysis(data: PlayerData) {
  const cacheKey = `analysis:${data.playerId}:${data.gameweek}`;

  // Layer 1: Check cache (saves 90% of AI calls)
  const cached = await env.KV.get(cacheKey);
  if (cached) return cached;

  // Layer 2: Rate limiting (prevent cost explosions)
  await rateLimiter.check(userId);

  // Layer 3: Multiple attempts with exponential backoff
  const analysis = await retryWithBackoff(async () => {
    return await ai.run('@cf/meta/llama-2-7b-chat-int8', {
      messages: [{ role: "user", content: optimizedPrompt }]
    });
  }, { maxAttempts: 3 });

  // Layer 4: Validation (AI output isn't always valid JSON)
  const validated = validateAnalysis(analysis);

  // Layer 5: Cache for next time
  await env.KV.put(cacheKey, validated, { expirationTtl: 3600 });

  return validated;
}
Enter fullscreen mode Exit fullscreen mode

That's the orchestration layer nobody's teaching.

2. Production Experience as Moat

Boris Cherny (TypeScript expert, Anthropic engineer) responded to Karpathy's tweet with something fascinating:

"newer coworkers and even new grads that don't make all sorts of assumptions about what the model can and can't do — legacy memories formed when using old models — are able to use the model most effectively."

This is both terrifying and liberating.

The Terror: Your hard-earned debugging skills might be baggage if they make you reach for old tools when AI could solve it faster.

The Liberation: The playing field is being leveled. New grads without "legacy assumptions" can be more effective.

But here's the nuance: Boris and Karpathy work at frontier AI companies. They have immediate access to the latest models, problems perfectly suited to AI assistance, and teams already fluent in AI-first workflows.

Most of us are building in a different context:

  • Clients who don't understand AI capabilities
  • Systems needing reliability guarantees
  • Infrastructure constraints (I work primarily on Cloudflare Workers)
  • Real-world failure modes

And this is where production experience becomes valuable.

3. The Patterns That Transfer

Models change every 30 days. APIs evolve. Frameworks come and go.

But these patterns compound:

Pattern Recognition for Reliability

  • How AI systems fail in production
  • When to use AI vs traditional code
  • Cost/performance tradeoffs at scale
  • Retry logic and fallback strategies

System Design Principles

  • Orchestrating multiple AI calls efficiently
  • Managing context windows at scale
  • Building observable AI systems
  • Edge computing constraints with AI

Production Debugging

  • What actually breaks in prod
  • How users interact with AI features
  • Cost scaling at real traffic volumes
  • Non-deterministic failure modes

Example: My $5/month RAG system (which became a top post on DEV) works because I understand:

  • Where to cache (KV vs R2 vs Vectorize)
  • When embeddings are worth the cost
  • How to batch operations efficiently
  • Which queries can be served from cache

That knowledge doesn't come from prompting - it comes from running production systems and watching the bills.

The "New Technical Class"

While I was writing this article, Ben Tossell (Factory/Droid team, ex-Zapier acquisition) published something that perfectly captures what Karpathy describes in practice.

Ben spent 3 billion tokens in four months building production systems - crypto trackers, custom CLIs, video generation systems - without being "technical" in the traditional sense.

His journey mirrors Karpathy's thesis:

"There's a new programmable layer involving agents, subagents, prompts, contexts, memory, skills, hooks..."

Ben's approach:

  • Uses CLI agents exclusively (Droid)
  • Iterates on agents.md configuration files
  • Ships real products with real users
  • Learns by building ahead of his capability

Is he a "programmer"? Not traditionally.

Is he "technical"? Absolutely.

Ben calls this the "new technical class" - and I think he's onto something. It's not "vibe coding" (a term that misses the point). It's systems orchestration.

The parallel to MCP:

Ben's agents.md files are portable configuration that travels between projects - defining how agents should behave, what tools they have access to, what patterns to follow.

Sound familiar? That's exactly what MCP servers do. Different implementation, same concept: portable instructions for AI systems.

From his article:

"My agents.md setup... says to explicitly set up each new repo with what to do and not to do, how to do things with GitHub, how to commit... Running tests, end-to-end tests... I often look at others' agents.md files to see what I can borrow for my own."

This is configuration-as-code for the AI era. Not Python. Not JavaScript. Agent orchestration.

What Ben discovered matches what I see in production:

Simple, single-purpose wins:

  • Ben: One CLI tool per task
  • Me: One MCP server per function

Complex, over-configured loses:

  • Ben: "Looking at Twitter you see people really optimizing or potentially over-optimizing their system"
  • Me: Multi-agent systems with 5+ config files get abandoned

The Unix philosophy applies to AI:

  • Do one thing well
  • Make tools composable
  • Keep configuration minimal

Ben's not replacing engineers. He's augmenting himself with AI to solve real problems. He understands systems, recognizes patterns, and ships production code.

That's the new technical class Karpathy references. The abstraction layer isn't the programming language - it's the orchestration layer.

Read Ben's full article: How I code with agents, without being 'technical'

What This Means For You (Right Now)

The Uncomfortable Truth

There's a window closing. Right now, in early 2026, you can still build production experience with AI systems before the next wave of developers figures this out.

By mid-2026? The "missing out factor" will be much higher.

What To Learn First

Don't start with the hype. Start with the infrastructure.

  1. Pick One AI Deployment Platform
    • Cloudflare Workers AI (my choice - edge computing + AI)
    • Vercel AI SDK (if you're in the Next.js ecosystem)
    • LangChain/LlamaIndex (if you need flexibility)

Master ONE platform deeply. Don't platform-hop.

  1. Build Something Real

    • Not a demo. Not a tutorial project.
    • Something with actual users or actual cost constraints
    • Let it fail. Learn from the failures.

  2. Learn MCP (Model Context Protocol)

    • This is becoming the standard for AI tool integration
    • ChatGPT, Claude, and others are adopting it
    • Get ahead while it's still early

  3. Master Observability

    • AI systems fail silently
    • Learn to instrument everything
    • Track: latency, cost per request, failure rates, user satisfaction

  4. Understand Vector Databases

    • Not just "how to use them"
    • When they're worth the cost
    • How to optimize for your use case
    • When traditional databases are better

The Positioning Opportunity

Here's what I did, and it's working:

Before: "Full Stack Developer" (competing on price)
After: "Cloudflare Workers AI Specialist" (competing on expertise)

The shift made me more expensive and more in-demand.

Why? Because I can tell clients:

  • Here's how AI will reduce your support load (with real numbers)
  • Here's why your current approach will cost 10x more at scale
  • Here's how to build this reliably, not just impressively

That's not prompt engineering. That's systems thinking applied to AI.

The Synthesis

Karpathy's "magnitude 9 earthquake" metaphor is perfect. The ground is shifting under our feet.

But here's what I've learned from building through the earthquake:

The new skill isn't "stop using traditional tools" - it's knowing when to delegate to AI vs when to use traditional engineering.

Could I prompt Claude to "build an MCP server with affiliate tracking"? Maybe.

Would it know:

  • How to structure it for Cloudflare Workers specifically
  • The right patterns for Vectorize integration
  • Cost-effective embedding strategies
  • Proper error handling for production use

Probably not without extensive iteration.

The skill becomes: Use AI to accelerate implementation, but apply your production knowledge to architecture and constraints.

Boris Cherny's "200 PRs, every single line by Opus 4.5" is impressive. But someone still had to:

  • Define the architecture
  • Make technology choices
  • Set the constraints
  • Review the output

That's where experience matters.

Your Next Steps

If you're feeling behind (and honestly, who isn't?), here's my advice:

This Week

  1. Pick one small project to rebuild with AI assistance
  2. Notice where AI helps vs where you need traditional engineering
  3. Document what fails and why

This Month

  1. Build one production AI feature (even if small)
  2. Instrument it properly (observability is key)
  3. Watch it under real traffic
  4. Write about what you learned

This Quarter

  1. Master one AI infrastructure platform deeply
  2. Build your mental model for reliability patterns
  3. Start teaching others (articles, tweets, videos)
  4. Position yourself around your specific expertise

The Meta-Game

The developers who will thrive aren't the ones with the most AI knowledge - they're the ones who can bridge AI capabilities with production constraints.

That's a teachable skill. And there's never been a better time to learn it.

Resources

If you want to go deeper:

Cloudflare Workers AI:

My Other Writing:

Follow My Journey:

Final Thought

Eric S. Raymond (legendary open source developer) recently shared a meditation on 50 years of programming - from punch cards to AI that passes the Turing test.

His closing line: "Everything I've lived through and learned was just prologue."

We're living through a compression of that 50-year journey into 2-3 years. It's disorienting. It's overwhelming. It's also unprecedented opportunity.

Karpathy feels behind because he's intellectually honest about the magnitude of change.

But feeling behind isn't the same as being behind.

The people who will succeed are the ones who:

  1. Acknowledge the earthquake
  2. Roll up their sleeves
  3. Build through the chaos
  4. Document what works

You don't need to master everything. You need to master something - deeply, in production, with real constraints.

Then teach others.

The manual for this alien tool? We're writing it together, one production system at a time.

What's your experience been with the new AI development stack? Drop a comment below - I'm genuinely curious what patterns you're discovering in production.

P.S. - If you're building production AI systems and want to compare notes, reach out. The best learning happens in conversation with other practitioners.

Top comments (2)

Collapse
 
frong1979 profile image
Marta •

Really insightful post! It’s valuable to see practical lessons from building production AI systems rather than just theory. Love how you break down what truly works versus hype—it’s a great guide for anyone aiming to deploy AI effectively

Collapse
 
dannwaneri profile image
Daniel Nwaneri •

Thanks Marta. The biggest surprise for me has been how much "AI engineering" is actually just systems engineering with new failure modes. The fundamentals—caching, rate limiting, observability, fallback strategies—matter more than ever because AI components fail in non-obvious ways.

Example: Traditional API fails → you get an error code. AI fails → you get plausible-looking garbage that passes validation but confuses users. That requires completely different debugging approaches.
Are you deploying AI features in production yourself? Would love to hear what unexpected challenges you've hit.