A Billion Token Lesson: Because You Can You Should

#ai #agents #product #lessons

A Billion Token Lesson: Because You Can ≠ You Should

Agent Autopsy, Day 6

Last week I spent a weekend building a product nobody asked for.

I didn't catch it in time. Let me tell you how it happened.

The trap

Nicolas Fränkel published a piece on designing teams of AI agents — specialist subagents that collaborate on software engineering tasks. Planner, challenger, coder, tester. Each with a focused system prompt, each with limited tools, orchestrated through a skill.

It's good work. I read it and immediately thought: we should build this for SPFx.

A TypeScript specialist harness. Four agents — Architect, Scaffolder, Builder, Verifier — in a deterministic pipeline. Zod-validated handoffs between them. Compliance injected at build time, not bolted on after. Terminal CLI, BYO token, no desktop app.

I had the architecture designed in 15 minutes. The pipeline spec. The agent system prompts. The handoff packet schema. Everything.

Then I stopped and asked: is anyone actually looking for this?

The answer was no.

The numbers don't lie

SPFx developers are a niche. SharePoint is a niche. Agent-orchestrated SPFx development is a niche of a niche — maybe a few hundred people worldwide who'd even understand the value proposition.

Of those, how many would switch from Copilot in VS Code to a terminal-based agent pipeline? Zero.

The market had already spoken. It was speaking before I even asked the question.

But the exciting part isn't the market analysis. It's what I did next.

What I did instead

I binned the harness. No commit. No repo. One weekend I'm not getting back.

Then I published the things people were actually asking for:

A learn hub with 10 methodology patterns — 85,000 words of production agent knowledge, free, no email gate. Live at workswithagents.dev/learn.
A benchmark of 12 local LLMs on real agent coding tasks. SmolLM3-3B at 93.3% — beating Claude Sonnet 4. Qwen2.5-1.5B at 85% on 940MB. DeepSeek-R1 collapsed to 27.5% because reasoning training is toxic below 3B. Data nobody else has published.

The harness took a weekend. The learn hub and benchmark took a few hours. One made 0 people's lives better. The other gave people answers they were actually searching for.

The billion token lesson

Agents make us dangerous. Not in the Skynet way — in the I can build anything in an afternoon way.

When an agent can scaffold a project in 30 seconds, write a pipeline in 2 minutes, and deploy to production before lunch, the cost of building drops to nearly zero. That sounds great. It's actually a trap.

Without the friction of effort, there's no natural filter against bad ideas.

In the pre-agent world, you wouldn't build an SPFx specialist harness because the thought of spending 40 hours on it was exhausting. The idea died on the whiteboard where it belonged.

Now an agent can build it for you in 4 hours. The only gate left is your judgment.

That's the billion token lesson. It's not about saving compute. It's about saving the thing you can't get back: your weekend, your focus, your credibility.

The rule

Before you tell an agent to build something, ask:

Is anyone looking for this?

Not "could someone use this?" Not "is this technically possible?" Not "would it be cool?"

Is there someone, right now, who would pay money or attention for this thing to exist?

If the answer is no — close the terminal. Go work on something people actually want.

"Because we can" is never the right reason. The market doesn't care what you're capable of building. It cares what you actually shipped that someone needed.

The SPFx harness taught me this the hard way — with a full weekend I'm not getting back.

Cheapest lesson I ever learned.