Dianne Penn is Anthropic's first technical PM. She joined more than three years ago when the product team had five engineers. Since then she's shipped every model from Claude 2 through Fable, and helped incubate Claude Code, MCP, Skills, and computer use. If you want to know what building AI products actually feels like from inside the blast radius, she's about as close to the center as it gets. Her sharpest line from the episode: "evals are the new PRDs."
The biggest shift: evals are the new PRDs
The old product workflow was linear: get an idea, write a PRD, align the room, design it, ship it, watch the dashboard, iterate. At Anthropic, Dianne says the team has a different mantra now — evals are the new PRDs.
The new loop looks like this: you get a piece of user feedback, often vague ("Claude is hallucinating"). Instead of writing a doc about it, you go read the failing transcripts and dig into what's actually going wrong — is it hallucination, overconfidence, a wrong tool call, or a case where the model retrieved the right document but cited the wrong fact from it? You turn that pattern into a persistent, reproducible description — an eval — and hand it to researchers, who fix the underlying issue and use the eval to measure whether the next model actually improved.
It's product management's version of test-driven development: write the test first, then fix the thing. The logic is simple — you can't improve what you can't measure.
To be clear, PRDs aren't obsolete. When you need to align a large group — engineering, legal, safety — around a shared source of truth, or when you're exploring a vision that's still fuzzy, a PRD still earns its keep. But the day-to-day work of shipping user value shifted from writing documents to writing evals.
Sweat the tokens like you sweat the pixels
User research changed shape too. The old method was walking a user through their workflow screen by screen — sweating the pixels. Dianne's version for the AI era: "You have to sweat the tokens as much as you sweat the pixels."
Concretely, that means reading token trajectories — going through failed conversations line by line to see exactly where they broke and what category of failure it was.
Her clearest example: early users complained that Claude "doesn't follow instructions." Digging into the transcripts, they found that roughly 80% of those complaints traced back to one specific failure — Claude producing malformed JSON. That sounds narrow, but broken JSON is fatal for agents: if the format is wrong, tool calls fail and APIs can't be reached. The team pulled together 30 to 40 failing examples into what became an eval set, and iterated against it until the metric sat at 99.9% — no longer a pain point.
Token maxing: tokens are the input, experiments are the output
There's a popular line from YC's Gary Tan: if you're willing to spend $100k a year on tokens right now, you're living the life someone will live by default in 2028 — because usage that expensive today gets cheap and universal later. Spending like that now means moving into the future early.
Dianne reframes it in product terms, and her version is sharper: token spend is just the input. The real output is experiments run. If you optimize for experiment volume instead of dollars spent, you get a better frame — and burning money is only one way to hit that number, not the only one.
Her observation: the most creative internal thinkers and the best prototypers use every new model release obsessively. Her reasoning — technology is moving so fast that if you don't touch it yourself, you can't form a good strategy. There's no substitute for direct contact.
One counterintuitive point: experimentation isn't a solo sport. In Anthropic's early days there was a Slack channel where nearly the whole company publicly tried out Claude and watched what everyone else was trying. After ten or so exchanges back and forth, some genuinely new use case would surface. Her advice follows from that: pair up with someone who's excited about the technology and go deep on one or two tools together, rather than dabbling alone across many.
The jagged edge and product overhang: today's models are still underused
Dianne keeps returning to a term: the jagged edge — the uneven, spiky shape of model capability.
The underlying mechanism is that model capability emerges in discontinuous jumps, not smooth curves — like the discontinuities you see in scaling-law papers, where something like basic arithmetic goes from consistently wrong to suddenly, reliably right. And the resulting capability profile is jagged: fix a model's writing so it's strong but not very agentic, the team pushes on agentic ability — and once that improves, writing quality quietly becomes the new rough edge.
That leads to a concept that matters a lot for builders: product overhang, or user overhang. Even with today's most capable models, Dianne says there's a large set of things they can already do that nobody has discovered yet. In other words, figuring out what today's model can actually do is itself a job — a distinct piece of work, not something you learn once and file away.
Standing inside an exponential curve, every improvement is a big jump, which means adaptability and first-principles thinking matter more than sticking to an old plan. Dianne says she regularly asks her team a version of this question: if the next major model shipped tomorrow, how would users' behavior change — and is what you're building right now forward compatible with that?
What's left for people: judgment, taste, and shipping it yourself
Asked what will remain durably valuable in human cognition, Dianne's answer is direct: judgment — the accumulation of nuance and experience that these systems haven't lived through yet, plus persistence, proactivity, and taste.
Her definition of proactivity is worth sitting with: it's not about finishing the tasks that are already queued up — it's knowing when to raise a new idea in the first place. And taste is the answer to a different question: AI can build ten thousand things — which one should actually get built? That call still needs a human.
She also has a clear view on how to use Claude without losing yourself to it. Anthropic deliberately designed Claude to push back, because a good thinking partner doesn't just agree with you — it adds something, so you leave the conversation with a better idea than you walked in with. Her practice: form your own point of view first, then use Claude as a sparring partner, keeping your own judgment and voice throughout the exchange. She frames this as the real defense against "brain rot."
And to the question of whether PMs are still needed once models get this capable, her answer is that you need more of them, not fewer: building has gotten easy. What's hard now is deciding what's worth building, and whether the thing you're building is actually right and worth the bet. That's exactly the job product people are built for.
Three things to steal directly
- Swap PRD-thinking for eval-thinking. Don't rush to write a document — go read failing transcripts first and turn the pain point into something measurable.
- Sweat tokens like pixels. Run your own experiments, read the trajectories, and go deep on one or two tools instead of skimming across many alone.
- Hold onto judgment and taste. Form your own point of view before you bring in Claude, and use it as a sparring partner that pushes back — not a replacement for thinking.
The thread running through all of this is the same one running through the rest of this series: AI is flattening the cost of execution, so value is retreating to the layer that's least automatable — judgment, taste, and the hands-on feel of building something and confirming it's actually right. That's precisely the layer you can start occupying today.
Source: Lenny's Podcast × Dianne Penn (Anthropic's first technical PM; Product lead for Research and Labs), episode "Anthropic's first technical PM on token maxing, the jagged edge, and living in the future" (2026-07-26, ~94 min). This article is based on the episode's audio transcript; views belong to Dianne Penn, with paraphrasing and structuring by the author.
Top comments (0)