Originally published on The AI Prism
The 1,306-point claim that split Hacker News
In coverage that drew 1,306 points on Hacker News, Sean Goedecke advanced a deceptively simple thesis: working with an LLM amplifies the skilled and exposes the unskilled. Expertise, he argued, is rewarded rather than replaced (Goedecke, “LLMs reward expertise”). The post resonated because it flatly contradicts both panic narratives — that AI will erase knowledge workers — and triumphalist ones — that anyone can now produce expert output by typing a sentence.
The argument deserves scrutiny because it makes a falsifiable empirical claim, not a philosophical one. Does the data support the idea that expertise is the variable that determines how much value a person extracts from an LLM? Or does the evidence point somewhere more nuanced? Goedecke himself flagged the risk in his own comment thread: some readers, he noted, are “rightly suspicious of a view that’s reassuring them about how they’re still valuable.” That honesty is the right starting point. We should test the claim against studies, not vibes.
What Goedecke actually argues
Goedecke’s core mechanic is straightforward. Before LLMs, a technical gap — say, not knowing CSS — forced you to either recruit a skilled colleague or hope a matching answer already existed online. Today the same person can delegate that gap to a model and produce “sort-of-okay” output. Everyone becomes a generalist (Goedecke).
From this, a tempting conclusion follows: if everyone talks to the same model, prompting skill is irrelevant and expertise no longer matters. Goedecke rejects that. His central claim is that the most important skill in prompting is expertise in the domain you are prompting about. A novice and an expert may get similar first drafts, but only the expert can steer, prune, and verify the result. He extends this to codebases specifically: if you hold a strong “theory of your codebase,” you can push the LLM far harder than someone with no familiarity, because you have a prior sense of what a good solution looks like.
The mechanism he proposes is information retrieval, not magic. The answer is “in the model” already; the scarce resource is the human ability to pull the right answer out. That reframes expertise as a compression and filtering skill: knowing which of the model’s many plausible lines to keep, and which to throw away before they calcify into a confident mistake.
The Terence Tao conversation, and what it really shows
His flagship exhibit is Terence Tao’s public conversation with ChatGPT about a recently discovered counterexample to the Jacobian Conjecture (Tao’s shared chat). Tao does not merely prompt; he makes leaps, proposes reformulations, and pushes back when outputs “look weird.” Goedecke notes the model shifts into “talking-to-mathematicians” mode for Tao, producing terser, denser replies than a layperson receives.
The lesson is not that Tao has mastered a secret prompt syntax. It is that domain knowledge lets you pull the right idea out of a multi-paragraph response and discard the rest. As we explored in our own analysis of Tao’s method, the genius sees the shape of the problem before the model finishes speaking, and he almost never takes the model’s advice about where to go next (TheAIprism: what a genius sees that we don’t). The model is a sparring partner, not an authority.
This maps onto a broader pattern Goedecke observes in his own engineering work: familiarity with concrete specifics beats generic principles. He can ask sharp questions about the systems he owns at GitHub that he could never ask about abstract mathematics. Expertise, in other words, is local — and LLMs reward the locality. The same person can be expert and novice in the same afternoon, depending on the domain, which is why the claim “expertise is rewarded” is true only relative to a specific task.
What the controlled studies actually say
Goedecke’s claim is anecdotal. The cleanest counter-evidence comes from randomized experiments. In a 2023 study, Shakked Noy and Whitney Zhang gave 453 professionals incentivized writing tasks, randomly assigning ChatGPT access (Noy & Zhang, Science). Output quality rose and completion time fell — but the gains were concentrated among lower-ability workers. The productivity distribution compressed rather than spread.
A large field study of customer-service agents reached the same pattern at scale. Brynjolfsson, Li, and Raymond studied thousands of agents before and after AI deployment and found an average 15% productivity lift, but a 34% lift for novice and low-skilled workers, with minimal effect on the best performers (Brynjolfsson, Li & Raymond, “Generative AI at Work”). Their interpretation: AI transmits the best practices of top performers downward, lifting the floor for everyone below them.
A 2024 age-classification experiment by Caplin et al. compounds the point: AI raised performance across ability levels but reduced dispersion most when users were well calibrated about their own skill (Law & Economics Center review). The recurring result across writing, support, and classification tasks is skill compression, not elite-only reward. If the only evidence were these three papers, Goedecke’s thesis would look wrong — which is exactly why the next study matters.
The mechanism behind compression is best-practice transmission. A junior who has never seen a strong example of the task suddenly has one on tap, every time. The senior, who already embodied those practices, gains little from re-encountering them. That is why the floor moves and the ceiling barely does — at least on the tasks these studies measured, which tend to be bounded and verifiable.
The jagged technological frontier
The most important caveat comes from the BCG–Harvard study of 758 consultants and 18 realistic tasks (Dell’Acqua et al., “Navigating the Jagged Technological Frontier”). Within the model’s capabilities, GPT-4 users completed 12.2% more tasks, 25.1% faster, and 40% produced higher-quality work. But on tasks just outside that “jagged frontier,” AI users were 19% less likely to reach a correct answer than people with no AI at all.
This is the crux. AI does not fail uniformly; it fails on a ragged boundary the user cannot see. The researchers distinguish “centaurs” (clean human/AI task splits) from “cyborgs” (constant interaction) — both work, but both depend on the human knowing where the frontier sits. Lakhani’s blunt warning captures it: “This is not Google.” Treating the model as a search box is exactly how users fell 19% behind on the hard tasks.
The frontier finding does something subtle to Goedecke’s thesis. It suggests the expert’s advantage is not merely producing better drafts — it is knowing which tasks to hand the model at all. Outside the frontier, more delegation is worse. That is a meta-judgment the novice lacks, and it is invisible in aggregate productivity numbers that average over easy and hard tasks alike. The expert’s reward, in other words, shows up as avoidance of catastrophe rather than headline speed gains.
Two effects, not one: compression and amplification
Set the studies side by side and a cleaner picture emerges. There are two distinct effects running in opposite directions:
Compression at the floor. AI lifts weaker workers most. Noy & Zhang, Brynjolfsson et al., and Caplin et al. all find performance dispersion shrinks, especially when users are well calibrated about their own skill. The floor rises fast.
Amplification at the ceiling. Experts extract more at the top. Goedecke’s Tao example and the BCG finding — that outside-frontier failure depends on user judgment — both imply the expert’s edge grows precisely where tasks are hard and the model is silent or wrong.
Goedecke is right that expertise is rewarded. He understates how much AI compresses the gap below. The honest synthesis is asymmetric: the floor rises faster than the ceiling. A junior with a model can now mimic a competent senior on routine work, but no amount of model access converts a novice into Tao on the frontier.
Consider a concrete split. On a bounded writing task — summarize this memo, draft this email — the novice-plus-model and the expert-plus-model land close, because the frontier encloses the task and the model supplies the missing structure. On an open mathematical proof or a subtle production incident, the novice gets fluent nonsense the expert immediately flags. The same tool, two regimes: compression where the frontier is generous, amplification where it is thin.
One caveat tempers the whole comparison: the frontier is not fixed. As models improve, tasks that were once outside it migrate inside, and the compression effect expands with them. If the boundary keeps moving outward, the era in which expertise is decisively rewarded at the ceiling may shrink to a thinner and thinner sliver of remaining-hard problems — unless expertise itself is what defines where the frontier currently lies.
Why calibration is the new bottleneck
If AI both lifts the floor and rewards expertise at the ceiling, what exactly does the skilled person contribute? The BCG data points to one scarce skill: calibration — knowing when to trust the model and when to ignore it. Outside the frontier, over-trust was actively harmful (−19% correctness). The human who suspects “this looks more complex than I hoped” and reroutes is the human who stays accurate.
Goedecke’s own phrasing fits: “the human is the bottleneck, not the model,” because the hard part is communicating exactly what solution you want (Goedecke). I would sharpen that: the bottleneck is judgment about the model’s limits, a meta-skill that sits above raw domain expertise. Domain expertise helps you recognize a wrong answer; calibration tells you whether to ask at all. Both are human, neither is automatable yet.
This also explains the Hacker News skeptic’s objection — that anyone can now feel rewarded. Feeling rewarded and being right are different. The model happily confirms the incompetent, which is precisely why calibration, not confidence, separates the expert from the amateur. Calibration is buildable: it grows from repeated, consequential feedback where wrong answers carry a cost the model cannot absorb for you. That is another reason expertise, earned through consequences, stays relevant.
Implications for knowledge work and hiring
For organizations, the data argues against two instincts. First, do not assume AI erases the need for senior talent; you need experts precisely to set direction and catch errors outside the frontier. Second, do not assume juniors are now interchangeable with seniors — AI narrows the gap but does not close it, and someone must still validate the output. The realistic play is mixed teams where experts handle the frontier and juniors, augmented, handle the floor.
This reframes the jobs debate away from “will AI replace us” toward “who can steer it” — a theme we examine in our broader review of what is actually happening to jobs (TheAIprism: separating AI hype from reality). The scarce role is the editor of the machine, not its operator. Hiring should weight demonstrated calibration — can this person tell good model output from fluent nonsense? — above raw output volume, because volume is now nearly free and discernment is not.
The same logic reshapes internal metrics. If AI narrows the spread between your best and worst contributors on routine work, average throughput becomes a worse signal of talent. Managers should monitor the tail — the hard, frontier cases where only calibration prevents regressions — rather than aggregate speed, or they will reward the person who delegates most and ships the most plausible errors.
Implications for education and evaluating AI output
The synthesis also reshapes how we should teach and assess. If AI compresses the floor, drilling rote execution matters less; teaching calibration matters more. Students need to learn not “how to write the essay” but “how to tell whether the essay the model wrote is correct.” That is a higher-order skill, and it is exactly the one experts already possess. Education that skips the fundamentals in favor of pure prompt reliance risks producing adults who cannot catch the model when it is wrong.
For evaluation, the lesson is uncomfortable: we can no longer grade the artifact without grading the process. A flawless draft may be expert-steered or expert-blind. The differentiator is whether the author can defend every line — a capacity AI cannot fake and expertise alone supplies. Assessment must move toward oral defense, source tracing, and revision history rather than the final product alone.
There is a Carnegie-style lesson here too. Just as cognitive tools historically offloaded routine computation, LLMs offload routine composition — and in both cases the expert’s value migrated to the parts the tool could not do. The frontier, not the floor, is where expertise lives, and curricula that teach only floor-level execution are teaching the part the machine now owns. The goal of training shifts from producing flawless executors to producing reliable judges.
The verdict, and an open question
Goedecke’s thesis survives contact with the data, but in a revised form. Expertise is rewarded — yet so is the absence of it, because AI raises the floor for everyone. The net effect is not replacement but reorganization: execution cheapens, judgment appreciates. The people who thrive are those who treat the model as a brilliant, unreliable junior colleague rather than an oracle, and who invest in the calibration the studies show is decisive.
The practical takeaway for knowledge workers is unglamorous. Spend less energy on prompt incantations and more on deepening the domain sense the model cannot fake; build feedback loops where your mistakes are visible; and reserve the model for the tasks inside its frontier while you guard the boundary yourself. Expertise is not obsolete. It is redistributed toward the places the model cannot reach.
That leaves the question the studies have not settled: if AI keeps lifting the floor while the expert’s edge persists mainly at the frontier, will deep expertise become a smaller share of total value — or the only part that still commands a premium?
References
• Goedecke, S. (2026). LLMs reward expertise. seangoedecke.com. https://www.seangoedecke.com/llms-reward-expertise/
• Dell’Acqua, F., Lakhani, K. R., McFowland, E., et al. (2023). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality. Harvard Business School / BCG. Coverage
• Noy, S., & Zhang, W. (2023). Experimental evidence on the productivity effects of generative artificial intelligence. Science, 381(6654), 187–192. https://www.science.org/doi/10.1126/science.adh2586
• Brynjolfsson, E., Li, D., & Raymond, L. (2023). Generative AI at Work. NBER Working Paper 31161. https://www.nber.org/papers/w31161
• Law & Economics Center (2024). AI, Productivity, and Labor Markets: A Review of the Empirical Evidence. George Mason University. https://laweconcenter.org/resources/ai-productivity-and-labor-markets-a-review-of-the-empirical-evidence/
The post ‘LLMs Reward Expertise’: What the Data Actually Shows appeared first on The AI Prism.
Cross-posted from theaiprism.com — Cutting Through the AI Noise 🧊
Top comments (0)