DEV Community

Cover image for Prompt Engineering Won't Survive — the Skill Underneath It Will
Jason Lau
Jason Lau

Posted on AI-assisted

Prompt Engineering Won't Survive — the Skill Underneath It Will

#promptengineering #llm #career #machinelearning

For about two years, "prompt engineer" was the job title recruiters attached to six-figure listings and career-changers scrambled to add to a resume. Then the interest mostly stopped converting into jobs. The Wall Street Journal reported that Indeed searches for the role spiked to 144 per million U.S. searches in April 2023 and have since flatlined at 20-30 per million — and, per Indeed VP of AI Hannah Calhoon, that interest was never matched by actual employer job postings in the first place: "Maybe they talked about the value of prompt engineers, but they weren't then actually hiring for that." A title that rose and mostly stalled out inside two model generations. Microsoft's Jared Spataro put the reason bluntly: "You don't have to have the perfect prompt."

That's a fair obituary for the "prompt engineer" job title. It's a bad obituary for the underlying work, and the mistake is treating both as the same thing.

What actually died was a set of tricks tuned to specific models

Open any 2023-era prompt-engineering article and it's mostly mantras: "think step by step," phrase it as a role assignment, wrap the instruction in triple quotes, promise the model a tip. These worked because they guided a specific model around a specific weakness — and weaknesses don't persist across model generations.

Chain-of-thought is the clearest case, because it's the most-cited prompting technique there is and it's now been re-measured against current models. A 2025 technical report from Wharton's Generative AI Labs ran chain-of-thought prompting against eight models spanning both non-reasoning and reasoning types. The results don't point one direction. Gemini 2.0 Flash gained 13.5% and Sonnet 3.5 gained 11.7% — real improvements. GPT-4o-mini gained 4.4%, a difference the authors couldn't call statistically significant. Gemini Pro 1.5's rate of getting every part of a problem fully correct dropped 17.2 percentage points with chain-of-thought turned on. Among the reasoning models — the ones already trained to produce their own extended reasoning — o3-mini and o4-mini picked up a marginal 2.9-3.1%, while Gemini Flash 2.5 lost 3.3%. All of this arrived with a real tax: 35-600% longer response times for non-reasoning models, 20-80% longer for reasoning ones. The paper's own framing is careful rather than dramatic — "its effectiveness depends significantly on model type and specific use case" — but read against the trend, that's a technique whose payoff is shrinking and inverting as the models it was compensating for get better at reasoning on their own.

A 2025 arXiv paper on what it calls the "Prompting Inversion" makes the same point sharper by comparing one technique across two model generations directly. The authors built "Sculpting," a heavily constrained prompting method, and pitted it against plain chain-of-thought. On GPT-4o, Sculpting won clearly — 97% accuracy against chain-of-thought's 93%. On GPT-5, the ranking flips: Sculpting drops to 94.00% while plain chain-of-thought reaches 96.36%. Their explanation is that the same constraints acting as guardrails for a mid-tier model become "handcuffs" on a more capable one — the scaffolding that stopped GPT-4o from wandering off just gets in GPT-5's way. A technique doesn't get more true as models improve; it can flip sign entirely.

The instability was already visible in how fragile the tricks were

None of this should be surprising if you look at how sensitive prompt outputs already were to changes that had nothing to do with meaning. Sclar, Choi, Tsvetkov, and Suhr measured how much purely cosmetic prompt formatting — whitespace, separator characters, capitalization — could move accuracy on the same task with the same content, and found swings of up to 76 accuracy points on LLaMA-2-13B. Critically, they found this sensitivity "remains even when increasing model size, the number of few-shot examples, or performing instruction tuning" — it isn't a small-model problem that scale quietly fixes.

That finding reframes what a lot of "prompt engineering" tips actually were. If reformatting a prompt with no change in meaning can swing accuracy by 76 points, then a technique's measured 10-point gain on some benchmark was never a stable property of the technique — it was one sample from a distribution of formatting accidents that happened to land favorably on that model, at that snapshot in time. Chasing that kind of gain by memorizing the exact phrasing that worked last week is chasing noise, and noise doesn't transfer to the next model release. You have to separate the parts of a prompt that are doing real work (the schema, the goal, the constraints) from the parts that are stylistic residue — which is exactly the split SophiArch's AI-Assisted Data Science course spends its second lesson on, precisely because the residue is what breaks first when your model evolves.

What was actually being taught the whole time

Strip out the mantras and what's left in a good prompt is closer to a specification than a sacred phrase. A 2024 peer-reviewed framework in Frontiers in Education defines prompt engineering as "the skill of communicating the problem, its context, and the constraints imposed on the desirable solution to an LLM to solve it correctly as fast as possible" — and its authors explicitly push back on the idea that this reduces to generic communication skill, arguing it's closer to task decomposition: knowing what you actually want closely enough to state it, before you ever open a chat window.

That's the part that doesn't expire when a new model ships, because it isn't a fact about models — it's a fact about the gap between what's in your head and what's on the page. Consider the difference between two requests to deduplicate a customer table:

"Clean up the duplicate customers in this table."

versus:

"Two rows are the same customer if email matches exactly (case-insensitive) OR both phone and last_name match. When rows conflict, keep the one with the most recent last_order_date; if that's tied, keep the one with more non-null fields. Do not merge rows that share only a last_name — that's a coincidence, not a duplicate. Return a mapping from removed customer_id to the customer_id it was merged into, so the removal is auditable."

Neither of these is a prompting trick. No chain-of-thought cue, no role-play framing, no promised tip. The second one just doesn't leave the model — or a junior analyst, for that matter — a single ambiguous decision to make on its own: what counts as a match, which record wins a conflict, what NOT to merge, and how the result gets checked. A model that's gotten better at reasoning will make smarter guesses at filling those gaps than last year's model did. It will still be guessing, and on a dataset that decides refund eligibility or fraud flags, "the model guessed well" is not the standard anyone should be signing off on.

That's the actual content of "prompt engineering" once you take the model-specific mantras out of it, and it's also why the skill was never new — it's requirements-writing and problem specification, wearing a 2023 label because a chat interface made the gap between vague and precise suddenly visible to everyone, all day, on every request. The interface is what's new. The gap it exposed is not, and closing it is a durable skill for exactly the reason the mantras weren't: it doesn't depend on which model happens to be running underneath.

References

Top comments (0)