How to hurt yourself with AI

Summary

34x more work than it should have been.

Timeline

I write technical articles, and I like experimenting with something new in each one. For one of them, the experiment was an AI pipeline: it would draft the piece and pass it through several rounds of automated review before I saw it.

That pipeline produced 1,948 words while processed 65,930 in total which is ~34x more.

The idea was to base the writing on careful review and a feedback loop. So I ended up reviewing in person and wasting 34x words/tokens more than what I could actually produce.

Root cause

I am not sure what the exact cause is. There are several candidate explanations, and I do not think any one of them is complete on its own:

Not enough context to judge whether a cut was a real improvement.
The AI outputs what is locally coherent, what "sounds right," with no guarantee that coherent and applicable are the same thing.
The AI was built to fill in gaps rather than flag them, and what it fills in with is often an assumption, usually one that turns out false.

What a human has that the AI does not is a felt sense of consequence: what a decision actually costs or breaks, in a specific world, learned from having been wrong before. "Experience" is the word for that, and it is overused to the point of meaning nothing, but I do not have a better one.

If I compress all of that into one line: the pipeline optimized for the wrong thing. It optimized for passing a check. It could not optimize for "is this piece actually good," because nothing in it could feel the cost of being wrong.

The aim behind building it was sound and I would keep it: automate whatever a computer can do so a human does not have to. What failed was assuming editorial judgment was one of those things.

The debt

Engineers already track several kinds of debt that are not financial.
One of them is getting real attention now: cognitive debt, which is what you take on when you outsource tracking why your own system does what it does to an LLM. Two recent sources define it the same way: a
MIT Media Lab study on AI-assisted writing, and getdx.com's framing for software teams specifically.

This pipeline is a worked example. I could tell you the ratio, the gate verdicts, the timeline. For a while I could not have told you, from memory, what any single piece actually argued.

The more AI gets pushed into everything, the more this particular kind of debt matters. On a prototype, nobody bothers tracking it. On something meant to hold up, not understanding what you built is a deal-breaker,
and "just ship it, iterate later" is the same old fallacy wearing an
AI-shaped coat: speed and understanding traded off as if they were opposites, when the trade was optional.

What changes

The intent behind the pipeline was to be the reviewer: read a draft,
judge it, hand back a correction, and only step in myself when the correction failed too. In practice, I was correcting constantly, sentence by sentence, round after round, and each correction only produced more issues.

That does not work, and it should not. Scale it up and the failure gets clear: imagine ten thousand articles like this shipping out daily, all with a human correcting sentence by sentence, all costing that person a 34x reading tax.

Going forward, AI in this workflow does small, bounded jobs: formatting, a lookup, a narrow research question with a checkable answer. Things that do not load much cognitively.

Question for you

What have you noticed, using AI or watching it get used badly? Two data points I found while researching this piece:

Different domain, same shape as this piece: a check that looked like it was working, until someone read closely enough to notice it wasn't.

DEV Community