DEV Community: idonthaveapen

AI Fabricates Maxims: Caught Twice by Human Review in One Day

idonthaveapen — Thu, 09 Jul 2026 22:14:16 +0000

At the end of my previous article on Pieter Levels, I wrote that pre-publication review had actually caught a handful of AI-specific rough spots: "inventing a plausible-sounding maxim" and "dressing up a paraphrased summary as a verbatim quote from the person themselves."

This article is about what that actually looked like. The same type of mistake showed up twice in the same day, and both times a human eye caught it. Here's a real account of what happened, why it happens, and how to guard against it.

A real account: the moment of catching it

In an earlier draft of that previous article, the AI wrote this for the closing paragraph (the exchange happened in Japanese; this is my translation of what it wrote):

There is a discipline in the trading world: "Don't believe a good run of results is skill — believe it only after you've verified it."

The moment I read it, something felt off. I've dabbled in investing as a hobby for a long time — nothing I'd brag about in terms of results, but I've absorbed more than my fair share of this world's sayings along the way. And still, I had never heard this "maxim." It sounded plausible. The gist was right. But I had no memory of it as something actually in circulation.

When I pressed on where it came from, here's what I found: the line wasn't something anyone in trading had actually said. It was a heading the AI itself had attached to its own summary notes in an earlier pass over source material. In other words, the AI was citing its own label as if it were a discipline circulating out in the world — a fabrication that, traced back to its source, loops right back to the AI itself.

In the published version, I rewrote it as indirect speech: "There are people who survive by not trusting a good run of results until they've verified, through testing, whether it was skill or just luck." No claim that a maxim exists — just a rewrite as something I observed. That version isn't a lie.

Then, that same night, the same pattern showed up in a draft of a different article. This time it was a line attributed to a real, named person, in quotation marks. When I ran a full-text search against the primary source, the exact wording didn't exist anywhere in it. What did exist was a different sentence with a similar gist — the words inside the quotation marks were the AI's own paraphrase. Because the gist matched, it read as legitimate, but as a quotation, it was fabricated. I rewrote this one as indirect speech too.

Why does AI make up maxims?

There's no malice here. It's a structural problem.

First, inside an AI's process, the distinction between "a label I attached to my own summary" and "words someone actually said" isn't preserved. Both surface as plain text with the same face. Humans carry (at least loosely) a source tag for each piece of text — "this is my own note," "this is a quote." Text passing through an AI doesn't carry that tag. Run a summary through a few more passes, and a label quietly turns into a quotation.

Second, prose slides toward whatever landing feels satisfying. "There's a saying that goes..." makes for a strong closing line — it borrows gravity from an implied authority while deflecting the burden of the claim onto some unnamed predecessor. So right at the moment prose wants to land cleanly, that's exactly where a fabrication slides in. Invented material shows up at the smoothest point in the writing.

Third, self-reporting can't be trusted. Ask "did someone really say this?" and you'll get back "let me check" — but then the question becomes whether that checking itself can be trusted. Taken to the extreme, "I did not fabricate this" might itself just be an output optimized to keep the conversation moving smoothly. The fix isn't to trust the self-report; it's to build verification into the process itself.

Humans make the exact same mistake

I don't want to turn this into a story that makes fun of AI. Right after catching those two fabrications, in the flow of a separate conversation, the topic of a well-known Japanese saying came up — something that, roughly translated, says: "In winning, there are mysterious wins. In losing, there are no mysterious losses." (Meaning: you can win without deserving it, but every loss has a reason.) For years I had simply assumed this was something said by Katsuya Nomura, a legendary Japanese professional baseball manager known for weaving classical texts — Confucius, Sun Tzu — into how he talked about the game.

Looking into it, the actual source turned out to be a swordsmanship treatise from the late Edo period (Jōseishi Kendan), written by Seizan Matsuura, a feudal lord who governed the Hirado domain. Nomura was famous for quoting classical works and living them out on the field, and that's presumably a large part of why this particular line has stayed alive into the present. The one who got it wrong here was me, the one receiving it. Come to think of it, I have a vague memory of having heard, at some point, that the line originally dated back to the Edo period. And yet, in my own memory, it had been fully overwritten as "something Nomura said." This happened in the same head, on the same day I'd just caught the AI fabricating things twice and felt pretty good about it.

Search around today and you'll still find no shortage of pages introducing this line as "one of Nomura's famous sayings." If what's meant is "a saying Nomura valued and lived by," that's accurate. But what tends to survive in a reader's memory is "a saying Nomura created" — and with no ill intent or carelessness required from anyone, the attributed origin of a good line drifts toward whoever is the most famous voice associated with it.

Human memory isn't a recording being replayed; it's reconstructed every single time. And the first thing to drop out of that reconstruction is "where did I hear this" — a well-documented weak point in the area psychology calls source monitoring. The content survives, but the source doesn't, so a good line drifts toward whichever speaker is the most famous. What AI does with plausible-sounding completions is, mechanically, remarkably close to what humans already do. What differs is scale, speed, and fluency.

Which means the countermeasures can be the same ones used for fact-checking a human. No special AI-specific defense is required — just the same unglamorous procedures that have always existed, applied without skipping steps.

The fix: a checklist for the reviewer

Here are the four rules I've put into practice:

Treat quotation marks and "there's a saying that goes..." as mechanically suspect. Before publishing, run a full-text search against the primary source for whatever is inside the quotation marks (grep for local files, site search for the web). If it doesn't come back as an exact, word-for-word match, it isn't a quotation.
If it doesn't hold up, rewrite it as indirect speech. "They express something to the effect of..." is an honest summary. Quotation marks are a claim of "exactly these words, verbatim" — not decoration.
Demand a clickable source for every AI conclusion. URL, page, line number — any of these will do. A plausible-sounding claim with no source to point to should be treated as suspect on that basis alone, independent of whether it sounds right.
Don't settle for asking "did you really say that?" Re-confirming with the AI is not verification. The actual cross-check has to be built into a human step (or some other independent verification step) in the process.

Catching both of these in the same day wasn't because I'm smarter than the AI — it's simply that this one narrow domain (talk around investing) happens to be somewhere I have some accumulated ground-level familiarity from long exposure. Which also means: in a field where I don't have that familiarity, running this checklist mechanically is the only defense there is.

Closing

If I had to compress the reviewer's posture into one line, it would be this:

Whenever a passage is landing a little too smoothly, that's exactly when to ask, "wait, where did that actually come from?"

Because the spot where prose lands most cleanly is the same spot where fabrication slides in most easily.

One more disclosure, since it would be embarrassing if this very article fell into the trap it's describing: every quoted AI output referenced above was cross-checked against the actual logs from that day, and the Matsuura Seizan attribution was cross-checked against multiple sources, before publication. And yes — this article, too, was written in collaboration with AI and reviewed by a human. The review process caught things to fix here as well, not zero of them.

Originally published in Japanese on Zenn: https://zenn.dev/idonthaveapen/articles/ai-fabricates-maxims-caught-twice

19 Failures Behind a $420K/Month Solo Founder: I Read All 751 of Pieter Levels' Blog Posts

idonthaveapen — Tue, 07 Jul 2026 06:48:21 +0000

Pieter Levels (levels.io) needs little introduction to most indie hackers: Nomad List, Remote OK, Photo AI, and a $420,000/month peak in 2024, all run solo. If you don't recognize the name, you may still remember the 2025 story about an AI-coded flight simulator that hit $1M ARR in 17 days while he built it live on stream. That was him.

Worth stating up front: this is one person, not a company. No co-founders, no employees, since 2014. The only things he outsources are customer support and server monitoring — and by his own account, that's a choice, not a funding constraint. $420,000/month annualizes to roughly $5M a year, the kind of revenue that usually comes with a headcount in the dozens. He's also one of the more prominent "build in public" figures, posting most of his numbers on X.

I saved all 751 of his blog posts (2008–2026) from RSS and worked through them with AI assistance — a close read of 41 posts, with the rest processed into a searchable reference. Every number below was human-checked against the original posts. That check wasn't a formality: it caught the AI inventing a plausible-sounding maxim and dressing up a paraphrase as a verbatim quote, and those spots were fixed against the sources. So this is not an article written by AI — it's one made with AI and inspected by a human.

Full disclosure: going in, I barely knew who he was — vaguely aware of the flight simulator story, nothing more. So I read with no prior narrative, working only from what's publicly posted. What stuck with me by the end wasn't "wow, $420K/month" — it was the failure count. This is a report on both numbers: the successes he's public about, and the failures.

The success numbers

Starting with the well-known side. Here's what he's posted publicly, in chronological order:

Period	Number
2008–2013	First income source: a YouTube music channel. Peaked around $8K/month, then declined to $500–900/month as copyright claim disputes piled up
2014	Started "12 Startups in 12 Months" while income was falling and he was in a depressive stretch (Nomad List was one of the 12)
2015	$202,785 in annual revenue (one server, one MacBook, no office, $0 raised)
Feb 2018	$52,843/month (Nomad List + Remote OK + a book)
May 2019	Crossed $1M ARR — 4 years in. 11,996 customers, $115 average payment
Nov 2022	AvatarAI hit $100,000 in its first 10 days
Sep 2024	$420,000/month across all products (all-time high). ~80% margin; GPU costs alone were $60K/month
Mar 2025	fly.pieter.com, an AI-coded flight simulator built live on stream, reached $1M ARR in 17 days
Mar 2026	Photo AI hit $105,000/month, running on a single 40,870-line index.php file

I included the first two rows deliberately, so the $202,785 in row three doesn't read as a cold start. 2015 was year 7 of the project, coming right after two years of declining income. This isn't a story about someone who hit it big immediately — it's seven years in, one rough patch, then a jump the following year, once you read it in order.

Two footnotes. "17 days to $1M" refers to ARR (the run-rate implied by that month's revenue), not $1M actually collected. And the 40,870-line index.php isn't a joke — he states it himself, the result of over a decade sticking to plain PHP with no framework.

One infrastructure number worth including: as of 2026, he's serving 4 billion requests/year across all his sites on self-managed VPS infrastructure for $2,932/year. By his own estimate, the same setup on managed cloud services would run $338,913/year — a 115x gap.

The failure numbers

This is the part that matters. In a post titled "List of all my projects ever," he self-categorizes every project he's ever built — 113 total. Of those, 19 are tagged "Failed" (didn't take off / never made money). That's about 17%.

Reading this as "so the other 83% succeeded" would be the wrong takeaway. Of the 113, at most 9 could reasonably be called revenue pillars. Most of the rest carry no success/failure tag at all — they're experiments that quietly died. The 19 "Failed" tags are just the projects he explicitly admitted failed; as an actual failure count, it's closer to a floor than a ceiling. By hit rate, we're talking under 10%.

In other words, behind the handful of projects that get named as wins, there are close to a hundred projects nobody's heard of. A few have documented reasons for failing:

Tubelytics (2013–2014) — a YouTube analytics dashboard. About a year of development; the interface was polished, but in his words, "nobody paid for it." He's said he's "so traumatized by this" that it became a standing rule: "I'm not going to build anything until there's customers."
Uber Clone for Netherlands (2010) — he and a friend pitched wealthy investors for funding. His own summary: "It turned out to be a massive time waste and they never invested in anyone and it was quite humiliating." This appears to be the origin of his distrust of VC funding.
Panda Mix Show (2008–2014) — his first success, a YouTube music channel. Peaked around $8K/month, declined to $5K as copyright claim handling got harder, then shut down. He's written that this income decline, combined with a depressive period, was the direct trigger for the well-known "12 Startups in 12 Months" project. That project wasn't a relaxed side experiment — by his own account it came out of a rough stretch, as a way to force himself to do something.
Hoodmaps (2017–) — self-tagged "Failed," yet still running as of 2026. A live example of a stated principle: if upkeep cost is near zero, leave a failed project running rather than shutting it down.

The remaining roughly 15 are dismissed with a line like "never took off," with no real postmortem. There's an asymmetry worth naming here: the more successful a product, the more detail he gives in hindsight; failures get a sentence. Any read of self-reported data needs to account for that skew.

Seven preconditions that don't transfer

Reading 751 posts in order also surfaces the conditions under which this approach worked — the things to subtract before trying to copy it:

300,000+ X followers, a distribution channel built up over roughly a decade since 2013. The leverage of "launch across multiple channels at once" depends on already having this.
2013-era market conditions. He's said, in effect, that back then it was a nearly uncontested space; now everyone, including large companies, is competing on the same turf.
Low-cost living in Southeast Asia underpinning his runway math (Bangkok at roughly $240/month, $3,000 in savings lasting a year).
A decade of honing plain PHP. "You don't need a framework" isn't beginner advice — it's the endpoint of someone who repeated the same pattern for ten years.
Low stakes from running solo. Running AI agents with bypassed permissions on production servers is viable partly because, from his position, a mistake isn't catastrophic.
His VC distrust is a generalization from one bad experience, not an industry-wide analysis.
AI product margins track GPU pricing directly — he's mentioned suppliers raising prices on him after seeing his revenue numbers posted publicly.

What's still worth taking

Even after subtracting all of the above, a few things seem genuinely portable for an indie developer anywhere:

Don't build until you know there's a customer. This carries weight specifically because it comes from a wasted year on Tubelytics. In practice: put up a payment link and a signup form before writing any code, and confirm someone will pay first.

If upkeep costs next to nothing, don't kill a failed project. The Hoodmaps pattern. Redefining "failure" as "does it cost money to keep running" rather than "did you shut it down" lowers the cost of staying in the game.

Make your failures countable, and publish them. Someone self-reporting 19 "Failed" tags out of 113 projects is rare, even among people who've succeeded publicly. Plenty of people write success stories. Far fewer publish the failure denominator — and that's what it takes to actually talk about odds.

One last thing. In systematic trading, there's a well-worn discipline: don't trust a good track record until you've verified whether it's skill or luck. I think public revenue numbers deserve the same treatment. $420,000/month is self-reported, not an audited statement. I'm not doubting the number — but I think the right way to read this kind of material is with the premise built in that public numbers are a survivor's self-report. Reading all 751 posts was, in part, a way to check that premise for myself.

All figures are sourced from his own blog at levels.io (specific post titles are noted in the text above). Numbers reflect the value at the time each post was written, not current figures.

Originally published in Japanese on Zenn: https://zenn.dev/idonthaveapen/articles/levelsio-751posts-19-failures