In 1770, a Hungarian engineer named Wolfgang von Kempelen toured Europe with a chess-playing "automaton" that defeated Benjamin Franklin and, decades later, Napoleon — a cabinet of whirring gears that secretly concealed a human chess master folded inside. Two and a half centuries later, the cabinet has become a cloud dashboard, and the pattern of quietly disguising human labor as machine intelligence is the subject of a growing forensic literature, including a pointed examination of the staged demo and what it eventually costs once the audience finally opens the cabinet door. The last eighteen months handed developers two spectacular case studies in what happens when the person inside the machine is discovered — one ended in insolvency proceedings, the other in a quiet retreat from hundreds of grocery stores. Both are worth studying line by line, because the failure mode they share is one every engineering team is now one funding round away from repeating.
Seven Hundred Engineers Named Natasha
Builder.ai spent eight years selling a beautiful story. Founded in London in 2016 as Engineer.ai, the startup promised to make building custom software "as easy as ordering a pizza." Its star was Natasha, an AI assistant marketed as capable of assembling applications largely on its own. The story attracted more than $450 million from backers including Microsoft and Qatar's sovereign wealth fund, and a peak valuation of $1.5 billion.
The reality, pieced together by journalists and former employees, was that the heavy lifting behind Natasha was performed by roughly 700 human engineers, mostly in India, whose output was presented to customers as machine-generated. The warning signs were public remarkably early — The Wall Street Journal questioned the company's AI claims as far back as 2019 — but funding kept flowing anyway, because the demo kept working and nobody was contractually allowed to look inside the cabinet.
The unraveling, when it came, was fast and financial rather than technical. A new CEO discovered that reported revenue had been drastically overstated, and in May 2025 the company entered insolvency after Bloomberg reported that its lender seized $37 million from its accounts, leaving just $5 million to run a business that had once been a unicorn. Roughly 80 percent of the workforce — including the very engineers whose invisible labor had powered "the AI" — lost their jobs. The staged demo did not merely mislead customers; it set the valuation, the hiring plan, and the debt load, which meant that when the illusion collapsed, everything priced against it collapsed too.
A Thousand Reviewers Behind the Checkout Cameras
Amazon's Just Walk Out was a genuinely more honest venture, which is precisely what makes it instructive. The technology — cameras, shelf sensors, and computer vision that let shoppers grab items and leave without a checkout — was real and shipped in actual stores starting with Amazon Go. But reporting by The Information revealed that the system leaned on more than 1,000 workers in India who reviewed transactions and labeled video, and that in 2022 roughly 700 out of every 1,000 sales still required human verification, wildly above internal targets. Amazon disputed the framing, insisting the team primarily annotated footage to train models rather than acting as remote cashiers.
The distinction mattered less than the optics. In April 2024, CNBC reported that Amazon was pulling the cashierless system from its U.S. Fresh grocery stores in favor of smart shopping carts, and the "1,000 people in India watching you shop" narrative attached itself permanently to the product. Note the asymmetry: nobody was indicted, no revenue was faked, and the technology continues to run in airports and stadiums. Yet the gap between the promise ("AI does this") and the mechanism ("AI plus a large human backstop does this") was big enough to cost the flagship deployment and years of accumulated trust. The market punishes the undisclosed human in the loop even when the human is doing legitimate work.
Wizard of Oz Is a Method, Not a Business Model
Here is the uncomfortable nuance: hiding a human inside a prototype is a respected, decades-old research technique. UX practitioners call it Wizard of Oz testing — simulating a system's behavior with a concealed operator to learn whether the product is worth building at all. Startups call the same idea a concierge MVP. Used correctly, it is one of the cheapest ways to validate demand before writing expensive software. The technique only turns toxic when specific lines get crossed, and the two cases above show exactly where those lines sit:
- Duration. A wizard behind the curtain for a six-week usability study is research; the same wizard eight years into a commercial product is deception.
- Audience. Misleading a test participant who signed a consent form is ethics-board territory; misleading paying customers and investors is fraud territory.
- Money. The moment valuations, contracts, or debt are priced on the simulated capability, the simulation becomes a financial instrument — and financial instruments get audited.
- Disclosure. "Human-assisted, working toward automation" survives scrutiny indefinitely; "fully autonomous" survives only until the first leak, deposition, or disgruntled engineer.
What Builders Should Actually Do
For working developers, the practical lesson is not "never fake anything" — it is that the temporary human backend must be an explicitly tracked liability, like any other form of debt. Put it in writing: which capabilities are simulated, who knows, and what the automation roadmap and sunset date are. Push back when marketing collapses "human-in-the-loop" into "autonomous," because the engineers are always the first witnesses called when the gap surfaces. And when evaluating someone else's miracle demo, ask the one question the Mechanical Turk's audiences never did: what, exactly, is inside the cabinet, and how do I verify it myself?
The chess master hidden in von Kempelen's automaton was eventually exposed by curious journalists in the 1820s, and the machine itself burned in a museum fire. The modern versions end the same way, just faster — the curious journalists now have subpoena-ready sources, and the fire is a creditor with account access. Build the real thing, or label the person behind the curtain. Those are, and have always been, the only two durable options.
Top comments (0)