DEV Community: Rodrigo Nobrega

I Hid 40 Bugs in an Online Store and Let an AI Play QA Tester

Rodrigo Nobrega — Fri, 16 Oct 2026 11:06:18 +0000

Every team I've worked with has shipped something that passed its automated checks but still confused people. The button did its job, but nobody could find it. The form submitted, but only after you'd typed your phone number three different ways. Automated tests only catch what someone thought to write an assertion for. A good human QA tester is the one who notices what nobody wrote down, and says "this is annoying."

So I wanted to know: can an AI agent do that part of the job? Not "does the page load," but "would a real person get confused here?"

And if it can, how much does it depend on how you ask?

A coffee shop with 40 problems hidden in it

To test a tester, you need to know the right answers in advance. A real website doesn't give you that. If the agent reports five issues, you can't tell whether it missed ten or there were only five.

So I made a fake one: Ponto Coffee Roasters, a small online store with a shop page, product pages, a basket, a two-step checkout and a sign-up form. There is no real server behind it. A small piece of code pretends to be one, with random delays and an order that always fails on the first attempt, because how an app behaves while waiting and failing is a big part of user experience.

Then it got filled with problems, from two sources. Claude built the store and planted the first 20 bugs. I then asked Codex, a different AI, to introduce 20 more, and kept that request and the new bug list outside anything Claude could see. Claude was also the model doing the testing, so I didn't want all the bugs to come from the same model that would be hunting them.

The 40 bugs cover a wide range. There's pale grey text you can barely read, an "Add to bag" button that gives no sign it worked, and a review step that says shipping is "Free" while the total quietly includes €6.90. The basket's minus button actually adds coffee, the logo takes you to the basket, and on a phone the checkout spills off the screen. Some are quiet: the search box clears itself when you click away, and the surname field stops accepting letters after eight characters.

Three ways of asking

The experiment changed only one thing: the prompt.

Level 1 was the one-liner most people would type first:

You are a QA engineer. Test the website and look for UX errors.

Level 2 gave the agent a role (a senior QA tester who cares about user experience), some context about the store, a way of working (main journey first, then mistakes and small screens) and a report format.

Level 3 was level 2 plus a full checklist, built on Jakob Nielsen's 10 usability heuristics (a classic list of what makes interfaces easy to use) and the basics of WCAG, the web accessibility guidelines. It also told the agent to test in passes: desktop, mistakes, waiting and failure, phone size, keyboard only.

The checklist was written before any bug existed. Otherwise it might have described the answers without me noticing.

Keeping it fair

Every run used the same agent and model: Claude Code, Anthropic's coding agent, running Claude Sonnet 5.5. It drove a fresh browser through Playwright, a tool that lets an AI click, type and look at web pages. The browser was its only tool. It couldn't read files or use a terminal, so it couldn't peek at the source code or the answer key. It could only use the site the way a visitor does. Each level ran three times, because AI output varies from run to run.

Every report was scored against the answer key: a full point if a bug was clearly described, half a point if it was only touched on, nothing if it was missed. I'll call the result a weighted detection score. To reduce scoring subjectivity, a separate AI agent scored all nine reports against the answer key and scoring rules, without seeing my scores. Its numbers came within half a point of mine on every report.

The results

Prompt	Runs	Average score	Percentage	Time per run	Cost per run
Level 1: one-liner	24, 25, 25	24.7 / 40	62%	1.7 min	$0.51
Level 2: role + context	21, 22, 23.5	22.2 / 40	55%	1.9 min	$0.51
Level 3: full checklist	27.5, 31.5, 32	30.3 / 40	76%	3.5 min	$0.81

The first surprise is right there in the middle row.

Surprise one: the "better" prompt did worse

I expected each level to beat the one before. Level 2 lost to the one-liner.

Level 2 produced longer, more polished entries, with a title, location, impact and severity for every issue, and it scored lower. The one-liner runs wrote quick, dense lists. The reporting format may have contributed, but these runs don't isolate its effect from differences in how each agent explored the site. It's a pattern worth testing, not a conclusion.

Surprise two: a checklist changes behaviour, not just wording

Level 3 didn't only describe things better. It did different things.

It was the only level that ever pressed the Tab key. That's how you test a site for people who don't use a mouse, and it's how all three level 3 runs noticed there was no visible keyboard focus. None of the six other runs caught it.

Every single run, at every level, shrank the window to phone width at some point. But only level 3 caught the checkout overflowing on a phone, in two of its three runs. None of the level 1 or 2 runs reported it, and one of them stated there was "no horizontal scroll at 375px."

The checklist worked like a test plan. It didn't make the agent smarter. It made it go to places it otherwise skipped.

Surprise three: a checklist also narrows attention

The opposite happened too. The newsletter box in the footer has a spinner that never stops. One of the one-liner runs clearly reported it, and the other two got partial credit. None of the six runs with detailed instructions got any credit. They followed the journey they were given, and the footer wasn't on it.

I've seen this in people before. Give a tester a checklist and they'll cover the checklist. Freedom finds odd things, and structure finds thorough things.

The bugs nobody found

Five bugs were missed by all nine runs. Four came from Codex's set:

the search box only accepts 12 characters
the search clears itself when you click away
the surname field stops at 8 characters
the Huila Sunrise card on the shop page shows another coffee's picture

The fifth was one of Claude's: double-clicking "Add to bag" adds the item twice.

Most of these have one thing in common. They don't produce an obvious error message or broken layout. Finding them requires trying a boundary value or comparing behaviour across interactions. None of the reports identified the image mismatch between the shop and product pages.

The double-click bug is a messier case, and it taught me something about the experiment itself. Codex's set also made a single click add two items, so a double click added four. Because one click already added two items, the repeated-click issue was harder to distinguish. Planted bugs can hide each other.

The bugs everyone found

On the other hand, every run reported the logo leading to the basket and the password shown in plain text. The same went for the "Free" shipping that wasn't, the basket row that ignored quantity and the price that changed between pages: all nine runs caught these numerical inconsistencies, including the one-liner runs.

The scorer flagged at most one false alarm per run, mostly complaints about slow loading caused by my test environment rather than the app. Every run also found real problems nobody planted, like form fields with no autocomplete hints, error messages that screen readers wouldn't announce, and a header that still said "Create account" after you'd made one.

What I'd be careful about

This is a small experiment. Nine runs with one model is a hint, not proof, and another model could shift the numbers. The tests ran in a cloud sandbox that blocked web fonts, which isn't quite a real visitor's experience. All 40 bugs came from AIs, two different ones, so a human would probably plant different problems. And some planted bugs interacted: the clipped product description partly hid the wall-of-text problem, and the doubled "Add to bag" muddied the double-click bug. That changes how visible some bugs were, and it means the five universal misses don't all point to a single weakness in the agent.

What I'd try next: a checklist that learns

I haven't tested the workflow below yet. It's what I'd try next.

The checklist made the agent thorough, and the one-liner's freedom made it curious. So I'd run two complementary passes before each release: a checklist run, and a free-form run told to explore and report anything odd. The free run's job isn't to find more bugs. Looking back at my saved scores, I combined runs in pairs, giving each bug the best credit either run earned. One free-form run plus one checklist run averaged 31.6 out of 40, and two checklist runs averaged 32.3. That's a retrospective calculation, not a separately tested workflow, but it suggests a free run won't raise the score. Its job is to find the bugs that aren't on the checklist yet.

Those discoveries, along with what human testers find and what real users complain about, feed the checklist. Some become general habits, like "type an unusually long value into every field," and stay there because they need judgment.

Others are specific regressions, like "the newsletter spinner must stop." For those, the checklist is a cheap placeholder. Adding a line takes seconds. An end-to-end test (a script that clicks through the site the same way each time, written with a tool like Cypress) takes real attention to write. So the line goes on the checklist first. Later, when there's time or the list gets long, it graduates into a Cypress test and leaves the list.

Neither half is a guarantee. My own newsletter results show that an instruction in a prompt doesn't ensure the agent follows it, and scripted tests can be flaky too. But the split puts judgment where AI helps and repetition where scripts are cheaper.

So, can an AI be your QA tester?

In this small experiment, a browser-driven AI found many of the planted functional and usability problems, quickly and cheaply. The best setup scored 76% for under a dollar a run. A checklist improved coverage, but several quiet bugs went unnoticed in every run. I'd use it as a first pass before a release, not as a replacement for a person who types a long name, clicks away by accident and thinks "wait, didn't this look different a second ago?"

The thing I'll take away is about prompting. A prompt for an AI tester works a lot like a test plan. What you put in it tends to get covered carefully. What you leave out may never be visited. And like any good test plan, it should never be finished: every bug the agent misses today is a candidate line for tomorrow's checklist.

The whole experiment is open: the store, the 40 bugs, the three prompts, every agent's report and the scores. You're welcome to run your own prompt against it and see how many you can catch. github.com/rodrinobrega/agent-qa-experiment

Have you tried using AI agents for QA or UX reviews? I'd love to hear what they caught for you, and what they walked right past.

I built a blind AI pipeline to catch silent decisions in AI-written code. A single prompt did just as well.

Rodrigo Nobrega — Thu, 15 Oct 2026 12:14:58 +0000

TL;DR: I spent a few weeks building a pipeline that finds the decisions an AI coding agent makes without telling you. Its core idea was that the reader must never see the plan. Then I ran the comparison I'd been putting off: a single prompt that just asks a model to review the code against the plan. On my two test cases it found as many silent decisions, with better precision, in about a minute and at roughly a seventh of the cost. Here's the idea, how the pipeline works, and why it lost.

The problem: code that quietly decides things

You write a plan. An AI agent implements it. The code is clean and every test passes. Somewhere inside it, the agent decided that withdrawals are capped at 5,000 a day, that amounts round half-up, and that opening an account that already exists does nothing. None of that was in the plan.

I call these silent decisions. Each one is reasonable on its own. The agent had to pick something, and it picked without asking.

The tests don't catch them, because the tests came from the same reading of the same plan. If the agent decided overdrafts are rejected, its tests check that overdrafts are rejected. The suite is consistent with itself, and you find out it was the wrong call after deploy.

Code review doesn't reliably catch them either. A reviewer reads the code with the plan in mind, and a rule that looks sensible reads as intended. The question "did anyone actually decide this?" rarely gets asked.

The idea: start from the code, not the plan

Plan-compliance checks, like Traycer's verification pass or Qodo's ticket compliance, start from the plan and ask does the code do what was asked? I wanted to ask the opposite question: what does the code do that nobody asked for?

That means the thing you check has to be derived from the code, not from the plan. The plan tells you what was intended. The code is the only record of what was actually built.

It also changes the shape of the problem. "What's missing from this plan?" is an open-ended search over everything that could have been specified. "Here are 40 things this code does. Which of them can you point to in the plan?" is a matching problem over two concrete lists. Matching is something you can check.

The catch is that whoever describes the code must not have read the plan. If they have, they'll describe the code in the plan's words and every decision will look intended. So the describer has to be blind.

That was the hypothesis the whole pipeline is built on. As you'll see, the data didn't support it.

The pipeline

The tool is called silent-decisions. It runs after the agent has finished implementing, in six steps. The plan only enters at step 4, and even there no model sees more than one sentence of it at a time.

Prepare. Copy the source into a "clean room" with comments, tests, docs and git history removed. Agent-written code often carries the plan in its comments (// per spec: reject overdrafts), and that would break blindness. A plain AST walk also lists every decision point: literals, comparisons, rejection paths, rounding calls.
Extract, blind. A fresh agent that has never seen the plan reads the clean room and writes what the code does as concrete scenarios with real values. Each scenario comes with a contrary: the outcome a different, equally reasonable implementation would give.
Run everything. Every scenario becomes a test and is executed. Each also runs with a perturbed twin that asserts the wrong outcome, and the twin must fail. What survives is a verified description of behaviour, not a model's opinion.
Trace back to the plan. For every pair (plan sentence, scenario), a small decision-only classifier (Jev, from TypeSafe AI) answers one question: does this sentence require what the code does, require the contrary, or neither? It's asked twice with the two options swapped, and answers that move with the position are thrown away. A scenario with no sentence behind it is a silent decision.
Probe forwards. Every behavioural plan sentence gets a probe test. If the probe fails, the code dropped a requirement.
Control the run. A few plan sentences are hidden and the trace is repeated: behaviours that depended on them must come back unsourced. A canary string checks that the extractor really was blind. If a control fails, the run doesn't count.

What you review at the end is only the delta: the silent decisions and the dropped requirements. Each one is a card that asks you to choose:

Withdrawals are capped at 5,000 per day Given an account with balance 10,000 that has already withdrawn 4,000 When a further 1,500 is withdrawn Then the withdrawal is rejected with reason daily-limit

A. What the code does: rejected, balance stays 6,000 B. The alternative: accepted; there is no limit

You pick A or B, and your answer becomes a signed regression test. You read a handful of decisions instead of the whole diff.

The experiment

A pipeline this elaborate only makes sense if it finds things that cheaper methods miss. So I designed six "arms" to run on the same plan and code, and scored them against the same answer key.

Arm	What it does	Does the code reader see the plan?
a	Self-report: tell a model it implemented the code and ask "list the decisions you made that the plan didn't specify"	Yes
b	Sighted review: one pass that reads plan and code together and lists decisions	Yes
c	Full pipeline, but the extractor has read the plan	Yes
d	Full pipeline with an LLM tracer and adversary instead of the classifier	No
e	Full pipeline: blind extractor, execution, pairwise classifier, probes	No
f	Cheap static baseline: match each decision point from the AST walk against each plan sentence	No model reads the code

Arms a and b are the ones that matter. If a one-line prompt finds the same decisions, the rest of the machinery isn't earning its cost. Arm c tests the core idea directly: same pipeline, but the extractor is allowed to read the plan. If blindness matters, c should do worse than e.

There were two test cases, both small TypeScript ledgers with planted decisions:

ledger: 10 silent decisions (5 planted up front, 5 added to the key after the first run, from reading the pipeline's output), 1 dropped requirement, 2 decisions the plan does make (to catch tools that flag everything).
ledger-rounding: the same core plus rounding rules for fees, currency, interest and splits. 22 silent decisions, 3 dropped requirements, 8 planned rules.

Both cases were written on my side, with an AI agent doing most of the writing and running. Keep that in mind for everything below.

I developed the pipeline over five runs (arms e and f), then ran arms a to d in two more runs on the same code. Finally, every output from every arm was graded blind: a fresh model session got the plan, the code, the answer key and the shuffled lists, labelled L01 to L13, without being told which method produced which. Recall is the share of planted silent decisions found. Precision is the share of flagged items that are real unplanned decisions.

What happened

On the harder case, the one-minute prompts matched the full pipeline. These are blind decision-level grades on ledger-rounding (22 planted decisions):

Arm	Runs	Recall	Precision	Time per run	Cost per run*
a. Self-report	3	0.83	0.89	~1 min	~$0.30
b. Sighted review	3	0.82	0.93	~1 min	~$0.30
c. Pipeline, sighted extractor	1	0.86	0.82	~7 min	~$2.20
d. Pipeline, LLM tracer	1	0.91	0.77	~10 min	~$3.10
e. Full pipeline (blind)	3	0.83	0.85	~7 min	~$2 + classifier calls (not fully logged)
f. Static baseline	1	0	n/a	5 s	~$0

* API-equivalent cost reported by Claude Code for claude-opus-5-5. The real spend was subscription usage.

On the small ledger case every model-based arm scored between 0.90 and 1.0, so it can't separate them. Every arm caught every dropped requirement on both cases.

Arm f scores zero because of its format, not because it found nothing. It reports code locations ("a decision at DAILY_WITHDRAWAL_LIMIT = 5000") rather than rules, and the grader was told an item must state the decision. An earlier, more lenient grading gave it about 0.5.

A few details matter more than the averages:

The pipeline does catch something the prompts miss. "Re-opening an existing account does nothing" was missed by the prompts in 11 of 12 runs and found by the pipeline arms in 9 of 10. One caveat: this is exactly the miss I fixed after run 004, and every pipeline run graded on the harder case came after that fix.
The prompts found more real decisions that weren't in my answer key: about 9 to 10 per run against 0 to 4 for the pipeline. Some of that may be free text being easier to accept than an executed scenario, but the gap is large.
The hardest decision was missed by everyone. "Transfers don't count towards the daily withdrawal limit" was missed by every arm in every run. That's not a weakness of blind extraction. It's hard for any model reading this code.
The grader agreed with my earlier grades of the pipeline to within one decision per run, so the earlier results weren't inflated by my own grading.

Why it lost

Blindness didn't help. The whole design rests on one hypothesis: a model that has read the plan will describe the code as matching it, so the extractor must never see the plan. Arm c tested exactly that, and the pipeline with a sighted extractor did as well as the blind one (0.86 vs 0.83). The sighted review didn't rubber-stamp the code either. It had the best precision of any arm.

What the machinery buys is verification, not discovery. Every scenario the pipeline reports has been executed, and every dropped requirement it reports is confirmed by a failing test. The prompts' claims aren't checked against the code at all. That's worth something, and it's where the one consistent extra catch came from. On these cases it isn't worth seven times the time and seven to ten times the cost.

The limits are real, and they cut both ways:

Small, self-made cases. Both are a single file of about 140 lines, written on my side with an AI agent, with every rule an explicit if or constant. That's exactly where one read-through should do best. On real code, with rules spread across many files, the result could change.
One model grader. It was blind to the method, but it's a model, not a person, and the arms write differently (free text vs Given/When/Then), so it could have guessed which family a list came from.
Few runs and lots of variance. c and d ran once. Three runs of the same arm differed by up to about 0.2 recall.
Arm a isn't a real implementer. It's a fresh session told it wrote the code, with no memory of doing so.
The pipeline was tuned on these cases over five runs. The prompts got no tuning at all, and they still matched it.
On the small case, the answer key leans towards the pipeline. Half of ledger's 10 decisions were added after reading the pipeline's own output. On the 5 original ones, every model-based arm scored 1.0. The harder case was graded on the original key only, so the headline numbers aren't affected.

What I learned

Run the dumbest baseline first. I did build a cheap baseline, but the wrong one: a static scan that was easy to beat. The baseline that mattered was a one-line prompt. It took about a minute to run and it settled the question that five runs of pipeline tuning never could.

Write down what would prove you wrong, then actually run it. My own method write-up said that if the pipeline couldn't beat a one-line prompt, the prompt does the job and that's the finding. That turned out to be the finding. I could have known it weeks earlier.

Execution is the part I'd keep. Turning every claim into a test that actually runs, with a twin that must fail, worked in every run. Whatever replaces the rest of the pipeline, "show me a scenario that runs" is a better unit of review than "the model says".

Don't let the AI grade its own homework. An AI agent wrote a test case, ran the experiment, graded the results and proposed fixes, all competently. None of that could tell me whether the method worked. What settled it was running the comparison I'd been avoiding, and having every arm's output graded blind, shuffled and unlabelled, by a session that didn't know which method it was judging.

The closest published work is AssumptionMiner, which extracts implicit assumptions from LLM-generated code with the prompt in view and without running anything. Arm b is roughly that kind of approach, a single sighted pass, and on these cases it matched the pipeline.

What's next

The next step is probably a skill, not this pipeline: one well-prompted review pass (arm b's prompt is the starting point), with execution as an optional check on its claims. The cheapest experiment left is to feed arm b's list through the pipeline's execution and probe steps and see whether it matches the full pipeline's verified output.

The question that would actually settle this is independent cases: real repositories, with answer keys written by someone who hasn't seen any tool's output. That's where rules are spread out and harder to see in one pass, and where blind extraction plus execution might still pull ahead. Or might not.

The code, both test cases, the answer keys and a diary of every run (including what went wrong) are on GitHub: rodrinobrega/silent-decisions. Every run is tagged in git, and the blind grading packets are included so anyone can re-grade them. It's an experiment, not a product.

Code Preview: Reviewing the Plan Before the AI Writes the Code

Rodrigo Nobrega — Mon, 12 Oct 2026 13:45:58 +0000

AI made writing code cheap. It did not make reviewing code cheap. On my team, that gap became the real bottleneck. So I proposed a change to how we work: move the most important review upstream, and review the plan before any code exists. The team adopted it, and it is now part of how we build. We call it the code preview.

There is nothing revolutionary here. Design reviews, RFCs and architecture decision records have existed for decades. What changed is the economics. When a developer and an AI can produce a 1,500-line pull request in an afternoon, the expensive part is no longer typing. It is a reviewer trying to reverse-engineer intent from a wall of generated code, and discovering on line 900 that the whole approach was wrong.

Some teams are already living the worst version of this. A developer who had just joined a big company described it publicly: specs, tickets, code and reports all generated by AI, nobody reading anything, and no time given to understand where things are going.

The problem in that post isn't that the team uses AI. It's that nobody is reviewing intent anymore. Output goes out faster than anyone can understand it.

A widely shared response came from Mitchell Hashimoto, who proposed a benchmark he calls the "whiteboard defense":

He doesn't ask engineers to know every line. He asks them to explain how a system works and defend its decisions: why X instead of Y, what happens with a malicious actor, where it fails. And he draws a clear line: prototypes and demos can be fully generated, customer-facing work cannot ship without high-level understanding.

I agree with every word. The code preview is how my team tries to meet that bar in practice. The whiteboard defense is a test you can be given at any moment after shipping. The code preview builds the answers before any code is generated: the "why X instead of Y" lives in the plan's alternatives section, the failure modes in its risks, and the whole team has already challenged them.

Reviewing a plan written in plain language is much cheaper than reviewing the code that comes out of it. That is the whole idea. The rest of this post is about how we run it, what we learned in practice, and where it falls short.

How the code preview works

The plan is a first-class artifact in git, reviewed like code, before code is generated.

A developer picks up a business requirement and investigates it: the codebase, the constraints, the options.
They write a plan.md, usually with AI help, and commit it to a dedicated plan branch.
A short kickoff meeting presents the plan to the team. This is the only synchronous part. Its job is shared context, the why and the big picture, not line-by-line scrutiny.
The team reviews the plan asynchronously. Everyone has access to the plan branch and can point their own AI tools and skills at it: check it against the codebase, hunt for missing edge cases, compare it with existing patterns.
Once the plan is approved, the code is generated on a feature branch that the plan branch merges into.
Code review follows, against a plan the reviewer already understands.

Only the reasoning survives the trip to main; the plan itself is removed before the merge.

We originally called the meeting the "solution defense." I renamed it. "Defense" made it sound like the developer was on trial, and people who feel on trial argue instead of listening. "Preview" describes what it really is: a look at the code before it exists. The defense still happens, as Hashimoto describes it, but as a conversation the team has together before anything is built.

The part I find most interesting is step 4. AI sits on both sides of the plan. One AI helps write it, and several others, configured differently by different people, stress-test it. Reviewers with different setups catch different things.

The best catch wasn't a bug

The most valuable thing a code preview has caught for us wasn't a mistake. It was a missed opportunity.

A colleague presented his plan for a new third-party integration. It used an outbox pattern so the integration could deliver reliably even when the external API misbehaved. It was a solid, well-reasoned plan.

During the discussion, others realized the same problem existed well beyond his integration. Other parts of the system talk to third-party APIs too, and they need the same guarantees. The conversation shifted from "is this the right outbox for this integration?" to "why don't we have one outbox for the whole application?"

We left with a broader scope: a single outbox implementation that every third-party integration uses. Every integration gets the same robust behavior, and nobody has to reinvent it. His work became the foundation for the rest of the system instead of one more local solution.

A shared outbox isn't the right call for every system, and that trade-off was part of the debate. In our case, the integrations' contracts and delivery guarantees were close enough that one general implementation made sense. We agreed to start shared and split it only once a real need proves it.

A code review might have caught this too, but by then we would already have invested in the local solution. Catching it at the planning stage meant we could change direction before any code was written. AI coding agents can scan a codebase for reuse, but they usually lack the organizational context, the knowledge of what others are about to build, and the mandate to question a ticket's scope. That took several people with context on different parts of the system, looking at the plan together while changing course was still cheap.

One question decides everything

Every judgment call in this process comes down to the same question: is this architectural, or will other parts of the system depend on it? We ask it at three moments.

Should this task get a code preview at all? It is up to the developer. A new report can skip it. A different frontend component in an admin panel can skip it. An outbox pattern for a new integration should be presented and defended. A simple rule of thumb: preview anything that is expensive to undo or that other parts of the system will build on. It's the same line Hashimoto draws between a demo and customer-facing work.

Does a mid-course correction need re-approval? Implementation always teaches you something the plan missed. A naming convention change or a small database index doesn't need another round. Changing a deep integration pattern, for example adding a deduplication table where the plan had none, goes back for approval.

What survives into the permanent docs? The reasoning behind architectural decisions does. The step-by-step implementation does not (more on that below).

One principle applied at three moments is what keeps the process from turning into a pile of rules. It also leaves room for developer judgment, which is the point. We are not trying to replace judgment with ceremony.

Plans are transient, docs are permanent

A plan never reaches main. The plan branch merges into the feature branch, and the plan.md must be removed before the feature merges to main.

The reason is a distinction I think many teams blur. A plan describes how to build something. System documentation describes how something works. Once the feature ships, the first kind of document is history, and history that sits next to living docs becomes a liability. Repos fill up with stale, half-true markdown: plans that were changed during implementation, steps that no longer match the code.

This matters more with AI in the loop. Coding assistants read the repo as context, and they can't tell an outdated plan from current documentation. A stale plan.md on main is a prompt that confidently describes a system that doesn't exist.

That doesn't mean the plan's reasoning is thrown away. A good plan explores the alternatives and explains why they were discarded, for example: "we will call the third-party API asynchronously, so a slow or failing provider never blocks a user-facing request." That reasoning belongs in the final docs too, and the docs are part of the deliverable. So when a feature ships, the plan's scaffolding (the steps, the file lists, the task breakdown) is discarded, and its reasoning is promoted into permanent documentation.

Two gates, not a bottleneck

My biggest concern when we started was obvious: we were adding a second approval gate. Plan review, then code review. That sounds like twice the waiting.

In our experience, it worked the other way: reviewing the plan made the code reviews that followed more focused and easier to navigate.

Code review doesn't get lighter. Reviewers still check the code properly. It gets easier because the reviewer no longer has to learn the semantics of the change from the code itself. They already learned them from the plan, written in natural language, days earlier. Code review becomes a comparison: does this code do what we agreed it would do? That is a much easier question than "what is this code trying to do, and is it a good idea?"

The two questions also get answered by the right people at the right time. Whether the approach is right is debated while changing it costs nothing. Whether the implementation is right is checked against a reference everyone shares.

What makes a good plan.md

A good plan is short enough to review in one sitting and specific enough that the AI can generate code from it without guessing. The section that matters most is the one most plans skip: alternatives considered.

A skeleton that works:

# Plan: <feature name>

## Requirement
The business problem, in a few sentences.

## Proposed solution
The approach, the main components, how data flows.

## Alternatives considered
Each option and why it was discarded.

## Impact
What existing parts of the system change or depend on this.

## Risks and open questions
What could go wrong; what the team needs to decide.

## Implementation steps
The ordered steps the code generation will follow.

## Docs to update
Which system docs this feature changes.

The last two sections map directly onto the lifecycle above. Implementation steps are scaffolding and disappear with the plan. Alternatives considered and docs to update are what carry the reasoning into permanent documentation.

A plan written this way doubles as the best prompt you can give a code generator. The team effectively reviews the prompt before it runs.

Pros and cons

The trade is more upfront thinking in exchange for less wasted generation and faster code review. Here is how it has played out.

What works

Wrong approaches die early. They are caught in a markdown file, before tokens, review time or rework are spent on them.
Code reviews are more focused, because reviewers compare code against a plan they already understand.
The plan is the prompt. A plan the team has challenged produces better generated code than one person's first idea.
Developers keep thinking. Writing a plan someone will scrutinize forces you to understand the problem instead of delegating the thinking to the AI.
Diverse review. Each reviewer brings their own AI tools and skills, so the plan is tested from several angles.
Shared context. The kickoff means the whole team knows what is being built and why.
Cleaner docs. Transient plans stay off main, and the reasoning that matters is promoted into permanent documentation.

What costs

Upfront time. Writing and reviewing a plan takes time before any code exists. For small tasks it is pure overhead, which is why skipping is allowed.
It relies on judgment. Whether to preview, and whether a correction needs re-approval, are developer calls. That flexibility is deliberate, but different people will draw the line in different places.
Plans can look right and still be wrong. Some problems only appear in code. The process reduces surprises; it doesn't eliminate them.
Reviewing a plan is a skill. Reviewers have to engage with prose seriously, not skim it and approve.
The "why" depends on discipline. If no one promotes the reasoning into the docs before the plan is removed, it survives only in git history.

Review intent, not just output

AI moved the cost of software from writing code to understanding it. The code preview is our response: put human attention where decisions are made, in a plain-language plan, and let the generated code be checked against it.

If you want to try it, start small. Pick the next architectural change on your board, write the plan, put it on a branch, and ask the team to review it with their own tools before a single line is generated. Then notice how the code review feels.

I'd like to hear how other teams handle this. Do you review plans, prompts, or only the final code?

What LiveOps Really Is and What It Can Achieve

Rodrigo Nobrega — Mon, 08 Dec 2025 07:55:27 +0000

What LiveOps Really Is and What It Can Achieve

The foundation of a more dynamic, iterative, and resilient way to run software.

Introduction

“LiveOps” has become one of those phrases that gets thrown around so often that its meaning becomes blurry. Some think it’s just A/B testing. Others assume it’s a dashboard for operators. Many in gaming associate it with seasonal events and content drops. In SaaS, people use it interchangeably with “feature flags.”

But LiveOps is much more than any single tool or capability.

At its core, LiveOps is an operational philosophy: the practice of continuously tuning a product in real time based on how users behave.

It’s the shift from build → release → hope for the best to build → release → observe → iterate → optimize, continuously.

This post sets the foundation for the rest of the series by defining what LiveOps really is, why it matters, and what it can unlock for your product—whether you're running a game, SaaS platform, mobile app, or consumer marketplace.

The Evolution of Operating Software

From release cycles → continuous delivery → continuous operations

Software used to ship in big releases. You planned features, built them, released them, and maybe patched them if something went wrong.

Then CI/CD transformed how teams deliver code—releases became smaller, safer, more frequent.

LiveOps is the natural next step.

It doesn’t replace CI/CD; it extends it. Even with rapid deployments, real-world user behavior can’t be fully predicted. LiveOps acknowledges this and gives teams tools to adapt after code ships.

With LiveOps, the product no longer feels static between deployments. It becomes something that evolves in real time.

So What Is LiveOps?

Let’s clear up the confusion:

LiveOps is the capability to operate your product dynamically through configuration, experimentation, and real-time data—without shipping new code.

LiveOps is not:

a dashboard
a feature flag system
an analytics suite
an A/B tester
a content pipeline
an admin console

But it includes parts of all of these.

A more precise definition:

LiveOps is the closed-loop system that enables teams to:

Release features safely
Observe how they perform in the real world
Adjust configuration or content dynamically
Iterate based on live signals rather than assumptions

Users experience the product differently based on:

their segment
their behavior
ongoing events
dynamic feature configurations

This loop repeats—continuously.

The Core Capabilities of LiveOps

LiveOps systems differ across companies, but most share these fundamental components:

a. Real-time Feature Configuration

Turn features on/off, tune parameters, change thresholds, and modify behavior instantly.

Example: Change matchmaking rules on the fly based on load.

b. Experimentation (A/B Testing & Variants)

Test multiple versions of features or content. Measure real impact before rolling out globally.

Example: Two onboarding flows to learn which produces higher retention.

c. Segmentation & Personalization

Deliver different experiences to different groups of users.

Example: Target new users with special offers or reduced complexity.

d. Live Events & Time-Bound Changes

Schedule events, promotions, or content changes without code deployments.

Example: A weekend XP boost in a game; a 24-hour sale in an e-commerce app.

e. Observability & Decisioning

Use real-time telemetry to detect issues or opportunities and respond quickly.

Example: Auto-disable a feature if error rates spike.

What LiveOps Can Achieve

Companies adopt LiveOps for two reasons: resilience and growth.

1. Faster Iteration, Higher Confidence

LiveOps lets teams test ideas faster, expose them to small audiences, and roll out safely.In turn, this:

reduces risk
accelerates innovation
cuts development cost

No more waiting weeks for insights.

Dynamic, Personalized Experiences

Products feel more alive when content adapts to users.

This increases:

engagement
retention
conversion
satisfaction

LiveOps is one of the reasons modern mobile games retain users for years.

Operational Resilience

LiveOps acts as a safety net.

Example:

A feature causes increased latency? Disable it instantly.

Traffic spikes unexpectedly? Adjust configurations to shed load.

This reduces outages and downtime.

Revenue Uplift

For products with monetization (games, marketplaces, SaaS upsells), LiveOps enables:

targeted offers
optimized pricing experiments
usage-based triggers
seasonal campaigns

Small percent improvements compound dramatically at scale.

Real-World Examples of LiveOps in Action

Gaming

Games use LiveOps to:

run seasonal events
test content
tune difficulty
release dynamic challenges

Players keep coming back because the experience changes weekly.

SaaS

Product-led growth tools rely on LiveOps to:

optimize onboarding
enforce personalized UI flows
roll out new features to segments safely

E-Commerce

LiveOps powers:

flash sales
promotion targeting
dynamic pricing
real-time inventory reactions

Mobile Apps

Streaming, fitness, and finance apps run:

personalized recommendations
habit nudges
time-bound challenges