DEV Community: AI Flip Room

42 of our 71 regression guards stayed green when we broke the code on purpose

AI Flip Room — Sat, 10 Oct 2026 19:19:29 +0000

We build an AI virtual staging tool: a user uploads a photo of a room, an image model restyles it, and a set of checks compares the result with the original before anyone sees it. We wrote about those checks in a previous post. This one is about the layer under that: the scripts that make sure the checks, the billing rules and the consent logic stay the way we fixed them.

We call them guards. Each is a standalone Node script, .mjs or .mts, that asserts one behaviour and exits non-zero if it is gone. A fix ships with a guard. By late August we had about seventy, run together with npm run qa.

Then we asked each one a simple question: if we break the thing you protect, do you go red? For 42 of 71, the answer was no.

How we got there

The first problem was not quality but running them at all. Until 22 August there was no runner; the list lived in a Markdown file that named 33 of the 78 scripts on disk. A runner that compares its list with the files actually present found one guard that was red (15 of 18 checks) with nobody noticing, 27 live guards that had not been run for months, and nine that were on disk but not in git.

Now everything ran, and everything was green. That felt like proof. It was not.

Two days later an independent review tried something small: in the error-tracker configuration it renamed the environment variable that carries the DSN, the kind of typo you make during a rename. All 55 checks of the guard watching that configuration stayed green. In production that typo means the error tracker is silently a no-op, the exact failure the guard existed to prevent.

That was reason enough to do it to every guard.

The audit

Method, by hand, one guard at a time: edit the production file so the guarded behaviour is really broken, run the guard, restore the file, write down the numbers. A hole counted only if the guard stayed green on a real break.

Result: 42 holes across 71 guards, and 42 guards that stayed green on at least one real break and had to be rewritten (35 in the first batch, 7 of the last eight). These are two separate counts that happen to match: one guard can hide more than one hole, and a few weak guards only showed up while we were fixing the others.

Group	Guards	Holes
Error tracking, observability, infrastructure	16	7
Image engine and render pipeline	19	5
Billing, consent, privacy, SEO	28	21
The last eight (older zones, referrals, video, pins)	8	9 in 7 files
Total	71	42

Some individual findings:

One guard had no checks at all. It printed a green tick and exited 0 whatever the code looked like.
One check compared the position of two strings in a route file to assert that storage blobs are deleted before database rows (rows first would orphan a customer's files for good). The first string had been removed by a refactor more than three weeks earlier. indexOf returned -1, -1 is less than anything, and the check had passed every run since.
One guard tested a hand-written copy of the SQL instead of the production webhook.

The pattern behind most of the 42: the guards were written against text, not behaviour. We read the route file as a string and looked for names. Such a guard fails both ways: an honest refactor turns it red for nothing, and a real break slides past because the names are still there.

One guard, before and after

The consent route does three things in order: write the user's choice to the auth provider, append to the audit log, unpublish their renders from the public showcase. The guard checked that order with indexOf on the source. Nine checks, all green. The mutation that beat it, and the rewrite:

// production: awaited; a failure returns 500 to the user
await client.users.updateUserMetadata(userId, { ... });
// mutation: same call, same position in the file - all nine checks still green
void client.users.updateUserMetadata(userId, { ... }).catch(() => {});

// rewritten guard: cut out the statement itself, not a name near it
const i = ROUTE.indexOf("client.users.updateUserMetadata");
const stmt = ROUTE.slice(ROUTE.lastIndexOf("\n", i) + 1, ROUTE.indexOf(";", i) + 1).trim();
ok("write is awaited, not fired in the background", /^await client\.users\.updateUserMetadata\(/.test(stmt));
ok("a failed write is not swallowed", !/\.catch\(/.test(stmt));
ok("and not silenced with void", !/^void\b/.test(stmt));

Under the mutation the order of the three calls did not move, so the old checks passed. The meaning was gone: the write is fire-and-forget, the failure is swallowed, the audit log records "revoked", the user believes they opted out, and the showcase keeps publishing because the provider still says "allowed".

The rewrite is still text, but the text of the statement itself, and the mutation that defeated the old version is now a permanent check. Where the code can be imported, we go further: the guard loads the real function through a loader and runs it over a table of inputs.

The runner

Every one of those experiments was a throwaway script, fifty lines each, and each hit the same traps: CRLF on the Windows working copy versus LF in git, a snippet that matched twice, a "mutation" that deleted a line and could not be reverted by an empty pattern (one stayed in the code and was caught on a second pass).

On 28 August it became one tool. A job is JSON: the guard command plus named substitutions.

{
  "guard": "node tmp/qa/geometry-escalation-proof.mjs",
  "mutations": [
    {
      "file": "src/lib/draft-rank.ts",
      "name": "a judge that errored no longer ranks last - an empty verdict beats a checked image",
      "from": "  if (input.errored) return Number.MAX_SAFE_INTEGER - 1;\n",
      "to":   "  void input.errored;\n"
    },
    {
      "file": "src/app/api/generate/route.ts",
      "name": "escalation became unconditional - furniture-placement failures go to the expensive engine too",
      "from": "if (!economyMode && lastGeometryBroken && attempt < MAX_ATTEMPTS) {",
      "to":   "if (!economyMode && attempt < MAX_ATTEMPTS) {"
    }
  ]
}

The loop is short; the rules around it are what we paid for:

if (!runGuard()) fail("guard is already red; it proves nothing");          // 1. green before

for (const m of mutations) {
  if (!m.to.trim()) { bad(m, "empty replacement is a deletion, not a mutation"); continue; } // 2
  const src = original(m.file);
  const hits = src.match(rx(m.from, "g"));                      // rx() tolerates \r?\n  // 3
  if (!hits || hits.length !== 1) { bad(m, "snippet must occur exactly once"); continue; } // 4
  fs.writeFileSync(m.file, src.replace(rx(m.from), m.to));
  const stillGreen = runGuard();
  fs.writeFileSync(m.file, src);
  stillGreen ? bad(m, "GUARD STAYED GREEN") : good(m);
}
assertEveryFileRestoredByteForByte();                                         // 5

Mutations are named and semantic on purpose. Classic mutation testing flips operators automatically; we write each mutation as a sentence about how a customer would get hurt, because that sentence also documents why the guard exists. The trade-off is coverage: we test only the breaks we thought of.

What the mutations found in production code

The day the runner was born, a run on a new guard did something we did not expect. The function that decides whether a payment-return URL is stale had an explicit "payment time is in the future" branch. Six mutations turned the guard red. The seventh, removing that branch, changed nothing: a future timestamp gives a negative age, and a negative age is never above the threshold. The branch was a crutch the guard could not tell from working code. We removed it and wrote the reason above the function.

In the same session, 35 mutations across seven fixes found three guards checking the wrong thing: a parser that stopped at the first closing brace and broke on a nested object (four of six mutations passed on obviously broken code); a "the catch block logs" check that stayed green when the log left one branch because a neighbouring branch still had one; and a scanner for prototype-key guards that cleared a whole file if the guard appeared anywhere in it.

Where it stands

93 mutation jobs, 867 mutations, in the repo next to the guards.
171 guards in the runner's list, 12 named skips (they cost money, run on a calendar or need a fresh build); the list is compared with the disk on every commit.
Written rule: a new guard without a mutation run counts as not written.
The pre-commit hook checks only the registry, in 0.08 seconds. The full run takes minutes, needs the network and a database, and can go red from someone else's outage; a hook like that gets --no-verify on the first urgent fix, and a bypassed hook is worse than none. The full run stays manual and mandatory before a merge.

What still fails

Hand-written mutations come from the same person who wrote the guard, with the same blind spots. On 5 October a mutation named "only seven articles left in the links block" removed 5 of 17 links and the guard stayed green. The mutation was too weak, not the guard; it now removes every link. Nothing finds weak mutations except somebody noticing.

Many guards still read source text. Executed guards are better, but much of our logic sits in route handlers tied to auth and a database, and extracting it just to test it is a refactor we have not earned yet. Text guards are honest only as long as their mutation set is.

There is no CI. The guards run on a laptop before a merge.

And the first finding of all: 42 of 71 is not a number about carelessness. Every guard was written in good faith after a real bug. A green check is a claim, and until something tries to make it red, it is an unverified one.

We build this at AI Flip Room.

Correction, 11 October: an earlier version said 44 of 70. The table adds up to 71 guards and 42 holes, and 42 guards had to be rewritten; the headline mixed two counts. Thanks to Arhan Canli for doing the sums in the comments.

How we check every AI-generated room render before a user sees it

AI Flip Room — Sat, 03 Oct 2026 02:05:35 +0000

Image models are very good at restyling a room and quietly bad at leaving the room alone. Ask for a Japandi living room and you get beautiful furniture, and sometimes also:

a window that moved, shrank, or appeared over a kitchen sink where the photo shows a solid wall;
a closet door that came back as a curtain, a sliding screen, or plain wall;
a sofa parked across the only archway;
a new doorway at the frame edge, or a niche carved into a flat wall;
a camera that slid sideways;
or the opposite: a picture that is almost exactly the photo you uploaded.

Our users put these images in property listings, where a staged photo may change the furniture but not the windows, walls or doors (why that line exists). So every render is checked automatically before anyone sees it. Here is how, and what building it taught us.

The pipeline at a glance

For an indoor room, each render goes through three gates in parallel:

A vision-model judge compares the original photo with the render and answers a fixed list of yes/no questions.
A mechanical camera check measures whether the viewpoint moved, with no language model involved.
A crop-level presence check looks at each door, window and staircase found in the original, at full resolution.

If all three pass, the render is served. If not, we retry. If nothing passes, the user gets the best attempt, clearly marked as a draft.

Gate 1: a judge that only answers booleans

The judge gets the original and the render, both downscaled, and must return a fixed JSON object. We use structured output with a schema, so it cannot reply in prose. An earlier version asked for "only JSON" and scraped it out with a regex; when the output got long or cut off, the parse failed and the gate silently did nothing.

Indoors there are 16 checks, outdoors 9. Indoors they cover:

Openings: every door is still a door you could open; every passage has clear floor to walk through.
Windows: same count, position, size and shape; none invented; not hidden behind a tall wardrobe; the same view outside.
The room shell: ceiling, fireplace, no carved niches, features on the same walls, solid frame edges, no new rooms or corridors.
Finish consistency: no half-repainted rooms.
Content rules: no people, and no photo or realistic portrait of a person in the wall art.
Camera: same viewpoint, same room depth.

Outdoors: buildings, fence line and gates, terrain (no new pool), mature trees, background, and a usable driveway.

A simplified, illustrative verdict:

{
  "doors_preserved": true,
  "openings_clear": false,
  "windows_ok": true,
  "window_unblocked": true,
  "walls_uniform": true,
  "frame_edges_solid": true,
  "no_faces_in_art": true,
  "fail_reasons": ["sofa end stands on the archway floor"]
}

A render passes only if every counted boolean is true; the reasons feed logs and the retry prompt. Most of the work went into the wording:

Say what must pass. Restyling is the product, so the instructions list what is expected (new floors, paint, wallpaper, fixtures) and say "unsure means pass". The exception is a short list where unsure means fail, led by a window that may not have been in the original. An invented window is the worst defect a listing photo can have.

One question per field. While the frame-edge rule was a sub-clause of a broader "no phantom architecture" check, new doorways at the frame edge got through. Its own boolean made the model actually look.

Naming things primes the judge. An early niche check listed which shelves were allowed, and the judge started flagging every shelf. The current wording is purely geometric: is this flat wall still one plane?

Allow "nothing to compare". The "same view through the window" check kept firing on originals with blinds drawn or glass blown out to white. That case is now an explicit pass.

Wording is measurable. The old wall-art wording wrongly failed 10 of 16 good frames (stylised line drawings counted as faces). The rewrite failed 0 of 16 and still caught all 5 real portraits.

Gate 2: measuring the camera instead of asking about it

Early on, the vision model was close to blind to viewpoint changes: renders shot from a visibly different spot passed every neural check. Later, a stronger judge had the opposite problem: on 43 renders where the camera had not moved, it said it had 13 times. So indoors the judge's camera answers no longer decide pass or fail. A measurement does, with two channels:

Ceiling line. We trace the wall/ceiling boundary as a curve (dynamic programming over the vertical brightness gradient) and compare the original's curve with the render's.
Depth profile. A monocular depth model estimates both images. Per column we take how far the far wall reads, then compare the profiles with a robust fit, a rank correlation and a search for a sideways shift.

Each channel is fooled by something different. Paint fools the ceiling line (dark beams, art on a mantel). Glass fools depth (windows and mirrors are estimated differently each time). Paint is flat in depth, and glass does not move the ceiling line, so we flag a shift only when both agree: the minimum of the two normalised scores.

On 24 hand-labelled pairs from two rooms, this caught 5 of 5 real shifts with 0 false flags. The margin was thin, so a grey zone around the threshold is logged, not failed. Outdoors there is no ceiling to trace, so the mechanical check is off and the judge's camera question stays in charge.

Gate 3: crops, because a door is 80 pixels wide

At the judge's resolution a front door is a thin strip. So doors, windows and stairs are detected at upload, and after each render a model is asked about each crop at full resolution: is it still there? A mechanical channel also measures whether a window drifted or widened. On 87 labelled pairs, the crop judge caught 7 of 10 broken renders with no false alarms on the 75 clean ones; the mechanical channel caught the other 3.

When a render fails

Retries depend on the kind of defect.

Placement defects (a blocked passage, a face in the art, half-painted walls) are retried on the same image model with the judge's reasons added to the prompt.
Geometry defects go to a second, pricier image model. We measured a same-model retry fixing broken geometry about one time in five. In an earlier bake-off the two models were clean on 18 and 17 of 20 renders, and 19 of 20 together: their failures barely overlap.

The geometry retry gets a fixed, count-based instruction ("same set of openings, not one more, not one less"), never the judge's description of the defect. When a retry prompt named an arched opening at the left edge, the retry model painted exactly that.

If nothing passes, the user still gets the best attempt, under a "rough draft" banner. On paid plans the first four rejected drafts in a rolling week don't use a render; the free plan gets one. The cap is deliberate: a draft is a real, downloadable image, so unlimited forgiveness would be a free image farm.

The checks fail open. If the judge itself errors or times out, the user is not blocked: the image is served flagged, and an unverified render is not charged as a draft.

The near-copy trap

At first, "best attempt" meant the one with the fewest failed checks. The attempt that breaks the least is often the one where the model barely did anything. In one real case the first attempt was a convincing industrial restyle that failed on geometry, and the user was served the second: their old room, old wallpaper and all, plus a pair of lamps. The ranking was paying the model for doing nothing.

The fix is a change meter: shrink both images to a tiny thumbnail and count how much of the frame visibly changed. No model call, and we calibrated it on 110 real renders and 10 synthetic near-copies (recompressed, blurred, brightened, shifted, a couple of lamp-sized patches). The ranking now:

// simplified
function draftRank(a: Attempt): number {
  if (a.checkerErrored) return LAST;
  let rank = a.failedChecks.length;
  if (a.hasPeopleOrRealFacesInArt) rank += HARD_BAN;
  if (a.isNearCopy && a.intensity !== "light") rank += NEAR_COPY;
  return rank; // lowest is served
}

A light restage is exempt, because there the user asked us to keep the room.

Picking the judge model with a calibration set

The judge runs on every attempt, so a cheaper model is tempting. We keep a calibration set of 31 hand-labelled frames and run every candidate on it twice. A cheaper model at under half the price per call missed 6 defects per run against our current judge's 3. On 46 live renders with 8 defects marked by eye, it caught 0 and 1 across two runs; the current judge caught 6. A newer model wasn't clearly better either: fewer misses (2 vs 3), twice the false alarms (4 vs 2). We kept the current judge.

Takeaways

Ask the judge narrow yes/no questions, not one "is this good?".
Where geometry can be measured, measure it, with channels whose failure modes don't overlap.
Calibrate every wording change and model swap on a labelled set, twice.
Design the failure path as carefully as the check: ranking, draft label, who pays.
"Least broken" can mean "least changed". Measure change directly.

We build this at AI Flip Room.