How to Stop an Image Model From Ignoring Your Reference Image

A user uploaded a floor plan, typed "remove the room labels", and got back the same plan — labels still on it. Tried again, same result.

The interesting part: the model was not ignoring the instruction. Another line in our own prompt was quietly outranking it.

I work on FloorDrafter, an AI floor plan generator. A big part of the product takes a reference image — a photo of a hand sketch, a scanned blueprint, an existing plan — and applies one requested change to it. That turns out to be a much harder prompt problem than generating from scratch, because now the model has two masters: the picture and the sentence.

Here are six things we learned making it behave, plus how we measure whether a prompt change actually helped.

1. Separate "how to draw" from "what to draw"

Our style presets started life as single strings:

top-down 2D color floor plan: flat orthographic overhead view, color-filled rooms… clear room labels and dimension lines with measurements

Looks harmless. But that string got injected into the same block as the user's instruction, under a rule that said everything in the block must be delivered fully and visibly. So the block read:

REQUESTED CHANGE
- Representation: ...clear room labels and dimension lines...
- remove the room labels          <- the user's own words

Two contradictory requirements, both marked mandatory. The style description won, because it was more specific.

The fix was to split every style into two halves: render (viewpoint, line weight, materials, lighting — how to draw) and annotations (room labels, dimension lines — what content to include). The from-scratch branch names both. The edit-an-existing-image branch names only render, because labels and dimensions are content that already exists in the reference — whether they stay is the user's call, not the style preset's.

Generalizable version: if your prompt is assembled from fragments, audit every fragment for whether it smuggles in content requirements. A fragment that describes form is safe to inject anywhere. A fragment that describes content can collide with user intent.

2. Write the rule as "only change what's named", not as a list of prohibitions

The obvious way to protect a reference image is a MUST NOT list: don't move walls, don't resize rooms, don't change the window positions.

That list locks the user out. Someone who wants to "keep the layout but knock down the kitchen wall" now can't, because you preemptively banned it.

Inverting it works better: change only what the instruction names, leave everything else as it is in the reference. The granularity comes from the user's own sentence instead of from your list.

3. Explicitly authorize the requested change, or you get a copy

This one is counterintuitive. After we tightened the fidelity language, output quality went up — and then users started reporting that nothing changed at all. The model had become so conservative it was returning something close to the original image.

Fidelity instructions alone teach the model that changing things is risky. You have to also grant full permission to execute the named change, and add a closing self-check that confirms the change is actually visible. Both halves, or you trade one failure mode for the other.

4. Ban adding, never ban existing

Our closing prohibition originally read, roughly, "no text beyond room labels and dimensions."

Then a user uploaded a full drawing sheet — not a bare floor plan, but a title block, project name, legend, scale bar, north arrow, notes. The model read that prohibition literally and deleted all of it. A real case: a church site plan came back with the main title, the church name, the SIMBOLOGÍA legend, the scale bar, the NOTA block and five bottom notes all gone. The floor plan survived. Everything around it did not.

A prohibition on text cannot distinguish "text you invented" from "text that was already there". So the rule can only forbid adding. Preserving what exists needs its own affirmative clause — and it needs one more sentence that most people miss:

Changing the style is not a reason to delete content.

Without that, picking a monochrome CAD style gets read as "so only keep room labels and dimensions", and you are back to deleting the sheet.

Both clauses shipped, and both are asserted in the test suite described at the end — title blocks, legends, scale bars and note blocks are now on an explicit preserve list, copied verbatim.

5. Some constraints must be immune to your own exemption clauses

Once you have a rule that says "only change what's named", you will naturally write self-checks that begin "Setting aside what the change named…". That phrasing is a loophole.

A user photographed a C-shaped outline, 16 by 24, and wrote "2 units on 1 footprint". The model duplicated the entire C shape and butted the two copies together into a 32 by 24 building. Every dimension label was preserved perfectly. The geometry had doubled.

Two failures stacked here:

• The protection clause banned moving the footprint but not duplicating, scaling or extending it.
• The self-check started with the exemption phrase, so the model decided the footprint change fell under "what the change named" and skipped the check entirely.

The footprint check had to be rewritten to start with "Regardless of what the change named…". Geometric integrity is not negotiable by the instruction; content is.

There is a second lesson buried in that example. "2 units on 1 footprint" contains a goal (2 units) and a boundary (1 footprint) in one breath. Models reliably grab the goal and drop the boundary, so the prompt has to say out loud that the user's sentence may contain limits, not just targets.

6. A capability needs an exemption in every section that can override it

Back to the labels. "Remove the labels" failed because three separate parts of the prompt each independently reinstated them:

1. The rule authorized restyling and adding — but never mentioned deleting.
2. The sheet-protection clause said do not drop, blank out, condense or summarise any existing element.
3. The closing self-check saw missing annotations and told the model to redo them.

Patch any one of those and the other two still win. Deletion had to be named in all three places at once — and scoped tightly ("delete only what was named; deleting anything else is an error"), because a broad deletion permission takes you straight back to lesson 4.

The pattern: when you add a capability to a long prompt, grep the whole thing for sections that could veto it. Long prompts develop internal politics.

How we know any of this worked

Prompt changes are easy to rationalize and hard to verify. We ran fixed probes: same reference image, same instruction, repeated runs per variant, counting outcomes by hand against the previous prompt. The denominators below count elements rather than runs — dimension-chain segments in the reference, room labels across the probe set, and images in the sweep.

Metric	Before	After
Original dimension chain survives	~0.6 / 9	~7 / 9
Room labels preserved	~5 / 20	~15 / 20
Images with invented dimensions	9 / 10	0 / 10

Those numbers are small-sample and hand-counted. That is fine — they are decision-grade, not paper-grade. The point is that "this prompt feels better" is not a finding, and a fixed probe costs an hour.

Unit-testing a prompt

The six lessons above are all invariants: properties the assembled prompt must have, in every branch, forever. Nothing stops a future edit from quietly removing one — they are just strings in a template.

So they are unit tests now. The prompt builder is a pure function, which makes this cheap:

it('bans only added text, never existing text', () => {
  const p = buildPrompt({ hasReference: true, /* … */ });
  assert.match(p, ADD_ONLY_TEXT_RULE);       // "do not add … text"
  assert.doesNotMatch(p, LEGACY_TEXT_BAN);   // the old blanket ban
});

it('states that a style change is not a reason to delete content', () => { /* … */ });

it('makes the footprint check immune to the change exemption', () => {
  const p = buildPrompt({ hasReference: true, /* … */ });
  assert.match(p, FOOTPRINT_UNCONDITIONAL);  // "regardless of what the change named"
});

There are 28 of them. They do not test the model — they test that the string we send still contains the clauses we worked out. Each one encodes a specific failure mode we found and fixed, so it cannot come back unnoticed.

If you maintain a prompt longer than a screenful, this is the highest-value hour you can spend on it.

The short version

• Split prompt fragments into form and content; only form is safe to inject everywhere.
• Prefer "change only what is named" over a list of prohibitions.
• Grant the change explicitly, or fidelity language will suppress it.
• Forbid adding, never forbid existing.
• Make integrity constraints immune to your own exemption phrasing.
• New capabilities need an exemption in every section that could veto them.
• Probe with fixed inputs and count. Then lock the result in with tests.

None of this is specific to floor plans. Any time you send an image plus an instruction and expect the untouched parts to stay untouched, the same failure modes are waiting.