DEV Community: libersand

How to Stop an Image Model From Ignoring Your Reference Image

libersand — Fri, 21 Aug 2026 14:16:12 +0000

A user uploaded a floor plan, typed "remove the room labels", and got back the same plan — labels still on it. Tried again, same result.

The interesting part: the model was not ignoring the instruction. Another line in our own prompt was quietly outranking it.

I work on FloorDrafter, an AI floor plan generator. A big part of the product takes a reference image — a photo of a hand sketch, a scanned blueprint, an existing plan — and applies one requested change to it. That turns out to be a much harder prompt problem than generating from scratch, because now the model has two masters: the picture and the sentence.

Here are six things we learned making it behave, plus how we measure whether a prompt change actually helped.

1. Separate "how to draw" from "what to draw"

Our style presets started life as single strings:

top-down 2D color floor plan: flat orthographic overhead view, color-filled rooms… clear room labels and dimension lines with measurements

Looks harmless. But that string got injected into the same block as the user's instruction, under a rule that said everything in the block must be delivered fully and visibly. So the block read:

REQUESTED CHANGE
- Representation: ...clear room labels and dimension lines...
- remove the room labels          <- the user's own words

Two contradictory requirements, both marked mandatory. The style description won, because it was more specific.

The fix was to split every style into two halves: render (viewpoint, line weight, materials, lighting — how to draw) and annotations (room labels, dimension lines — what content to include). The from-scratch branch names both. The edit-an-existing-image branch names only render, because labels and dimensions are content that already exists in the reference — whether they stay is the user's call, not the style preset's.

Generalizable version: if your prompt is assembled from fragments, audit every fragment for whether it smuggles in content requirements. A fragment that describes form is safe to inject anywhere. A fragment that describes content can collide with user intent.

2. Write the rule as "only change what's named", not as a list of prohibitions

The obvious way to protect a reference image is a MUST NOT list: don't move walls, don't resize rooms, don't change the window positions.

That list locks the user out. Someone who wants to "keep the layout but knock down the kitchen wall" now can't, because you preemptively banned it.

Inverting it works better: change only what the instruction names, leave everything else as it is in the reference. The granularity comes from the user's own sentence instead of from your list.

3. Explicitly authorize the requested change, or you get a copy

This one is counterintuitive. After we tightened the fidelity language, output quality went up — and then users started reporting that nothing changed at all. The model had become so conservative it was returning something close to the original image.

Fidelity instructions alone teach the model that changing things is risky. You have to also grant full permission to execute the named change, and add a closing self-check that confirms the change is actually visible. Both halves, or you trade one failure mode for the other.

4. Ban adding, never ban existing

Our closing prohibition originally read, roughly, "no text beyond room labels and dimensions."

Then a user uploaded a full drawing sheet — not a bare floor plan, but a title block, project name, legend, scale bar, north arrow, notes. The model read that prohibition literally and deleted all of it. A real case: a church site plan came back with the main title, the church name, the SIMBOLOGÍA legend, the scale bar, the NOTA block and five bottom notes all gone. The floor plan survived. Everything around it did not.

A prohibition on text cannot distinguish "text you invented" from "text that was already there". So the rule can only forbid adding. Preserving what exists needs its own affirmative clause — and it needs one more sentence that most people miss:

Changing the style is not a reason to delete content.

Without that, picking a monochrome CAD style gets read as "so only keep room labels and dimensions", and you are back to deleting the sheet.

Both clauses shipped, and both are asserted in the test suite described at the end — title blocks, legends, scale bars and note blocks are now on an explicit preserve list, copied verbatim.

5. Some constraints must be immune to your own exemption clauses

Once you have a rule that says "only change what's named", you will naturally write self-checks that begin "Setting aside what the change named…". That phrasing is a loophole.

A user photographed a C-shaped outline, 16 by 24, and wrote "2 units on 1 footprint". The model duplicated the entire C shape and butted the two copies together into a 32 by 24 building. Every dimension label was preserved perfectly. The geometry had doubled.

Two failures stacked here:

The protection clause banned moving the footprint but not duplicating, scaling or extending it.
The self-check started with the exemption phrase, so the model decided the footprint change fell under "what the change named" and skipped the check entirely.

The footprint check had to be rewritten to start with "Regardless of what the change named…". Geometric integrity is not negotiable by the instruction; content is.

There is a second lesson buried in that example. "2 units on 1 footprint" contains a goal (2 units) and a boundary (1 footprint) in one breath. Models reliably grab the goal and drop the boundary, so the prompt has to say out loud that the user's sentence may contain limits, not just targets.

6. A capability needs an exemption in every section that can override it

Back to the labels. "Remove the labels" failed because three separate parts of the prompt each independently reinstated them:

The rule authorized restyling and adding — but never mentioned deleting.
The sheet-protection clause said do not drop, blank out, condense or summarise any existing element.
The closing self-check saw missing annotations and told the model to redo them.

Patch any one of those and the other two still win. Deletion had to be named in all three places at once — and scoped tightly ("delete only what was named; deleting anything else is an error"), because a broad deletion permission takes you straight back to lesson 4.

The pattern: when you add a capability to a long prompt, grep the whole thing for sections that could veto it. Long prompts develop internal politics.

How we know any of this worked

Prompt changes are easy to rationalize and hard to verify. We ran fixed probes: same reference image, same instruction, repeated runs per variant, counting outcomes by hand against the previous prompt. The denominators below count elements rather than runs — dimension-chain segments in the reference, room labels across the probe set, and images in the sweep.

Metric	Before	After
Original dimension chain survives	~0.6 / 9	~7 / 9
Room labels preserved	~5 / 20	~15 / 20
Images with invented dimensions	9 / 10	0 / 10

Those numbers are small-sample and hand-counted. That is fine — they are decision-grade, not paper-grade. The point is that "this prompt feels better" is not a finding, and a fixed probe costs an hour.

Unit-testing a prompt

The six lessons above are all invariants: properties the assembled prompt must have, in every branch, forever. Nothing stops a future edit from quietly removing one — they are just strings in a template.

So they are unit tests now. The prompt builder is a pure function, which makes this cheap:

it('bans only added text, never existing text', () => {
  const p = buildPrompt({ hasReference: true, /* … */ });
  assert.match(p, ADD_ONLY_TEXT_RULE);       // "do not add … text"
  assert.doesNotMatch(p, LEGACY_TEXT_BAN);   // the old blanket ban
});

it('states that a style change is not a reason to delete content', () => { /* … */ });

it('makes the footprint check immune to the change exemption', () => {
  const p = buildPrompt({ hasReference: true, /* … */ });
  assert.match(p, FOOTPRINT_UNCONDITIONAL);  // "regardless of what the change named"
});

There are 28 of them. They do not test the model — they test that the string we send still contains the clauses we worked out. Each one encodes a specific failure mode we found and fixed, so it cannot come back unnoticed.

If you maintain a prompt longer than a screenful, this is the highest-value hour you can spend on it.

The short version

Split prompt fragments into form and content; only form is safe to inject everywhere.
Prefer "change only what is named" over a list of prohibitions.
Grant the change explicitly, or fidelity language will suppress it.
Forbid adding, never forbid existing.
Make integrity constraints immune to your own exemption phrasing.
New capabilities need an exemption in every section that could veto them.
Probe with fixed inputs and count. Then lock the result in with tests.

None of this is specific to floor plans. Any time you send an image plus an instruction and expect the untouched parts to stay untouched, the same failure modes are waiting.

A Postgres driver silently shifted my timestamps by 8 hours — and every test stayed green

libersand — Mon, 17 Aug 2026 14:14:55 +0000

A chunk of the timestamp columns in my database were quietly wrong.

Not corrupted. Not null. Just — eight hours off. The values looked perfectly reasonable in psql. They looked reasonable in the app. They were reasonable, for a timezone I wasn't in.

Every test passed. CI was green. It had been green the whole time.

Here's what was actually happening, and why it's invisible to most of the people who could have warned me.

The setup

Nothing exotic:

Postgres
postgres.js as the driver
Drizzle ORM on top

The schema looked like this — and if you use Drizzle, yours probably does too:

export const user = pgTable('user', {
  id: text('id').primaryKey(),
  createdAt: timestamp('created_at').notNull(),
  updatedAt: timestamp('updated_at').notNull(),
});

That timestamp('created_at') is the important part. In Drizzle, timestamp() with no options generates a Postgres timestamp without time zone column — not timestamptz. The default is the unsafe one.

Two things that are individually fine

One: timestamp without time zone stores no offset.

That isn't a bug, it's the definition of the type. Postgres stores the wall-clock digits you gave it — 2026-08-17 10:00:00 — and nothing else. No zone, no offset. It has no idea whether that's 10am in Shanghai or 10am in Chicago, and it doesn't care.

Two: the driver has to guess on the way back.

When you hand postgres.js a JavaScript Date, it serializes it and sends it over. Postgres receives a value for a timestamp without time zone column, discards the offset, and stores the literal wall clock.

Then you read it back. Postgres hands the driver a bare string with no zone marker. The driver has to turn that into a JS Date, and it has no offset to work with — so it resolves the string against the runtime's local timezone.

Which is not necessarily the timezone the value was written in.

The round trip

My laptop is UTC+8. Trace one value through:

Step	Value
`new Date()` in my app	18:00 Beijing = 10:00 UTC
Driver serializes, Postgres strips the offset	stored as `10:00`
Read back: bare string, no zone	`2026-08-17 10:00:00`
Driver resolves against local time (UTC+8)	10:00 Beijing = 02:00 UTC

Wrote 10:00 UTC. Read 02:00 UTC.

Eight hours. And it compounds — write that value back and you lose another eight.

The number isn't special. It's just my UTC offset. In New York it would be five hours, in the other direction. In London during summer, one.

And in UTC it is zero.

Which is why every test passed

Read that last line again, because it's the whole story.

GitHub Actions runners: UTC
Your Docker container, unless you went out of your way: UTC
Your production server: almost certainly UTC

When the local offset is zero, the write-side and read-side interpretations agree exactly. The bug doesn't produce a small error under UTC. It produces no error at all. This isn't a flaky test or a rare edge case — in a UTC environment the behavior is genuinely, completely correct.

So my test suite wasn't failing to catch a bug. It was running in the one environment where the bug does not exist.

There is exactly one way to trigger it: run code that reads and writes timestamps from a machine whose clock isn't UTC. A developer laptop. Which is precisely what I did.

The expensive version

I needed to backfill a column — copy timestamps from an old table into a new one. Straightforward script: read rows, transform, write rows.

I ran it from my laptop.

Every row it touched moved eight hours. The script was correct. The logic was correct. It would have been fine on the server. It just happened to make a round trip through a JavaScript runtime sitting in UTC+8, and paid the toll on the way through.

The rule I now follow without exception:

A backfill must never round-trip a timestamp through the application layer. Copy it inside the database.

UPDATE new_table n
SET created_at = o.created_at
FROM old_table o
WHERE n.id = o.id;

Postgres moving a value from one column to another cannot get the timezone wrong, because no timezone is ever inferred. There is no JS runtime in the path to guess with.

Fixing it going forward

Use timestamptz. In Drizzle:

createdAt: timestamp('created_at', { withTimezone: true }).notNull(),

timestamptz doesn't actually store a timezone either — that's a common misconception; Postgres normalizes to UTC internally. But the wire format carries an offset in both directions, so the driver never has to guess. The round trip becomes lossless no matter what timezone anything is running in.

Audit what you already have. My schema turned out to be a mix: a handful of newer tables had withTimezone: true and everything older didn't. Nothing in the application code distinguishes them. This finds them:

SELECT table_name, column_name, data_type
FROM information_schema.columns
WHERE data_type LIKE 'timestamp%'
ORDER BY data_type, table_name;

Set TZ=UTC for your local runtime. This is a band-aid, not a fix — it makes your laptop behave like CI, which hides the bug rather than removing it. But it stops you from making things worse while you migrate.

What I actually took away

I had assumed a green test suite meant the code was correct. What it actually meant was: the code is correct in the environment the tests run in.

Timezone is an ambient property of the machine, not an input to my tests. So is locale. So is filesystem case sensitivity. So is CPU architecture. My CI wasn't wrong — it just isn't a laptop in UTC+8, and it never will be.

If your infrastructure is UTC top to bottom and your team isn't, that gap is where this class of bug lives. You don't close it by writing better tests. You close it by not letting the ambient environment participate in the answer — which, for timestamps, means timestamptz and doing data migrations in SQL.

I'm an indie developer running several small SaaS products solo. This one cost me a day and a data migration, so I figured I'd write it down.