Pinning Defects on the Photo Itself: Building 360 Hotspot Mapping

This is a deep-dive into one feature of Inspection OS, a SaaS I built for property-inspection teams. It's the piece I'm most proud of, and the one with the most interesting engineering behind it.

The problem: a photo is a bad way to describe where a defect is

Property inspectors take hundreds of site photos. The traditional report then describes each defect in prose: "Hollowness observed in kitchen floor tiles near the north wall." The reader has to mentally map that sentence back onto the photo. Multiply that by 69 defects across 16 rooms and the report becomes a wall of text that nobody can act on quickly.

The fix seems obvious once you see it: stop describing the location in words — pin it on the image.

Each red dot is a defect anchored to the exact pixel where it was found. Click one and you get its severity, status, and recommended remedy. The photo becomes the interface. That's a "hotspot," and here's how the system behind it works.

Design decision #1: store coordinates as fractions, not pixels

The first real decision is how to store a pin's position. The naive answer is pixel coordinates — "this defect is at (1840, 920)." That breaks the moment anything about the display changes: a different screen size, a zoomed view, a thumbnail, the PDF export. Pixels are tied to one specific rendering.

So hotspots are stored as normalized coordinates in the range [0, 1]:

// shared/schema.ts — the hotspots table (Drizzle + Postgres)
export const hotspots = spatial.table("hotspots", {
  id:        varchar("id").primaryKey().default(sql`gen_random_uuid()`),
  captureId: varchar("capture_id").notNull().references(() => captures.id, { onDelete: "cascade" }),
  x: numeric("x", { precision: 5, scale: 4 }).notNull(), // 0.0000 – 1.0000
  y: numeric("y", { precision: 5, scale: 4 }).notNull(),
  label:         text("label").notNull(),
  issueSeverity: text("issue_severity"),
  issueStatus:   text("issue_status"),
  notes:         text("notes"),
  resolvedPhoto: text("resolved_photo"),
  // ...
});

x = 0.5, y = 0.5 means dead centre of the image, whatever the image's resolution. Rendering then becomes trivial and resolution-independent — the pin is positioned as a percentage of its container:

<div style={{ left: `${x * 100}%`, top: `${y * 100}%` }} />

The same stored coordinate renders correctly on a retina iPad, a downscaled thumbnail, and the A4 PDF — no conversion tables, no per-device math. numeric(5,4) gives four decimal places of precision, which is sub-pixel on any realistic image.

Design decision #2: inverting the pan/zoom transform on click

The canvas isn't static — inspectors pan and zoom (0.3×–5×) to place pins precisely on small defects. That makes capturing the coordinate the tricky part. When the user clicks, the browser gives me a screen coordinate, but I need the coordinate in the original image's space, undoing whatever pan and zoom are currently applied.

The image is rendered with a CSS transform:

transform: `translate(${panX}px, ${panY}px) scale(${scale})`

So on click I apply the inverse of that transform to recover the true image-space point:

const rect = containerRef.current.getBoundingClientRect();
const rx = e.clientX - rect.left;   // click, relative to container
const ry = e.clientY - rect.top;

// undo the translate + scale to get original-image coordinates
const ox = (rx - cx - panX) / scale;
const oy = (ry - cy - panY) / scale;

Then ox, oy get normalized against the image dimensions and stored. Get this wrong and pins "drift" when you place them at anything other than 100% zoom — a classic and maddening bug. Getting the transform math right is what makes placement feel exact.

Design decision #3: pins that don't grow when you zoom

If the whole canvas scales by scale, the pins scale with it — zoom to 5× and your dots become giant blobs that cover the very defect they mark. The fix is to counter-scale each pin by the inverse of the zoom:

transform: `translate(-50%, -50%) scale(${1 / scale})`

The translate(-50%, -50%) centres the dot on its coordinate; the scale(1 / scale) cancels the parent's zoom so the pin stays a constant visual size at any zoom level. Small detail, big difference in how "solid" the tool feels.

Design decision #4: one coordinate model for flat photos and 360° panoramas

Inspectors also shoot 360° panoramas of whole rooms. A panorama isn't a flat plane — a point on it is really a direction (pitch and yaw) on a sphere. I could have built a second, separate hotspot system for panoramas... but I didn't want to.

Instead, the same normalized (x, y) is reinterpreted for 360° captures via a single mapping:

const { pitch, yaw } = toPitchYaw(parseFloat(pin.x), parseFloat(pin.y));

A captures.is_360 boolean flag decides which lens to view the coordinate through. Flat image? (x, y) is a position. Panorama? The same (x, y) maps to a spherical direction. One storage format, one placement flow, two renderers. The data model didn't have to know about the difference, which kept the schema — and my head — clean.

The data model: a hotspot is a defect

Notice the hotspot row carries issueSeverity, issueStatus, notes, and resolvedPhoto. A hotspot isn't just a marker that points at a separate "issue" record — it is the defect, anchored in space. This is a deliberate denormalization: the issue's display fields live on the hotspot so the canvas can render every pin (with its severity colour) in a single query, without joining and re-fetching issue records for each dot. For a view that paints dozens of pins at once, that read-path simplicity is worth the redundancy.

Everything cascades on delete — remove a capture and its hotspots go with it (onDelete: "cascade"), so there are no orphaned pins pointing at images that no longer exist.

What I'd do differently / open questions

• Precision vs. storage: numeric(5,4) is precise but heavier than a real. At current scale it's irrelevant; at millions of hotspots I'd benchmark it.
• Denormalized issue fields speed up reads but mean an issue edit has to update the hotspot too. A single-writer inspection flow makes this safe today; a collaborative one would need more care.
• 360° math currently assumes an equirectangular projection. Other panorama formats would need their own toPitchYaw.

Future work: the dataset nobody planned for

Here's the part I find most interesting, and where I want to take this next.

Every hotspot an inspector places is a human-labeled training example: an image, a precise (x, y) location, a defect category, and a severity — "at this spot, there is a major tile hollowness." The app has been quietly accumulating a labeled, spatially-grounded defect dataset as a byproduct of normal use.

That opens a genuine research direction: can a computer-vision model learn to propose hotspots automatically? Given a fresh site photo, suggest "likely crack here, likely dampness there," and let the inspector confirm or reject — turning placement from fully-manual into human-in-the-loop verification. The normalized-coordinate design already gives model output and human labels the exact same representation, so predictions and ground truth are directly comparable.

There are hard problems in the way — class imbalance (some defects are rare), domain shift across sites and lighting, and the fact that "hollowness" isn't visually obvious at all (it's found by tapping, not looking, which is a fascinating limit of a vision-only approach). But the data is real, the labels are honest, and the representation is already right. That's the thread I want to pull on next.

---

Inspection OS is built with React 19, Express, and PostgreSQL (Drizzle ORM). If the hotspot design or the CV direction interests you, the code is on GitHub.