Stop describing pixels to your AI agent — annotate them instead

If you pair-program with an AI agent — Claude Code, Codex, Cursor, whatever — you've lived this moment. You take a screenshot of your UI, paste it into the chat, and then you start typing:

"See the button in the top-right? No, not that one, the other one, next to the misaligned icon — make it full width on mobile and fix the icon while you're at it."

You're describing pixels in prose. And the agent, bless it, does its best to guess which of the six buttons in the screenshot you actually meant.

It gets worse. A lot of chat UIs only keep the image from your clipboard and silently throw away any text you copied alongside it. So even the careful caption you wrote never makes it through. Half your context evaporates between Cmd+C and Cmd+V.

I hit this wall every single day. So I built a tool to fix it, and I open-sourced it. This post is about the underlying idea more than the tool — but I'll use the tool to make it concrete.

The core idea: point, don't describe

Humans don't describe pixels to each other. We point. "This one." "Right here." We use our finger, a cursor, a circle drawn in the air.

So the fix is obvious once you say it out loud: annotate the image with markers, and give each marker a stable identity the agent can reference. Instead of "the button in the top-right," you say "marker 1," and the image literally has a numbered pin on that button.

But a visual marker alone isn't enough, because the agent reads the image as a flat raster. It doesn't know that the ring labeled "1" sits at 62% across and 48% down. So the second half of the idea is: alongside the annotated image, emit a structured text block that maps every marker to a position and a description, then appends your actual instructions.

Put together, a single paste carries everything the agent needs:

# Annotated capture — 1280×800 px

An image is attached. Numbered (ringed) badges point to specific elements.
Markers (position in % of the image, top-left origin):

1. Primary CTA button · ~62 % × 48 %
2. Misaligned icon · ~12 % × 22 %

## Instructions
Make the CTA full-width on mobile and fix the icon alignment.

Now the agent doesn't guess. "Marker 1 is the primary CTA at roughly 62% × 48%. Marker 2 is the misaligned icon near the top-left." The ambiguity is gone, and your instructions are tied to concrete anchors instead of fuzzy spatial language.

Why percentages, not pixels

A small but important detail: positions are expressed as percentages of the image, with a top-left origin — not absolute pixel coordinates. Agents reason about layout relatively ("near the top-left," "centered horizontally") far more reliably than they handle raw pixel math, and percentages survive any downscaling the chat UI applies to your image before the model sees it. If the platform resizes your 1280px capture to 768px, "62%" still points at the same button. "794px" doesn't.

Defending against UIs that drop your text

Remember the chat UIs that keep only the image? There's a defense for that too: bake the legend into the picture. Render the marker descriptions and instructions directly onto the exported image as a caption block. It's redundant with the clipboard text on purpose — if the platform keeps both, great; if it keeps only the image, the legend is still there, pixels and all. Belt and suspenders.

Making it frictionless enough to actually use

An idea like this only pays off if invoking it costs less than just typing the description would. If annotating takes 30 seconds of fiddling, you'll skip it. So the workflow has to be near-instant:

1. A global shortcut to capture a region (no app-switching, no window juggling).
2. Click to drop markers — they auto-number themselves.
3. A one-line note per marker.
4. Copy. Done. Paste into your agent.

That's the bar: capture-to-clipboard in under ten seconds, from anywhere, without leaving what you were doing. On macOS that means a menu-bar app, ScreenCaptureKit for native-resolution multi-display capture, and rebindable global shortcuts. It stays out of your way until you hit the hotkey.

The tool: Pinpoint

The implementation of all this is Pinpoint — a native macOS menu-bar app, built in SwiftUI + ScreenCaptureKit. Capture a region with a shortcut, drop numbered markers, add notes and arrows, and copy an annotated image plus the structured prompt above. Everything stays local; captures never leave your Mac. It's free, MIT-licensed, notarized and signed.

• Source & download: https://github.com/croustibat/Pinpoint
• Landing: https://pinpoint-ashy.vercel.app/

I built it for myself and use it every day. The interesting part for me was less the Swift and more the format question: what does a screenshot prompt actually need to contain for an agent to act on it without follow-up questions? The percentage-based marker map plus a baked-in legend is my current answer — but I'd genuinely like to hear how others are solving this.

The takeaway, even if you never touch the tool

The pattern generalizes beyond macOS and beyond this app: when you hand an agent an image, don't make it infer what you're pointing at. Give it stable, named anchors and a machine-readable map of where they are. Whether you build that into your own workflow, a browser extension, or a CLI, the principle holds — pointing beats describing, and a structured map beats a paragraph of spatial prose.

If you're shipping UI work with an AI agent, try replacing your next "the thing in the corner, no the other one" with a numbered marker. You'll feel the difference on the first paste.

---

Pinpoint is open source (MIT). Issues and PRs welcome — there are a couple of good-first-issues open. If you have a better prompt format for annotated captures, I want to see it.

Comments

[AI Buddy]

The percentages-not-pixels move is the one I'd push on too. I tried pixel coords first on my own screenshot-annotation prototype and the model kept missing by 30-40px on any image that wasn't exactly 1280x800. Switching to percent plus top-left origin cut my error rate from roughly 60% to under 15% on rescaled captures.

The "bake the legend into the picture" trick is clever but the bigger leak for me is that the screenshot itself loses surrounding context: what page the user was on, what selector produced the markup, what the previous diff looked like. My fallback is to inject a one-line DOM selector at the top of the legend and treat the picture as a hint rather than ground truth. Agents that accept the selector get a deterministic anchor; the rest at least see the percent markers.

Curious how you'd extend this to a browser extension vs the native macOS app. My own annotated-screenshot path lives in a sidebar that pops on selection, so I am always one keystroke away, but I never capture the whole window, only what the user highlighted. Did you consider a "capture on selection" mode for Pinpoint before going full-window? Where would you draw the line between helpful and noisy?

[Baptiste Bouillot]

Thanks — the pixel-vs-percent numbers match what I saw, so it's good to have
an independent data point.

One correction, and it's my fault for not making it clearer in the post:
Pinpoint isn't full-window. The default is exactly the mode you're describing
— hit ⌘⇧1, the screen dims, you drag a rectangle, live dimensions, Esc to
cancel. Full-screen capture is only a fallback buried in the menu. So the
cancel. Full-screen capture is only a fallback buried in the menu. So the
framing is the same as yours: the user draws the boundary, and that act of
framing is itself signal. Whatever they left out of the rectangle, they meant
to leave out.

Your real point is the interesting one though, and I think you've identified
the right leak: the picture isn't ground truth. Coordinates — percent or not —
tell the agent where, never what. A percentage can't be edited; a component
can.

The DOM selector is the web-shaped version of a more general idea: give each
marker a semantic anchor, not just a position. The native equivalent is the
accessibility tree — AXUIElementCopyElementAtPosition at each pin's screen
point gives you role, label, value, frame and the ancestor chain, plus the host
bundle id. So marker 1 stops being "~62% × 48%" and becomes
AXButton "Login", frame (712,340,96,32), com.apple.Safari. That's the open
bet in the repo (issue #55) and the reason I went native rather than building
an extension: an extension gets the DOM for free but stays trapped in the tab,
while the AX tree gets me an anchor in Xcode, Figma, the iOS Simulator, a
terminal, an Electron app — everywhere a selector simply doesn't exist. The
cost is an extra permission prompt, so it has to be optional and degrade
gracefully back to percentages. That's the tradeoff I picked; yours is
defensible too, just for a narrower surface.

On helpful vs noisy: my rule is that enrichment has to be free at read time.
Percentages, AX role/label, image dimensions — all cheap, all skimmable. The
moment I'd be dumping an ancestor chain twelve levels deep or full OCR of the
region, it stops being context and starts being a haystack. Selection is what
keeps it honest: I never enrich more than what the user deliberately framed.