How I Gave My AI Coding Agent Eyes: A Screenshot Feedback Loop for UI Work

TL;DR

My AI coding agent was great at backend work and consistently terrible at UI work — it could make every test pass while the page still looked broken. I fixed it by wiring a headless browser into the agent's loop so it renders the page, screenshots it, and reads its own output before it's allowed to say "done." Here's the harness, the prompt contract that makes it actually work, and five lessons — including the ones where visual feedback didn't help at all.

The Problem

I run a coding agent (Claude Code, v2.x, on Node.js 22.x) on real work most days. On backend tasks it's genuinely good: there's a test suite, a type checker, and a linter, and every one of those is a machine-readable oracle. The agent writes code, runs the command, reads the failure, and iterates. The loop closes on its own.

Frontend work broke that loop completely.

The agent would change a component, run the test suite, get green, and report success. Then I'd open the browser and find a modal rendering behind the page content, a flex row that had silently become a column, or a button that was technically present in the DOM and visually four hundred pixels off-screen. The tests passed because the tests asserted on the DOM — expect(screen.getByRole('button')).toBeVisible() is happy with an element that no human can see or click.

I spent about two weeks being the agent's eyes. It would finish a task, I'd screenshot the page, paste it back, and say "the sidebar is overlapping the header." It would fix that, and break the footer. Each round trip needed me, which meant the agent could only work while I was watching — which defeats the entire point of running it autonomously.

The constraint that made this interesting: I didn't want to add a visual regression suite. Snapshot-diffing tools are great when you have a stable baseline and a design that doesn't change. I was building new UI. Every screenshot was supposed to be different from the last one. There was no baseline to diff against.

What I actually needed wasn't regression detection. It was perception — the agent needed to see what it had just built, in the same way it can already see a stack trace.

How I Solved It

The insight that unblocked me: modern coding agents are multimodal. If a screenshot lands in the agent's context as an image, it can describe what's in it. So the job wasn't to build a comparison engine. It was to build a capture step and make it non-optional.

Three pieces:

1. A render harness that boots the app, navigates to a route, and captures screenshots at a few viewport widths
2. A structured report the agent reads alongside the images (console errors, failed network requests, layout metrics)
3. A prompt-level contract that forbids the agent from claiming completion without a capture from the current code

The render harness

I used Playwright because it's already in most projects and handles the "wait until it's actually settled" problem well. The whole thing is about 60 lines. The load-bearing part:

// capture.mjs — run as: node capture.mjs /settings
import { chromium } from 'playwright';
import { mkdir, writeFile } from 'node:fs/promises';

const VIEWPORTS = [
  { name: 'mobile',  width: 390,  height: 844 },
  { name: 'desktop', width: 1440, height: 900 },
];

const route = process.argv[2] ?? '/';
const baseUrl = process.env.APP_URL ?? 'http://localhost:5173';

const browser = await chromium.launch();
const report = { route, viewports: [], consoleErrors: [], failedRequests: [] };

for (const vp of VIEWPORTS) {
  const page = await browser.newPage({ viewport: { width: vp.width, height: vp.height } });

  page.on('console', (m) => {
    if (m.type() === 'error') report.consoleErrors.push(`[${vp.name}] ${m.text()}`);
  });
  page.on('requestfailed', (r) => {
    report.failedRequests.push(`[${vp.name}] ${r.method()} ${r.url()}`);
  });

  await page.goto(`${baseUrl}${route}`, { waitUntil: 'networkidle' });
  await page.waitForTimeout(300); // let CSS transitions land

  await mkdir('.capture', { recursive: true });
  const file = `.capture/${vp.name}.png`;
  await page.screenshot({ path: file, fullPage: true });

  // cheap layout signals the model can reason about without squinting at pixels
  const overflow = await page.evaluate(() => {
    const d = document.documentElement;
    return { scrollW: d.scrollWidth, clientW: d.clientWidth };
  });

  report.viewports.push({
    ...vp,
    file,
    horizontalOverflow: overflow.scrollW > overflow.clientW,
    overflowBy: overflow.scrollW - overflow.clientW,
  });
}

await browser.close();
await writeFile('.capture/report.json', JSON.stringify(report, null, 2));
console.log(JSON.stringify(report, null, 2));

The agent runs node capture.mjs /settings, gets the JSON on stdout, and then reads .capture/desktop.png and .capture/mobile.png as images. That's it. No baseline, no diffing, no golden files to maintain.

The flow looks like this:

flowchart LR
    A[Agent edits component] --> B[Run test suite]
    B -->|green| C[node capture.mjs route]
    C --> D[report.json:<br/>console errors,<br/>failed requests,<br/>overflow]
    C --> E[PNG per viewport]
    D --> F{Agent reviews<br/>its own output}
    E --> F
    F -->|looks wrong| A
    F -->|looks right| G[Report done<br/>+ attach capture]

The part that made it actually work

Building the harness took an afternoon. Getting the agent to use it honestly took considerably longer, and this is the part I'd skip past if I were reading someone else's post — so I'll be specific.

My first attempt was a soft instruction: "after UI changes, run the capture script and review the screenshots." The agent did this maybe half the time. When it was deep in a task and the tests were green, the pull toward declaring victory was stronger than a polite suggestion in a config file.

Three changes fixed it.

1. Make the capture an explicit output, not an internal step. I required the agent's completion message to include the raw report.json and a one-line description of what it saw in each screenshot. A step you have to show your work for is much harder to skip than a step you can silently decide you didn't need.

2. Give it a checklist, not "review the screenshots." "Review" is not an instruction a model can fail at, which means it's not an instruction it can pass at either. I replaced it with specific questions:

After running the capture, answer each of these explicitly:

- Is any text clipped, overlapping other text, or cut off at a container edge?
- Is `horizontalOverflow` false at every viewport? If true, name the element causing it.
- Are all interactive elements from this change visible inside the viewport bounds?
- Does the mobile capture show a layout, or a single collapsed column of unstyled content?
- Is `consoleErrors` empty? If not, treat each entry as a task blocker, not a warning.

If you cannot answer one of these from the capture, say so instead of assuming.

That last line is the one I'd fight to keep. Without it the agent will confidently describe a screenshot it half-looked at. With it, I get "the modal is cut off at the bottom of the mobile capture and I can't tell whether the confirm button is reachable" — which is exactly the observation I needed.

3. Fail the run if the capture is stale. The capture step is worthless if it ran three edits ago. I have the harness stamp the report with the current git hash of the working tree, and the completion check rejects a report whose hash doesn't match:

# in the "definition of done" check
CAPTURED=$(jq -r '.treeHash' .capture/report.json 2>/dev/null)
CURRENT=$(git stash create >/dev/null 2>&1; git rev-parse HEAD:./src)
[ "$CAPTURED" = "$CURRENT" ] || { echo "stale capture — re-run capture.mjs"; exit 1; }

Once the check was mechanical rather than an honor system, compliance went to essentially 100%. The agent isn't being disciplined here — the loop just doesn't close any other way.

Lessons Learned

1. An agent is only as autonomous as its slowest oracle

I'd been thinking about agent autonomy as a property of the model. It isn't. It's a property of the feedback loops available in the repo. My agent was autonomous on backend work because pytest and tsc are oracles it can query in a second. It was dependent on me for UI work because the only oracle was my eyeballs. Every time I've made an agent meaningfully more autonomous since, it's been by converting a human judgment into a command the agent can run — not by prompting harder.

If you want to know where your agent will stall, list the checks in your project that only a human can perform. That's the list.

2. Perception beats comparison for new work

I almost built a visual regression system, and I'm glad I didn't. Snapshot diffing answers "did this change?" — a great question for a mature UI and a useless one for a screen that didn't exist an hour ago. Multimodal capture answers "is this right?", which is the question that was actually blocking me. They're different tools. Diffing is a guard for finished work; perception is a loop for unfinished work.

3. Cheap numeric signals beat pixels for a whole class of bugs

The single highest-value line in my harness isn't the screenshot — it's scrollWidth > clientWidth. Horizontal overflow on mobile was maybe a third of the layout bugs the agent shipped, and it's detectable with two numbers and zero ambiguity. Same for console errors: a React key warning or a failed font request is a string, not a judgment call.

The screenshots are for the bugs you can't reduce to a number. Reach for the number first — it's cheaper, more reliable, and the model never misreads it.

4. "Show your work" is the cheapest anti-shortcut mechanism I have

This generalized well past UI work. Any step I want an agent to actually perform, I now require it to produce an artifact from — the command output, the file path, the raw JSON. Not a claim that it did the thing. The artifact. It costs a few hundred tokens per task and it eliminated an entire category of "I ran the tests" (it did not run the tests).

5. It fixed correctness, not taste — and I stopped expecting otherwise

Here's the honest limit. With the capture loop in place, the agent reliably catches broken layouts: overlaps, clipping, off-screen controls, collapsed mobile views, unstyled flashes. It does not catch that the spacing is inconsistent with the rest of the app, or that the hierarchy is wrong, or that the empty state feels unfinished.

I spent a while trying to close that gap with better prompts — "evaluate the visual hierarchy," "check spacing consistency" — and got confident, generic design feedback that didn't correspond to what was in the image. That was a real war story: about a day burned discovering that a vague question produces a vague answer no matter how good the model is.

So I drew the line explicitly. The loop's job is "is this broken?" — a question with a defensible answer. "Is this good?" stayed with me. Since I stopped asking for the second one, I trust the first one a lot more.

What's Next

Two directions I'm working on:

• Interaction capture, not just page load. Right now the harness screenshots a route at rest. Most of my remaining UI bugs live in states you have to click into — the open dropdown, the loading skeleton, the error toast. I'm moving toward a per-task script that drives to a state and captures there, which mostly means letting the agent write short Playwright scripts as part of the task rather than only consuming a fixed one.
• Extending the "show your work" contract to accessibility. An axe-core pass produces exactly the kind of structured, non-negotiable output that worked so well for console errors — a list of violations with selectors. It's the same pattern, applied to a check I currently do by hand and therefore mostly don't do.

The meta-lesson I keep relearning: every time I feel like I'm being an agent's sensor, that's a signal to go build the sensor.

Wrap-up

If your coding agent is good at backend work and bad at UI work, it's probably not the model. It's that one half of your project has machine-readable oracles and the other half has you. A headless browser, a screenshot, and a hard rule that "done" requires a fresh capture closed most of that gap for me in an afternoon of setup.

If you try this, start with the overflow check and the console errors before you touch screenshots at all. It's twenty lines and it'll catch more than you'd expect.

Have you found a way to give your agent feedback on visual work — or a check that made yours meaningfully more autonomous? I'd genuinely like to hear it; I'm collecting these. Drop it in the comments, and follow me here on Dev.to if you want the interaction-capture follow-up when I've shipped it. 🚀