I spent a day pointing an AI agent at a browser to publish one product across four marketplaces. Most of it worked. The parts that didn't work failed in the worst possible way: silently, with no error, no exception, and no log line.
Here are the five failure modes I hit, in the order I hit them, and the probe that turned the worst one from "this site is broken" into a two-line fix.
Everything below is from one working session. No hypotheticals.
The symptom: clicks that do nothing
I asked the agent to click a menu item. The tool reported success:
[computer:left_click] Clicked at (383, 734)
Nothing happened. No menu opened, no network request fired, no console error. I re-read the DOM — the element was there, visible, not disabled, not covered by an overlay:
const el = document.elementFromPoint(383, 734);
// => <span>削除</span> ← the exact element I wanted
So the coordinate was right, the element was right, and the click "succeeded". And nothing happened.
I burned close to an hour assuming the site was blocking synthetic input. It wasn't.
The probe: make the page answer for itself
Instead of reasoning about it, I made the page report whether a click ever arrived. Inject a button at a known position with a counter:
const b = document.createElement('button');
b.id = '__clicktest';
b.textContent = 'CLICK TEST';
b.style.cssText =
'position:fixed;top:200px;left:200px;width:220px;height:60px;' +
'z-index:2147483647;background:#f0f;';
b.addEventListener('click', (e) => {
window.__clickOK = (window.__clickOK || 0) + 1;
window.__trusted = e.isTrusted;
});
document.body.appendChild(b);
window.__clickOK = 0;
const r = b.getBoundingClientRect();
`center=${r.x + r.width / 2},${r.y + r.height / 2}`;
// => center=310,230
Then click that exact center and read the counter:
window.__clickOK; // => 0
Zero. The click was not landing on a button that occupied 220×60 pixels at a coordinate I had just measured. That rules out "the site blocks synthetic events" — a blocked event still arrives; it just gets ignored by the handler. Mine never arrived.
So the coordinate space was wrong.
The math: two coordinate spaces, one number
The automation tool takes coordinates in screenshot space. The page reports coordinates in CSS pixel space. If the screenshot is captured at a different scale than the viewport, every coordinate you read from the DOM is wrong by a constant factor — and the failure is invisible, because a click still happens, just somewhere else.
I measured the factor by bisecting. Device-pixel space (×2, since devicePixelRatio was 2) missed. Then:
window.__clickOK; // after clicking (232, 172) instead of (310, 230)
// => 1
That gives the ratio directly:
k = 232 / 310 = 0.7484
Viewport was 1702 CSS px wide; the screenshots came back ~1274 px. 1274 / 1702 ≈ 0.7485. It matched.
From then on, every click went through one conversion:
const K = 0.7484; // measure this yourself; do not copy mine
function clickPoint(el) {
const r = el.getBoundingClientRect();
return {
x: Math.round((r.x + r.width / 2) * K),
y: Math.round((r.y + r.height / 2) * K),
};
}
Every interaction that had failed for the previous hour started working on the first try — dropdowns, confirmation dialogs, menu items.
The lesson isn't the number. k depends on your window size, DPI, and tool. The lesson is that you should never trust a coordinate you didn't verify with a probe, because this class of bug produces no error at any layer.
Run the probe once at the start of a session. It costs three seconds.
Failure mode 2: the menu that was below the fold
With clicks fixed, one menu item still did nothing. Same probe logic applied — elementFromPoint returned null for it.
const r = item.getBoundingClientRect();
r.y; // => 981
window.innerHeight; // => 876
The dropdown extended past the bottom of the viewport. getBoundingClientRect() happily returns coordinates for content that isn't on screen, and a click at y=981 in a 876px viewport goes nowhere.
Scroll first, then re-measure. Never cache coordinates across a scroll:
el.scrollIntoView({ block: 'center' });
await new Promise(r => setTimeout(r, 300));
const r2 = el.getBoundingClientRect(); // re-read, always
Failure mode 3: file inputs that don't exist yet
One site had an upload drop zone with no file input in the DOM:
document.querySelectorAll('input[type=file]').length; // => 0
The input is created on demand when you click the zone. So: click the zone, then look. To catch the element before the app calls .click() on it (which opens a native dialog you can't drive), patch createElement first:
const orig = document.createElement.bind(document);
window.__created = [];
document.createElement = function (tag, ...rest) {
const el = orig(tag, ...rest);
if (String(tag).toLowerCase() === 'input') {
setTimeout(() => {
if (el.type === 'file') window.__created.push(el);
}, 0);
}
return el;
};
Now click the zone and the input is waiting for you in window.__created. Populate it with a DataTransfer and dispatch change:
const dt = new DataTransfer();
dt.items.add(file);
input.files = dt.files;
input.dispatchEvent(new Event('change', { bubbles: true }));
This worked on two of the sites I tried. On the third it didn't — see below.
Failure mode 4: synthetic drops that React ignores
Some drop zones aren't inputs at all. The obvious move is to synthesize the drop:
const dt = new DataTransfer();
dt.items.add(file);
for (const type of ['dragenter', 'dragover', 'drop']) {
const ev = new DragEvent(type, { bubbles: true, cancelable: true });
Object.defineProperty(ev, 'dataTransfer', { value: dt });
zone.dispatchEvent(ev);
}
I tried this against every plausible target — the drop zone, its ancestors, document, window. Nothing. The editor never registered a file.
To be sure the file wasn't the problem, I generated one in-page so there was no transfer step at all:
const c = document.createElement('canvas');
c.width = 1200; c.height = 800;
c.getContext('2d').fillRect(0, 0, 1200, 800);
const blob = await new Promise(r => c.toBlob(r, 'image/jpeg', 0.8));
const file = new File([blob], 'test.jpg', { type: 'image/jpeg' });
Same result. That's a real boundary: for that editor, drag-and-drop needs genuine OS-level input, and no amount of event synthesis substitutes for it. I stopped and did those uploads by hand.
Knowing where the wall is is worth more than another hour of clever attempts.
Failure mode 5: localhost is still mixed content
To avoid shipping image bytes through a JS payload, I served them locally with CORS enabled and fetched them from the page:
class H(http.server.SimpleHTTPRequestHandler):
def end_headers(self):
self.send_header('Access-Control-Allow-Origin', '*')
super().end_headers()
curl confirmed the header. From an https:// page, the fetch hung until it timed out:
await fetch('http://127.0.0.1:8941/img1.png'); // never resolves
An https page fetching http://127.0.0.1 is mixed content. The CORS header is irrelevant — the request never gets far enough to matter. Terminate that path early instead of debugging your server.
The preflight I now run first
Five minutes of setup that would have saved me most of a day:
// 1. Coordinate probe — is my click space the page's space?
// (inject the test button above, click its center, read window.__clickOK)
// 2. Is the target actually on screen?
const onScreen = (el) => {
const r = el.getBoundingClientRect();
return r.top >= 0 && r.bottom <= window.innerHeight && r.width > 0;
};
// 3. Does this page even have the input I assume it has?
document.querySelectorAll('input[type=file]').length;
// 4. Verify before every destructive click.
// Hover first, then confirm what is under the cursor:
document.elementFromPoint(x / K, y / K)?.textContent?.trim();
Step 4 matters more than it looks. In one menu, "Delete" sat directly below "Unpublish" — 24 pixels apart. With a 25% coordinate error, "unpublish" lands on "delete". Hovering and reading back the element text before committing turns a destructive misfire into a no-op.
What I'd tell my past self
The productive shift wasn't a better selector strategy. It was refusing to reason about why a click "didn't work" and instead making the page report what it actually received.
Every one of these five failures is invisible from the outside: success-shaped tool output, no exception, no console error. The only reliable signal came from instrumenting the page and reading a counter.
If your automation is mysteriously doing nothing, don't start with the selector. Start with the probe.
All measurements here come from a single session driving Chrome against live marketplace sites. The scaling factor k is specific to that window and tool — measure your own.
Top comments (0)