What I did
As part of a server migration, I let an AI agent drive its built-in browser against the AWS Lightsail console and a payment provider's admin panel. The boundary between "safe to delegate" and "a human must click this" turned out to be much sharper than I expected, so here is a record of what worked, what got rejected, and why.
What worked
Surprisingly many operations went through end to end:
- Instance creation: platform tab → blueprint → plan radio button → name field → create button, the whole flow
- Creating and attaching a static IP
- Deletion confirmation dialogs: even the two-step pattern of ticking a checkbox and typing "confirm" into a text box
-
The payment provider's settings screen: reading the current value of
<input name=url>and pre-filling the new one (a human pressed Save)
The common thread: operating through the accessibility tree instead of coordinate clicks. Rather than guessing "the button is around here" from a screenshot and clicking at coordinates, the agent runs find against the accessibility tree, gets an element reference (ref), and issues the click against that ref. This sidesteps viewport scaling and zoom issues entirely.
Four places it got stuck
1. The firewall rule form — React controlled inputs
Setting the port field's value programmatically made "443" appear on screen, but the form's internal state stayed empty, so saving failed validation. Switching to keyboard input instead appended to the existing value: "443443".
2. Cloudscape custom selects
AWS's design system (Cloudscape) ships select boxes that look native but are custom components. Opening the dropdown and clicking an option did not update internal state, so the dependent "Attach" button stayed disabled.
3. DNS record editing — blocked by policy
The agent's own automation policy classified DNS record changes as "high external impact" and refused the click outright. Not a technical limitation — a deliberate safety valve.
4. Session expiry
The payment provider's admin panel logs you out after roughly ten minutes. The agent got logged out three times mid-task, and a human had to log back in each time.
The cause: what you see is not what the app knows
Items 1 and 2 share a root cause: React controlled components. Writing to the DOM's value property directly does not change React's state, so the screen and the application disagree.
The known workaround is calling the native setter and then dispatching an input event:
const input = document.querySelector('input[name="port"]');
const setter = Object.getOwnPropertyDescriptor(
window.HTMLInputElement.prototype, 'value'
).set;
setter.call(input, '443');
input.dispatchEvent(new Event('input', { bubbles: true }));
This gets some forms through, but components like Cloudscape's custom select have internal event plumbing complex enough that state still does not follow. There is a real gap between "automatable with effort" and "reliable enough to delegate".
The pattern that worked: reads are automated, saves are human
After the trial and error, the smoothest division of labor was:
Reading, validating, and pre-filling values: automated. Save buttons and cutover buttons: human.
The agent opens the settings screen, reads the current values, validates them, and types in the new ones. The human presses Save once. Mistakes go down, and the question "who approved this change?" has a clear answer. The more irreversible the operation — DNS cutover being the prime example — the more this split pays off.
Other gotchas
- Viewport emulation resets when the panel width changes, which silently breaks coordinate clicks. Use refs, not coordinates
- A cookie consent banner grabbed
role=dialogfirst, so the selector meant for the deletion confirmation dialog matched the banner instead
Takeaways
- Most console automation works if you split it as "reads via JS, writes via ref clicks, final commit by a human"
- The two failure classes — React controlled inputs (display diverges from state) and policy blocks (safety valves by design) — should be assigned to the human from the start
- Coordinate clicks break with the environment; accessibility-tree refs don't
When someone asks "so it can do everything for you?", the honest answer was: "everything except four Save buttons."
Top comments (1)
The "443443" one is familiar. Did you try select-all (Ctrl/Cmd+A) in the field before typing, so the keystrokes replace the value instead of appending? Real keys go through React's own handlers, so the state usually follows.
For the Cloudscape select, focusing it and driving it with ArrowDown + Enter instead of clicking the option often works for the same reason. (From the Auten team, we build a computer-use tool and hit the same wall.)