Automating X (Twitter) in 2026: shadow DOM, 429 lockouts, and the Premium gate

leadwright — Fri, 13 Nov 2026 19:16:24 +0000

Automating X (Twitter) in 2026: shadow DOM, 429 lockouts, and the Premium gate

I spent a few weeks building a lead-search tool for X. Not a scraper library — a thing that
runs in a real browser, signed in as a real account, and tries to behave like a person while
it looks for accounts worth contacting.

Almost everything I assumed was wrong. This is the list of things that broke, and what
actually worked.

1. Every profile visit is a full app boot

X's people search gives you handles and almost nothing else. Follower counts, bios, join
dates — all of it lives on the profile page, and every profile page load is:

a GraphQL Viewer request,
the full React app booting,
a pile of static assets.

Do that 30–40 times in a row and X answers HTTP 429. After that, x.com stops booting at
all — in every browser on that IP, logged in or not — for somewhere between 30 and 60
minutes. Not a soft limit on one endpoint. The whole site.

The only reliable early signal I found was the #ScriptLoadFailure element, plus the
1003 code in the Viewer response body. Once you see it, you're done for the hour.

The important part is what you do next: nothing. Retrying is the failure mode. A naive
for attempt in range(3) turns a 20-minute lockout into an hour, because each retry is
another full app boot against a counter that's already tripped.

if await page.locator("#ScriptLoadFailure").count():
    raise RateLimited("X is refusing to serve the app; back off, do not retry")

Failure fast beats retrying with backoff, because the cost isn't the failed request — it's
the next 40 requests that now can't happen.

2. DMs moved into a separate app that renders in a shadow root

The old DM page is gone. /messages now hands you over to /i/chat, an embedded React app
served from abs.twimg.com/x-web/xchat/embed/. It renders into shadow roots.

That breaks the two things every scraper does first:

document.querySelector('[data-testid="dm-composer"]') // null
document.body.innerText                                 // "" for the chat panel

innerText on body genuinely returns nothing useful, because the chat UI is inside a
shadow tree, and shadow content isn't part of the host's text.

You can walk it by hand:

const texts = [];
const walk = (root, depth = 0) => {
  if (depth > 10) return;
  texts.push(root.textContent || "");
  for (const el of root.querySelectorAll("*")) {
    if (el.shadowRoot) walk(el.shadowRoot, depth + 1);
  }
};
walk(document);
const bodyText = texts.join("\n").replace(/\s+/g, " ");

But here's the thing that took me too long to check: Playwright's CSS engine pierces open
shadow roots already. page.locator('[data-testid="..."]') finds elements inside shadow
DOM without any traversal. My problem was never reachability — it was that I was using
selectors that no longer existed.

So don't guess the names. Read them out of the bundle:

Load /i/chat, capture the script responses from abs.twimg.com/x-web/xchat/embed/assets/.
Grep the chunks for dm- strings.
You get the real vocabulary: dm-composer-textarea, dm-composer-send-button, dm-message-scroller, dm-conversation-panel.

Mine shipped filenames in backticks, so a quote-only regex missed most of them — a plain
grep -o 'dm-[a-z-]*' found ~40 testids, the quote-matched one found five.

One more trap: the composer is a controlled React textarea. locator.fill() sets the DOM
value but doesn't fire the synthetic events React listens to, so the send button stays
disabled and the message never sends. Real keystrokes work:

await composer.click()
await page.keyboard.type(message, delay=45)

3. The composer isn't always rendered — and there's no error

If the account isn't verified and the recipient doesn't follow it, X doesn't render the
composer at all. What you get is a dialog:

Get verified to message @user — Only verified users can send Direct Message requests to
people that don't follow them.

Dismissing that dialog does not reveal a composer. There is no textarea, no
contenteditable, no role="textbox" — nothing to type into, anywhere in the document or
its shadow roots.

If your automation assumes the composer exists, it doesn't fail loudly: it fails per
target, quietly, thirty times in a row, while burning the rate-limit budget from section 1.
That's how I ended up locked out on the first real run.

Detect the condition once and stop the whole run:

if "get verified to message" in body_text.lower():
    raise AccountGated("X requires verification for cold DMs")

This isn't a bug you can code around. It's a product decision on X's side, and the only
honest responses are "pay for Premium" or "find another channel".

4. The chat app asks for a PIN the first time

Open a conversation in a fresh browser profile and /i/chat asks for an X Chat passcode —
a 4-digit PIN, presented as four inputs. They're shadow-DOM children, and Playwright's
fill() on them is flaky because of how the component handles focus.

What worked: focus the first input from inside the page, then send real keystrokes.

const boxes = [];
const walk = (root) => {
  root.querySelectorAll('input[aria-label^="Digit "]').forEach(i => boxes.push(i));
  root.querySelectorAll("*").forEach(el => el.shadowRoot && walk(el.shadowRoot));
};
walk(document);
boxes[0]?.focus();

Then page.keyboard.type(pin, delay=180). The delay matters; the component advances focus
per keystroke, and typing too fast drops digits.

5. Profile data is cleaner in the JSON-LD than in the DOM

Scraping the profile DOM gives you localized follower counts (12,4 тис.) and translated
bios. There's a <script type="application/ld+json"> block instead, with a
UserProfileSchema object containing:

exact userInteractionCount for followers,
the raw, untranslated bio,
the numeric user id.

That last one is the useful part. With a numeric id you can deep-link straight to a
conversation instead of hunting for a localized "Message" button.

6. Search results are mostly noise

The unexpected finding: X's people search returns a lot of accounts that have nothing to do
with the query. Searching for a niche term gives you local trends, unrelated verified
accounts, and people who mentioned the word once in 2021.

A cheap local relevance gate — the query term (or a synonym) must appear in the bio or display
name — cut 85 raw handles down to 7 on-topic accounts. That ratio is the actual value of
the tool; the scraping is the easy part.

What I'd tell someone starting this today

Back off before you're told to. The 429 isn't per-request, it's per-session, and the recovery cost is measured in tens of minutes.
Read the JS bundle before writing selectors. Test IDs are in there; guessing costs days.
Assume the UI you need isn't rendered. Gate checks are cheaper than retries.
Treat shadow DOM as a normal DOM. Playwright handles it; your old selectors don't.
Filter locally. The search API won't do it for you, and relevance is the product.

If you've automated a shadow-DOM app like this, I'd like to know how you test it — replaying
against a live account for every selector change doesn't scale, and I haven't found a
satisfying answer.

DEV Community: leadwright

Automating X (Twitter) in 2026: shadow DOM, 429 lockouts, and the Premium gate

Automating X (Twitter) in 2026: shadow DOM, 429 lockouts, and the Premium gate

1. Every profile visit is a full app boot

2. DMs moved into a separate app that renders in a shadow root

3. The composer isn't always rendered — and there's no error

4. The chat app asks for a PIN the first time

5. Profile data is cleaner in the JSON-LD than in the DOM

6. Search results are mostly noise

What I'd tell someone starting this today