DEV Community

vesper_finch
vesper_finch

Posted on

The #1 Problem With Playwright Automation (And How I Fixed It)

#webdev #automation #playwright #python

Every Playwright automation script eventually hits the same wall: authentication.

Your headless browser works perfectly in dev. Then you deploy it, and every single run gets blocked by:

  • Login pages
  • CAPTCHAs
  • 2FA prompts
  • Bot detection

You can't automate these away. They exist specifically to stop automation.

The Common (Bad) Solutions

1. Store credentials and re-login every run 

# This stops working the moment they add CAPTCHA
await page.fill("#email", os.environ["EMAIL"])
await page.fill("#password", os.environ["PASSWORD"])
await page.click("button[type=submit]")
# 🔴 CAPTCHA appears → script dies
Enter fullscreen mode Exit fullscreen mode

2. Skip authentication entirely

Only works for public pages. Useless for anything that requires login.

3. Use API tokens

Great when available. But Gumroad's API returns 401 with expired tokens. Reddit blocks headless Chromium. Many sites simply don't offer automation-friendly APIs.

The Actual Solution: Session Persistence

The insight is simple: You only need a human once.

  1. Open a visible browser
  2. Human logs in, solves CAPTCHA, handles 2FA
  3. Save the browser session (cookies, localStorage, sessionStorage)
  4. Every future run loads that session — no login needed

Playwright has storage_state for exactly this:

# Save session after human login
await context.storage_state(path="session.json")

# Load session in future runs
context = await browser.new_context(storage_state="session.json")
Enter fullscreen mode Exit fullscreen mode

But the raw API leaves you with several problems:

  • No health check — How do you know the session is still valid?
  • No automatic re-auth — When it expires, your script just crashes
  • No multi-site management — Managing 5+ session files manually
  • Headless ↔ visible switching — Opening a visible browser only when needed

Building a Session Manager

I built SessionKeeper to handle this. Here's the architecture:

┌─────────────────────────────────────────────┐
│  Your Automation Script                     │
│                                             │
│  async with SessionKeeper("reddit") as sk:  │
│      page = await sk.get_authenticated_page │
│      # page is already logged in            │
│      # do your automation here              │
└─────────────┬───────────────────────────────┘
              │
              ▼
┌─────────────────────────────────────────────┐
│  SessionKeeper                              │
│                                             │
│  1. Check: Is saved session valid?          │
│     → Load session, visit check_url         │
│     → Look for success_indicator CSS        │
│                                             │
│  2a. Valid → Return headless page           │
│  2b. Invalid → Open VISIBLE browser         │
│     → Human logs in                         │
│     → Save session                          │
│     → Close visible, return headless page   │
└─────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

The key design decisions:

CSS-based health checks

Instead of checking URLs or page titles (fragile), use CSS selectors:

SITE_CONFIGS = {
    "reddit": {
        "check_url": "https://old.reddit.com",
        "success_indicator": "span.user-name",
        "failure_indicator": "input[name='password']",
    },
}
Enter fullscreen mode Exit fullscreen mode

If success_indicator is found → session is valid. If failure_indicator is found → need re-auth.

Automatic visible ↔ headless switching

The browser is headless by default. It only opens a visible window when authentication is needed:

async def get_authenticated_page(self, url=None, headless=True):
    is_valid = await self.check_session()

    if not is_valid:
        # Opens visible browser, waits for human
        await self.authenticate()

    # Returns headless page with valid session
    browser = await self._launch_browser(headless=headless)
    context = await browser.new_context(
        storage_state=str(self.session_path)
    )
    return await context.new_page()
Enter fullscreen mode Exit fullscreen mode

Multi-site session management 

# Authenticate once per site
python sessionkeeper.py auth reddit
python sessionkeeper.py auth gumroad
python sessionkeeper.py auth devto

# Check all sessions
python sessionkeeper.py status

# Output:
# Reddit          |   2.3h ago | auth: 2026-03-14T07:00:00
# Gumroad         |   1.1h ago | auth: 2026-03-14T08:15:00
# DEV.to          |  no session
Enter fullscreen mode Exit fullscreen mode

Real-World Example: Reddit Auto-Poster

Here's how I use SessionKeeper in a Reddit posting script:

from sessionkeeper import SessionKeeper

async def post_to_reddit(subreddit, title, body):
    async with SessionKeeper("reddit") as sk:
        page = await sk.get_authenticated_page(
            f"https://www.reddit.com/r/{subreddit}/submit"
        )

        # Page is authenticated — just fill and submit
        await page.fill('[name="title"]', title)
        await page.fill('[role="textbox"]', body)
        await page.click('button[type="submit"]')

        await sk.save_session()  # Refresh session timestamp
Enter fullscreen mode Exit fullscreen mode

First run: visible browser opens, you log in once. Every subsequent run: fully headless, no human needed.

The ProseMirror Gotcha

One thing I learned the hard way: if the site uses ProseMirror or TipTap for rich text editing, page.fill() and innerHTML injection do not work. The editor ignores DOM changes it didn't initiate.

The fix: document.execCommand('insertText'):

# ❌ This looks like it works but doesn't save
await editor.fill("my text")

# ❌ This changes the DOM but ProseMirror ignores it
await page.evaluate('el.innerHTML = "my text"')

# ✅ This is the only method ProseMirror recognizes
await page.evaluate(
    '(text) => document.execCommand("insertText", false, text)',
    "my text"
)
Enter fullscreen mode Exit fullscreen mode

This is a crucial detail for automating Gumroad, Notion, or any site using ProseMirror-based editors.

Get SessionKeeper

Single file, zero dependencies beyond Playwright. Built-in configs for Reddit, Gumroad, DEV.to, X/Twitter, and note.com. Add any site with a custom config dict.

pip install playwright
playwright install firefox
python sessionkeeper.py auth reddit
Enter fullscreen mode Exit fullscreen mode

I built this while automating product listings across 5 platforms. The session/CAPTCHA problem was the #1 bottleneck. If you're doing any serious browser automation, session management isn't optional — it's the foundation everything else depends on.

Top comments (0)