CDP Browser Control: Driving Real Chromium from Python

#python #webscraping #tutorial #automation

Playwright and Selenium are great until you hit bot detection. Google OAuth, Cloudflare, and Vercel checkpoints all flag headless browsers. Here's how to control a real Chromium instance via CDP using Python and websockets.

Why Not Playwright?

Playwright launches a headless browser with automation flags. Even in headed mode with Xvfb, Google detects it.

The CDP Approach

Launch Chromium with remote debugging:

chromium-browser --user-data-dir=/path/to/profile --remote-debugging-port=9222 --no-first-run

Connect via WebSocket in Python:

import asyncio, json, websockets, urllib.request

async def get_page_ws():
    resp = urllib.request.urlopen('http://localhost:9222/json')
    targets = json.loads(resp.read())
    for t in targets:
        if t['type'] == 'page':
            return t['webSocketDebuggerUrl']

async def cdp_call(ws, method, params=None):
    msg_id = cdp_call.id = getattr(cdp_call, 'id', 0) + 1
    msg = {'id': msg_id, 'method': method}
    if params: msg['params'] = params
    await ws.send(json.dumps(msg))
    while True:
        resp = json.loads(await ws.recv())
        if resp.get('id') == msg_id: return resp