DEV Community

Cover image for From Spawn to JSON: Launching Chrome and Attaching CDP in Pure Python
Saiful Islam
Saiful Islam

Posted on

From Spawn to JSON: Launching Chrome and Attaching CDP in Pure Python

Launch Chrome, attach to CDP, and extract structured data with a zero-dependency Python service

Most Python browser automation tutorials begin with pip install playwright and end with page.click(). That is a good default. Playwright is mature, it handles waits, and it lets you forget that a browser is a separate operating-system process.

But sometimes you need to see the layer underneath. Maybe you are debugging a scraper that hangs once every thousand pages. Maybe you need the JSON response behind an infinite scroll, not the DOM that happens to be rendered. Maybe you want to attach to a Chrome instance that is already running with a logged-in profile. Or maybe you just want to stop treating one of the most complicated programs on your machine as a black box.

This tutorial builds that layer from scratch.

You will launch Chrome yourself, discover the port it chose, open one WebSocket, multiplex commands and events over it, attach to a tab, navigate, evaluate JavaScript, and turn the result into structured Python data. The runtime dependency list is empty: Python's standard library is enough.

The project uses uv and a small service layer, so each file owns one boundary instead of one Browser class owning the process, the socket, the protocol, and your scraping logic at the same time.

What you need: Python 3.12+, uv, and Chrome, Chromium, Edge, or Brave installed. A Linux server without a desktop is fine; use headless mode.


What the Chrome DevTools Protocol actually is

The Chrome DevTools Protocol (CDP) is the interface Chrome exposes for tools like DevTools, Puppeteer, and Playwright. It is a set of commands and events grouped into domains:

Domain What it is for
Target Browser contexts, tabs, and sessions
Page Navigation, lifecycle events, screenshots
Runtime Evaluating JavaScript in a page
Network Requests, responses, cookies, response bodies
Emulation User agent, viewport, device metrics
Fetch Intercepting requests, including proxy authentication

The transport is a WebSocket carrying JSON messages. A command looks like this:

{
  "id": 1,
  "method": "Page.navigate",
  "params": { "url": "https://example.com" }
}
Enter fullscreen mode Exit fullscreen mode

A reply carries the same id:

{ "id": 1, "result": { "frameId": "ABC", "loaderId": "DEF" } }
Enter fullscreen mode Exit fullscreen mode

An event has no id and arrives whenever Chrome feels like it:

{ "method": "Page.loadEventFired", "params": { "timestamp": 1234.5 } }
Enter fullscreen mode Exit fullscreen mode

When you attach to a tab, Chrome adds a sessionId to both directions, so one socket can carry traffic for many tabs.

That is almost the whole mental model. The complexity is in the lifecycle and the races:

  • Chrome is a child process that can crash, ignore signals, or start slowly.
  • The debugging port is not always the one you asked for.
  • A fast page can finish loading before you subscribe to the load event.
  • A dead socket must wake every caller that is waiting for a reply, or your worker hangs forever.
  • Headless Chrome sometimes gets a different page than the one you expected.

The service layer exists to make those problems explicit.


The shape of the service

Create a fresh project. We are deliberately not copying a large application layout; this is the smallest set of boundaries that stays testable.

chrome-cdp/
├── pyproject.toml
└── src/chrome_cdp/
    ├── __init__.py
    ├── browser.py      # find, launch, and own the Chrome process
    ├── connection.py   # multiplex CDP commands and events
    ├── page.py         # attach to a tab, navigate, evaluate
    ├── transport.py    # RFC 6455 WebSocket, text frames only
    ├── errors.py       # ChromeError, CDPError
    └── demo.py         # extraction example
Enter fullscreen mode Exit fullscreen mode

Each service has one job:

  • BrowserService owns the operating-system process and its temporary profile.
  • CDPConnection owns the protocol: request IDs, replies, events, timeouts.
  • PageService owns one tab session and the operations you actually want.
  • WebSocket owns bytes on the wire, not browser concepts.
  • errors.py gives every layer the same failure vocabulary.

That separation matters when something breaks. A timeout is a CDP problem. A failed launch is a process problem. A missing selector is a page problem. Do not make all three look like RuntimeError: something went wrong.


Step 0: Create the project with uv

uv init --package chrome-cdp
cd chrome-cdp
uv add --dev pytest
Enter fullscreen mode Exit fullscreen mode

uv init --package creates a src/chrome_cdp/ layout and a build configuration. Replace the generated pyproject.toml with this:

[project]
name = "chrome-cdp"
version = "0.1.0"
description = "Launch Chrome and talk to it over the DevTools Protocol"
readme = "README.md"
requires-python = ">=3.12"
dependencies = []

[project.scripts]
cdp-demo = "chrome_cdp.demo:main"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.build.targets.wheel]
packages = ["src/chrome_cdp"]
Enter fullscreen mode Exit fullscreen mode

There are no runtime dependencies. pytest is the only development dependency. Run:

uv sync
Enter fullscreen mode Exit fullscreen mode

If you would rather not hand-roll the WebSocket layer, add uv add websockets and use its synchronous client. The CDP multiplexing ideas in this article still apply. The zero-dependency version is included here because it makes the protocol visible and because it is small enough to audit.


Step 1: Find a Chrome binary

Do not hard-code /usr/bin/google-chrome. People run Chrome, Chromium, Edge, and Brave, on macOS, Linux, Windows, and inside containers. A locator service keeps that decision in one place.

# src/chrome_cdp/browser.py

import os
import platform
import shutil

CANDIDATES = {
    "Darwin": [
        "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
        "/Applications/Chromium.app/Contents/MacOS/Chromium",
        "/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge",
        "/Applications/Brave Browser.app/Contents/MacOS/Brave Browser",
    ],
    "Linux": [
        "google-chrome",
        "google-chrome-stable",
        "chromium",
        "chromium-browser",
        "microsoft-edge",
        "brave-browser",
    ],
    "Windows": [
        r"%ProgramFiles%\Google\Chrome\Application\chrome.exe",
        r"%ProgramFiles(x86)%\Google\Chrome\Application\chrome.exe",
        r"%ProgramFiles%\Microsoft\Edge\Application\msedge.exe",
        r"%ProgramFiles%\BraveSoftware\Brave-Browser\Application\brave.exe",
    ],
}


def find_chrome() -> str:
    for candidate in CANDIDATES.get(platform.system(), []):
        candidate = os.path.expandvars(candidate)
        path = candidate if os.sep in candidate else (shutil.which(candidate) or "")
        if path and os.path.isfile(path):
            return path
    raise ChromeError(
        "No Chrome/Chromium found. Install Google Chrome, or pass its path."
    )
Enter fullscreen mode Exit fullscreen mode

Two practical notes:

  1. Let the caller override the path. BrowserService(path="/opt/chrome/chrome") is how you support a downloaded Chrome for Testing binary, a container image, or a corporate install.
  2. On Ubuntu, skip the snap build of Chromium. Its private /tmp means the DevToolsActivePort file lands somewhere your process cannot see. Use the .deb package, Google Chrome, or Chrome for Testing instead.

For a server, install Chrome once or download Chrome for Testing and verify the archive's SHA-256 before running it. A browser binary is executable code; treat it like one.


Step 2: Launch Chrome and discover its port

Here is the launch sequence:

  1. Choose an isolated --user-data-dir.
  2. Ask Chrome to choose a free debugging port with --remote-debugging-port=0.
  3. Start the process with about:blank.
  4. Poll for the DevToolsActivePort file in the profile directory.
  5. Read the port and the browser-level WebSocket path from that file.

That file is the reliable handshake. It is written by Chrome once the DevTools endpoint is ready, and it contains two whitespace-separated tokens:

51234
/devtools/browser/6f1c...
Enter fullscreen mode Exit fullscreen mode

The first token is the port. The second is the path of the browser endpoint. Do not scrape stderr for a URL, and do not assume port 9222.

# src/chrome_cdp/browser.py

import os
import shutil
import subprocess
import tempfile
import time
from pathlib import Path

from .connection import CDPConnection
from .errors import ChromeError


class BrowserService:
    def __init__(
        self,
        path: str | None = None,
        headless: bool = True,
        timeout: float = 30.0,
    ) -> None:
        self.path = path or find_chrome()
        self.timeout = timeout
        self.profile = tempfile.mkdtemp(prefix="chrome-cdp-")
        self.proc: subprocess.Popen | None = None
        self.connection: CDPConnection | None = None

        # A file, not a PIPE: a full stderr pipe can block Chrome.
        self._stderr = Path(self.profile, "chrome-stderr.log").open("w+b")
        try:
            args = self._args(headless)
            port, ws_path = self._launch(args)
            self.connection = CDPConnection(
                f"ws://127.0.0.1:{port}{ws_path}", timeout=timeout
            )
            self._fix_user_agent()
        except Exception:
            self.close()
            raise

    def _args(self, headless: bool) -> list[str]:
        args = [
            self.path,
            "--remote-debugging-port=0",
            f"--user-data-dir={self.profile}",
            "--no-first-run",
            "--no-default-browser-check",
            "--disable-extensions",
            "--disable-background-networking",
            "--disable-dev-shm-usage",
            "--disable-blink-features=AutomationControlled",
            "--window-size=1280,900",
        ]
        if headless:
            args.append("--headless=new")
        if hasattr(os, "geteuid") and os.geteuid() == 0:
            # Chrome refuses to run as root with its sandbox.
            args.append("--no-sandbox")
        return args

    def _launch(self, args: list[str]) -> tuple[str, str]:
        port_file = Path(self.profile, "DevToolsActivePort")
        port_file.unlink(missing_ok=True)
        self.proc = subprocess.Popen(
            args + ["about:blank"],
            stdout=subprocess.DEVNULL,
            stderr=self._stderr,
            stdin=subprocess.DEVNULL,
        )

        deadline = time.monotonic() + self.timeout
        while time.monotonic() < deadline:
            if self.proc.poll() is not None:
                self._stderr.seek(0)
                raise ChromeError(
                    f"Chrome exited with {self.proc.returncode}: "
                    f"{self._stderr.read().decode(errors='replace')[-2000:]}"
                )
            try:
                port, ws_path = port_file.read_text().split()
                return port, ws_path
            except (OSError, ValueError):
                time.sleep(0.1)
        raise ChromeError(
            f"Chrome published no DevTools port within {self.timeout}s"
        )
Enter fullscreen mode Exit fullscreen mode

Why these flags

  • --remote-debugging-port=0 asks the operating system for a free port. This avoids collisions when several workers launch browsers at once.
  • --user-data-dir=... gives this run its own cookies, cache, and history. It also keeps the browser from touching your real profile.
  • Chrome 136 and newer changed the rules. Remote debugging is ignored when Chrome uses its default profile directory. If you see “port 9222 refused” after an upgrade, the missing --user-data-dir is usually the reason. Always pass one.
  • --headless=new is the current headless mode. The old headless implementation is gone from modern Chrome.
  • --disable-dev-shm-usage avoids /dev/shm exhaustion in containers.
  • --no-sandbox is not a default. Only use it when you are root in a container or when your environment cannot provide a sandbox. It weakens process isolation.
  • --disable-blink-features=AutomationControlled removes a common automation signal. It does not make you invisible. Treat anti-bot work as a separate problem.

Why stderr goes to a file

If Chrome writes a lot of diagnostics and nobody reads the pipe, the pipe buffer fills and Chrome blocks. A file also lets you show the last 2,000 bytes when startup fails, which is usually enough to identify a missing shared library or a profile lock.

Startup failure should name the step

Wrap each phase in BrowserService with a step label. Instead of OSError, report Could not start Chrome while connecting to Chrome's DevTools port. When a scraper fails at 3 a.m., the difference between “failed” and “failed here” is the difference between a five-minute fix and an hour of guessing.


Step 3: A WebSocket client you can read in one sitting

CDP only needs text frames. It does not need extensions, compression, or wss:// for a local browser. That means a small RFC 6455 client is practical.

The handshake is an HTTP GET with Upgrade: websocket. The server replies 101 Switching Protocols and proves it understood the key by sending:

Sec-WebSocket-Accept: base64(SHA1(client_key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"))
Enter fullscreen mode Exit fullscreen mode

After that, each frame has an opcode, a length, and (for clients) a mask. Text frames use opcode 0x1; ping and pong are 0x9 and 0xA; close is 0x8.

# src/chrome_cdp/transport.py

import base64
import hashlib
import os
import socket
import struct
import threading
import urllib.parse

GUID = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"


class WebSocketError(OSError):
    pass


class WebSocketClosed(WebSocketError):
    pass


class WebSocket:
    def __init__(self, url: str, timeout: float = 10.0) -> None:
        parsed = urllib.parse.urlparse(url)
        if parsed.scheme != "ws" or not parsed.hostname:
            raise WebSocketError(f"expected ws://host..., got {url!r}")

        self.sock = socket.create_connection(
            (parsed.hostname, parsed.port or 80), timeout=timeout
        )
        self.sock.settimeout(timeout)
        self._send_lock = threading.Lock()
        self._buffer = b""
        self._handshake(parsed)

    def _handshake(self, parsed: urllib.parse.ParseResult) -> None:
        key = base64.b64encode(os.urandom(16)).decode()
        path = parsed.path or "/"
        if parsed.query:
            path += f"?{parsed.query}"
        host = parsed.hostname or "127.0.0.1"
        port = parsed.port or 80

        self.sock.sendall(
            (
                f"GET {path} HTTP/1.1\r\n"
                f"Host: {host}:{port}\r\n"
                "Upgrade: websocket\r\n"
                "Connection: Upgrade\r\n"
                f"Sec-WebSocket-Key: {key}\r\n"
                "Sec-WebSocket-Version: 13\r\n\r\n"
            ).encode()
        )

        head = b""
        while b"\r\n\r\n" not in head:
            chunk = self.sock.recv(4096)
            if not chunk:
                raise WebSocketClosed("closed during the handshake")
            head += chunk
        head, self._buffer = head.split(b"\r\n\r\n", 1)

        status, _, headers = head.decode("latin-1").partition("\r\n")
        if " 101 " not in status:
            raise WebSocketError(f"upgrade refused: {status}")

        expected = base64.b64encode(
            hashlib.sha1((key + GUID).encode()).digest()
        ).decode()
        accepts = [
            value.strip()
            for name, _, value in (
                line.partition(":") for line in headers.split("\r\n")
            )
            if name.lower() == "sec-websocket-accept"
        ]
        if accepts != [expected]:
            raise WebSocketError("bad Sec-WebSocket-Accept")

        # The reader thread may block indefinitely while the browser is quiet.
        # Per-command timeouts live in CDPConnection instead.
        self.sock.settimeout(None)

    def _read_exact(self, size: int) -> bytes:
        while len(self._buffer) < size:
            chunk = self.sock.recv(max(65536, size - len(self._buffer)))
            if not chunk:
                raise WebSocketClosed("socket closed")
            self._buffer += chunk
        head, self._buffer = self._buffer[:size], self._buffer[size:]
        return head

    def _send_frame(self, opcode: int, payload: bytes) -> None:
        mask = os.urandom(4)
        header = bytes([0x80 | opcode])
        length = len(payload)
        if length < 126:
            header += bytes([0x80 | length])
        elif length < 65536:
            header += bytes([0x80 | 126]) + struct.pack(">H", length)
        else:
            header += bytes([0x80 | 127]) + struct.pack(">Q", length)

        masked = bytes(byte ^ mask[i & 3] for i, byte in enumerate(payload))
        with self._send_lock:
            self.sock.sendall(header + mask + masked)

    def send(self, text: str) -> None:
        self._send_frame(0x1, text.encode())

    def recv(self) -> str:
        parts: list[bytes] = []
        started = False
        while True:
            first, second = self._read_exact(2)
            fin = first & 0x80
            opcode = first & 0x0F
            length = second & 0x7F
            if length == 126:
                length = struct.unpack(">H", self._read_exact(2))[0]
            elif length == 127:
                length = struct.unpack(">Q", self._read_exact(8))[0]

            mask = self._read_exact(4) if second & 0x80 else None
            data = self._read_exact(length)
            if mask:
                data = bytes(byte ^ mask[i & 3] for i, byte in enumerate(data))

            if opcode == 0x8:
                raise WebSocketClosed("closed by the peer")
            if opcode == 0x9:  # ping -> pong
                self._send_frame(0xA, data)
                continue
            if opcode == 0xA:  # pong
                continue
            if opcode in (0x1, 0x2):
                parts, started = [data], True
            elif opcode == 0x0 and started:
                parts.append(data)
            else:
                raise WebSocketError(f"unexpected opcode {opcode}")

            if fin:
                return b"".join(parts).decode("utf-8")

    def close(self) -> None:
        try:
            self._send_frame(0x8, struct.pack(">H", 1000))
        except OSError:
            pass
        try:
            self.sock.close()
        except OSError:
            pass
Enter fullscreen mode Exit fullscreen mode

This is intentionally minimal. It handles masked client frames, fragmented messages, ping/pong, and clean closes. It does not do compression, TLS, extensions, or subprotocol negotiation. CDP over ws://127.0.0.1 needs none of those.

If you keep the reader on one thread and set a socket timeout, be careful: after the handshake the socket timeout is cleared. Otherwise a quiet browser would look like a dead one. Timeouts belong at the command layer, not the byte layer.


Step 4: Multiplex commands and events over one socket

Here is the problem with a naive client:

send Page.navigate
wait for reply
Enter fullscreen mode Exit fullscreen mode

While you wait, Chrome may send Page.loadEventFired, Network.requestWillBeSent, Network.responseReceived, and a dozen other events. If your loop expects the next message to be the reply to your command, it will misread an event as a reply and deadlock.

A real CDP client needs two routes:

  • replies routed by id to the caller waiting on that command;
  • events routed by (sessionId, method) to whichever subscriber asked for them.

One background reader thread handles both.

# src/chrome_cdp/connection.py

import json
import queue
import threading
from typing import Any

from .errors import CDPError
from .transport import WebSocket

DEAD = object()


class CDPConnection:
    def __init__(self, ws_url: str, timeout: float = 30.0) -> None:
        self.ws = WebSocket(ws_url, timeout=timeout)
        self.timeout = timeout
        self.closed: Exception | None = None
        self.user_agent: str | None = None

        self._next_id = 0
        self._id_lock = threading.Lock()
        self._pending: dict[int, queue.Queue[dict]] = {}
        self._subs: dict[tuple[str | None, str], list[queue.Queue[Any]]] = {}
        self._subs_lock = threading.Lock()

        self._reader = threading.Thread(
            target=self._read_loop, name="cdp-reader", daemon=True
        )
        self._reader.start()

    def _read_loop(self) -> None:
        try:
            while True:
                message = json.loads(self.ws.recv())

                if "id" in message:
                    waiter = self._pending.pop(message["id"], None)
                    if waiter:
                        waiter.put(message)
                    continue

                key = (message.get("sessionId"), message.get("method", ""))
                with self._subs_lock:
                    listeners = tuple(self._subs.get(key, ()))
                for listener in listeners:
                    listener.put(message.get("params", {}))
        except Exception as error:
            self.closed = error
            reply = {"error": {"message": f"connection lost: {error}"}}
            for waiter in list(self._pending.values()):
                waiter.put(reply)
            with self._subs_lock:
                everyone = [q for qs in self._subs.values() for q in qs]
            for listener in everyone:
                listener.put(DEAD)

    def call(
        self,
        method: str,
        params: dict | None = None,
        session_id: str | None = None,
        timeout: float | None = None,
    ) -> dict:
        if self.closed:
            raise CDPError(f"{method}: connection lost: {self.closed}")

        with self._id_lock:
            self._next_id += 1
            message_id = self._next_id

        request = {"id": message_id, "method": method, "params": params or {}}
        if session_id:
            request["sessionId"] = session_id

        waiter: queue.Queue[dict] = queue.Queue(1)
        self._pending[message_id] = waiter
        try:
            self.ws.send(json.dumps(request))
        except OSError as error:
            self._pending.pop(message_id, None)
            raise CDPError(f"{method}: send failed: {error}") from error

        try:
            reply = waiter.get(timeout=timeout or self.timeout)
        except queue.Empty:
            self._pending.pop(message_id, None)
            raise CDPError(
                f"{method}: no reply within {timeout or self.timeout}s"
            )

        if "error" in reply:
            raise CDPError(f"{method}: {reply['error'].get('message')}")
        return reply.get("result", {})

    def subscribe(self, method: str, session_id: str | None = None) -> queue.Queue:
        q: queue.Queue[Any] = queue.Queue()
        with self._subs_lock:
            self._subs.setdefault((session_id, method), []).append(q)
        return q

    def unsubscribe(
        self, method: str, q: queue.Queue, session_id: str | None = None
    ) -> None:
        with self._subs_lock:
            listeners = self._subs.get((session_id, method), [])
            if q in listeners:
                listeners.remove(q)

    def wait_for(
        self,
        method: str,
        session_id: str | None = None,
        timeout: float | None = None,
        q: queue.Queue | None = None,
    ) -> dict:
        own = q is None
        q = q or self.subscribe(method, session_id)
        try:
            event = q.get(timeout=timeout or self.timeout)
        except queue.Empty:
            raise CDPError(
                f"no {method} within {timeout or self.timeout}s"
            )
        finally:
            if own:
                self.unsubscribe(method, q, session_id)

        if event is DEAD:
            raise CDPError(f"{method}: connection lost")
        return event

    def close(self) -> None:
        self.ws.close()
Enter fullscreen mode Exit fullscreen mode

The parts worth stealing even if you rewrite everything else:

One queue per pending command

self._pending[message_id] is a Queue(1). The reader puts exactly one reply into it. The caller blocks on get() with a timeout. There is no global condition variable and no chance of one reply waking the wrong caller.

Event subscriptions are queues, not callbacks

Callbacks run on the reader thread. If a callback blocks, every command and event stops. Queues let the consumer decide when to process the event, and they make tests trivial: push a dict, assert what the consumer did.

The death sentinel

When the reader dies, every pending call gets an error reply and every subscriber gets DEAD. This is the difference between a clean failure and a worker that waits forever after Chrome crashes. Never rely on the operating system to close the socket promptly; it may not.

Timeouts at the call layer

The socket blocks indefinitely after the handshake. Each call() gets a timeout. A page can take 40 seconds to load without making the whole connection look dead. If a command times out, remove its pending entry so a late reply does not leak a queue.

Thread safety

Incrementing the ID is guarded by a lock. The pending dictionary is only touched by the caller that owns the ID and by the reader, and the Queue makes that handoff safe. Multiple worker threads can call CDPConnection.call() concurrently; command IDs keep the replies separate.

One thing this minimal version does not do: validate that a CDP reply's sessionId matches the command. If you care about a malicious or buggy browser, add that check. For local Chrome, it is not the first problem you will hit.


Step 5: Attach to a tab

A browser-level CDP connection can create tabs and manage contexts, but it cannot evaluate JavaScript in a page directly. You need a target and a session:

  1. Target.createTarget creates a tab and returns a targetId.
  2. Target.attachToTarget with flatten: true returns a sessionId.
  3. Enable the domains you need on that session.
  4. Send page commands with the sessionId.

flatten: true is important. Without it, Chrome wants a separate WebSocket connection for each target. Flattening lets you multiplex every session over the browser connection, which is exactly what a worker pool wants.

# src/chrome_cdp/page.py

import time
from typing import Any

from .connection import CDPConnection
from .errors import CDPError


class PageService:
    def __init__(
        self,
        connection: CDPConnection,
        url: str = "about:blank",
        context_id: str | None = None,
    ) -> None:
        self.connection = connection
        params: dict = {"url": "about:blank"}
        if context_id:
            params["browserContextId"] = context_id

        self.target_id = connection.call("Target.createTarget", params)["targetId"]
        self.session_id = connection.call(
            "Target.attachToTarget",
            {"targetId": self.target_id, "flatten": True},
        )["sessionId"]

        for domain in ("Page", "Runtime", "Network"):
            self.call(f"{domain}.enable")

        if connection.user_agent:
            self.call(
                "Emulation.setUserAgentOverride",
                {"userAgent": connection.user_agent},
            )

        if url != "about:blank":
            self.navigate(url)

    def call(
        self,
        method: str,
        params: dict | None = None,
        timeout: float | None = None,
    ) -> dict:
        return self.connection.call(method, params, self.session_id, timeout)

    def navigate(self, url: str, timeout: float = 45.0) -> None:
        # Subscribe BEFORE navigating: a fast page can fire the load event
        # between Page.navigate returning and a later subscribe.
        loaded = self.connection.subscribe("Page.loadEventFired", self.session_id)
        try:
            result = self.call("Page.navigate", {"url": url}, timeout=timeout)
            if result.get("errorText"):
                raise CDPError(f"navigate {url}: {result['errorText']}")
            self.connection.wait_for(
                "Page.loadEventFired", self.session_id, timeout, q=loaded
            )
        finally:
            self.connection.unsubscribe(
                "Page.loadEventFired", loaded, self.session_id
            )

    def evaluate(self, expression: str, timeout: float | None = None) -> Any:
        result = self.call(
            "Runtime.evaluate",
            {
                "expression": expression,
                "returnByValue": True,
                "awaitPromise": True,
            },
            timeout,
        )
        if "exceptionDetails" in result:
            detail = result["exceptionDetails"]
            message = (
                detail.get("exception", {}).get("description")
                or detail.get("text")
            )
            raise CDPError(f"JavaScript: {message}")
        return result.get("result", {}).get("value")

    def wait_for_truthy(
        self, expression: str, timeout: float = 15.0, interval: float = 0.25
    ) -> Any:
        deadline = time.monotonic() + timeout
        while time.monotonic() < deadline:
            try:
                value = self.evaluate(expression)
            except CDPError:
                if self.connection.closed:
                    raise
                value = None
            if value:
                return value
            time.sleep(interval)
        return None

    def close(self) -> None:
        try:
            self.connection.call(
                "Target.closeTarget", {"targetId": self.target_id}
            )
        except (CDPError, OSError):
            pass
Enter fullscreen mode Exit fullscreen mode

The load-event race

This is the bug that makes people say “CDP is flaky”:

page.call("Page.navigate", {"url": url})
page.wait_for("Page.loadEventFired")   # too late on a fast page
Enter fullscreen mode Exit fullscreen mode

If the page finishes between those two lines, the event is gone and you wait until the timeout. Subscribe first, navigate second, then consume from the queue. The code above does that.

Runtime.evaluate is the workhorse

returnByValue: True asks Chrome to serialize the JavaScript result as JSON instead of returning a remote object reference. awaitPromise: True lets you evaluate an async expression and wait for it. If the expression throws, exceptionDetails is returned; turn that into a real exception in Python, including the JavaScript stack when available.

wait_for_truthy is a pragmatic poll, not a lifecycle model

Polling document.querySelector(...) every 250 ms is fine for “did the feed render?” It is not a substitute for event-driven waits when a precise event exists. Use polling for application state that Chrome does not expose as an event. Make the timeout explicit, and treat a None return as a failure the caller must handle.

User agent fixup

Headless Chrome reports HeadlessChrome/ in its user agent. Some sites use that token to serve a reduced UI. The fix is not to invent a fake UA from scratch; take the browser's native UA and replace only that token:

native = connection.call("Browser.getVersion")["userAgent"]
if "HeadlessChrome/" in native:
    connection.user_agent = native.replace("HeadlessChrome/", "Chrome/")
Enter fullscreen mode Exit fullscreen mode

Apply it with Emulation.setUserAgentOverride before the first navigation, so the first real request already sees the corrected value. Do not claim this defeats anti-bot systems. It removes one accidental signal.


Step 6: Extract structured data

Now the part everyone came for. Use a site that exists for scraping practice: books.toscrape.com.

The JavaScript runs inside the page and returns a plain JSON-able array. Keep the browser-side code small and boring: query, map, return.

(() => {
  const text = (node) => (node?.textContent || "").trim().replace(/\s+/g, " ");

  return [...document.querySelectorAll("article.product_pod")].map((card) => ({
    title: card.querySelector("h3 a")?.title ?? null,
    price: text(card.querySelector(".price_color")),
    in_stock: text(card.querySelector(".instock.availability")),
    url: card.querySelector("h3 a")?.href ?? null,
  }));
})();
Enter fullscreen mode Exit fullscreen mode

The Python side:

# src/chrome_cdp/demo.py

import json

from .browser import BrowserService
from .page import PageService

EXTRACT_BOOKS = """
(() => {
  const text = (node) => (node?.textContent || "").trim().replace(/\\s+/g, " ");
  return [...document.querySelectorAll("article.product_pod")].map((card) => ({
    title: card.querySelector("h3 a")?.title ?? null,
    price: text(card.querySelector(".price_color")),
    in_stock: text(card.querySelector(".instock.availability")),
    url: card.querySelector("h3 a")?.href ?? null,
  }));
})()
"""


def main() -> None:
    with BrowserService(headless=True) as browser:
        print("Chrome:", browser.version())
        page = PageService(
            browser.connection,
            "https://books.toscrape.com/",
        )
        try:
            page.wait_for_truthy("document.readyState === 'complete'")
            books = page.evaluate(EXTRACT_BOOKS)
            print(json.dumps(books[:5], indent=2))
            print(f"\nExtracted {len(books)} books")
        finally:
            page.close()


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Run it:

uv run cdp-demo
Enter fullscreen mode Exit fullscreen mode

Output:

Chrome: Chrome/157.0.8103.78
[
  {
    "title": "A Light in the Attic",
    "price": "£51.77",
    "in_stock": "In stock",
    "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
  },
  {
    "title": "Tipping the Velvet",
    "price": "£53.74",
    "in_stock": "In stock",
    "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"
  }
]

Extracted 20 books
Enter fullscreen mode Exit fullscreen mode

Rules for reliable extraction

  • Return JSON, not DOM nodes. returnByValue cannot serialize a DOM element usefully. Map to strings, numbers, booleans, arrays, and plain objects inside JavaScript.
  • Resolve URLs in the browser. a.href is already absolute; a.getAttribute("href") is not. Let the browser do that work.
  • Normalize whitespace. HTML is full of newlines and indentation. replace(/\s+/g, " ") makes the output stable.
  • Use optional chaining. A missing card should produce null, not a JavaScript exception that kills the whole batch.
  • Keep one expression per page shape. If the page changes, you want one obvious thing to fix.
  • Bound the result size. Runtime.evaluate with returnByValue marshals the whole result through the protocol. Extract 50 rows per call rather than 50,000.

Other useful CDP calls

Goal Call
Full rendered HTML Runtime.evaluate with document.documentElement.outerHTML
DOM tree with node IDs DOM.getDocument, then DOM.getOuterHTML
Screenshot Page.captureScreenshot (format: "png"), then base64-decode
Set a cookie Network.setCookie
Mobile viewport Emulation.setDeviceMetricsOverride
Isolated context Target.createBrowserContext
Attach to another tab Target.attachToTarget

A screenshot is useful when extracting a value fails and you want to see what the browser saw:

import base64

png = page.call("Page.captureScreenshot", {"format": "png"})
open("page.png", "wb").write(base64.b64decode(png["data"]))
Enter fullscreen mode Exit fullscreen mode

Step 7: The faster path — capture the JSON the page already fetched

DOM scraping is a last resort for modern applications. Often the data you want arrived as JSON, and the page used it to render. If you can capture that response, you skip brittle selectors and get the fields the site actually uses.

Network.enable is already on in PageService. Subscribe to Network.responseReceived before the action that triggers the request:

responses = page.connection.subscribe(
    "Network.responseReceived", page.session_id
)

page.navigate("https://example.com/search?q=widgets")

while True:
    event = responses.get(timeout=15)
    response = event["response"]
    if "/api/search" not in response["url"]:
        continue
    request_id = event["requestId"]
    break

body = page.call("Network.getResponseBody", {"requestId": request_id})
raw = body["body"]
if body.get("base64Encoded"):
    import base64
    raw = base64.b64decode(raw).decode()
data = json.loads(raw)
Enter fullscreen mode Exit fullscreen mode

The same pattern powers an infinite scroll:

  1. Navigate to the search page.
  2. Subscribe to network responses.
  3. Scroll the feed with Runtime.evaluate.
  4. Drain new responseReceived events.
  5. Filter for the API endpoint that returns the next batch.
  6. Call Network.getResponseBody, parse, append, deduplicate.

Two timing details matter:

  • Subscribe before the trigger. Events are not replayed for subscribers that arrive late.
  • Get the body while the page is alive. Chrome may evict it. If Network.getResponseBody fails because the response is not complete yet, wait for Network.loadingFinished for that requestId first, then call it.

This is usually faster, smaller, and more stable than reading rendered HTML. It is also more coupled to an internal API, so version your extractor and expect it to change.


Step 8: Attach to a Chrome that is already running

Launching a browser per job is not always what you want. Sometimes you have a Chrome window with a logged-in session, a debugging session, or a browser started by another process. CDP is designed for this.

Start Chrome with a dedicated profile and a fixed port:

google-chrome \
  --remote-debugging-port=9222 \
  --user-data-dir="$PWD/.chrome-profile" \
  about:blank
Enter fullscreen mode Exit fullscreen mode

Then ask Chrome for its browser endpoint:

curl -s http://127.0.0.1:9222/json/version
Enter fullscreen mode Exit fullscreen mode
{
  "Browser": "Chrome/157.0.8103.78",
  "Protocol-Version": "1.3",
  "webSocketDebuggerUrl": "ws://127.0.0.1:9222/devtools/browser/6f1c..."
}
Enter fullscreen mode Exit fullscreen mode

Connect to webSocketDebuggerUrl:

import json
import urllib.request

from chrome_cdp.connection import CDPConnection
from chrome_cdp.page import PageService

info = json.load(
    urllib.request.urlopen("http://127.0.0.1:9222/json/version")
)
connection = CDPConnection(info["webSocketDebuggerUrl"])
print(connection.call("Browser.getVersion")["product"])

page = PageService(connection, "https://books.toscrape.com/")
print(page.evaluate("document.title"))
page.close()
connection.close()
Enter fullscreen mode Exit fullscreen mode

That creates a new tab in the running browser. If you want to drive an existing tab, list targets through the browser connection:

targets = connection.call("Target.getTargets")["targetInfos"]
tab = next(t for t in targets if t["type"] == "page")
session_id = connection.call(
    "Target.attachToTarget",
    {"targetId": tab["targetId"], "flatten": True},
)["sessionId"]

for domain in ("Page", "Runtime", "Network"):
    connection.call(f"{domain}.enable", session_id=session_id)

title = connection.call(
    "Runtime.evaluate",
    {"expression": "document.title", "returnByValue": True},
    session_id=session_id,
)["result"]["value"]
print(title)
Enter fullscreen mode Exit fullscreen mode

Two cautions:

  • Do not close a tab you do not own. Use Target.detachFromTarget or close your connection. Target.closeTarget will close the user's tab.
  • Do not expose the debugging port. CDP can execute arbitrary JavaScript in every tab. Bind it to localhost, keep it behind a firewall, and use an SSH tunnel if you need remote access.

The Chrome 136 profile rule

Recent Chrome versions refuse to enable remote debugging when the browser uses its default user-data directory. This is a security change: an attacker should not be able to attach to your everyday, logged-in browser just because a debugging flag was set. Pass a non-default --user-data-dir for every automated or attachable instance. The browser will still work normally; it just will not use your personal profile.

If port 9222 is not listening, check this before you debug anything else.


Step 9: The production details that decide whether it works

The happy path is short. The difference between a demo and a scraper that survives a week is in the edges.

Lifecycle: terminate, wait, then kill

BrowserService.close() should be safe to call twice and should not leave an orphan process:

def close(self) -> None:
    if self.connection:
        try:
            self.connection.close()
        except Exception:
            pass
    if self.proc and self.proc.poll() is None:
        self.proc.terminate()
        try:
            self.proc.wait(5)
        except subprocess.TimeoutExpired:
            self.proc.kill()
            self.proc.wait()
    self._stderr.close()
    shutil.rmtree(self.profile, ignore_errors=True)
Enter fullscreen mode Exit fullscreen mode

Chrome can ignore SIGTERM while it is shutting down a renderer. Give it a few seconds, then escalate. Always reap the process; a zombie is still a process slot.

On Windows, the profile directory can stay locked for a moment after Chrome exits. Retry the deletion a handful of times before giving up. Do not fail the whole job because a temp directory outlived the browser.

One browser, many sessions

A browser process is expensive: hundreds of megabytes, many child processes, a warm network stack. If you run twenty workers, do not launch twenty browsers by default. Launch one and attach many sessions, each with its own target or browser context.

The trade-off is blast radius: one crash takes every session down. A shared-browser service should detect a dead connection and restart once, with a lock so ten workers do not all launch a replacement at the same time. BrowserService owns the process; a SharedBrowser wrapper owns the “ensure one is alive” policy.

For isolation, create a browser context per job:

context_id = connection.call(
    "Target.createBrowserContext", {"disposeOnDetach": True}
)["browserContextId"]

page = PageService(connection, context_id=context_id)
# ...
page.close()
connection.call(
    "Target.disposeBrowserContext", {"browserContextId": context_id}
)
Enter fullscreen mode Exit fullscreen mode

Contexts behave like separate incognito profiles: cookies, cache, and storage do not leak between jobs.

Proxies

CDP splits proxy configuration in two:

  • the proxy server is set when creating a browser context (proxyServer);
  • the proxy credentials are supplied in response to Fetch.authRequired events.

Install a Fetch interceptor on the tab, continue normal requests, and answer the first auth challenge for each requestId with Fetch.continueWithAuth. If the same request is challenged again, treat it as a rejection: cancel and mark the credential as refused. This gives you a clear “proxy refused auth” state instead of an endless retry loop.

Headless is not a separate browser

--headless=new is a mode of the same browser, but some sites still branch on the user agent or on missing window APIs. Test the exact mode you will run in production. If a page behaves differently in headless mode, compare the user agent and the viewport first, then the feature detection.

Timeouts should be layered

  • Socket connect and handshake: seconds.
  • CDP command reply: 10–30 seconds.
  • Page load: up to 45 seconds.
  • Whole job: a budget enforced by the caller.

A single global 30-second timeout is either too short for a slow page or too long for a dead browser. Make the layers explicit and log which one fired.

DevTools is remote code execution

Treat an open debugging port like a shell. A local process that can reach it can evaluate JavaScript in every tab, read cookies, and navigate anywhere. Do not expose it on 0.0.0.0. Do not ship it in a container image without a network policy. When you are done, close the connection and the process.


When you should use Playwright instead

Build the raw CDP layer when:

  • you need to attach to an existing Chrome instance;
  • you want precise control over network events and response bodies;
  • you need a tiny runtime footprint or cannot add a browser-automation dependency;
  • you are debugging why an automation framework is slow or flaky;
  • you want to understand the protocol that your tools sit on.

Use Playwright when:

  • you need cross-browser support;
  • you want automatic waiting, selectors, tracing, video, and test runner integration;
  • your team is more productive with a high-level API;
  • you are building a product, not a protocol experiment.

The two are not enemies. Playwright is a well-engineered CDP client with a lot of extra machinery. Knowing what is underneath makes you better at using it.


The checklist

If you build this yourself, the failure modes fit on one page:

  • [ ] Find Chrome explicitly; support an override path.
  • [ ] Launch with --remote-debugging-port=0 and a non-default --user-data-dir.
  • [ ] Poll DevToolsActivePort; do not parse logs.
  • [ ] Send stderr to a file, not an unread pipe.
  • [ ] Run one reader thread; route replies by id and events by (sessionId, method).
  • [ ] Wake every waiter when the socket dies.
  • [ ] Put timeouts on commands, not on the idle socket.
  • [ ] Attach with flatten: true and enable the domains you use.
  • [ ] Subscribe to Page.loadEventFired before navigating.
  • [ ] Evaluate with returnByValue and handle exceptionDetails.
  • [ ] Subscribe to network events before the action that triggers them.
  • [ ] Close targets, contexts, connections, and the process.
  • [ ] Never expose the debugging port.

Once the service layer is in place, the scraping code becomes ordinary Python: call a method, get a dict, validate it, store it. That is the point. The browser is a dependency, not the architecture.

The next useful step is to make the extractor survive change: run the same job against a fixture page, assert the shape of the output, and fail loudly when a selector disappears. The browser will still be there in the morning. Your assumptions might not be.


If you build this, the two files worth reading twice are connection.py and page.py. One is about not hanging. The other is about not racing. Everything else is plumbing.


Top comments (0)