Launch Chrome, attach to CDP, and extract structured data with a zero-dependency Python service
Most Python browser automation tutorials begin with pip install playwright and end with page.click(). That is a good default. Playwright is mature, it handles waits, and it lets you forget that a browser is a separate operating-system process.
But sometimes you need to see the layer underneath. Maybe you are debugging a scraper that hangs once every thousand pages. Maybe you need the JSON response behind an infinite scroll, not the DOM that happens to be rendered. Maybe you want to attach to a Chrome instance that is already running with a logged-in profile. Or maybe you just want to stop treating one of the most complicated programs on your machine as a black box.
This tutorial builds that layer from scratch.
You will launch Chrome yourself, discover the port it chose, open one WebSocket, multiplex commands and events over it, attach to a tab, navigate, evaluate JavaScript, and turn the result into structured Python data. The runtime dependency list is empty: Python's standard library is enough.
The project uses uv and a small service layer, so each file owns one boundary instead of one Browser class owning the process, the socket, the protocol, and your scraping logic at the same time.
What you need: Python 3.12+,
uv, and Chrome, Chromium, Edge, or Brave installed. A Linux server without a desktop is fine; use headless mode.
What the Chrome DevTools Protocol actually is
The Chrome DevTools Protocol (CDP) is the interface Chrome exposes for tools like DevTools, Puppeteer, and Playwright. It is a set of commands and events grouped into domains:
| Domain | What it is for |
|---|---|
Target |
Browser contexts, tabs, and sessions |
Page |
Navigation, lifecycle events, screenshots |
Runtime |
Evaluating JavaScript in a page |
Network |
Requests, responses, cookies, response bodies |
Emulation |
User agent, viewport, device metrics |
Fetch |
Intercepting requests, including proxy authentication |
The transport is a WebSocket carrying JSON messages. A command looks like this:
{
"id": 1,
"method": "Page.navigate",
"params": { "url": "https://example.com" }
}
A reply carries the same id:
{ "id": 1, "result": { "frameId": "ABC", "loaderId": "DEF" } }
An event has no id and arrives whenever Chrome feels like it:
{ "method": "Page.loadEventFired", "params": { "timestamp": 1234.5 } }
When you attach to a tab, Chrome adds a sessionId to both directions, so one socket can carry traffic for many tabs.
That is almost the whole mental model. The complexity is in the lifecycle and the races:
- Chrome is a child process that can crash, ignore signals, or start slowly.
- The debugging port is not always the one you asked for.
- A fast page can finish loading before you subscribe to the load event.
- A dead socket must wake every caller that is waiting for a reply, or your worker hangs forever.
- Headless Chrome sometimes gets a different page than the one you expected.
The service layer exists to make those problems explicit.
The shape of the service
Create a fresh project. We are deliberately not copying a large application layout; this is the smallest set of boundaries that stays testable.
chrome-cdp/
├── pyproject.toml
└── src/chrome_cdp/
├── __init__.py
├── browser.py # find, launch, and own the Chrome process
├── connection.py # multiplex CDP commands and events
├── page.py # attach to a tab, navigate, evaluate
├── transport.py # RFC 6455 WebSocket, text frames only
├── errors.py # ChromeError, CDPError
└── demo.py # extraction example
Each service has one job:
-
BrowserServiceowns the operating-system process and its temporary profile. -
CDPConnectionowns the protocol: request IDs, replies, events, timeouts. -
PageServiceowns one tab session and the operations you actually want. -
WebSocketowns bytes on the wire, not browser concepts. -
errors.pygives every layer the same failure vocabulary.
That separation matters when something breaks. A timeout is a CDP problem. A failed launch is a process problem. A missing selector is a page problem. Do not make all three look like RuntimeError: something went wrong.
Step 0: Create the project with uv
uv init --package chrome-cdp
cd chrome-cdp
uv add --dev pytest
uv init --package creates a src/chrome_cdp/ layout and a build configuration. Replace the generated pyproject.toml with this:
[project]
name = "chrome-cdp"
version = "0.1.0"
description = "Launch Chrome and talk to it over the DevTools Protocol"
readme = "README.md"
requires-python = ">=3.12"
dependencies = []
[project.scripts]
cdp-demo = "chrome_cdp.demo:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/chrome_cdp"]
There are no runtime dependencies. pytest is the only development dependency. Run:
uv sync
If you would rather not hand-roll the WebSocket layer, add uv add websockets and use its synchronous client. The CDP multiplexing ideas in this article still apply. The zero-dependency version is included here because it makes the protocol visible and because it is small enough to audit.
Step 1: Find a Chrome binary
Do not hard-code /usr/bin/google-chrome. People run Chrome, Chromium, Edge, and Brave, on macOS, Linux, Windows, and inside containers. A locator service keeps that decision in one place.
# src/chrome_cdp/browser.py
import os
import platform
import shutil
CANDIDATES = {
"Darwin": [
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
"/Applications/Chromium.app/Contents/MacOS/Chromium",
"/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge",
"/Applications/Brave Browser.app/Contents/MacOS/Brave Browser",
],
"Linux": [
"google-chrome",
"google-chrome-stable",
"chromium",
"chromium-browser",
"microsoft-edge",
"brave-browser",
],
"Windows": [
r"%ProgramFiles%\Google\Chrome\Application\chrome.exe",
r"%ProgramFiles(x86)%\Google\Chrome\Application\chrome.exe",
r"%ProgramFiles%\Microsoft\Edge\Application\msedge.exe",
r"%ProgramFiles%\BraveSoftware\Brave-Browser\Application\brave.exe",
],
}
def find_chrome() -> str:
for candidate in CANDIDATES.get(platform.system(), []):
candidate = os.path.expandvars(candidate)
path = candidate if os.sep in candidate else (shutil.which(candidate) or "")
if path and os.path.isfile(path):
return path
raise ChromeError(
"No Chrome/Chromium found. Install Google Chrome, or pass its path."
)
Two practical notes:
- Let the caller override the path.
BrowserService(path="/opt/chrome/chrome")is how you support a downloaded Chrome for Testing binary, a container image, or a corporate install. - On Ubuntu, skip the snap build of Chromium. Its private
/tmpmeans theDevToolsActivePortfile lands somewhere your process cannot see. Use the.debpackage, Google Chrome, or Chrome for Testing instead.
For a server, install Chrome once or download Chrome for Testing and verify the archive's SHA-256 before running it. A browser binary is executable code; treat it like one.
Step 2: Launch Chrome and discover its port
Here is the launch sequence:
- Choose an isolated
--user-data-dir. - Ask Chrome to choose a free debugging port with
--remote-debugging-port=0. - Start the process with
about:blank. - Poll for the
DevToolsActivePortfile in the profile directory. - Read the port and the browser-level WebSocket path from that file.
That file is the reliable handshake. It is written by Chrome once the DevTools endpoint is ready, and it contains two whitespace-separated tokens:
51234
/devtools/browser/6f1c...
The first token is the port. The second is the path of the browser endpoint. Do not scrape stderr for a URL, and do not assume port 9222.
# src/chrome_cdp/browser.py
import os
import shutil
import subprocess
import tempfile
import time
from pathlib import Path
from .connection import CDPConnection
from .errors import ChromeError
class BrowserService:
def __init__(
self,
path: str | None = None,
headless: bool = True,
timeout: float = 30.0,
) -> None:
self.path = path or find_chrome()
self.timeout = timeout
self.profile = tempfile.mkdtemp(prefix="chrome-cdp-")
self.proc: subprocess.Popen | None = None
self.connection: CDPConnection | None = None
# A file, not a PIPE: a full stderr pipe can block Chrome.
self._stderr = Path(self.profile, "chrome-stderr.log").open("w+b")
try:
args = self._args(headless)
port, ws_path = self._launch(args)
self.connection = CDPConnection(
f"ws://127.0.0.1:{port}{ws_path}", timeout=timeout
)
self._fix_user_agent()
except Exception:
self.close()
raise
def _args(self, headless: bool) -> list[str]:
args = [
self.path,
"--remote-debugging-port=0",
f"--user-data-dir={self.profile}",
"--no-first-run",
"--no-default-browser-check",
"--disable-extensions",
"--disable-background-networking",
"--disable-dev-shm-usage",
"--disable-blink-features=AutomationControlled",
"--window-size=1280,900",
]
if headless:
args.append("--headless=new")
if hasattr(os, "geteuid") and os.geteuid() == 0:
# Chrome refuses to run as root with its sandbox.
args.append("--no-sandbox")
return args
def _launch(self, args: list[str]) -> tuple[str, str]:
port_file = Path(self.profile, "DevToolsActivePort")
port_file.unlink(missing_ok=True)
self.proc = subprocess.Popen(
args + ["about:blank"],
stdout=subprocess.DEVNULL,
stderr=self._stderr,
stdin=subprocess.DEVNULL,
)
deadline = time.monotonic() + self.timeout
while time.monotonic() < deadline:
if self.proc.poll() is not None:
self._stderr.seek(0)
raise ChromeError(
f"Chrome exited with {self.proc.returncode}: "
f"{self._stderr.read().decode(errors='replace')[-2000:]}"
)
try:
port, ws_path = port_file.read_text().split()
return port, ws_path
except (OSError, ValueError):
time.sleep(0.1)
raise ChromeError(
f"Chrome published no DevTools port within {self.timeout}s"
)
Why these flags
-
--remote-debugging-port=0asks the operating system for a free port. This avoids collisions when several workers launch browsers at once. -
--user-data-dir=...gives this run its own cookies, cache, and history. It also keeps the browser from touching your real profile. -
Chrome 136 and newer changed the rules. Remote debugging is ignored when Chrome uses its default profile directory. If you see “port 9222 refused” after an upgrade, the missing
--user-data-diris usually the reason. Always pass one. -
--headless=newis the current headless mode. The old headless implementation is gone from modern Chrome. -
--disable-dev-shm-usageavoids/dev/shmexhaustion in containers. -
--no-sandboxis not a default. Only use it when you are root in a container or when your environment cannot provide a sandbox. It weakens process isolation. -
--disable-blink-features=AutomationControlledremoves a common automation signal. It does not make you invisible. Treat anti-bot work as a separate problem.
Why stderr goes to a file
If Chrome writes a lot of diagnostics and nobody reads the pipe, the pipe buffer fills and Chrome blocks. A file also lets you show the last 2,000 bytes when startup fails, which is usually enough to identify a missing shared library or a profile lock.
Startup failure should name the step
Wrap each phase in BrowserService with a step label. Instead of OSError, report Could not start Chrome while connecting to Chrome's DevTools port. When a scraper fails at 3 a.m., the difference between “failed” and “failed here” is the difference between a five-minute fix and an hour of guessing.
Step 3: A WebSocket client you can read in one sitting
CDP only needs text frames. It does not need extensions, compression, or wss:// for a local browser. That means a small RFC 6455 client is practical.
The handshake is an HTTP GET with Upgrade: websocket. The server replies 101 Switching Protocols and proves it understood the key by sending:
Sec-WebSocket-Accept: base64(SHA1(client_key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"))
After that, each frame has an opcode, a length, and (for clients) a mask. Text frames use opcode 0x1; ping and pong are 0x9 and 0xA; close is 0x8.
# src/chrome_cdp/transport.py
import base64
import hashlib
import os
import socket
import struct
import threading
import urllib.parse
GUID = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"
class WebSocketError(OSError):
pass
class WebSocketClosed(WebSocketError):
pass
class WebSocket:
def __init__(self, url: str, timeout: float = 10.0) -> None:
parsed = urllib.parse.urlparse(url)
if parsed.scheme != "ws" or not parsed.hostname:
raise WebSocketError(f"expected ws://host..., got {url!r}")
self.sock = socket.create_connection(
(parsed.hostname, parsed.port or 80), timeout=timeout
)
self.sock.settimeout(timeout)
self._send_lock = threading.Lock()
self._buffer = b""
self._handshake(parsed)
def _handshake(self, parsed: urllib.parse.ParseResult) -> None:
key = base64.b64encode(os.urandom(16)).decode()
path = parsed.path or "/"
if parsed.query:
path += f"?{parsed.query}"
host = parsed.hostname or "127.0.0.1"
port = parsed.port or 80
self.sock.sendall(
(
f"GET {path} HTTP/1.1\r\n"
f"Host: {host}:{port}\r\n"
"Upgrade: websocket\r\n"
"Connection: Upgrade\r\n"
f"Sec-WebSocket-Key: {key}\r\n"
"Sec-WebSocket-Version: 13\r\n\r\n"
).encode()
)
head = b""
while b"\r\n\r\n" not in head:
chunk = self.sock.recv(4096)
if not chunk:
raise WebSocketClosed("closed during the handshake")
head += chunk
head, self._buffer = head.split(b"\r\n\r\n", 1)
status, _, headers = head.decode("latin-1").partition("\r\n")
if " 101 " not in status:
raise WebSocketError(f"upgrade refused: {status}")
expected = base64.b64encode(
hashlib.sha1((key + GUID).encode()).digest()
).decode()
accepts = [
value.strip()
for name, _, value in (
line.partition(":") for line in headers.split("\r\n")
)
if name.lower() == "sec-websocket-accept"
]
if accepts != [expected]:
raise WebSocketError("bad Sec-WebSocket-Accept")
# The reader thread may block indefinitely while the browser is quiet.
# Per-command timeouts live in CDPConnection instead.
self.sock.settimeout(None)
def _read_exact(self, size: int) -> bytes:
while len(self._buffer) < size:
chunk = self.sock.recv(max(65536, size - len(self._buffer)))
if not chunk:
raise WebSocketClosed("socket closed")
self._buffer += chunk
head, self._buffer = self._buffer[:size], self._buffer[size:]
return head
def _send_frame(self, opcode: int, payload: bytes) -> None:
mask = os.urandom(4)
header = bytes([0x80 | opcode])
length = len(payload)
if length < 126:
header += bytes([0x80 | length])
elif length < 65536:
header += bytes([0x80 | 126]) + struct.pack(">H", length)
else:
header += bytes([0x80 | 127]) + struct.pack(">Q", length)
masked = bytes(byte ^ mask[i & 3] for i, byte in enumerate(payload))
with self._send_lock:
self.sock.sendall(header + mask + masked)
def send(self, text: str) -> None:
self._send_frame(0x1, text.encode())
def recv(self) -> str:
parts: list[bytes] = []
started = False
while True:
first, second = self._read_exact(2)
fin = first & 0x80
opcode = first & 0x0F
length = second & 0x7F
if length == 126:
length = struct.unpack(">H", self._read_exact(2))[0]
elif length == 127:
length = struct.unpack(">Q", self._read_exact(8))[0]
mask = self._read_exact(4) if second & 0x80 else None
data = self._read_exact(length)
if mask:
data = bytes(byte ^ mask[i & 3] for i, byte in enumerate(data))
if opcode == 0x8:
raise WebSocketClosed("closed by the peer")
if opcode == 0x9: # ping -> pong
self._send_frame(0xA, data)
continue
if opcode == 0xA: # pong
continue
if opcode in (0x1, 0x2):
parts, started = [data], True
elif opcode == 0x0 and started:
parts.append(data)
else:
raise WebSocketError(f"unexpected opcode {opcode}")
if fin:
return b"".join(parts).decode("utf-8")
def close(self) -> None:
try:
self._send_frame(0x8, struct.pack(">H", 1000))
except OSError:
pass
try:
self.sock.close()
except OSError:
pass
This is intentionally minimal. It handles masked client frames, fragmented messages, ping/pong, and clean closes. It does not do compression, TLS, extensions, or subprotocol negotiation. CDP over ws://127.0.0.1 needs none of those.
If you keep the reader on one thread and set a socket timeout, be careful: after the handshake the socket timeout is cleared. Otherwise a quiet browser would look like a dead one. Timeouts belong at the command layer, not the byte layer.
Step 4: Multiplex commands and events over one socket
Here is the problem with a naive client:
send Page.navigate
wait for reply
While you wait, Chrome may send Page.loadEventFired, Network.requestWillBeSent, Network.responseReceived, and a dozen other events. If your loop expects the next message to be the reply to your command, it will misread an event as a reply and deadlock.
A real CDP client needs two routes:
- replies routed by
idto the caller waiting on that command; - events routed by
(sessionId, method)to whichever subscriber asked for them.
One background reader thread handles both.
# src/chrome_cdp/connection.py
import json
import queue
import threading
from typing import Any
from .errors import CDPError
from .transport import WebSocket
DEAD = object()
class CDPConnection:
def __init__(self, ws_url: str, timeout: float = 30.0) -> None:
self.ws = WebSocket(ws_url, timeout=timeout)
self.timeout = timeout
self.closed: Exception | None = None
self.user_agent: str | None = None
self._next_id = 0
self._id_lock = threading.Lock()
self._pending: dict[int, queue.Queue[dict]] = {}
self._subs: dict[tuple[str | None, str], list[queue.Queue[Any]]] = {}
self._subs_lock = threading.Lock()
self._reader = threading.Thread(
target=self._read_loop, name="cdp-reader", daemon=True
)
self._reader.start()
def _read_loop(self) -> None:
try:
while True:
message = json.loads(self.ws.recv())
if "id" in message:
waiter = self._pending.pop(message["id"], None)
if waiter:
waiter.put(message)
continue
key = (message.get("sessionId"), message.get("method", ""))
with self._subs_lock:
listeners = tuple(self._subs.get(key, ()))
for listener in listeners:
listener.put(message.get("params", {}))
except Exception as error:
self.closed = error
reply = {"error": {"message": f"connection lost: {error}"}}
for waiter in list(self._pending.values()):
waiter.put(reply)
with self._subs_lock:
everyone = [q for qs in self._subs.values() for q in qs]
for listener in everyone:
listener.put(DEAD)
def call(
self,
method: str,
params: dict | None = None,
session_id: str | None = None,
timeout: float | None = None,
) -> dict:
if self.closed:
raise CDPError(f"{method}: connection lost: {self.closed}")
with self._id_lock:
self._next_id += 1
message_id = self._next_id
request = {"id": message_id, "method": method, "params": params or {}}
if session_id:
request["sessionId"] = session_id
waiter: queue.Queue[dict] = queue.Queue(1)
self._pending[message_id] = waiter
try:
self.ws.send(json.dumps(request))
except OSError as error:
self._pending.pop(message_id, None)
raise CDPError(f"{method}: send failed: {error}") from error
try:
reply = waiter.get(timeout=timeout or self.timeout)
except queue.Empty:
self._pending.pop(message_id, None)
raise CDPError(
f"{method}: no reply within {timeout or self.timeout}s"
)
if "error" in reply:
raise CDPError(f"{method}: {reply['error'].get('message')}")
return reply.get("result", {})
def subscribe(self, method: str, session_id: str | None = None) -> queue.Queue:
q: queue.Queue[Any] = queue.Queue()
with self._subs_lock:
self._subs.setdefault((session_id, method), []).append(q)
return q
def unsubscribe(
self, method: str, q: queue.Queue, session_id: str | None = None
) -> None:
with self._subs_lock:
listeners = self._subs.get((session_id, method), [])
if q in listeners:
listeners.remove(q)
def wait_for(
self,
method: str,
session_id: str | None = None,
timeout: float | None = None,
q: queue.Queue | None = None,
) -> dict:
own = q is None
q = q or self.subscribe(method, session_id)
try:
event = q.get(timeout=timeout or self.timeout)
except queue.Empty:
raise CDPError(
f"no {method} within {timeout or self.timeout}s"
)
finally:
if own:
self.unsubscribe(method, q, session_id)
if event is DEAD:
raise CDPError(f"{method}: connection lost")
return event
def close(self) -> None:
self.ws.close()
The parts worth stealing even if you rewrite everything else:
One queue per pending command
self._pending[message_id] is a Queue(1). The reader puts exactly one reply into it. The caller blocks on get() with a timeout. There is no global condition variable and no chance of one reply waking the wrong caller.
Event subscriptions are queues, not callbacks
Callbacks run on the reader thread. If a callback blocks, every command and event stops. Queues let the consumer decide when to process the event, and they make tests trivial: push a dict, assert what the consumer did.
The death sentinel
When the reader dies, every pending call gets an error reply and every subscriber gets DEAD. This is the difference between a clean failure and a worker that waits forever after Chrome crashes. Never rely on the operating system to close the socket promptly; it may not.
Timeouts at the call layer
The socket blocks indefinitely after the handshake. Each call() gets a timeout. A page can take 40 seconds to load without making the whole connection look dead. If a command times out, remove its pending entry so a late reply does not leak a queue.
Thread safety
Incrementing the ID is guarded by a lock. The pending dictionary is only touched by the caller that owns the ID and by the reader, and the Queue makes that handoff safe. Multiple worker threads can call CDPConnection.call() concurrently; command IDs keep the replies separate.
One thing this minimal version does not do: validate that a CDP reply's sessionId matches the command. If you care about a malicious or buggy browser, add that check. For local Chrome, it is not the first problem you will hit.
Step 5: Attach to a tab
A browser-level CDP connection can create tabs and manage contexts, but it cannot evaluate JavaScript in a page directly. You need a target and a session:
-
Target.createTargetcreates a tab and returns atargetId. -
Target.attachToTargetwithflatten: truereturns asessionId. - Enable the domains you need on that session.
- Send page commands with the
sessionId.
flatten: true is important. Without it, Chrome wants a separate WebSocket connection for each target. Flattening lets you multiplex every session over the browser connection, which is exactly what a worker pool wants.
# src/chrome_cdp/page.py
import time
from typing import Any
from .connection import CDPConnection
from .errors import CDPError
class PageService:
def __init__(
self,
connection: CDPConnection,
url: str = "about:blank",
context_id: str | None = None,
) -> None:
self.connection = connection
params: dict = {"url": "about:blank"}
if context_id:
params["browserContextId"] = context_id
self.target_id = connection.call("Target.createTarget", params)["targetId"]
self.session_id = connection.call(
"Target.attachToTarget",
{"targetId": self.target_id, "flatten": True},
)["sessionId"]
for domain in ("Page", "Runtime", "Network"):
self.call(f"{domain}.enable")
if connection.user_agent:
self.call(
"Emulation.setUserAgentOverride",
{"userAgent": connection.user_agent},
)
if url != "about:blank":
self.navigate(url)
def call(
self,
method: str,
params: dict | None = None,
timeout: float | None = None,
) -> dict:
return self.connection.call(method, params, self.session_id, timeout)
def navigate(self, url: str, timeout: float = 45.0) -> None:
# Subscribe BEFORE navigating: a fast page can fire the load event
# between Page.navigate returning and a later subscribe.
loaded = self.connection.subscribe("Page.loadEventFired", self.session_id)
try:
result = self.call("Page.navigate", {"url": url}, timeout=timeout)
if result.get("errorText"):
raise CDPError(f"navigate {url}: {result['errorText']}")
self.connection.wait_for(
"Page.loadEventFired", self.session_id, timeout, q=loaded
)
finally:
self.connection.unsubscribe(
"Page.loadEventFired", loaded, self.session_id
)
def evaluate(self, expression: str, timeout: float | None = None) -> Any:
result = self.call(
"Runtime.evaluate",
{
"expression": expression,
"returnByValue": True,
"awaitPromise": True,
},
timeout,
)
if "exceptionDetails" in result:
detail = result["exceptionDetails"]
message = (
detail.get("exception", {}).get("description")
or detail.get("text")
)
raise CDPError(f"JavaScript: {message}")
return result.get("result", {}).get("value")
def wait_for_truthy(
self, expression: str, timeout: float = 15.0, interval: float = 0.25
) -> Any:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
value = self.evaluate(expression)
except CDPError:
if self.connection.closed:
raise
value = None
if value:
return value
time.sleep(interval)
return None
def close(self) -> None:
try:
self.connection.call(
"Target.closeTarget", {"targetId": self.target_id}
)
except (CDPError, OSError):
pass
The load-event race
This is the bug that makes people say “CDP is flaky”:
page.call("Page.navigate", {"url": url})
page.wait_for("Page.loadEventFired") # too late on a fast page
If the page finishes between those two lines, the event is gone and you wait until the timeout. Subscribe first, navigate second, then consume from the queue. The code above does that.
Runtime.evaluate is the workhorse
returnByValue: True asks Chrome to serialize the JavaScript result as JSON instead of returning a remote object reference. awaitPromise: True lets you evaluate an async expression and wait for it. If the expression throws, exceptionDetails is returned; turn that into a real exception in Python, including the JavaScript stack when available.
wait_for_truthy is a pragmatic poll, not a lifecycle model
Polling document.querySelector(...) every 250 ms is fine for “did the feed render?” It is not a substitute for event-driven waits when a precise event exists. Use polling for application state that Chrome does not expose as an event. Make the timeout explicit, and treat a None return as a failure the caller must handle.
User agent fixup
Headless Chrome reports HeadlessChrome/ in its user agent. Some sites use that token to serve a reduced UI. The fix is not to invent a fake UA from scratch; take the browser's native UA and replace only that token:
native = connection.call("Browser.getVersion")["userAgent"]
if "HeadlessChrome/" in native:
connection.user_agent = native.replace("HeadlessChrome/", "Chrome/")
Apply it with Emulation.setUserAgentOverride before the first navigation, so the first real request already sees the corrected value. Do not claim this defeats anti-bot systems. It removes one accidental signal.
Step 6: Extract structured data
Now the part everyone came for. Use a site that exists for scraping practice: books.toscrape.com.
The JavaScript runs inside the page and returns a plain JSON-able array. Keep the browser-side code small and boring: query, map, return.
(() => {
const text = (node) => (node?.textContent || "").trim().replace(/\s+/g, " ");
return [...document.querySelectorAll("article.product_pod")].map((card) => ({
title: card.querySelector("h3 a")?.title ?? null,
price: text(card.querySelector(".price_color")),
in_stock: text(card.querySelector(".instock.availability")),
url: card.querySelector("h3 a")?.href ?? null,
}));
})();
The Python side:
# src/chrome_cdp/demo.py
import json
from .browser import BrowserService
from .page import PageService
EXTRACT_BOOKS = """
(() => {
const text = (node) => (node?.textContent || "").trim().replace(/\\s+/g, " ");
return [...document.querySelectorAll("article.product_pod")].map((card) => ({
title: card.querySelector("h3 a")?.title ?? null,
price: text(card.querySelector(".price_color")),
in_stock: text(card.querySelector(".instock.availability")),
url: card.querySelector("h3 a")?.href ?? null,
}));
})()
"""
def main() -> None:
with BrowserService(headless=True) as browser:
print("Chrome:", browser.version())
page = PageService(
browser.connection,
"https://books.toscrape.com/",
)
try:
page.wait_for_truthy("document.readyState === 'complete'")
books = page.evaluate(EXTRACT_BOOKS)
print(json.dumps(books[:5], indent=2))
print(f"\nExtracted {len(books)} books")
finally:
page.close()
if __name__ == "__main__":
main()
Run it:
uv run cdp-demo
Output:
Chrome: Chrome/157.0.8103.78
[
{
"title": "A Light in the Attic",
"price": "£51.77",
"in_stock": "In stock",
"url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
},
{
"title": "Tipping the Velvet",
"price": "£53.74",
"in_stock": "In stock",
"url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"
}
]
Extracted 20 books
Rules for reliable extraction
-
Return JSON, not DOM nodes.
returnByValuecannot serialize a DOM element usefully. Map to strings, numbers, booleans, arrays, and plain objects inside JavaScript. -
Resolve URLs in the browser.
a.hrefis already absolute;a.getAttribute("href")is not. Let the browser do that work. -
Normalize whitespace. HTML is full of newlines and indentation.
replace(/\s+/g, " ")makes the output stable. -
Use optional chaining. A missing card should produce
null, not a JavaScript exception that kills the whole batch. - Keep one expression per page shape. If the page changes, you want one obvious thing to fix.
-
Bound the result size.
Runtime.evaluatewithreturnByValuemarshals the whole result through the protocol. Extract 50 rows per call rather than 50,000.
Other useful CDP calls
| Goal | Call |
|---|---|
| Full rendered HTML |
Runtime.evaluate with document.documentElement.outerHTML
|
| DOM tree with node IDs |
DOM.getDocument, then DOM.getOuterHTML
|
| Screenshot |
Page.captureScreenshot (format: "png"), then base64-decode |
| Set a cookie | Network.setCookie |
| Mobile viewport | Emulation.setDeviceMetricsOverride |
| Isolated context | Target.createBrowserContext |
| Attach to another tab | Target.attachToTarget |
A screenshot is useful when extracting a value fails and you want to see what the browser saw:
import base64
png = page.call("Page.captureScreenshot", {"format": "png"})
open("page.png", "wb").write(base64.b64decode(png["data"]))
Step 7: The faster path — capture the JSON the page already fetched
DOM scraping is a last resort for modern applications. Often the data you want arrived as JSON, and the page used it to render. If you can capture that response, you skip brittle selectors and get the fields the site actually uses.
Network.enable is already on in PageService. Subscribe to Network.responseReceived before the action that triggers the request:
responses = page.connection.subscribe(
"Network.responseReceived", page.session_id
)
page.navigate("https://example.com/search?q=widgets")
while True:
event = responses.get(timeout=15)
response = event["response"]
if "/api/search" not in response["url"]:
continue
request_id = event["requestId"]
break
body = page.call("Network.getResponseBody", {"requestId": request_id})
raw = body["body"]
if body.get("base64Encoded"):
import base64
raw = base64.b64decode(raw).decode()
data = json.loads(raw)
The same pattern powers an infinite scroll:
- Navigate to the search page.
- Subscribe to network responses.
- Scroll the feed with
Runtime.evaluate. - Drain new
responseReceivedevents. - Filter for the API endpoint that returns the next batch.
- Call
Network.getResponseBody, parse, append, deduplicate.
Two timing details matter:
- Subscribe before the trigger. Events are not replayed for subscribers that arrive late.
-
Get the body while the page is alive. Chrome may evict it. If
Network.getResponseBodyfails because the response is not complete yet, wait forNetwork.loadingFinishedfor thatrequestIdfirst, then call it.
This is usually faster, smaller, and more stable than reading rendered HTML. It is also more coupled to an internal API, so version your extractor and expect it to change.
Step 8: Attach to a Chrome that is already running
Launching a browser per job is not always what you want. Sometimes you have a Chrome window with a logged-in session, a debugging session, or a browser started by another process. CDP is designed for this.
Start Chrome with a dedicated profile and a fixed port:
google-chrome \
--remote-debugging-port=9222 \
--user-data-dir="$PWD/.chrome-profile" \
about:blank
Then ask Chrome for its browser endpoint:
curl -s http://127.0.0.1:9222/json/version
{
"Browser": "Chrome/157.0.8103.78",
"Protocol-Version": "1.3",
"webSocketDebuggerUrl": "ws://127.0.0.1:9222/devtools/browser/6f1c..."
}
Connect to webSocketDebuggerUrl:
import json
import urllib.request
from chrome_cdp.connection import CDPConnection
from chrome_cdp.page import PageService
info = json.load(
urllib.request.urlopen("http://127.0.0.1:9222/json/version")
)
connection = CDPConnection(info["webSocketDebuggerUrl"])
print(connection.call("Browser.getVersion")["product"])
page = PageService(connection, "https://books.toscrape.com/")
print(page.evaluate("document.title"))
page.close()
connection.close()
That creates a new tab in the running browser. If you want to drive an existing tab, list targets through the browser connection:
targets = connection.call("Target.getTargets")["targetInfos"]
tab = next(t for t in targets if t["type"] == "page")
session_id = connection.call(
"Target.attachToTarget",
{"targetId": tab["targetId"], "flatten": True},
)["sessionId"]
for domain in ("Page", "Runtime", "Network"):
connection.call(f"{domain}.enable", session_id=session_id)
title = connection.call(
"Runtime.evaluate",
{"expression": "document.title", "returnByValue": True},
session_id=session_id,
)["result"]["value"]
print(title)
Two cautions:
-
Do not close a tab you do not own. Use
Target.detachFromTargetor close your connection.Target.closeTargetwill close the user's tab. - Do not expose the debugging port. CDP can execute arbitrary JavaScript in every tab. Bind it to localhost, keep it behind a firewall, and use an SSH tunnel if you need remote access.
The Chrome 136 profile rule
Recent Chrome versions refuse to enable remote debugging when the browser uses its default user-data directory. This is a security change: an attacker should not be able to attach to your everyday, logged-in browser just because a debugging flag was set. Pass a non-default --user-data-dir for every automated or attachable instance. The browser will still work normally; it just will not use your personal profile.
If port 9222 is not listening, check this before you debug anything else.
Step 9: The production details that decide whether it works
The happy path is short. The difference between a demo and a scraper that survives a week is in the edges.
Lifecycle: terminate, wait, then kill
BrowserService.close() should be safe to call twice and should not leave an orphan process:
def close(self) -> None:
if self.connection:
try:
self.connection.close()
except Exception:
pass
if self.proc and self.proc.poll() is None:
self.proc.terminate()
try:
self.proc.wait(5)
except subprocess.TimeoutExpired:
self.proc.kill()
self.proc.wait()
self._stderr.close()
shutil.rmtree(self.profile, ignore_errors=True)
Chrome can ignore SIGTERM while it is shutting down a renderer. Give it a few seconds, then escalate. Always reap the process; a zombie is still a process slot.
On Windows, the profile directory can stay locked for a moment after Chrome exits. Retry the deletion a handful of times before giving up. Do not fail the whole job because a temp directory outlived the browser.
One browser, many sessions
A browser process is expensive: hundreds of megabytes, many child processes, a warm network stack. If you run twenty workers, do not launch twenty browsers by default. Launch one and attach many sessions, each with its own target or browser context.
The trade-off is blast radius: one crash takes every session down. A shared-browser service should detect a dead connection and restart once, with a lock so ten workers do not all launch a replacement at the same time. BrowserService owns the process; a SharedBrowser wrapper owns the “ensure one is alive” policy.
For isolation, create a browser context per job:
context_id = connection.call(
"Target.createBrowserContext", {"disposeOnDetach": True}
)["browserContextId"]
page = PageService(connection, context_id=context_id)
# ...
page.close()
connection.call(
"Target.disposeBrowserContext", {"browserContextId": context_id}
)
Contexts behave like separate incognito profiles: cookies, cache, and storage do not leak between jobs.
Proxies
CDP splits proxy configuration in two:
- the proxy server is set when creating a browser context (
proxyServer); - the proxy credentials are supplied in response to
Fetch.authRequiredevents.
Install a Fetch interceptor on the tab, continue normal requests, and answer the first auth challenge for each requestId with Fetch.continueWithAuth. If the same request is challenged again, treat it as a rejection: cancel and mark the credential as refused. This gives you a clear “proxy refused auth” state instead of an endless retry loop.
Headless is not a separate browser
--headless=new is a mode of the same browser, but some sites still branch on the user agent or on missing window APIs. Test the exact mode you will run in production. If a page behaves differently in headless mode, compare the user agent and the viewport first, then the feature detection.
Timeouts should be layered
- Socket connect and handshake: seconds.
- CDP command reply: 10–30 seconds.
- Page load: up to 45 seconds.
- Whole job: a budget enforced by the caller.
A single global 30-second timeout is either too short for a slow page or too long for a dead browser. Make the layers explicit and log which one fired.
DevTools is remote code execution
Treat an open debugging port like a shell. A local process that can reach it can evaluate JavaScript in every tab, read cookies, and navigate anywhere. Do not expose it on 0.0.0.0. Do not ship it in a container image without a network policy. When you are done, close the connection and the process.
When you should use Playwright instead
Build the raw CDP layer when:
- you need to attach to an existing Chrome instance;
- you want precise control over network events and response bodies;
- you need a tiny runtime footprint or cannot add a browser-automation dependency;
- you are debugging why an automation framework is slow or flaky;
- you want to understand the protocol that your tools sit on.
Use Playwright when:
- you need cross-browser support;
- you want automatic waiting, selectors, tracing, video, and test runner integration;
- your team is more productive with a high-level API;
- you are building a product, not a protocol experiment.
The two are not enemies. Playwright is a well-engineered CDP client with a lot of extra machinery. Knowing what is underneath makes you better at using it.
The checklist
If you build this yourself, the failure modes fit on one page:
- [ ] Find Chrome explicitly; support an override path.
- [ ] Launch with
--remote-debugging-port=0and a non-default--user-data-dir. - [ ] Poll
DevToolsActivePort; do not parse logs. - [ ] Send stderr to a file, not an unread pipe.
- [ ] Run one reader thread; route replies by
idand events by(sessionId, method). - [ ] Wake every waiter when the socket dies.
- [ ] Put timeouts on commands, not on the idle socket.
- [ ] Attach with
flatten: trueand enable the domains you use. - [ ] Subscribe to
Page.loadEventFiredbefore navigating. - [ ] Evaluate with
returnByValueand handleexceptionDetails. - [ ] Subscribe to network events before the action that triggers them.
- [ ] Close targets, contexts, connections, and the process.
- [ ] Never expose the debugging port.
Once the service layer is in place, the scraping code becomes ordinary Python: call a method, get a dict, validate it, store it. That is the point. The browser is a dependency, not the architecture.
The next useful step is to make the extractor survive change: run the same job against a fixture page, assert the shape of the output, and fail loudly when a selector disappears. The browser will still be there in the morning. Your assumptions might not be.
If you build this, the two files worth reading twice are connection.py and page.py. One is about not hanging. The other is about not racing. Everything else is plumbing.
Top comments (0)