DEV Community: Branch42 Labs

Your crawler sees an empty div. Crawling a JavaScript SPA with Playwright

Parth V Square — Mon, 26 Oct 2026 17:01:12 +0000

TL;DR: On a client-rendered SPA, the HTML the server sends is a script tag and an empty <div id="root">. An HTTP crawler parses that, finds no links and stops. Load each page in a headless browser (Playwright or Puppeteer), wait for it to render, read a[href] from the live DOM and queue what you find. If you want the data rather than the pages, the JSON API the app calls is usually the better thing to crawl.

The empty shell

I pointed a plain crawler at a React dashboard and got exactly one page back. Here's why:

$ curl -s https://app.example.com/
<!doctype html>
<html>
  <head><script type="module" src="/assets/index-4f2a.js"></script></head>
  <body><div id="root"></div></body>
</html>

Every link on that site exists only after the bundle runs, hits the API and renders. wget --mirror, Scrapy or fetch plus cheerio all see the shell above. You need something that executes JavaScript, which means a browser engine you can drive from code.

The crawler

Setup:

npm i playwright robots-parser
npx playwright install chromium

crawl.mjs:

import { chromium } from "playwright"
import robotsParser from "robots-parser"

const start = new URL(process.argv[2])
const MAX_PAGES = 50
const HASH_ROUTES = false // true for apps with URLs like /#/settings
const SKIP = /logout|sign-?out|delete|remove/i

const robotsUrl = new URL("/robots.txt", start).href
const robotsTxt = await fetch(robotsUrl).then((r) => (r.ok ? r.text() : ""), () => "")
const robots = robotsParser(robotsUrl, robotsTxt)
const DELAY_MS = Math.max(1000, (robots.getCrawlDelay("*") ?? 0) * 1000)

const key = (href) => {
  const url = new URL(href)
  if (!(HASH_ROUTES && url.hash.startsWith("#/"))) url.hash = ""
  return url.href
}

const browser = await chromium.launch()
const page = await browser.newPage()
const apiCalls = new Set()
page.on("request", (req) => {
  if (["fetch", "xhr"].includes(req.resourceType())) apiCalls.add(req.method() + " " + req.url())
})

const queue = [key(start.href)]
const seen = new Set(queue)
const pages = []

while (queue.length > 0 && pages.length < MAX_PAGES) {
  const url = queue.shift()
  if (robots.isAllowed(url, "*") === false) continue
  try {
    await page.goto(url, { waitUntil: "domcontentloaded", timeout: 30_000 })
    // Wait for the app's own API calls to settle, but don't hang on polling.
    await page.waitForLoadState("networkidle", { timeout: 10_000 }).catch(() => {})
  } catch (error) {
    console.error("failed " + url + ": " + error.message)
    continue
  }

  const hrefs = await page.$$eval("a[href]", (links) => links.map((a) => a.href))
  pages.push({ url, finalUrl: page.url(), title: await page.title(), links: hrefs.length })

  for (const href of hrefs) {
    if (!href.startsWith("http")) continue
    const next = key(href)
    if (new URL(next).origin !== start.origin || SKIP.test(next) || seen.has(next)) continue
    seen.add(next)
    queue.push(next)
  }
  await page.waitForTimeout(DELAY_MS)
}

await browser.close()
console.log(JSON.stringify({ pages, apiCalls: [...apiCalls] }, null, 2))

Run it with node crawl.mjs https://your-site/. I tested it against a small pushState SPA whose links only appear 300 ms after load. It found every route, skipped the Disallowed one and stayed away from /logout.

What each part is doing:

BFS with a seen set, capped at MAX_PAGES, so it can't run away on a big site.
networkidle with a timeout. It resolves after 500 ms with no network connections. The Playwright docs discourage it for tests, and they're right, but for a crawler it's a decent "the app has fetched and rendered" signal. The timeout covers apps that poll forever.
a.href instead of getAttribute("href"), because the browser hands you the resolved absolute URL.
robots.txt and Crawl-delay via robots-parser, plus a one-second floor between pages.
The apiCalls set. This one turns out to be the most useful output (more on that below).

Using Puppeteer? Same loop. Use waitUntil: "networkidle2" on goto and the same $$eval.

Things that will bite you

Hash routing. With /#/settings-style URLs the fragment never reaches the server and most crawlers strip it, so every route looks like one page. Flip HASH_ROUTES to keep #/ fragments.
Buttons pretending to be links. onClick={() => router.push("/reports")} on a <div> has no href, so a link crawler never finds it. Click candidates and check page.url() afterwards, or read /sitemap.xml, or find the route config in the source.
Infinite scroll. Loop await page.mouse.wheel(0, 5000) plus a short wait until the page height stops changing.
Login. Sign in once, await context.storageState({ path: "auth.json" }), then browser.newContext({ storageState: "auth.json" }) for every crawl. And keep the logout skip, because one stray logout click and you're crawling the login page 50 times.
Query string explosions. Sort and filter params can multiply one list into thousands of URLs. Strip what you don't need in key().

Does Google have the same problem?

Less than it used to. Per Google's JavaScript SEO basics, Googlebot crawls the HTML, queues the page for rendering in an evergreen headless Chromium, and then extracts links from the rendered HTML. Rendering can lag behind crawling, it only follows <a> elements with an href, and Google tells you not to use # fragments to load different content. Other bots may not run JavaScript at all, which is why SSR or pre-rendering is still the advice.

Often you want the API, not the HTML

An SPA builds its pages from JSON. If you're after the data, GET /api/projects?page=2 is more stable than whatever CSS classes this quarter's redesign uses. Watch the Network panel with the Fetch/XHR filter on while you use the app, or just read the apiCalls the crawler logged.

That's also why I ended up using Netraptor (I work on it) for my own apps. Its crawler runs in my real Chrome tab, so the SPA renders with my logged-in session and there's no headless browser to log in. It reads the rendered a[href] links from the start page and visits them one level deep on the same origin. It skips logout and delete links and records the API calls per page into an endpoint map. It doesn't click buttons, scroll or follow hash routes, so the Playwright script above still has a place.

Only crawl sites you own or are allowed to test.

The full guide, with an FAQ, is at netraptor.dev/guides/crawl-javascript-website.

Scraping a site behind a login in Python (and why I go for the API instead)

Parth V Square — Sun, 25 Oct 2026 21:23:47 +0000

TL;DR: Log in once in code and reuse the session. For a plain form login, that's a requests.Session that posts the form with its CSRF token. For a JavaScript login or 2FA, use Playwright, log in (by hand if needed) and save storage_state to a file. Then, before you write a single CSS selector, open DevTools and look for the JSON API the logged-in page calls. Requesting that with the same session is usually less code and breaks less often.

First, the boring but important bit

I only do this on my own apps, for my own account's data, or on sites whose terms allow automated access. Read robots.txt and the terms, go slowly, and check for an official API or export first. Don't share credentials, and don't try to get around CAPTCHAs or paywalls. A CAPTCHA on the login page means the site doesn't want this.

Step 0: watch the login in DevTools

Before writing code, I log in by hand with the Network panel open:

F12, Network tab, tick Preserve log (the login redirect would wipe the list otherwise).
Log in.
Click the login request. The Payload tab shows the field names the form really sends, including the CSRF token. The Headers tab shows the Set-Cookie that comes back.
Click a request made after login. If it has an Authorization: Bearer ... header, the app uses a token and cookies alone won't get you in.

Ten minutes here saves an hour of guessing field names.

Option 1: requests.Session + BeautifulSoup

pip install requests beautifulsoup4

The trick is to GET the login page with the same session first, then copy every named input from the form. That picks up the CSRF token (csrf_token, _token, authenticity_token, csrfmiddlewaretoken, whatever the framework calls it) without hard-coding the name.

import os
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

BASE = "https://app.example.com"

session = requests.Session()
session.headers["User-Agent"] = "orders-export/1.0 (you@example.com)"

resp = session.get(f"{BASE}/login", timeout=30)
resp.raise_for_status()
form = BeautifulSoup(resp.text, "html.parser").find("form")

payload = {
    f["name"]: f.get("value", "")
    for f in form.find_all("input")
    if f.get("name")
}
payload["email"] = os.environ["APP_EMAIL"]        # match the Payload tab
payload["password"] = os.environ["APP_PASSWORD"]

login = session.post(urljoin(resp.url, form.get("action") or resp.url), data=payload, timeout=30)
login.raise_for_status()

if "/login" in login.url:
    raise SystemExit("Login failed")  # lots of sites return 200 + the login page

page = session.get(f"{BASE}/account/orders", timeout=30)
soup = BeautifulSoup(page.text, "html.parser")
for row in soup.select("table.orders tbody tr"):
    print([td.get_text(strip=True) for td in row.find_all("td")])

If the token is in <meta name="csrf-token">, read it from there and send it as an X-CSRF-Token header instead.

To avoid logging in every run, dump session.cookies.get_dict() to a JSON file and load it back with session.cookies.update(...). Log in again only when it stops working.

Option 2: Playwright + storage_state

pip install playwright && playwright install chromium

When the login is JavaScript-driven, or the data only shows up after render, I use a real browser:

import os
from pathlib import Path
from playwright.sync_api import sync_playwright

BASE = "https://app.example.com"
STATE = Path("state.json")

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)

    if not STATE.exists():
        ctx = browser.new_context()
        page = ctx.new_page()
        page.goto(f"{BASE}/login")
        page.get_by_label("Email").fill(os.environ["APP_EMAIL"])
        page.get_by_label("Password").fill(os.environ["APP_PASSWORD"])
        page.get_by_role("button", name="Sign in").click()
        page.wait_for_url("**/dashboard")
        ctx.storage_state(path=STATE)
        ctx.close()

    ctx = browser.new_context(storage_state=STATE)
    page = ctx.new_page()
    page.goto(f"{BASE}/account/orders")
    print(page.locator("table.orders tbody tr").all_inner_texts())
    browser.close()

storage_state saves cookies and localStorage. It doesn't save sessionStorage, which bites you if the app keeps its token there.

2FA and SSO

Don't automate it. Log in by hand once:

page.goto(f"{BASE}/login")
input("Log in in the browser window, then press Enter")
ctx.storage_state(path=STATE)

Reuse state.json until the session dies, then do it again.

Option 3 (my default): call the JSON endpoint

Most logged-in apps build their pages from JSON calls. Asking for that JSON gives you named fields instead of table cells, and a redesign doesn't break your script.

DevTools → Network → Fetch/XHR.
Reload the page with the data.
Check the Preview tab of each request until you find the data. Cmd/Ctrl+F in the Network panel searches response bodies, so search for a value you can see on screen.
Note the URL, query params and any custom headers.

Then use the session you already have:

import time

items, page = [], 1
while True:
    r = session.get(f"{BASE}/api/orders", params={"page": page}, timeout=30)
    if r.status_code == 429:
        time.sleep(int(r.headers.get("Retry-After", "30")))
        continue
    if r.status_code in (401, 403):
        raise SystemExit("Session expired, log in again")
    r.raise_for_status()
    batch = r.json()["items"]
    if not batch:
        break
    items.extend(batch)
    page += 1
    time.sleep(1)

From Playwright, ctx.request.get(f"{BASE}/api/orders") sends the context's cookies for you. Or move the cookies over to requests:

import json

for c in json.loads(STATE.read_text())["cookies"]:
    session.cookies.set(c["name"], c["value"], domain=c["domain"], path=c["path"])

A quick way to prototype: right-click the request in DevTools → Copy as cURL, run it in a terminal, then convert it to Python (by hand, or with curlconverter). That command includes your cookies, so keep it out of Slack and issues.

Don't commit your session

state.json and cookies.json work like a password until they expire. Put them and .env in .gitignore before the first commit, read credentials from env vars, and use a separate test account for anything on a cron.

When it's your own app

Finding every endpoint by clicking through the Network tab gets old fast. For my own apps I use Netraptor, a Chrome extension I work on. It records the fetch/XHR calls the logged-in app makes and groups them into endpoints like GET /api/orders/{id}, with schemas and real responses. It can also crawl the same-origin links on a start page in your logged-in tab to find more (one level deep, and it skips logout/delete-style links). It copies requests as cURL, PowerShell or fetch (no Python, so curlconverter still helps) and exports Postman or HAR.

The full guide, with more on CSRF tokens, Selenium cookies and rate limits, is here: How to scrape a website that requires a login.

How I find all pages on a website (and the ones every tool misses)

Parth V Square — Sun, 25 Oct 2026 21:21:13 +0000

TL;DR: Check /sitemap.xml (or the Sitemap: line in /robots.txt), run a site:example.com search, then crawl the site. Add the Wayback Machine for old URLs and Search Console if it's yours. None of these see pages behind a login or routes that only exist in your app's JavaScript. For those, crawl from a browser tab where you're already signed in.

I get asked "how many pages does this site actually have?" every time I inherit a project. The honest answer is that no single source knows. Here's the order I check things in, with the commands I use.

1. The sitemap

curl -s https://example.com/robots.txt | grep -i '^sitemap:'
curl -s https://example.com/sitemap.xml | grep -oE '<loc>[^<]+' | sed 's/<loc>//'

If the first <loc> entries point at more .xml files, you're looking at a sitemap index. Run the second command on each child. One sitemap file maxes out at 50,000 URLs or 50 MB uncompressed (sitemaps.org), so big sites always split.

What it misses: anything the owner didn't list. Apps often only list their marketing pages.

2. Google `site:` search

site:example.com, or site:example.com/docs for one section. Good for a quick look, useless for a count. Google's own docs say it "doesn't necessarily return all the URLs that are indexed."

3. Search Console (your own site only)

Indexing → Pages. The Page indexing report splits indexed from not-indexed URLs and gives the reason for each group. Click a reason, then Export. Example lists cap at 1,000 rows, but the not-indexed groups are where I usually find forgotten pages.

4. Crawl it

Screaming Frog is free up to 500 URLs per crawl. If you'd rather stay in the terminal:

wget --spider --recursive --level=inf --no-verbose \
  --output-file=crawl.log https://example.com/

grep -oE 'https?://[^ ]+' crawl.log | sort -u > urls.txt

wget respects robots.txt in recursive mode and doesn't run JavaScript. Keep that second part in mind.

5. The Wayback Machine for URLs that used to exist

https://web.archive.org/cdx/search/cdx?url=example.com/*&fl=original&collapse=urlkey&filter=statuscode:200

/* is a prefix match, fl=original returns just URLs, and collapse=urlkey drops duplicate captures. Expect noise. Check what still loads. Docs: CDX server API.

About "hidden pages"

Usually these are orphan pages: on the server, linked from nowhere. On your own site, diff your CMS or access logs against the crawl output. Wayback and site: sometimes surface pages that were linked once. Wordlist-guessing URLs (directory brute forcing) is for your own site or a scoped, authorised test. On someone else's site it's a good way to get blocked or worse.

Where all of this falls over: SPAs and logged-in apps

My last two projects were a React dashboard and an admin app behind SSO. Every method above returned the login page and a handful of marketing URLs.

In a single-page app the server sends an empty shell. Routes live in JavaScript, and some are only reachable through onClick handlers instead of <a href>.
Behind a login, a crawler without your session gets redirected on every URL. Pasting cookies into a crawler works until the session expires.

For a quick look at one rendered page, I paste this into the DevTools Console. It reads the live DOM, so JS-rendered links count:

[...new Set([...document.querySelectorAll('a[href]')].map(a => a.href))].join('\n')

For the whole app I use Netraptor, a Chrome extension I work on. Its crawler runs in my own logged-in tab: it reads the <a href> links on the start page after render, keeps same-origin ones, skips anything that looks like logout or delete, and loads each in turn while recording the API calls every page makes. It goes one level from the start page (start from a section index or use a path prefix to go deeper), and it doesn't follow #/hash routes. Free crawls 25 pages per run. I only point it at apps I build or am allowed to test.

Full guide with the Search Console and crawler details: How to find all pages on a website.

What a CORS error actually means, and how I fix each Chrome message

Parth V Square — Sun, 25 Oct 2026 13:56:57 +0000

TL;DR: A CORS error means the browser refused to give your page a response from another origin, because the server didn't send an Access-Control-Allow-Origin header that allows you. The fix lives on the server, or in a dev proxy so the browser only talks to its own origin. The text after has been blocked by CORS policy: in the Console tells you which header to fix. If the request sends JSON or an Authorization header, look at the OPTIONS preflight first. The Network panel files it under Other, so the Fetch/XHR filter hides it.

The 30-second version of CORS

An origin is scheme + host + port. localhost:5173 and localhost:8080 are different origins.
The browser enforces CORS. Postman and curl don't, which is why "it works in Postman" tells you nothing.
The server opts in by sending Access-Control-Allow-Origin and friends. Your frontend can't grant itself access.
JSON bodies, Authorization headers and methods other than GET/HEAD/POST trigger a preflight: an OPTIONS request, sent without cookies or tokens, asking whether the real request is allowed.

Read the message, not the TypeError

In code you only get TypeError: Failed to fetch (or Axios's Network Error). The browser hides the reason from scripts on purpose. The Console has the real message:

Access to fetch at 'https://api.example.com/orders' from origin 'http://localhost:5173'
has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present
on the requested resource.

Here's what I do for each ending Chrome prints. The strings come from Chromium's cors_error_string.cc:

Message ends with	Fix
`No 'Access-Control-Allow-Origin' header is present on the requested resource.`	Send the header. If the response was a 401/404/500, your error path probably skips the CORS middleware.
`It does not have HTTP ok status.` (preflight)	The OPTIONS request got a non-2xx response. Usually auth middleware rejecting a preflight that has no token. Answer OPTIONS with 204 before auth.
`Redirect is not allowed for a preflight request.`	The preflight hit a 301/302/307. Use the final URL (trailing slash, https, login redirect).
`...must not be the wildcard '*' when the request's credentials mode is 'include'.`	You send cookies. Echo the exact origin, add `Vary: Origin`.
`...'Access-Control-Allow-Credentials' header in the response is '' which must be 'true'...`	Add `Access-Control-Allow-Credentials: true`.
`Request header field authorization is not allowed by Access-Control-Allow-Headers in preflight response.`	Add the header to `Access-Control-Allow-Headers`.
`Method PUT is not allowed by Access-Control-Allow-Methods in preflight response.`	Add the method to `Access-Control-Allow-Methods`.
`...has a value '...' that is not equal to the supplied origin.`	Allow list has the wrong origin. Check port, scheme, `www`.
`...contains multiple values ', ', but only one is allowed.`	Both nginx and the app add the header. Remove one.

Finding the preflight in DevTools

Open DevTools before reproducing. Tick Preserve log if a redirect clears the list.
In the Network panel, pick All instead of Fetch/XHR.
The blocked request shows CORS error in the Status column (Chrome 88+). Hover it for the reason code, e.g. PreflightInvalidStatus.
The preflight is its own OPTIONS row with Type preflight. Check its status, then compare Access-Control-Request-Method / -Headers in the request with Access-Control-Allow-* in the response.
No preflight row? The request was simple, or Chrome used a cached result (Access-Control-Max-Age, capped at 7200 s in Chrome). Tick Disable cache and retry.

Then I replay the preflight with curl, which shows exactly what the server returns:

curl -i -X OPTIONS 'https://api.example.com/orders' \
  -H 'Origin: http://localhost:5173' \
  -H 'Access-Control-Request-Method: POST' \
  -H 'Access-Control-Request-Headers: authorization, content-type'

To test a fix before touching the server, Chrome 113+ lets you override response headers locally: Network > Headers > Response Headers, edit the value, reload.

Server fixes

Express, mounted before auth:

import cors from "cors";

app.use(cors({
  origin: ["http://localhost:5173", "https://app.example.com"],
  credentials: true, // only if you send cookies
}));
app.use(authMiddleware);

Next.js App Router route handler:

// app/api/orders/route.ts
const ALLOWED = new Set(["http://localhost:5173", "https://app.example.com"]);

const corsHeaders = (origin: string | null): Record<string, string> => ({
  ...(origin && ALLOWED.has(origin) ? { "Access-Control-Allow-Origin": origin } : {}),
  "Access-Control-Allow-Methods": "GET, POST, OPTIONS",
  "Access-Control-Allow-Headers": "Content-Type, Authorization",
  Vary: "Origin",
});

export const OPTIONS = (req: Request) =>
  new Response(null, { status: 204, headers: corsHeaders(req.headers.get("origin")) });

export const GET = async (req: Request) =>
  Response.json({ orders: [] }, { headers: corsHeaders(req.headers.get("origin")) });

For lots of routes, do it once in proxy.ts (middleware.ts before Next 16).

nginx. The always matters, because without it your 401s and 500s go out without the header and surface as CORS errors:

location /api/ {
    if ($request_method = OPTIONS) {
        add_header Access-Control-Allow-Origin "https://app.example.com" always;
        add_header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS" always;
        add_header Access-Control-Allow-Headers "Authorization, Content-Type" always;
        return 204;
    }
    add_header Access-Control-Allow-Origin "https://app.example.com" always;
    add_header Vary Origin always;
    proxy_pass http://backend;
}

On localhost, skip CORS with a proxy

Vite:

// vite.config.ts
export default defineConfig({
  server: {
    proxy: { "/api": { target: "http://localhost:8080", changeOrigin: true } },
  },
});

Next.js:

// next.config.js
module.exports = {
  async rewrites() {
    return [{ source: "/api/:path*", destination: "https://api.example.com/:path*" }];
  },
};

Call fetch("/api/orders") and the browser never makes a cross-origin request. Remember the Vite proxy is dev-only.

Things that look like fixes but aren't

mode: "no-cors" gives you an opaque response: status 0, no headers, empty body. It also silently drops Authorization and Content-Type: application/json.
--disable-web-security or a "CORS unblock" extension fixes your browser only. Your users still get the error, and your profile is now readable by any site.
strict-origin-when-cross-origin is Chrome's default Referrer-Policy (since Chrome 85). It shows up in every request's Headers tab and has nothing to do with CORS.

A faster way to find the preflights

I work on Netraptor, a Chrome extension with its own DevTools panel. In live capture it lists the same requests DevTools records, preflights included, so I filter method:OPTIONS to see every preflight on the page, or is:error for failed calls and 4xx/5xx together. Right-click > Copy > Copy as cURL (bash) gives me the call to replay in a terminal, where CORS doesn't apply.

The full guide, with steps and FAQ: What is a CORS error, and how do you fix it?

Forgot to open DevTools before the bug? Here's what you can still do

Parth V Square — Sun, 25 Oct 2026 13:56:38 +0000

TL;DR: You can't see requests that happened before you opened DevTools. Chrome's Network panel only records while DevTools is open, so they were never logged. What you can do is set things up so the next one gets caught: reproduce with DevTools open, launch Chrome with --auto-open-devtools-for-tabs, record the whole browser with chrome://net-export, or check the server logs.

The moment

A one-off 500 flashes up. Or the login redirect loops once and then works. You hit F12, click Network, and the panel is empty.

The Chrome docs are clear that DevTools records network requests "so long as DevTools is open." There's no background buffer to recover from. Browser history has the page URL, not the API calls the page made.

So the job becomes catching it next time.

First, make sure the tab is really empty for that reason

I've lost time to these more than once:

A filter is still on. Something in the filter box, a type chip other than All, Invert ticked, or Hide data URLs / Hide extension URLs under More filters.
Recording is off. The round button at the left of the action bar should be red. Cmd+E / Ctrl+E toggles it.
Wrong tab. A link that opened a new tab or popup has its own DevTools. Iframe requests do show in the parent page's panel.
A service worker is involved. It may have answered from its cache, or sent the request itself (background sync and similar). Inspect the worker from chrome://inspect/#service-workers.

If none of those apply, the request ran before DevTools opened.

Option 1: reproduce with DevTools open

Open DevTools on the tab and go to Network.
Tick Preserve log so redirects don't wipe the list.
Tick Disable cache if the bug only happens on a cold load.
Reload or repeat the steps.
Export a HAR before closing DevTools, because the log dies with the window.

I wrote up Preserve log and its limits separately.

Option 2: let Chrome open DevTools for you

Quit Chrome fully, then:

# macOS
open -a "Google Chrome" --args --auto-open-devtools-for-tabs

# Linux
google-chrome --auto-open-devtools-for-tabs

# Windows
"C:\Program Files\Google\Chrome\Application\chrome.exe" --auto-open-devtools-for-tabs

Every new tab now opens with DevTools attached, so first-load requests are recorded. The switch is ignored if Chrome is already running. I add --user-data-dir=/tmp/chrome-debug so it gets its own profile and my normal windows stay normal.

If the problem is only popups (OAuth, payment windows), there's a setting for that: DevTools Settings > Preferences > Global > Auto-open DevTools for popups.

Option 3: chrome://net-export

This records a NetLog for the whole browser, no DevTools needed. The steps from Chromium's docs:

Open chrome://net-export in a new tab.
Click Start Logging to Disk.
Reproduce in another tab. Leave the net-export tab open or logging stops.
Click Stop Logging.

It also only records from when you press Start. On days I'm chasing something intermittent, I start it in the morning and leave it running. Cap the size with --net-log-max-size-mb if that worries you.

The default level strips cookies and credentials. Include raw bytes adds the actual bytes, which can contain personal data.

Open the file in the NetLog viewer. It isn't a HAR, and it reads more like a connection-level trace (DNS, TLS, sockets) than a request inspector. Great for "why did this connection fail", less great for "what JSON came back".

Option 4: the server already logged it

If it already happened and won't happen again, check access logs, your error tracker or APM, and traces around that timestamp. If your API returns a request ID header, start logging it on the client too, so the next report maps straight to a server line.

The gap is requests that never reached the server: CORS failures, blocked calls, client timeouts. For those you need one of the browser options.

What I do now

For apps I work on every day, I don't want to depend on remembering F12. I use Netraptor, a Chrome extension I'm building, and turn on Keep history for this site for those apps. It hooks fetch and XHR in the page from the start of the load and saves those calls to IndexedDB with DevTools closed. When a 500 shows up, I open the panel, pick the saved session and filter status-code:5xx.

It only records API calls (no documents, scripts or images), only on sites I've turned it on for, and only from that point on. So it won't rescue yesterday's bug, but it does catch the next one.

Full guide, with the Network-tab-empty checklist and FAQ: https://netraptor.dev/guides/see-requests-before-devtools-opened

Why my coding agent couldn't see my app's API calls, and the 3 fixes I use

Parth V Square — Sun, 25 Oct 2026 13:56:10 +0000

TL;DR: Your agent runs in a terminal or editor. Your app's API calls run in a Chrome tab, behind your login. Nothing connects the two until you add something. You have three options:

Paste it (Copy as cURL or HAR). Zero setup. Manual, and it leaks tokens if you don't clean it.
Give the agent a browser (Chrome DevTools MCP, Playwright MCP, Claude in Chrome, Cursor's browser). Best for reproducing bugs. Sees only what it loads in that session.
Let it read traffic you already recorded through a local MCP server. Best for writing code against the real API. Read-only, covers only what you recorded.

The problem

I kept asking Claude Code to write a client for an internal API nobody had documented. It read the frontend source, found the URLs, and invented the response shapes. Then it tried curl from the terminal and got a 401, because the terminal doesn't have my session cookies.

The agent did nothing wrong. The data it needed lived in a browser tab it had no way into.

Fix 1: paste the request

For one failing call this is still the fastest thing:

DevTools → Network, reproduce the problem.
Right-click the request → Copy → Copy as cURL.
Delete the Cookie and Authorization values.
Paste it into the chat with the response body.

Watch out for:

Copy as cURL includes your headers, so tokens go straight into the chat unless you remove them.
Chrome's default HAR export is sanitized, but that only drops Cookie, Set-Cookie and Authorization. Tokens in URLs, custom headers like X-Api-Key and bodies stay in.
A HAR of a few minutes of clicking has hundreds of entries. That's a lot of context spent on images and scripts.

Fix 2: give the agent its own browser

# Chrome DevTools MCP (network, console, perf traces, Lighthouse)
claude mcp add chrome-devtools --scope user -- npx chrome-devtools-mcp@latest

# Playwright MCP (multi-browser, accessibility tree, request mocking)
claude mcp add playwright npx @playwright/mcp@latest

# Claude in Chrome extension: uses your Chrome and its login state
claude --chrome

In Cursor, the built-in browser in Agent already reads console messages and monitors network requests for the pages it opens.

This is the one I reach for when the agent has to do something: reproduce a bug, check a UI fix, read console errors, profile a slow page. It's also the only option that can see a page nobody has opened yet.

Where it gets awkward is "tell me about the whole API":

The request list is per page. Chrome DevTools MCP resets it on navigation and includePreservedRequests reaches back 3 navigations.
DevTools MCP and Playwright MCP start their own browser profile, so logged-in pages mean logging in there or attaching to your real session.
Every page the agent walks costs tool calls and tokens.
/orders/1042 and /orders/1043 come back as two unrelated URLs.

I wrote up DevTools MCP setup and attach mode separately: Chrome DevTools MCP guide.

Fix 3: let the agent read what you already recorded

The other way round: you use the app normally, something records the calls, and the agent queries that recording.

This is what I built Netraptor for. It's a Chrome extension that records your app's fetch/XHR calls in your own signed-in Chrome (turn on Keep history for this site so sessions are saved). Pro includes a local, read-only MCP server with five tools: list_sessions, list_endpoints, get_endpoint, search and get_request. Endpoints come back as templates like /orders/{id} with an inferred schema and a couple of real examples.

Pair once from Settings → Local MCP server (it gives you an npx -y @branch42-labs/netraptor-mcp pair NR-… command), then add it to your client:

# Claude Code
claude mcp add netraptor -- npx -y @branch42-labs/netraptor-mcp

# Codex (CLI and IDE extension share ~/.codex/config.toml)
codex mcp add netraptor -- npx -y @branch42-labs/netraptor-mcp

// Cursor: ~/.cursor/mcp.json or .cursor/mcp.json
{
  "mcpServers": {
    "netraptor": {
      "command": "npx",
      "args": ["-y", "@branch42-labs/netraptor-mcp"]
    }
  }
}

Honest limits:

It only reads saved sessions. Live traffic in the DevTools panel without Keep history isn't visible to the agent.
Sensitive headers, URL params and top-level JSON fields (Authorization, Cookie, anything named like token, secret, session, key) are redacted. Nested JSON fields aren't, so I use a test account.
It records from inside the page, so browser-added headers like Cookie aren't there at all. DevTools MCP shows them.
It can't click or send anything. If you never opened a page, its calls aren't there.
The MCP server is Pro ($29.99 once). Recording and Postman/HAR/curl export are free.

Prompts that work well once it's connected:

"What does GET /orders/{id} return? Show me a real response."
"Write a typed client for the endpoints the checkout page calls."
"Which calls returned 401 in my last session, and what did the token refresh request return just before?"
"Generate a mock server for /api/billing from the recorded responses."

What I actually do

One request, one question: Copy as cURL, minus tokens.
Reproduce or verify: Chrome DevTools MCP, or Playwright MCP for flows and other browsers.
Write code against the real API: recorded traffic over MCP.
Usually both: recorded traffic tells the agent what the API looks like, and the browser server lets it check the fix.

The full comparison, with setup for each client and an FAQ, is on the original guide: Why your coding agent can't see your app's API calls.

Chrome DevTools MCP in Claude Code (setup, attach mode, vs Playwright MCP)

Parth V Square — Sun, 25 Oct 2026 08:55:21 +0000

TL;DR: Chrome DevTools MCP is the Chrome DevTools team's open-source MCP server. It lets Claude Code, Cursor, Codex or Gemini CLI drive a real Chrome and use DevTools features: network requests, console, screenshots, performance traces, Lighthouse and heap snapshots. Install it with claude mcp add chrome-devtools --scope user -- npx chrome-devtools-mcp@latest. Add --autoConnect (Chrome 144+) to let it attach to the Chrome you're already signed in to. It's great for debugging the page in front of it. It's a poor fit for "map this app's whole API".

Why I wanted it

My agent kept writing frontend fixes it couldn't check. With a browser MCP server it opens the page, reads the console, looks at the failing request and tells me what broke. The Chrome team shipped this as a public preview in September 2025, and it now has around 60 tools (some behind opt-in flags).

Setup

You need Node.js LTS and current stable Chrome.

Claude Code

claude mcp add chrome-devtools --scope user -- npx chrome-devtools-mcp@latest

Or the plugin, which also installs skills for a11y, cookies, LCP and memory leaks:

/plugin marketplace add ChromeDevTools/chrome-devtools-mcp
/plugin install chrome-devtools-mcp@chrome-devtools-plugins

Cursor (Settings → MCP → New MCP Server) and most other clients:

{
  "mcpServers": {
    "chrome-devtools": {
      "command": "npx",
      "args": ["-y", "chrome-devtools-mcp@latest"]
    }
  }
}

Codex / Gemini CLI

codex mcp add chrome-devtools -- npx chrome-devtools-mcp@latest
gemini mcp add chrome-devtools npx chrome-devtools-mcp@latest

First prompt to try: Check the performance of https://developer.chrome.com. You'll see a Chrome window open and a trace get recorded.

The network tools

There are two:

list_network_requests: requests for a page since the last navigation. Filter by resourceTypes, paginate with pageSize/pageIdx, and set includePreservedRequests to reach back 3 navigations.
get_network_request: one request with headers (including Cookie and Set-Cookie) and bodies. responseFilePath writes a big body to disk instead of into the context window.

The trick I like most: call get_network_request with no reqid and it returns whatever request you have selected in the DevTools Network panel. I click the broken call, then type "why did this 403?".

If headers carry tokens you'd rather not send to a model, start the server with --redactNetworkHeaders.

Attach mode (your signed-in browser)

By default the server launches Chrome with its own profile in ~/.cache/chrome-devtools-mcp/chrome-profile. --isolated uses a throwaway profile instead. To debug something behind your real login:

Open chrome://inspect/#remote-debugging in Chrome 144+ and allow incoming connections.
Add the flag:

   "args": ["-y", "chrome-devtools-mcp@latest", "--autoConnect"]

Approve the permission dialog Chrome shows each time the server connects. A "Chrome is being controlled by automated test software" banner stays up while it's attached.

No --autoConnect (sandboxes, older Chrome)? Start Chrome yourself and point the server at it:

/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \
  --remote-debugging-port=9222 --user-data-dir=/tmp/chrome-profile-stable

"args": ["chrome-devtools-mcp@latest", "--browser-url=http://127.0.0.1:9222"]

Heads up: attach mode gives the agent your cookies and signed-in accounts. Google's own docs say to only use it with agents you trust. Also, usage statistics are on by default (--no-usage-statistics), and performance tools can send trace URLs to CrUX (--no-performance-crux).

Chrome DevTools MCP vs Playwright MCP

Playwright MCP (claude mcp add playwright npx @playwright/mcp@latest) overlaps a lot. How I pick:

	Chrome DevTools MCP	Playwright MCP
Browsers	Chrome, Chrome for Testing	Chrome, Firefox, WebKit, Edge
Page model	screenshots + text snapshot	accessibility tree, no vision needed
Perf insights / Lighthouse / heap	yes	Playwright traces via `--caps=devtools`, no Lighthouse or heap tools listed
Network	list + get (headers, bodies)	list (URL regex) + get, mocking via `--caps=network`
Existing browser	`--autoConnect`, `--browser-url`	`--extension`, `--cdp-endpoint`

User flows and cross-browser checks: Playwright. "Why is this page slow or erroring in Chrome": DevTools MCP. Playwright's README also notes that coding agents may get on better with its CLI + skills, since those calls cost fewer tokens than MCP.

Where it stops for API work

Both servers only see what the browser loads while the agent is connected. When I want the agent to understand the whole API of an app, that hurts:

The request list resets per navigation (3 back at most), so the agent has to click through every screen.
Each screen costs tool calls and context.
/orders/1042 and /orders/1043 come back as separate URLs, so the agent has to figure out they're one endpoint.
Real logged-in traffic means attach mode and handing over my session.
Close the browser and the traffic is gone.

For that job I use Netraptor, a Chrome extension I work on. It records the app's fetch/XHR calls while I browse normally (or crawls same-site pages), groups them into endpoints like /orders/{id}, and its Pro plan has a local, read-only MCP server so the agent can ask "what does GET /orders/{id} return?" from real recorded responses. It has its own gap: it records from inside the page, so browser-added headers like Cookie aren't in its copy. DevTools MCP shows those. So I run both: Netraptor for "what does the API look like", DevTools MCP for "go reproduce this".

The full guide, with links to every doc above, is at netraptor.dev/guides/chrome-devtools-mcp.

Chrome won't let you paste into the console? Type "allow pasting" (no quotes)

Parth V Square — Sun, 25 Oct 2026 08:55:06 +0000

TL;DR: Paste once so the warning shows up. Then type allow pasting in the Console prompt, without quotes, and press Enter. Paste again. Chrome remembers the choice for that profile.

I set up a fresh Chrome profile last week, pasted a snippet into the Console, and got this instead of my code:

Warning: Don't paste code into the DevTools Console that you don't understand or haven't reviewed yourself. This could allow attackers to steal your identity or take control of your computer. Type "allow pasting" below and press Enter to allow pasting.

If you landed here from Google with that exact text in the search box, here is the short version and the parts that trip people up.

The fix

Open the Console (Ctrl+Shift+J, or Cmd+Option+J on a Mac).
Paste your code. Chrome blocks it and logs the warning.
Type allow pasting and press Enter.
Paste again. It lands in the prompt this time.

In the Sources panel (for example a snippet under Sources > Snippets) you get a dialog with a text field instead of a Console message. Type the same phrase there and confirm.

Why "allow pasting" sometimes doesn't work

I looked at the DevTools source to see what it checks. It compares the prompt text against the phrase exactly, and only while the warning is active. So:

Paste first. Typing the phrase before the warning appears does nothing.
No double quotes. The warning shows the phrase in double quotes, but "allow pasting" is just a JS string. The Console echoes it and you are still blocked. Plain allow pasting works, and so does 'allow pasting' with single quotes.
No extra spaces or capitals. Watch for autocomplete adding things.
Non-English DevTools uses a translated phrase. Type the one in your warning.

Why you don't see it on your main profile

Chrome only shows the warning to profiles that look new to DevTools. If your Console history has at least 5 entries, it skips the warning entirely. The Chrome team wrote this up in How Chrome DevTools helps to defend against self-XSS attacks.

It comes back when the stored setting is gone:

a new Chrome profile or a fresh install
Settings > Preferences > Restore defaults and reload
a rare bug mentioned in the Chrome post, where a new profile is the suggested fix

For CI and automation, launch Chrome with:

--unsafely-disable-devtools-self-xss-warnings

Chrome for Testing already has the warnings off. Edge is Chromium, so it behaves the same and takes the same flag.

The reason it exists: self-XSS

Self-XSS is the "paste this in your console to get free credits" scam. Whatever you paste runs as you on that site. It can:

read localStorage, sessionStorage and any cookie that isn't HttpOnly, which is where a lot of apps keep tokens
call the site's API with your session
fetch all of it off to someone else's server

For us the phrase is a one-time speed bump. The habit worth keeping is reading every paste before you hit Enter.

Safer habits for code you copied

Use a Snippet for anything you run twice. Sources > Snippets > New snippet, paste, save, then Ctrl+Enter / Cmd+Enter. You can also run it from the Command Menu (Ctrl+Shift+P, delete the >, type !snippet-name).

Read a copied fetch before running it. Check that:

the URL is the host you expect
the method is what you want (POST and DELETE change data)
the body is readable JSON or form data
nothing extra wraps the call, like eval, atob, or a second fetch that ships document.cookie somewhere

Prefer cURL in a terminal if you only need the response. It runs outside the page, so it can't touch page storage. It still carries your cookies and auth headers, so don't drop it into Slack.

Skipping the paste entirely

Half the time I'm pasting because I copied a request as fetch and want to replay it with a tweak. I use Netraptor for that now (disclosure: it's ours). It's a DevTools panel where you right-click a request and pick Resend, or Edit and resend as fetch to change the method, URL, headers or body first. It runs the request with fetch inside the inspected page using the page's cookies, and the result shows up tagged Resent. The same menu copies as cURL (bash/cmd), PowerShell, fetch or Node.js fetch. It only works on live traffic, since it needs the page you're inspecting.

Full guide, with the Sources dialog and FAQ: Allow pasting in the Chrome console

How to create a HAR file in Chrome without leaking your tokens

Parth V Square — Sun, 25 Oct 2026 08:54:55 +0000

TL;DR: Open DevTools > Network, check Preserve log and Disable cache, clear, reload, reproduce the bug, then click Export HAR (sanitized).... Since Chrome 130 that export drops Cookie, Set-Cookie and Authorization. Tokens in URLs, custom headers like X-Api-Key, and request and response bodies are still in the file. Run a grep over it before you attach it to a ticket.

Every support team eventually asks for "a HAR file". Most of the bad ones I've seen came from the same three mistakes: DevTools opened too late, Preserve log off, or a filter left on. Here's the version that works on the first try.

The steps

Go to the page with the problem and open DevTools (F12, Ctrl+Shift+I, or Cmd+Option+I on macOS). Click Network. Do this before you reproduce anything, because Chrome only logs requests while DevTools is open.
Check Preserve log (so redirects and logins don't wipe the list) and Disable cache (so you see what a first-time visitor sees).
Click the clear button, then reload. Now the log starts from a clean page load.
Reproduce the bug. Keep DevTools open until it has happened.
Clear the Filter box and select All. A leftover Fetch/XHR chip is an easy way to ship half a file.
Click the download icon, Export HAR (sanitized)..., and save. Right-clicking any request gives you Save all as HAR (sanitized) too.

If the flow opens a popup or a new tab (hello, SSO), those requests land in that tab's DevTools. Turn on Auto-open DevTools for popups in DevTools settings first.

"Sanitized" covers three headers

Chrome 130 changed the default export (release notes). The sanitized file leaves out:

Cookie
Set-Cookie
Authorization

That's the whole list. If you actually need those headers, say to debug your own auth flow, go to DevTools Settings > Preferences > Network, check Allow to generate HAR with sensitive data, then long-click the Export button and pick Export HAR (with sensitive data).

Edge behaves the same since it's the same DevTools. Firefox (Network Monitor gear menu > Persist Logs, then Save All As HAR) and Safari (Web Inspector > Network > Preserve Log, then Export) put cookies and auth headers in the file along with everything else.

What's still in a sanitized HAR

Things I've found in "sanitized" HAR files:

X-Api-Key, X-CSRF-Token, X-Auth-Token headers
?access_token=... and ?api_key=... in URLs, including inside Referer and Location
OAuth ?code= values on the login redirect
The password in a login request body, in plain text
Refresh tokens in token-exchange calls and fresh access tokens in login responses
Names, emails and addresses in ordinary API responses

Scrubbing it

First, record less. Log in, clear the log, then reproduce only the failing steps.

Then strip the headers you know about. With jq:

jq '.log.entries[] |= (
  .request.headers |= map(select(.name | test("^(cookie|authorization|x-api-key)$"; "i") | not))
  | .response.headers |= map(select(.name | test("^set-cookie$"; "i") | not))
  | .request.cookies = []
  | .response.cookies = []
)' session.har > session.scrubbed.har

Headers are the easy part. Now look for everything else:

grep -n -i -E 'token|secret|password|api[_-]?key|bearer|eyJ' session.scrubbed.har

eyJ catches JWTs, because {" base64-encodes to eyJ. Every hit is a value to replace by hand or a step to leave out of the recording. Google's HAR Analyzer can also download a redacted copy. Whatever you couldn't remove, log out or rotate the key once the ticket is closed.

A sanitizer that also reads URLs and bodies

I work on Netraptor, a Chrome extension that adds its own Network panel to DevTools, and its Save all as HAR (sanitized) goes further than Chrome's. It drops any header with a credential-like name (token, secret, password, session, auth, csrf, jwt, key, and so on), sets matching query and fragment params plus OAuth code to [redacted], redacts matching keys at any depth in JSON and form bodies, and removes bodies it can't parse (HTML, text, binary) with a comment saying so. It saves only the rows you have filtered to, so you can export just the failing calls. It matches on field names, so an email under email or a token in a URL path still gets through. Run the grep anyway.

The full guide, with FAQ and a section on why HAR files come out empty, is here: How to create a HAR file in Chrome.

Chrome's Network tab keeps clearing on reload. Here's how to stop it

Parth V Square — Sun, 25 Oct 2026 08:54:33 +0000

TL;DR: In Chrome DevTools, open the Network panel and tick Preserve log. Requests then survive reloads and redirects. They still vanish when you close DevTools, anything before DevTools opened is missing, and every page load lands in one unlabelled list. In Firefox the option is Persist Logs, under the gear icon in the Network Monitor.

The problem

You're debugging a login. The form posts, the server redirects, and the request you needed is gone. Chrome's Network panel clears its list every time the main frame loads a new document. That covers reloads, links, form posts and server redirects.

Turn on Preserve log

Open DevTools before loading the page (F12, or Cmd+Option+I on macOS). Chrome only records requests while DevTools is open.
Click the Network tab.
Tick Preserve log in the action bar at the top.
Reproduce the reload or redirect. The old requests stay put and new ones are added below them.
Before you close DevTools, export a HAR (download icon in the action bar) if you want to keep any of it.

The Console has its own Preserve log in its settings. Turn that on too if you want console messages to survive the reload.

Where it falls short

I hit these regularly:

Closing DevTools wipes it. The preserved log lives in the DevTools window. Close it and the log is gone.
The first load is often missing. If the page loaded before you opened DevTools, those requests were never recorded. Reload with DevTools open. For popups, start Chrome with:

  google-chrome --auto-open-devtools-for-tabs

If you need a capture that doesn't depend on DevTools at all, chrome://net-export writes a browser-level NetLog.

One flat list. After five reloads you have five copies of /api/me and nothing telling you which load each one belongs to. You end up reading the order of the document requests and the waterfall.
SPAs never clear anyway. A history.pushState route change isn't a document load, so the log never clears. With or without Preserve log, every route's API calls pile into the same list with no marker between them.

Disable cache is a different job

Disable cache sits right next to it. Preserve log decides whether the list survives a reload. Disable cache decides whether the requests in it hit the network or come from the HTTP cache (only while DevTools is open). If a request seems missing after a reload, check whether it came from the memory or disk cache before you assume it never ran.

Firefox: Persist Logs

Open the Network Monitor (Ctrl+Shift+E, or Cmd+Option+E on macOS).
Click the gear icon at the right end of the Network toolbar.
Select Persist Logs.

The Web Console has its own Persist Logs setting too. The limits are the same as Chrome's.

What I use when the flat list gets in the way

I work on Netraptor, a Chrome extension with its own DevTools panel, and this was one of the gaps we wanted to close. When you open the panel it pulls in what DevTools already logged for the page. After that, each reload, navigation or SPA route change (pushState, replaceState, back/forward) starts a group labelled "Reloaded" or "Navigated to" with the path, time and request count. Earlier loads fold away rather than being deleted. If you turn on "Keep history for this site", it saves that site's fetch/XHR calls to IndexedDB, so they're still there after you close DevTools.

The full guide, with steps and FAQ, is here: Preserve log in Chrome: keep network requests after a page reload.

What is a HAR file, and how to get one from Chrome

Parth V Square — Fri, 23 Oct 2026 19:04:44 +0000

A HAR file (HTTP Archive) is a JSON file that records every network request a browser made while a page was open, including URLs, headers, request bodies, responses and timings. Support teams ask for one because it shows exactly what your browser sent and received when something broke. You can generate a HAR file in Chrome from the Network panel in DevTools in under a minute.

Disclosure: I build Netraptor, which this guide mentions as one option. The steps work with plain Chrome and free tools.

Steps

Open the Network panel. Go to the page with the problem and open DevTools with F12, Ctrl+Shift+I on Windows and Linux, or Cmd+Option+I on macOS. Click the Network tab.
Turn on Preserve log. Check Preserve log so requests are kept across page loads and redirects. Check Disable cache too if the problem only shows up on a fresh load.
Reproduce the problem. Reload the page and repeat the exact steps that cause the issue. DevTools only records while it is open, so keep it open the whole time.
Export the HAR file. Click the download icon, Export HAR (sanitized)..., in the action bar at the top of the Network panel, and choose where to save the file.
Check the file before sharing. Open the .har file in a text editor and search for tokens, emails or other secrets before you send it to anyone.

What is a HAR file?

HAR stands for HTTP Archive. A HAR file is plain JSON with a .har extension that describes a browser's network activity: one entry per request, grouped under the pages that triggered them. Chrome, Edge, Firefox and Safari can all export it, and most debugging tools can read it.

The format was drafted as HAR 1.2 for the W3C Web Performance Working Group. That draft was never published as a standard and is now marked as abandoned, but browsers still write files that follow it. A trimmed HAR file looks like this:

{
  "log": {
    "version": "1.2",
    "creator": { "name": "WebInspector", "version": "537.36" },
    "pages": [{ "id": "page_1", "title": "https://app.example.com/" }],
    "entries": [
      {
        "pageref": "page_1",
        "startedDateTime": "2026-10-16T09:12:03.120Z",
        "time": 184.2,
        "request": { "method": "GET", "url": "https://app.example.com/api/orders" },
        "response": { "status": 500, "statusText": "Internal Server Error" },
        "timings": { "blocked": 1.1, "dns": 0, "connect": 0, "send": 0.2, "wait": 176.4, "receive": 6.5 }
      }
    ]
  }
}

What's inside a HAR file

Each entry in log.entries captures one request and its response. The fields you will look at most:

request: method, full URL, query string, headers, cookies and the request body (postData).
response: status code, headers, cookies, MIME type and usually the response body in content.text.
timings: how long the request spent blocked, on DNS, connecting, sending, waiting for the server (wait) and downloading (receive), in milliseconds.
startedDateTime and time: when the request began and how long it took in total.
pages: the page loads the requests belong to, so you can tell which navigation triggered what.

Because it holds full headers and bodies, a HAR file is often the fastest way for someone else to see a failed API call, a slow request or a bad redirect without access to your account.

Export HAR (sanitized) vs. with sensitive data

Since Chrome 130, the default HAR export is sanitized: it leaves out the Cookie, Set-Cookie and Authorization headers. This change was made to cut down on accidental leaks of login sessions when people attach HAR files to support tickets.

If you need those headers, for example to debug your own auth flow, open DevTools Settings > Preferences > Network and turn on Allow to generate HAR with sensitive data. The Export HAR button then offers both a sanitized and a sensitive-data option. See the Network features reference for the current labels.

Sanitized only removes those three headers. Tokens in URLs, request bodies, custom headers like X-Api-Key and response bodies are still in the file.

How to get a HAR file in Edge, Firefox and Safari

Microsoft Edge uses the same DevTools as Chrome. Follow the steps above.
Firefox: open the Network Monitor (Ctrl+Shift+E or Cmd+Option+E), reproduce the issue, then right-click any request and choose Save All As HAR.
Safari: enable the Develop menu in Safari settings, open Web Inspector, go to the Network tab, reproduce the issue and click Export.

Whichever browser you use, start recording before you reproduce the problem. A HAR file only contains requests made while the network tools were open.

Is it safe to share a HAR file?

Treat a HAR file like a password. A file exported with sensitive data contains session cookies and bearer tokens, and anyone who has it can often act as you until those expire. Even a sanitized file can include personal data from API responses, IDs in URLs and API keys in custom headers.

Before you share one:

Use the sanitized export unless the recipient specifically needs auth headers.
Record in a fresh tab with only the steps needed to reproduce the issue, so unrelated traffic stays out of the file.
Search the file for token, key, password, secret and your email address.
Log out after sending a file that included cookies. That invalidates the session on most sites.

Google's HAR Analyzer also has a redaction button that downloads a copy with sensitive values removed.

Doing this with Netraptor

Chrome's Network panel records everything, but only while DevTools is open, and you have to remember to turn on Preserve log first. If you capture the same app often, Netraptor can record it in the background instead.

In the Netraptor toolbar popup, turn on Record this site. It records the fetch and XHR API traffic for that domain and its subdomains across page loads, with or without DevTools open.
Open the Netraptor panel in DevTools to browse the recorded requests in a Chrome-style Requests table, or group them by endpoint in API Map.
On the Export tab, export a HAR file. Strip auth headers and cookies is on by default, so the file is safer to share.

HAR export is part of the Free plan, along with Postman and curl exports. Everything stays in your browser and no account is needed. Netraptor records API calls only, so if you need images, scripts and the full page-load waterfall in the file, use Chrome's own export.

What to do with a HAR file next

Once you have the file, you can load it back into Chrome DevTools, open it in an online viewer or query it with jq. See how to open and analyze a HAR file. If you want to replay the requests you captured, turn them into a Postman collection.

FAQ

What does HAR stand for?

HAR stands for HTTP Archive. It is a JSON-based format for logging a browser's network requests and responses.

Why is my HAR file empty?

DevTools only records network traffic while it is open. Open the Network panel first, check Preserve log, then reload the page and reproduce the problem before exporting.

Can I open a HAR file in Notepad?

Yes. A HAR file is plain JSON text, so any text editor can open it. For a readable request list, import it into the Chrome DevTools Network panel or a HAR viewer.

Does a HAR file contain my password?

It can. If you log in while recording, the login request body may include your password in plain text. Record after logging in, or check the file before sharing it.

Why are cookies missing from my HAR file?

Chrome exports sanitized HAR files by default and strips the Cookie, Set-Cookie and Authorization headers. Turn on Allow to generate HAR with sensitive data in DevTools settings if you need them.

Originally published at netraptor.dev/guides/what-is-a-har-file. More guides: netraptor.dev/guides.

What is an OpenAPI spec, and how to generate one from your web app's traffic

Parth V Square — Fri, 23 Oct 2026 19:04:12 +0000

An OpenAPI spec is a JSON or YAML document that describes an HTTP API in a standard, language-agnostic format, listing its paths, methods, parameters, request bodies and responses. If your API has no spec yet, you can generate a first draft from real traffic by recording the requests your web app makes in Chrome, exporting them as a HAR file, and converting that file with an open-source tool such as mitmproxy2swagger. You then review the draft by hand, because traffic only shows what was actually called.

Disclosure: I build Netraptor, which this guide mentions as one option. The steps work with plain Chrome and free tools.

Steps

Record the app's API calls in Chrome. Open DevTools (F12, or Cmd+Option+I on a Mac), go to the Network panel, check Preserve log, click the Fetch/XHR filter and reload the page.
Use every feature you want in the spec. Sign in with a test account and walk through each screen. Create, edit and delete test records so every endpoint and HTTP method shows up at least once.
Export a HAR file. Click Export HAR (sanitized) in the Network panel toolbar and save the file as app.har. The sanitized export leaves out Cookie, Set-Cookie and Authorization headers.
Run a first pass with mitmproxy2swagger. Install it with pip install mitmproxy2swagger, then run mitmproxy2swagger -i app.har -o openapi.yaml -p https://api.example.com. The output lists every discovered path under x-path-templates, each prefixed with ignore:.
Pick the paths and run it again. Remove the ignore: prefix from the paths you want, fix templates such as /users/{id}, and run the same command again. The second pass writes the operations into paths.
Validate and edit the result. Paste the file into Swagger Editor or run openapi-generator-cli validate -i openapi.yaml. Then add descriptions, required fields, error responses and auth by hand.

What is OpenAPI?

The OpenAPI Specification (OAS) defines a standard, programming-language-agnostic way to describe HTTP APIs. A tool or a person can read an OpenAPI document and learn which endpoints exist, what each one accepts and what it returns, without reading the server code.

OpenAPI started as the Swagger Specification. SmartBear donated it to the OpenAPI Initiative under the Linux Foundation in late 2015, and it was renamed in early 2016. Today "OpenAPI" is the name of the format, and "Swagger" usually refers to SmartBear's tools such as Swagger UI and Swagger Editor.

The published versions are 2.0, the 3.0.x line, the 3.1.x line and the 3.2.x line, with 3.2.1 the newest. Version 3.1 aligned its Schema Object with JSON Schema draft 2020-12, and most current tools read 3.0 and 3.1. If a tool you rely on rejects a newer version, check its docs before choosing.

What does an OpenAPI spec look like? (JSON and YAML)

An OpenAPI document is one JSON or YAML file, conventionally named openapi.json or openapi.yaml. It must have an openapi version field and an info object, plus at least one of paths, components or webhooks. Path parameters are written in curly braces, and every placeholder must match a parameter definition.

openapi: 3.1.0
info:
  title: Example app API
  version: 1.0.0
paths:
  /users/{id}:
    get:
      summary: Get a user
      parameters:
        - name: id
          in: path
          required: true
          schema:
            type: string
      responses:
        "200":
          description: The user
          content:
            application/json:
              schema:
                type: object
                properties:
                  id: { type: string }
                  email: { type: string }

The JSON form holds the same structure with braces and quotes. YAML is easier to edit by hand, JSON is easier to generate and parse, and most tools accept either.

What a spec generated from traffic will miss

Recorded traffic shows real requests and responses, which makes it a good starting point. It also has gaps you need to fill before anyone treats the file as documentation.

Endpoints you never triggered are missing. Admin screens, error paths and rarely used features only appear if you exercise them.
Schemas are inferred from examples. A field that was null in every response, or an optional field you never sent, will be typed loosely or left out.
Path parameters need checking. Generators guess which segments are IDs, so /users/42 and /users/me can end up merged or split the wrong way.
Auth is usually missing. Add a securitySchemes entry under components (for example a bearer token) and a security requirement yourself.
Enums, formats, limits and descriptions come from you, not the traffic.

Treat the generated file as a draft. One careful review pass usually turns it into something your team can rely on.

Is it safe to generate a spec from production traffic?

Be careful with what goes into the HAR file and the spec. Chrome's sanitized HAR export removes Cookie, Set-Cookie and Authorization headers, but request and response bodies stay in the file, and so do tokens passed in URLs or custom headers.

mitmproxy2swagger adds example data only when you pass --examples (and headers with --headers). Its README warns that this can include sensitive data. If you want examples in the spec, record against a test account with fake data, then read the examples before you commit or share the file.

Record traffic only from apps you build, test or are authorised to test, and prefer a staging environment over production.

What to do with an OpenAPI spec once you have one

Publish docs. Swagger UI and similar tools render the file as browsable reference pages.
Generate clients and server stubs. OpenAPI Generator takes -i (the spec), -g (a generator such as typescript-fetch) and -o (an output folder).
Test against it. The spec becomes a contract you can check the real API against in CI.
Import it into Postman. Postman imports OpenAPI 3.0 and 3.1 as a collection. See Create a Postman collection from a website's requests.

If you only need the list of endpoints and not a full spec, Find every API endpoint your web app calls covers the DevTools side in more detail.

Faster: generate the spec with Netraptor

Netraptor is a Chrome extension with its own DevTools panel. It shortens the record, convert and template steps above, and everything stays local in your browser.

Turn on "Record this site" in the toolbar popup. Netraptor records fetch/XHR API traffic for that domain and its subdomains across page loads, even with DevTools closed.
Optionally run "Crawl this site" to visit same-origin links in your logged-in tab. It skips logout and delete-looking links. Free crawls cover 25 pages, Pro up to 1000.
Check the API Map tab. Endpoints are grouped and IDs are already parameterized, for example /users/{id}.
In the Export tab, export OpenAPI 3.1 as YAML or JSON, with optional response samples. "Strip auth headers and cookies" is on by default.

OpenAPI export is part of Pro ($29.99 one-time, no subscription). The Free plan exports HAR, Postman Collection v2.1 and curl, so you can export a HAR from Netraptor and run it through mitmproxy2swagger instead. You still review the generated spec by hand either way.

FAQ

Is OpenAPI the same as Swagger?

OpenAPI is the specification, and it was called the Swagger Specification until 2016. Swagger now refers to SmartBear's tools, such as Swagger UI and Swagger Editor, that work with OpenAPI files.

Should I use OpenAPI 3.0, 3.1 or 3.2?

Use 3.1 unless a tool in your pipeline only supports 3.0. Version 3.1 matches JSON Schema 2020-12, and 3.2 is newer, so check tool support before adopting it.

What is OpenAPI Generator?

OpenAPI Generator is an open-source CLI that turns an OpenAPI document into client SDKs, server stubs and docs for many languages. It also has a validate command for checking a spec.

Can I generate an OpenAPI spec from code instead of traffic?

Yes, if you own the backend. Many frameworks, FastAPI for example, produce a spec from route definitions, and that is usually more complete than one inferred from traffic.

What is the difference between openapi.json and openapi.yaml?

They hold the same document in two syntaxes. The spec allows both, and most tools accept either.

Originally published at netraptor.dev/guides/generate-openapi-spec-from-traffic. More guides: netraptor.dev/guides.

DEV Community: Branch42 Labs

Your crawler sees an empty div. Crawling a JavaScript SPA with Playwright

The empty shell

The crawler

Things that will bite you

Does Google have the same problem?

Often you want the API, not the HTML

Scraping a site behind a login in Python (and why I go for the API instead)

First, the boring but important bit

Step 0: watch the login in DevTools

Option 1: requests.Session + BeautifulSoup

Option 2: Playwright + storage_state

2FA and SSO

Option 3 (my default): call the JSON endpoint

Don't commit your session

When it's your own app

How I find all pages on a website (and the ones every tool misses)

1. The sitemap

2. Google site: search

3. Search Console (your own site only)

4. Crawl it

5. The Wayback Machine for URLs that used to exist

About "hidden pages"

Where all of this falls over: SPAs and logged-in apps

What a CORS error actually means, and how I fix each Chrome message

The 30-second version of CORS

Read the message, not the TypeError

Finding the preflight in DevTools

Server fixes

On localhost, skip CORS with a proxy

Things that look like fixes but aren't

A faster way to find the preflights

Forgot to open DevTools before the bug? Here's what you can still do

The moment

First, make sure the tab is really empty for that reason

Option 1: reproduce with DevTools open

Option 2: let Chrome open DevTools for you

Option 3: chrome://net-export

Option 4: the server already logged it

What I do now

Why my coding agent couldn't see my app's API calls, and the 3 fixes I use

The problem

Fix 1: paste the request

Fix 2: give the agent its own browser

Fix 3: let the agent read what you already recorded

What I actually do

Chrome DevTools MCP in Claude Code (setup, attach mode, vs Playwright MCP)

Why I wanted it

Setup

The network tools

Attach mode (your signed-in browser)

Chrome DevTools MCP vs Playwright MCP

Where it stops for API work

Chrome won't let you paste into the console? Type "allow pasting" (no quotes)

The fix

Why "allow pasting" sometimes doesn't work

Why you don't see it on your main profile

The reason it exists: self-XSS

Safer habits for code you copied

Skipping the paste entirely

How to create a HAR file in Chrome without leaking your tokens

The steps

"Sanitized" covers three headers

What's still in a sanitized HAR

Scrubbing it

A sanitizer that also reads URLs and bodies

Chrome's Network tab keeps clearing on reload. Here's how to stop it

The problem

Turn on Preserve log

Where it falls short

Disable cache is a different job

Firefox: Persist Logs

What I use when the flat list gets in the way

What is a HAR file, and how to get one from Chrome

Steps

What is a HAR file?

What's inside a HAR file

Export HAR (sanitized) vs. with sensitive data

How to get a HAR file in Edge, Firefox and Safari

Is it safe to share a HAR file?

2. Google `site:` search