Scraping a site behind a login in Python (and why I go for the API instead)

TL;DR: Log in once in code and reuse the session. For a plain form login, that's a requests.Session that posts the form with its CSRF token. For a JavaScript login or 2FA, use Playwright, log in (by hand if needed) and save storage_state to a file. Then, before you write a single CSS selector, open DevTools and look for the JSON API the logged-in page calls. Requesting that with the same session is usually less code and breaks less often.

First, the boring but important bit

I only do this on my own apps, for my own account's data, or on sites whose terms allow automated access. Read robots.txt and the terms, go slowly, and check for an official API or export first. Don't share credentials, and don't try to get around CAPTCHAs or paywalls. A CAPTCHA on the login page means the site doesn't want this.

Step 0: watch the login in DevTools

Before writing code, I log in by hand with the Network panel open:

1. F12, Network tab, tick Preserve log (the login redirect would wipe the list otherwise).
2. Log in.
3. Click the login request. The Payload tab shows the field names the form really sends, including the CSRF token. The Headers tab shows the Set-Cookie that comes back.
4. Click a request made after login. If it has an Authorization: Bearer ... header, the app uses a token and cookies alone won't get you in.

Ten minutes here saves an hour of guessing field names.

Option 1: requests.Session + BeautifulSoup

pip install requests beautifulsoup4

The trick is to GET the login page with the same session first, then copy every named input from the form. That picks up the CSRF token (csrf_token, _token, authenticity_token, csrfmiddlewaretoken, whatever the framework calls it) without hard-coding the name.

import os
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

BASE = "https://app.example.com"

session = requests.Session()
session.headers["User-Agent"] = "orders-export/1.0 ([e-mail])"

resp = session.get(f"{BASE}/login", timeout=30)
resp.raise_for_status()
form = BeautifulSoup(resp.text, "html.parser").find("form")

payload = {
    f["name"]: f.get("value", "")
    for f in form.find_all("input")
    if f.get("name")
}
payload["email"] = os.environ["APP_EMAIL"]        # match the Payload tab
payload["password"] = os.environ["APP_PASSWORD"]

login = session.post(urljoin(resp.url, form.get("action") or resp.url), data=payload, timeout=30)
login.raise_for_status()

if "/login" in login.url:
    raise SystemExit("Login failed")  # lots of sites return 200 + the login page

page = session.get(f"{BASE}/account/orders", timeout=30)
soup = BeautifulSoup(page.text, "html.parser")
for row in soup.select("table.orders tbody tr"):
    print([td.get_text(strip=True) for td in row.find_all("td")])

If the token is in <meta name="csrf-token">, read it from there and send it as an X-CSRF-Token header instead.

To avoid logging in every run, dump session.cookies.get_dict() to a JSON file and load it back with session.cookies.update(...). Log in again only when it stops working.

Option 2: Playwright + storage_state

pip install playwright && playwright install chromium

When the login is JavaScript-driven, or the data only shows up after render, I use a real browser:

import os
from pathlib import Path
from playwright.sync_api import sync_playwright

BASE = "https://app.example.com"
STATE = Path("state.json")

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)

    if not STATE.exists():
        ctx = browser.new_context()
        page = ctx.new_page()
        page.goto(f"{BASE}/login")
        page.get_by_label("Email").fill(os.environ["APP_EMAIL"])
        page.get_by_label("Password").fill(os.environ["APP_PASSWORD"])
        page.get_by_role("button", name="Sign in").click()
        page.wait_for_url("**/dashboard")
        ctx.storage_state(path=STATE)
        ctx.close()

    ctx = browser.new_context(storage_state=STATE)
    page = ctx.new_page()
    page.goto(f"{BASE}/account/orders")
    print(page.locator("table.orders tbody tr").all_inner_texts())
    browser.close()

storage_state saves cookies and localStorage. It doesn't save sessionStorage, which bites you if the app keeps its token there.

2FA and SSO

Don't automate it. Log in by hand once:

page.goto(f"{BASE}/login")
input("Log in in the browser window, then press Enter")
ctx.storage_state(path=STATE)

Reuse state.json until the session dies, then do it again.

Option 3 (my default): call the JSON endpoint

Most logged-in apps build their pages from JSON calls. Asking for that JSON gives you named fields instead of table cells, and a redesign doesn't break your script.

1. DevTools → Network → Fetch/XHR.
2. Reload the page with the data.
3. Check the Preview tab of each request until you find the data. Cmd/Ctrl+F in the Network panel searches response bodies, so search for a value you can see on screen.
4. Note the URL, query params and any custom headers.

Then use the session you already have:

import time

items, page = [], 1
while True:
    r = session.get(f"{BASE}/api/orders", params={"page": page}, timeout=30)
    if r.status_code == 429:
        time.sleep(int(r.headers.get("Retry-After", "30")))
        continue
    if r.status_code in (401, 403):
        raise SystemExit("Session expired, log in again")
    r.raise_for_status()
    batch = r.json()["items"]
    if not batch:
        break
    items.extend(batch)
    page += 1
    time.sleep(1)

From Playwright, ctx.request.get(f"{BASE}/api/orders") sends the context's cookies for you. Or move the cookies over to requests:

import json

for c in json.loads(STATE.read_text())["cookies"]:
    session.cookies.set(c["name"], c["value"], domain=c["domain"], path=c["path"])

A quick way to prototype: right-click the request in DevTools → Copy as cURL, run it in a terminal, then convert it to Python (by hand, or with curlconverter). That command includes your cookies, so keep it out of Slack and issues.

Don't commit your session

state.json and cookies.json work like a password until they expire. Put them and .env in .gitignore before the first commit, read credentials from env vars, and use a separate test account for anything on a cron.

When it's your own app

Finding every endpoint by clicking through the Network tab gets old fast. For my own apps I use Netraptor, a Chrome extension I work on. It records the fetch/XHR calls the logged-in app makes and groups them into endpoints like GET /api/orders/{id}, with schemas and real responses. It can also crawl the same-origin links on a start page in your logged-in tab to find more (one level deep, and it skips logout/delete-style links). It copies requests as cURL, PowerShell or fetch (no Python, so curlconverter still helps) and exports Postman or HAR.

The full guide, with more on CSRF tokens, Selenium cookies and rate limits, is here: How to scrape a website that requires a login.