curl

zhenye yu — Fri, 19 Jun 2026 07:57:30 +0000

From Browser cURL to Clean Python Requests Code, Step by Step

zhenye yu — Fri, 19 Jun 2026 07:53:45 +0000

If you've ever opened your browser's DevTools, right-clicked a request in the Network tab, and hit Copy as cURL, you already know the good part: you've captured exactly what the browser sent — headers, cookies, auth, body and all.

The annoying part is what comes next. You wanted Python, and instead you're holding a 30-line shell command full of -H flags and escaped quotes.

This post walks through porting a real curl command to requests by hand, one flag at a time, so the mapping is clear. Do it once like this and you'll recognize every piece next time.

The raw cURL

Here's a representative command copied straight out of Chrome:

curl 'https://api.example.com/v2/search?lang=en&page=2' \
  -H 'accept: application/json' \
  -H 'authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJ1aWQiOjQyfQ.sig' \
  -H 'content-type: application/json' \
  -H 'referer: https://app.example.com/dashboard' \
  -H 'user-agent: Mozilla/5.0' \
  -b 'session=abc123; theme=dark' \
  --data-raw '{"q":"nginx","filters":["open"]}'

Let's take it apart.

1. Method and URL

There is no -X flag here, but there is a body (--data-raw). That detail decides the method: --data / --data-raw makes curl use POST by default, unless you explicitly override the method with flags like -X GET. With -G, curl moves the data into the query string instead.

So don't read the missing -X as "GET" — the presence of a body is what matters.

The URL also carries a query string, ?lang=en&page=2. We'll handle that as real parameters in a moment.

import requests

url = "https://api.example.com/v2/search"

2. Headers → a dict

Every -H 'key: value' becomes one entry in a headers dict. The one rule that bites people: split each header on the first colon, not every colon.

Header values can legitimately contain colons. Look at the referer here, whose value is a full URL: https://app.example.com/dashboard. A blanket split on : would mangle it into pieces; split(":", 1) keeps the value intact.

headers = {
    "accept": "application/json",
    "authorization": "Bearer eyJhbGciOiJIUzI1NiJ9.eyJ1aWQiOjQyfQ.sig",
    "content-type": "application/json",
    "referer": "https://app.example.com/dashboard",
    "user-agent": "Mozilla/5.0",
}

3. Cookies (`-b`) → their own dict

The -b flag is a semicolon-separated cookie string. Don't jam it into the headers dict as a raw Cookie line — hand it to requests as a proper cookies argument and let the library do the encoding:

cookies = {"session": "abc123", "theme": "dark"}

4. The JSON body: `json=` vs `data=`

The content-type is application/json, so the body is JSON. In requests, that means you pass it as json= — not data=:

payload = {"q": "nginx", "filters": ["open"]}

Why the distinction matters:

json=payload serializes the dict to a JSON string and sets Content-Type: application/json for you when you do not override it.
data=payload form-encodes it as q=nginx&filters=open, which a JSON endpoint will reject or misread.

If the original request had been application/x-www-form-urlencoded, you'd flip to data=. Reading the content type tells you which one to use.

5. Query params

You could leave ?lang=en&page=2 glued onto the URL string, but pulling it into a params dict is cleaner and lets requests handle the encoding:

params = {"lang": "en", "page": "2"}

One caution: if the params in the original URL are already percent-encoded — you see %20, %3D, etc. — don't decode them and pass them through params as well, or you'll double-encode. Either keep the encoded values in the URL string, or pass the decoded values through params, but not both.

6. Putting it together

import requests

url = "https://api.example.com/v2/search"

params = {"lang": "en", "page": "2"}

headers = {
    "accept": "application/json",
    "authorization": "Bearer eyJhbGciOiJIUzI1NiJ9.eyJ1aWQiOjQyfQ.sig",
    "content-type": "application/json",
    "referer": "https://app.example.com/dashboard",
    "user-agent": "Mozilla/5.0",
}

cookies = {"session": "abc123", "theme": "dark"}

payload = {"q": "nginx", "filters": ["open"]}

resp = requests.post(
    url,
    params=params,
    headers=headers,
    cookies=cookies,
    json=payload,
)

resp.raise_for_status()
print(resp.json())

Because json= sets the content type for you when you do not override it, you could often drop the explicit content-type header and get the same request on the wire.

The gotchas that eat an afternoon

A few things that don't show up until they break:

Assuming GET because there's no -X. If --data / --data-raw is present, curl uses POST by default unless you override it with -X GET or use -G to move data into the query string.
json= vs data=. The single most common mistake. Match it to the content type.
Multipart uploads (-F). These map to files=, not data=. Mixing them up gives you a malformed boundary and a baffling 400.
Splitting headers on every colon. Split once; values like URLs and timestamps contain colons of their own.
Double-encoding query params. Don't both decode the URL and pass the same values through params.
Compression headers. requests handles common decompression for you, and brotli support depends on your installed stack, so you can usually drop accept-encoding unless you are testing compression behavior specifically.

When you just want it done

Doing this by hand once is worth it — now you know what each flag maps to. After that, it's mostly tax, especially for requests with twenty-plus headers.

For the repetitive cases I keep a small browser-based converter open: paste the cURL, get requests code back. It runs entirely client-side, so the command — and any live auth tokens in it — never leaves the browser:

curl to Python requests converter

If you're dropping the request into a crawler rather than a one-off script, there's also a low-key Feapder spider mode that emits a ready-to-run spider class.

What's the gnarliest cURL you've had to port to Python by hand? Mine had a multipart body and eleven cookies. Curious what trips other people up most — json= and data=, or something else?

cron

zhenye yu — Fri, 19 Jun 2026 06:10:02 +0000

DEV Community: zhenye yu