Note: this article was originally published on my personal blog. Check it out here.
When working around APIs, whether it is building or playing with, you often end up making API requests over and over that all look alike, and yet there are some stuff that you can't remember. Like how would you provide form data with cURL? Or basic authentication with requests
? This article serves as a cheatsheet on all those common stuff.
- Selecting the HTTP method
- Headers
- Authentication
- Query parameters
- Body parameters
- Sending files
- Disabling SSL verification
- 3XX redirects
- Logging
Selecting the HTTP method
requests
(Python)
3 possible ways:
- Using the
requests.request
method:
resp = requests.request('GET', 'https://httpbin.org/get')
resp = requests.request('POST', 'https://httpbin.org/post')
resp = requests.request('PUT', 'https://httpbin.org/put')
resp = requests.request('PATCH', 'https://httpbin.org/patch')
resp = requests.request('DELETE', 'https://httpbin.org/delete')
resp = requests.request('HEAD', 'https://httpbin.org/get')
- Using the
requests.<method>
methods:
resp = requests.get('https://httpbin.org/get')
resp = requests.post('https://httpbin.org/post')
resp = requests.put('https://httpbin.org/put')
resp = requests.patch('https://httpbin.org/patch')
resp = requests.delete('https://httpbin.org/delete')
resp = requests.head('https://httpbin.org/get')
- Using a prepared request:
req = requests.Request('GET', 'https://httpbin.org/get')
r = req.prepare()
s = requests.Session()
s.send(r)
From requests.PreparedRequest
documentation.
cURL
curl -X GET https://httpbin.org/get
curl -X POST https://httpbin.org/post
curl -X PUT https://httpbin.org/put
curl -X PATCH https://httpbin.org/patch
curl -X DELETE https://httpbin.org/delete
If the -X/--request
is not provided, cURL will default to GET
.
Note on the HEAD
requests: to make a proper HEAD
request (ie. requesting only the headers), the -X HEAD
option will not work, you will need to use -I/--head
instead:
> curl -X HEAD https://httpbin.org/get
Warning: Setting custom HTTP method to HEAD with -X/--request may not work the
Warning: way you want. Consider using -I/--head instead.
curl: (18) transfer closed with 255 bytes remaining to read
> curl -I https://httpbin.org/get
HTTP/2 200
date: Sun, 19 Dec 2021 11:03:31 GMT
content-type: application/json
content-length: 255
server: gunicorn/19.9.0
access-control-allow-origin: *
access-control-allow-credentials: true
(Tested on cURL 7.64.0 and 7.77.0)
Headers
requests
(Python)
requests.get(
'https://httpbin.org/headers',
headers={
'X-My-Header': 'Value',
}
)
cURL
curl -H 'X-My-Header: Value' https://httpbin.org/headers
Authentication
HTTP Basic
requests
(Python)
2 possible ways:
- Using the
auth
option
requests.get(
'https://httpbin.org/basic-auth/user/passwd',
auth=('user', 'passwd')
)
- Using the headers option
requests.get(
'https://httpbin.org/basic-auth/user/passwd',
headers={
'Authorization': 'Basic dXNlcjpwYXNzd2QK',
}
)
from base64 import b64encode
requests.get(
'https://httpbin.org/basic-auth/user/passwd',
headers={
'Authorization': f'Basic {b64encode('user:passwd')}',
}
)
Note on using this along with a .netrc
file: Authorization headers that are provided this way will be overriden by credentials set in the .netrc
file [doc].
As a workaround you can either temporarily comment the authentication part in your .netrc
, set the NETRC
env var to an empty file, or temporarily set requests.utils.NETRC_FILES = []
[source code] .
cURL
2 possible ways:
- Using the
-u/--user
option
curl -u user:passwd https://httpbin.org/basic-auth/user/passwd
Few things to note when using this option:
- cURL will split the username and password on the first colon. This means you can't have one in the username, but you can in the password. If needed, use the solution below with the
-H/--header
option. - If only the username is provided, cURL will prompt for the password before sending the request
- Using the
-H/--header
option
curl -H 'Authorization: Basic dXNlcjpwYXNzd2QK' https://httpbin.org/basic-auth/user/passwd
curl -H "Authorization: Basic $(openssl base64 <<< user:passwd)" https://httpbin.org/basic-auth/user/passwd
HTTP Bearer
requests
(Python)
requests.get(
'https://httpbin.org/bearer',
headers={
'Authorization': 'Basic dGhpc0lzQUJlYXJlclRva2VuCg==',
}
)
cURL
curl -H 'Authorization: Bearer dGhpc0lzQUJlYXJlclRva2VuCg==' https://httpbin.org/bearer
Query parameters
requests
(Python)
requests.get(
'https://httpbin.org/anything',
params={
'arg1': 'val1',
'arg2': 'val2',
}
)
cURL
No useful options here unfortunately
curl https://httpbin.org/anything?arg1=val1&arg2=val2
Body parameters
requests
(Python)
There are multiple ways to provide a body, each with their side consequences:
- Providing a dict in the
data
parameter: this will set theContent-Type
header toapplication/x-www-form-urlencoded
, unless told otherwise by theheaders
parameter.
requests.post(
"https://httpbin.org/post",
data={'key1': 'value1', 'key2': 'value2'},
)
- Providing a raw string in the
data
parameter: this will not set any value to theContent-Type
header.
requests.post(
"https://httpbin.org/post",
data='theBodyOfMyRequest',
)
- Providing a dict in the
json
parameter: this will dumps the provided payload as JSON and set theContent-Type
header toapplication/json
, unless told otherwise by theheaders
parameter. The following two request are equivalents:
import json
requests.post(
"https://httpbin.org/post",
data=json.dumps({'key1': 'value1 and value2', 'key2': 'value2'}),
headers={'Content-Type': 'application/json'},
)
requests.post(
"https://httpbin.org/post",
json={'key1': 'value1 and value2', 'key2': 'value2'},
)
cURL
cURL offers multiple options to send a request with a body, each with their side consequences as well.
- Using the
-F/--form
option
curl -F key1=value1 -F key2=value2 https://httpbin.org/post
Few things to note when using this option:
- it forces the method to
POST
, unless told otherwise with-X/--request
option - it sets the
Content-Type
header tomultipart/form-data
, unless told otherwise aswell with-H/--header
option
Using this option, it is possible to send files or read input from a file by prefixing the filename with @
or <
, respectively. This means that if you need to have either a @
or <
in the key and/or the value, you need to use the --form-string
option.
- Using
-d/--data
option
curl -d key1=value1 -d key2=value2 https://httpbin.org/post
Few things to note when using this option:
- it forces the method to
POST
, unless told otherwise with-X/--request
option - it sets the
Content-Type
header toapplication/x-www-form-urlencoded
, unless told otherwise aswell with-H/--header
option
Using this option, it is possible to read input from a file by prefixing the filename with @
. This means that if you need to have a @
in the key and/or the value you need to use the --data-raw
option.
It is also possible to URL-encode the parameter using the --data-urlencode
option.
[doc of --data-urlencode
option]
Sending files
requests
(Python)
requests.post(
'https://httpbin.org/post',
files={'file': ('index.html', open('index.html', 'rb'), 'text/html', {'Expires': '0'})}
)
cURL
curl -F "file=@index.html" https://httpbin.org/post
curl -F "file=@index.html;type=text/html" https://httpbin.org/post
Disabling SSL verification
requests
(Python)
resp = requests.get(
'https://httpbin.org/get',
verify=False,
)
cURL
curl -k https://httpbin.org/get
3XX redirects
requests
(Python)
requests
default behavior is to follow redirect, unless making a HEAD
request (see below for more informations). Use the parameter allow_redirects
to disable it:
resp = requests.get(
'https://httpbin.org/redirect-to?url=bastien-antoine.fr&status_code=302',
allow_redirects=False,
)
Note on redirects with HEAD requests: when making HEAD
requests, requests
will not follow 3XX redirects in most case, depending on how the request is made:
How the request is made | Follow 3XX on default |
---|---|
requests.head(...) |
No |
requests.Session(...).head(...) |
No |
requests.request('HEAD', ...) |
Yes |
cURL
cURL's default behavior is to not follow 3XX redirects. To enable this use the -L/--location
option:
curl -L https://httpbin.org/redirect-to?url=bastien-antoine.fr&status_code=302
Logging
requests
(Python)
import requests
import logging
import http.client
http.client.HTTPConnection.debuglevel = 1
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True
cURL
curl -v https://httpbin.org/get
Note that this option only shows requests headers and full response. If you want to see the whole process, including request body, checkout the --trace
and --trace-ascii
options.
Top comments (0)