DEV Community

Bastien ANTOINE
Bastien ANTOINE

Posted on

Http Requests Cheatsheet

#cheatsheet #httprequests

Note: this article was originally published on my personal blog. Check it out here.

When working around APIs, whether it is building or playing with, you often end up making API requests over and over that all look alike, and yet there are some stuff that you can't remember. Like how would you provide form data with cURL? Or basic authentication with requests? This article serves as a cheatsheet on all those common stuff.

Selecting the HTTP method

requests (Python)

3 possible ways:

  1. Using the requests.request method: 
resp = requests.request('GET', 'https://httpbin.org/get')
resp = requests.request('POST', 'https://httpbin.org/post')
resp = requests.request('PUT', 'https://httpbin.org/put')
resp = requests.request('PATCH', 'https://httpbin.org/patch')
resp = requests.request('DELETE', 'https://httpbin.org/delete')
resp = requests.request('HEAD', 'https://httpbin.org/get')
Enter fullscreen mode Exit fullscreen mode
  1. Using the requests.<method> methods: 
resp = requests.get('https://httpbin.org/get')
resp = requests.post('https://httpbin.org/post')
resp = requests.put('https://httpbin.org/put')
resp = requests.patch('https://httpbin.org/patch')
resp = requests.delete('https://httpbin.org/delete')
resp = requests.head('https://httpbin.org/get')
Enter fullscreen mode Exit fullscreen mode
  1. Using a prepared request: 
req = requests.Request('GET', 'https://httpbin.org/get')
r = req.prepare()
s = requests.Session()
s.send(r)
Enter fullscreen mode Exit fullscreen mode

From requests.PreparedRequest documentation.

cURL 

curl -X GET https://httpbin.org/get
curl -X POST https://httpbin.org/post
curl -X PUT https://httpbin.org/put
curl -X PATCH https://httpbin.org/patch
curl -X DELETE https://httpbin.org/delete
Enter fullscreen mode Exit fullscreen mode

If the -X/--request is not provided, cURL will default to GET.

[doc of -X/--request option]

Note on the HEAD requests: to make a proper HEAD request (ie. requesting only the headers), the -X HEAD option will not work, you will need to use -I/--head instead:

> curl -X HEAD https://httpbin.org/get
Warning: Setting custom HTTP method to HEAD with -X/--request may not work the
Warning: way you want. Consider using -I/--head instead.
curl: (18) transfer closed with 255 bytes remaining to read
> curl -I https://httpbin.org/get
HTTP/2 200
date: Sun, 19 Dec 2021 11:03:31 GMT
content-type: application/json
content-length: 255
server: gunicorn/19.9.0
access-control-allow-origin: *
access-control-allow-credentials: true
Enter fullscreen mode Exit fullscreen mode

(Tested on cURL 7.64.0 and 7.77.0)

Headers

requests (Python) 

requests.get(
  'https://httpbin.org/headers',
  headers={
      'X-My-Header': 'Value',
  }
)
Enter fullscreen mode Exit fullscreen mode

[doc of headers parameter]

cURL 

curl -H 'X-My-Header: Value' https://httpbin.org/headers
Enter fullscreen mode Exit fullscreen mode

[doc of -H/--headers option]

Authentication

HTTP Basic

requests (Python)

2 possible ways:

  1. Using the auth option 
requests.get(
  'https://httpbin.org/basic-auth/user/passwd',
  auth=('user', 'passwd')
)
Enter fullscreen mode Exit fullscreen mode
  1. Using the headers option 
requests.get(
  'https://httpbin.org/basic-auth/user/passwd',
  headers={
      'Authorization': 'Basic dXNlcjpwYXNzd2QK',
  }
)

from base64 import b64encode
requests.get(
  'https://httpbin.org/basic-auth/user/passwd',
  headers={
      'Authorization': f'Basic {b64encode('user:passwd')}',
  }
)
Enter fullscreen mode Exit fullscreen mode

Note on using this along with a .netrc file: Authorization headers that are provided this way will be overriden by credentials set in the .netrc file [doc].

As a workaround you can either temporarily comment the authentication part in your .netrc, set the NETRC env var to an empty file, or temporarily set requests.utils.NETRC_FILES = [] [source code] .

cURL

2 possible ways:

  1. Using the -u/--user option 
curl -u user:passwd https://httpbin.org/basic-auth/user/passwd
Enter fullscreen mode Exit fullscreen mode

Few things to note when using this option:

  • cURL will split the username and password on the first colon. This means you can't have one in the username, but you can in the password. If needed, use the solution below with the -H/--header option.
  • If only the username is provided, cURL will prompt for the password before sending the request
  1. Using the -H/--header option 
curl -H 'Authorization: Basic dXNlcjpwYXNzd2QK' https://httpbin.org/basic-auth/user/passwd
curl -H "Authorization: Basic $(openssl base64 <<< user:passwd)" https://httpbin.org/basic-auth/user/passwd
Enter fullscreen mode Exit fullscreen mode

HTTP Bearer

requests (Python) 

requests.get(
  'https://httpbin.org/bearer',
  headers={
      'Authorization': 'Basic dGhpc0lzQUJlYXJlclRva2VuCg==',
  }
)
Enter fullscreen mode Exit fullscreen mode

cURL 

curl -H 'Authorization: Bearer dGhpc0lzQUJlYXJlclRva2VuCg==' https://httpbin.org/bearer
Enter fullscreen mode Exit fullscreen mode

Query parameters

requests (Python) 

requests.get(
  'https://httpbin.org/anything',
  params={
      'arg1': 'val1',
      'arg2': 'val2',
  }
)
Enter fullscreen mode Exit fullscreen mode

[doc of params parameter]

cURL

No useful options here unfortunately

curl https://httpbin.org/anything?arg1=val1&arg2=val2
Enter fullscreen mode Exit fullscreen mode

Body parameters

requests (Python)

There are multiple ways to provide a body, each with their side consequences:

  1. Providing a dict in the data parameter: this will set the Content-Type header to application/x-www-form-urlencoded, unless told otherwise by the headers parameter. 
requests.post(
  "https://httpbin.org/post",
  data={'key1': 'value1', 'key2': 'value2'},
)
Enter fullscreen mode Exit fullscreen mode
  1. Providing a raw string in the data parameter: this will not set any value to the Content-Type header. 
requests.post(
  "https://httpbin.org/post",
  data='theBodyOfMyRequest',
)
Enter fullscreen mode Exit fullscreen mode
  1. Providing a dict in the json parameter: this will dumps the provided payload as JSON and set the Content-Type header to application/json, unless told otherwise by the headers parameter. The following two request are equivalents: 
import json
requests.post(
  "https://httpbin.org/post",
  data=json.dumps({'key1': 'value1 and value2', 'key2': 'value2'}),
  headers={'Content-Type': 'application/json'},
)
requests.post(
  "https://httpbin.org/post",
  json={'key1': 'value1 and value2', 'key2': 'value2'},
)
Enter fullscreen mode Exit fullscreen mode

[doc]

cURL

cURL offers multiple options to send a request with a body, each with their side consequences as well.

  1. Using the -F/--form option 
curl -F key1=value1 -F key2=value2 https://httpbin.org/post
Enter fullscreen mode Exit fullscreen mode

Few things to note when using this option:

  • it forces the method to POST, unless told otherwise with -X/--request option
  • it sets the Content-Type header to multipart/form-data, unless told otherwise aswell with -H/--header option

Using this option, it is possible to send files or read input from a file by prefixing the filename with @ or <, respectively. This means that if you need to have either a @ or < in the key and/or the value, you need to use the --form-string option.

[doc of -F/--form option]

[doc of --form-string option]

  1. Using -d/--data option 
curl -d key1=value1 -d key2=value2 https://httpbin.org/post
Enter fullscreen mode Exit fullscreen mode

Few things to note when using this option:

  • it forces the method to POST, unless told otherwise with -X/--request option
  • it sets the Content-Type header to application/x-www-form-urlencoded, unless told otherwise aswell with -H/--header option

Using this option, it is possible to read input from a file by prefixing the filename with @. This means that if you need to have a @ in the key and/or the value you need to use the --data-raw option.

It is also possible to URL-encode the parameter using the --data-urlencode option.

[doc of -d/--data option]

[doc of --data-raw option]

[doc of --data-urlencode option]

Sending files

requests (Python) 

requests.post(
  'https://httpbin.org/post',
  files={'file': ('index.html', open('index.html', 'rb'), 'text/html', {'Expires': '0'})}
)
Enter fullscreen mode Exit fullscreen mode

[doc]

cURL 

curl -F "file=@index.html" https://httpbin.org/post
curl -F "file=@index.html;type=text/html" https://httpbin.org/post
Enter fullscreen mode Exit fullscreen mode

[doc of -F/--form option]

Disabling SSL verification

requests (Python) 

resp = requests.get(
    'https://httpbin.org/get',
    verify=False,
)
Enter fullscreen mode Exit fullscreen mode

[doc of verify parameter]

cURL 

curl -k https://httpbin.org/get
Enter fullscreen mode Exit fullscreen mode

[doc of -k/--insecure option]

3XX redirects

requests (Python)

requests default behavior is to follow redirect, unless making a HEAD request (see below for more informations). Use the parameter allow_redirects to disable it:

resp = requests.get(
    'https://httpbin.org/redirect-to?url=bastien-antoine.fr&status_code=302',
    allow_redirects=False,
)
Enter fullscreen mode Exit fullscreen mode

Note on redirects with HEAD requests: when making HEAD requests, requests will not follow 3XX redirects in most case, depending on how the request is made:

How the request is made Follow 3XX on default
requests.head(...) No
requests.Session(...).head(...) No
requests.request('HEAD', ...) Yes

cURL

cURL's default behavior is to not follow 3XX redirects. To enable this use the -L/--location option:

curl -L https://httpbin.org/redirect-to?url=bastien-antoine.fr&status_code=302
Enter fullscreen mode Exit fullscreen mode

[doc of -L/--location option]

Logging

requests (Python) 

import requests
import logging
import http.client

http.client.HTTPConnection.debuglevel = 1

logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True
Enter fullscreen mode Exit fullscreen mode

[source]

cURL 

curl -v https://httpbin.org/get
Enter fullscreen mode Exit fullscreen mode

Note that this option only shows requests headers and full response. If you want to see the whole process, including request body, checkout the --trace and --trace-ascii options.

[doc of -v/--verbose option]

Top comments (0)