hil for SerpApi

Posted on Dec 12, 2023 • Originally published at serpapi.com

How to use curl in Python (and it's alternative)

#python #curl #webscraping

Curl is a handy tool for moving data through URLs. If you like using Python, you might want to use curl with it. But there's a better way to do what curl does inside Python. We'll show you how to handle data transfer in Python both with Curl and without Curl.

How to use cURL in Python?

We have two options to run cURL in the Python code:

- Using subprocess to run cURL. It's like using a command prompt (Terminal) from Python.

- Using pycurl. It's a library that acts as an interface between Python and libcurl.

Even though we can run cURL with those options, it's not the recommended way if you just want to fetch data or request a web page. There's a better alternative like: requests or built-in http.client.

If you're interested in how to utilize cURL for web scraping and its basic commands, feel free to read this post.

Here are the links to the tools mentioned:

- subprocess python

- pycurl

- requests

- http.client

Let's take a look at how to use curl with each of the methods above.

How to use curl with Python's subprocessor

Here is an example of how you can use curl using the subprocess.

import subprocess

# The command you would type in the terminal
command = ["curl", "https://www.example.com"]

# Run the command
result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)

# Get the output and error message (if any)
output = result.stdout
error = result.stderr

# Check if it was successful
if result.returncode == 0:
    print("Success:")
    print(output)
else:
    print("Error:")
    print(error)

How to use cURL with pycurl library

pycurl is a Python interface to the libcurl library, which is what curl is used under the hood. It allows you to use curl's capabilities directly within Python. Here's a simple example of how to use pycurl to perform a GET request to a given URL and print the response:

import pycurl
from io import BytesIO

# The URL to send the request to
url = "https://www.example.com"

# Create a BytesIO object to capture the output
buffer = BytesIO()

# Initialize a pycurl object
c = pycurl.Curl()

# Set the URL
c.setopt(c.URL, url)

# Write the output to the BytesIO buffer
c.setopt(c.WRITEDATA, buffer)

# Perform the request
c.perform()

# Get the HTTP response code
http_code = c.getinfo(c.RESPONSE_CODE)

# Close the pycurl object
c.close()

# Get the content from the BytesIO buffer
body = buffer.getvalue().decode('utf-8')

# Check if it was successful
if http_code == 200:
    print("Success:")
    print(body)
else:
    print("Error:")
    print(http_code)

There's a high possibility that you'll face HTTPS/SSL issues when using Pycurl. It's recommended to install and use certifi alongside pycurl.

Now that we've seen two "raw" methods of using cURL with Python. Now let's take a look at the easy way to request a web page using requests.

Using `requests` as an alternative to cURL

"Requests is an elegant and simple HTTP library for Python, built for human beings." - requests documentation.

We can perform a GET, POST, or other HTTP request in Python requests instead of cURL. It has an easy-to-use interface to work with.

Is the library requests using curl under the hood?

No, the requests library in Python does not use curl or libcurl under the hood. Instead, requests is built on top of urllib3, which is another Python library that provides HTTP client functionality.

Online tool converter

I found an interactive online tool where you can convert cURL command to any programming language including Python, which will convert the command intro requests library: https://curlconverter.com/

Here are a few requests code samples

Basic GET request in `requests`

import requests

response = requests.get('http://example.com')

It's equivalent to: curl example.com

I'll skip the import requests for the rest of the sample.

Save the raw HTML in a file

Since we're writing in a programming language, we now have the flexibility to mix and match this code with other actions as well. For example, in order to save a file, we can use the open and write method from Python.

response = requests.get('http://example.com')

with open('index.html', 'wb') as f:
    f.write(response.content)

Equivalent to curl example.com -o "index.html"

Send a post request

Here's how we can send a post request with the requests library. We can slowly start seeing the benefits of using this library.


data = {
    'param1': 'value1',
    'param2': 'value2',
}

response = requests.post('http://example.com', data=data)

Equivalent to: curl -d "param1=value1&param2=value2" -X POST http://example.com

Requests with authentication

from requests.auth import HTTPBasicAuth

basic = HTTPBasicAuth('user', 'pass')
requests.get('https://httpbin.org/basic-auth/user/pass', auth=basic)

Equivalent to: curl -u username:password http://example.com

Send a JSON data

headers = {
    # Already added when you pass json=
    # 'Content-Type': 'application/json',
    'Accept': 'application/json',
}

json_data = {
    'tool': 'curl',
}

response = requests.post('https://example.com/', headers=headers, json=json_data)

Equivalent to: curl --json '{"tool": "curl"}' https://example.com/

Any header information can be put in the header key.


headers = {
    'X-Custom-Header': 'value',
}

response = requests.get('http://example.com', headers=headers)

Equivalent to: curl -H "X-Custom-Header: value" http://example.com

Simulate using a user agent

Like above, user agent information can be attached to the headers.


headers = {
    'User-Agent': 'User-Agent-String',
}

response = requests.get('http://example.com', headers=headers)

Equivalent to: curl -A "User-Agent-String" http://example.com

Using proxies

The requests library has the proxies\ key as a parameter.


proxies = {
    'http': 'http://proxyserver:port',
    'https': 'http://proxyserver:port',
}

response = requests.get('http://example.com', proxies=proxies)

Equivalent to: curl -x http://proxyserver:port http://example.comacts

What's the benefit of using requests instead of curl?

Using requests in Python instead of curl for making HTTP requests has several benefits:

Ease of Use: requests is a Python library, making it more intuitive to use within a Python environment. Its syntax is straightforward and user-friendly, especially for those already familiar with Python. This contrasts with curl, which is a command-line tool requiring a different syntax that can be less intuitive for those not accustomed to command-line interfaces.
Readability: Code written using requests is generally more readable and maintainable. This is particularly beneficial in larger projects or when working in a team, as it makes it easier for others to understand and contribute to the code.
Integration with Python Code: Since requests is a Python library that integrates seamlessly with other Python code and features. This includes error handling through exceptions, integration with Python's debugging tools, and compatibility with other Python libraries.
Support for Pythonic Idioms: requests supports Pythonic idioms, like context managers for session management, which makes it more in line with common Python practices.
Flexibility and Features: requests offer a high degree of flexibility and comes with many features out of the box, like support for various authentication methods, connection pooling, and more.

While curl is a powerful tool with its own set of advantages, especially in non-Python environments or for quick command-line use, requests is often more suitable for Python applications due to its ease of use, readability, and integration with Python's ecosystem.