Mastering cURL: A Comprehensive Guide

What is cURL?

cURL stands for "Client URL." It is a free tool that lets you send and receive data from the internet using your computer's command line. It was created by a human named Daniel Stenberg in 1997. Yes, literally there were no Claude, ChatGPT, Gemini, or Grok at that time.

It works everywhere. You can use it on Windows, macOS, Linux, and other systems.

It speaks many languages and understands many internet protocols (ways computers talk to each other), such as HTTP, FTP, and email protocols.

By the way, if you're not sure what protocols are, check out my article explaining what a protocol is.

It is already on your computer. If you have a modern Windows (10/11) or Linux computer, cURL is probably already installed.

In simple terms, cURL is like a messenger that can fetch information from websites and online services. Instead of clicking buttons in a browser, you type commands to tell it what to get or send.

Mastering cURL is extensive on its own, and if you want to truly master it, I would prefer the original eBook everything curl rather than this article. However, as a developer, this article can serve as a one-stop solution. I always prefer to learn using a T-shaped approach.

T-shaped learning refers to a skill development model that combines deep expertise (the vertical bar of the "T") in a specific domain with broad knowledge across related disciplines (the horizontal bar). This approach enables individuals to leverage specialized skills while collaborating effectively across functions and solving complex, interdisciplinary problems.

Now, let's understand cURL.

cURL is purely command-line based (no GUI). This can be an advantage (for scripting and automation) or a drawback (due to the learning curve), depending on your needs. Its default output is printed to standard output (terminal), which can be piped or saved to files. Because it has no graphical interface, beginners must learn its options, but advanced users appreciate the level of control and scriptability it provides.

Why cURL Matters

Before diving into commands, understand why cURL is ubiquitous ( Ever-Present ) :

• Pre-installed: Ships with most Linux distributions, macOS, and Windows 10/11

• Protocol Support: HTTP/HTTPS, FTP/FTPS, SFTP, SCP, SMTP, IMAP, POP3, LDAP, TELNET, and more

• Scriptable: Perfect for automation, CI/CD pipelines, and testing

• Universal: Works identically across platforms (with minor syntax differences)

• Powerful: Handles authentication, cookies, certificates, proxies, rate limiting, and complex workflows

Installation Check

Verify cURL is installed:

curl --version

You should see output like:

curl 8.x.x (platform) libcurl/8.x.x ...
Protocols: dict file ftp ftps gopher http https imap imaps ...
Features: AsynchDNS HSTS HTTP2 HTTPS-proxy IPv6 SSL ...

Windows Note: In PowerShell, curl might be aliased to Invoke-WebRequest. Use curl.exe explicitly or remove the alias:

Remove-Item alias:curl

I know must of you are using Windows, don’t worry i am showing you my Linux output:

curl --version
curl 8.5.0 (x86_64-pc-linux-gnu) libcurl/8.5.0 OpenSSL/3.0.13 zlib/1.3 brotli/1.1.0 zstd/1.5.5 libidn2/2.3.7 libpsl/0.21.2 (+libidn2/2.3.7) libssh/0.10.6/openssl/zlib nghttp2/1.59.0 librtmp/2.3 OpenLDAP/2.6.7
Release-Date: 2023-12-06, security patched: 8.5.0-2ubuntu10.6
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Let’s see the Basic Operations

1. Simple GET Request

The most basic operation fetches a URL and prints it to stdout (terminal):

curl https://httpbin.org/get

What happens: cURL makes an HTTP GET request and displays the response body. The output will be:

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/8.5.0", 
    "X-Amzn-Trace-Id": "Root=1-696ef48b-4c3c88435e7fa591119e1c4f"
  }, 
  "origin": "160.250.255.74", 
  "url": "https://httpbin.org/get"
}

Quick Advice: While reading this article, please test the commands in your terminal and understand their output. If you get stuck, use AI for assistance, as I am not going to include all the outputs in this article, it would make it too long.

2. Saving Output

Save to a file with -o (lowercase):

bash

curl -o response.json https://httpbin.org/json

Or use -O (uppercase) to use the remote filename:

curl -O https://httpbin.org/image/png

This saves the file as png in the current directory.

3. Following Redirects

Many URLs redirect (HTTP 301/302). Follow them with -L:

curl -L https://github.com

Without -L, you'd just get the redirect response, not the final page.

4. Viewing Headers

See response headers with -I (HEAD request):

curl -I https://httpbin.org/get

Or use -i to include headers with the body:

curl -i https://httpbin.org/get

5. Verbose Mode

Debug your requests with -v:

curl -v https://httpbin.org/get

This shows:

• DNS resolution

• Connection establishment

• TLS handshake

• Request headers sent

• Response headers received

Pro tip: Use -v when troubleshooting API issues.

HTTP Methods

cURL defaults to GET, but supports all HTTP methods.

GET (Default)

curl https://jsonplaceholder.typicode.com/posts/1

POST with Form Data

Using -d automatically changes the method to POST:

curl -d "title=My Post&body=This is content&userId=1" \
  https://jsonplaceholder.typicode.com/posts

Important: When you use -d, cURL:

1. Changes method from GET to POST

2. Sets Content-Type: application/x-www-form-urlencoded

3. Sends data in the request body

POST with JSON

Specify JSON content type explicitly:

curl -X POST https://jsonplaceholder.typicode.com/posts \
  -H "Content-Type: application/json" \
  -d '{
    "title": "My Post",
    "body": "This is the content",
    "userId": 1
  }'

From file:

curl -X POST https://jsonplaceholder.typicode.com/posts \
  -H "Content-Type: application/json" \
  -d @data.json

The @ prefix tells cURL to read from a file.

PUT (Update)

curl -X PUT https://jsonplaceholder.typicode.com/posts/1 \
  -H "Content-Type: application/json" \
  -d '{
    "id": 1,
    "title": "Updated Title",
    "body": "Updated content",
    "userId": 1
  }'

PATCH (Partial Update)

curl -X PATCH https://jsonplaceholder.typicode.com/posts/1 \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Only updating title"
  }'

DELETE

curl -X DELETE https://jsonplaceholder.typicode.com/posts/1

Headers and Authentication

Custom Headers

Add headers with -H:

curl https://httpbin.org/headers \
  -H "User-Agent: MyApp/1.0" \
  -H "Accept: application/json" \
  -H "X-Custom-Header: value"

Basic Authentication

Use -u for HTTP Basic Auth:

curl -u username:password https://httpbin.org/basic-auth/username/password

Security Warning: Passwords in command history are visible. Better approaches:

Prompt for password:

curl -u username https://httpbin.org/basic-auth/username/password
# cURL will prompt for password

Bearer Token (OAuth/JWT)

curl https://api.github.com/user \
  -H "Authorization: Bearer YOUR_GITHUB_TOKEN"

Real example with GitHub API:

# Public endpoint (no auth needed)
curl https://api.github.com/users/github

# Authenticated endpoint (requires token)
curl -H "Authorization: Bearer ghp_your_token_here" \
  https://api.github.com/user

Working with Forms

URL-Encoded Form (Default)

curl -d "name=John&email=[e-mail]" \
  https://httpbin.org/post

Multipart Form (File Upload)

Use -F for multipart/form-data:

# Create a test file
echo "Sample content" > test.txt

# Upload it
curl -F "file=@test.txt" \
  -F "description=My test file" \
  https://httpbin.org/post

Multiple files:

curl -F "file1=@image.png" \
  -F "file2=@document.pdf" \
  -F "note=Uploading two files" \
  https://httpbin.org/post

Specify content type:

curl -F "data=@data.json;type=application/json" \
  https://httpbin.org/post

Download Controls

Resume Interrupted Downloads

curl -C - -O https://releases.ubuntu.com/22.04/ubuntu-22.04.3-desktop-amd64.iso

The -C - tells cURL to automatically figure out where to resume.

Rate Limiting

Limit bandwidth to avoid saturating your connection:

# Limit to 100 KB/s
curl --limit-rate 100k -O https://httpbin.org/image/png

# Limit to 1 MB/s
curl --limit-rate 1m -O https://speed.hetzner.de/100MB.bin

Timeouts

Set maximum time for the operation:

# Maximum 10 seconds for entire operation
curl -m 10 https://httpbin.org/delay/5

# Connection timeout only
curl --connect-timeout 5 https://httpbin.org/delay/1

Progress Bar

Use -# for a simple progress bar instead of the default meter:

curl -# -O https://httpbin.org/image/png

Use -s (silent) to suppress all output:

curl -s https://httpbin.org/get > output.json

Cookies

Save Cookies

curl -c cookies.txt https://httpbin.org/cookies/set?session=abc123

Send Cookies

curl -b cookies.txt https://httpbin.org/cookies

Set Cookie Directly

curl -b "session=abc123" https://httpbin.org/cookies

Output Formatting and Processing

Extract Specific Data with -w

The --write-out or -w option lets you extract transfer information:

# Get HTTP status code
curl -s -o /dev/null -w "%{http_code}" https://httpbin.org/status/200

# Get response time
curl -s -o /dev/null -w "Time: %{time_total}s\n" https://httpbin.org/delay/1

# Multiple variables
curl -s -o /dev/null -w "Status: %{http_code}\nTime: %{time_total}s\n" \
  https://httpbin.org/get

Useful variables:

• %{http_code} - HTTP status code

• %{time_total} - Total time in seconds

• %{time_connect} - Time to establish connection

• %{size_download} - Bytes downloaded

• %{speed_download} - Download speed (bytes/sec)

• %{url_effective} - Final URL (after redirects)

Parse JSON with jq

# Install jq first (package manager)
curl -s https://api.github.com/users/github | jq '.name, .bio'

# Extract specific field
curl -s https://jsonplaceholder.typicode.com/posts/1 | jq '.title'

# Filter array
curl -s https://jsonplaceholder.typicode.com/posts | jq '.[0:3] | .[] | .title'

I know you got confused with these commands, don’t worry i still don’t remember these commands. Treat this article like a cheat sheet.

Let’s see few Real-World Examples

1. Health Check Script

#!/bin/bash
URL="https://httpbin.org/status/200"
STATUS=$(curl -s -o /dev/null -w "%{http_code}" $URL)

if [ $STATUS -eq 200 ]; then
  echo "Service is up (HTTP $STATUS)"
  exit 0
else
  echo "Service is down (HTTP $STATUS)"
  exit 1
fi

In the above code curl is responsible only for making the HTTP request and returning the status code.

The rest of the logic (variable assignment, condition checking, if statement, and exit codes) is handled by Bash. It is an scripting language.

2. API Testing with Multiple Endpoints

#!/bin/bash
ENDPOINTS=(
  "https://jsonplaceholder.typicode.com/posts/1"
  "https://jsonplaceholder.typicode.com/users/1"
  "https://jsonplaceholder.typicode.com/comments/1"
)

for endpoint in "${ENDPOINTS[@]}"; do
  echo "Testing: $endpoint"
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$endpoint")
  echo "Response: HTTP $STATUS"
  echo "---"
done

3. Bulk Download

#!/bin/bash
# Download multiple files from a list
while IFS= read -r url; do
  curl -O "$url"
done < urls.txt

4. POST Data from CSV

#!/bin/bash
# Read CSV and POST each row
tail -n +2 users.csv | while IFS=, read -r name email; do
  curl -X POST https://httpbin.org/post \
    -H "Content-Type: application/json" \
    -d "{\"name\":\"$name\",\"email\":\"$email\"}"
  echo "Posted: $name"
done

5. Monitor API Performance

#!/bin/bash
for i in {1..10}; do
  TIME=$(curl -s -o /dev/null -w "%{time_total}" https://httpbin.org/get)
  echo "Request $i: ${TIME}s"
  sleep 1
done

6. Webhook Testing

# Test a webhook endpoint
curl -X POST https://webhook.site/your-unique-id \
  -H "Content-Type: application/json" \
  -d '{
    "event": "user.created",
    "data": {
      "id": 123,
      "name": "John Doe"
    },
    "timestamp": "'$(date -u +%Y-%m-%dT%H:%M:%SZ)'"
  }'

Error Handling and Exit Codes

Understanding Exit Codes

cURL sets exit codes to indicate success or failure:

• 0 - Success

• 1 - Unsupported protocol

• 3 - URL malformed

• 6 - Couldn't resolve host

• 7 - Failed to connect

• 22 - HTTP error (4xx/5xx)

• 28 - Operation timeout

Fail on HTTP Errors

Use -f to make cURL exit with error on HTTP errors:

curl -f https://httpbin.org/status/404
echo "Exit code: $?"  # Will be 22

Silent Fail

curl -sf https://httpbin.org/status/500 || echo "Request failed"

Retry on Failure

# Retry up to 3 times with 5 second delay
curl --retry 3 --retry-delay 5 https://httpbin.org/status/500

Configuration File

Create ~/.curlrc to set default options:

# ~/.curlrc
user-agent = "MyApp/1.0"
header = "Accept: application/json"
location
silent
show-error

Now these options apply to all curl commands. Override with -: (disable .curlrc).

Advanced Protocols

FTP Download

curl ftp://ftp.gnu.org/README

FTP Upload

curl -T localfile.txt ftp://ftp.example.com/remote/path/ \
  -u username:password

SFTP (SSH)

curl -u username sftp://example.com/path/to/file

Send Email (SMTP)

curl smtp://mail.example.com:587 \
  --mail-from [e-mail] \
  --mail-rcpt [e-mail] \
  --user 'username:password' \
  -T email.txt

Where email.txt contains:

From: [e-mail]
To: [e-mail]
Subject: Test Email

This is the email body.

Comparing cURL with Alternatives

cURL vs Postman

cURL Advantages:

• Already installed everywhere

• Perfect for scripting and automation

• Lightweight and fast

• Works on servers without GUI

• Version control friendly (commands are text)

Postman Advantages:

• Visual interface for beginners

• Request collections and environments

• Built-in testing framework

• Team collaboration features

• Auto-generated code snippets

Use cURL when: automating, scripting, CI/CD, quick tests, server environments

Use Postman when: exploring APIs, team collaboration, complex test suites, documentation

cURL vs HTTPie

HTTPie is a modern CLI alternative designed for human friendliness:

# cURL
curl -X POST https://httpbin.org/post \
  -H "Content-Type: application/json" \
  -d '{"name":"John"}'

# HTTPie
http POST httpbin.org/post name=John

HTTPie Advantages:

• Simpler syntax

• Colorized output by default

• Pretty-printed JSON

• Intuitive form/JSON handling

cURL Advantages:

• Pre-installed everywhere

• More protocols (FTP, SMTP, etc.)

• More mature and stable

• Wider adoption in documentation

Best Practices

1. Don't Put Secrets in Commands

Bad:

curl -u admin:password123 https://api.example.com

Good:

curl -u admin https://api.example.com  # Prompts for password
# OR
curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com

2. Use -f in Scripts

Always use -f to catch HTTP errors:

if curl -f -s https://api.example.com/health > /dev/null; then
  echo "Service healthy"
else
  echo "Service down"
  exit 1
fi

3. Set User-Agent

Identify your application:

curl -A "MyApp/1.0 ([e-mail])" https://api.example.com

4. Handle Redirects

Always use -L unless you specifically want to see redirects:

curl -L https://bit.ly/some-shortlink

5. Timeout Operations

Prevent hanging:

curl -m 30 https://slow-api.example.com  # Max 30 seconds

6. Validate Certificates in Production

Never use -k in production:

# Development only
curl -k https://localhost:8443

# Production
curl https://api.example.com  # Validates certificates

7. Use Configuration Files

For repeated options, use .curlrc:

# ~/.curlrc for API testing
location
silent
show-error
max-time = 30
retry = 3

Troubleshooting Guide

Problem: "Could not resolve host"

Cause: DNS resolution failure

Solution:

# Check DNS
nslookup example.com

# Use different DNS
curl --dns-servers 8.8.8.8 https://example.com

# Check for typos in URL
curl -v https://exampl.com  # Missing 'e'

Problem: "SSL certificate problem"

Cause: Invalid/expired/self-signed certificate

Solution:

# See certificate details
curl -v https://expired.badssl.com 2>&1 | grep -A 5 "SSL certificate"

# For testing only
curl -k https://self-signed.example.com

# For production, fix the certificate

Problem: "Empty reply from server"

Cause: Server closed connection

Solution:

# Check if server is running
curl -v http://localhost:8080

# Try different protocol version
curl --http1.1 https://example.com

Problem: "Connection refused"

Cause: Nothing listening on that port

Solution:

# Verify port is correct
netstat -tuln | grep 8080

# Check firewall
telnet example.com 8080

Quick Reference

Essential Commands (Just learn this commands), use all the above commands just to understand the cURL.

# Basic GET
curl https://example.com

# Save to file
curl -o file.html https://example.com

# Follow redirects
curl -L https://bit.ly/short-url

# POST JSON
curl -X POST https://api.example.com/data \
  -H "Content-Type: application/json" \
  -d '{"key":"value"}'

# Upload file
curl -F "file=@document.pdf" https://example.com/upload

# Basic auth
curl -u user:pass https://example.com

# Bearer token
curl -H "Authorization: Bearer TOKEN" https://api.example.com

# View headers
curl -I https://example.com

# Verbose output
curl -v https://example.com

# Silent mode
curl -s https://example.com

# Get HTTP status
curl -s -o /dev/null -w "%{http_code}" https://example.com

Common Options

Option	Long Form	Description
-o	--output	Write to file
-O	--remote-name	Use remote filename
-L	--location	Follow redirects
-d	--data	Send POST data
-F	--form	Multipart form data
-H	--header	Custom header
-u	--user	Authentication
-x	--proxy	Use proxy
-v	--verbose	Verbose output
-s	--silent	Silent mode
-i	--include	Include headers in output
-I	--head	HEAD request only
-X	--request	Specify method
-f	--fail	Fail on HTTP errors
-k	--insecure	Skip certificate verification
-w	--write-out	Output format

Finally Conclusion after a decades of reading:

cURL is an indispensable tool for developers, system administrators, and DevOps engineers. Its power lies in its ubiquity (ever-present ), flexibility, and scriptability. While it has a learning curve compared to GUI tools, mastering cURL pays dividends in automation capabilities and troubleshooting speed.

Next Steps:

• Read the official documentation at everything.curl.dev

• Practice with public APIs like httpbin.org and JSONPlaceholder

• Create a .curlrc file with your preferred defaults

• Build scripts that integrate cURL into your workflow

Remember: cURL is a tool that grows with you. Start with basic GET requests, gradually add headers and authentication, then move to automation and complex workflows. The command line might seem daunting at first, but cURL's consistency and power make it worth the investment.

Happy curling! and Huge thanks to read this article because in this era of AI, almost nobody wants to read this type of verbose article