DEV Community

Cover image for Mastering File Retrieval in Linux: wget vs. curl
BrendahKiragu
BrendahKiragu

Posted on

Mastering File Retrieval in Linux: wget vs. curl

Introduction

When it comes to retrieving files and interacting with the web from the command line, wget and curl are two of the most powerful tools in Linux. Both allow users to download files, fetch web pages, and interact with APIs, but they serve different purposes.

In this blog, we’ll break down how to use wget and curl, their differences, and real-world use cases to help you choose the right tool for your needs.

Table of Contents

  1. What is wget?
  2. Using wget
  3. What is curl?
  4. Using curl
  5. Comparison: wget vs. curl
  6. Bonus: Installing AWS CLI with curl
  7. Conclusion

What is wget?

wget is a command-line utility used to download files from the web. It supports HTTP, HTTPS, and FTP protocols and is best for downloading files in the background or recursively retrieving entire directories.

Using wget

Basic File Download with wget

To download a file from a URL:

wget https://example.com/file.zip
Enter fullscreen mode Exit fullscreen mode

Resuming Interrupted Downloads

If your download was interrupted, you can resume it with:

wget -c https://example.com/large-file.zip
Enter fullscreen mode Exit fullscreen mode

Downloading an Entire Website

To mirror a website for offline access:

wget -r --no-parent https://example.com/
Enter fullscreen mode Exit fullscreen mode

What is curl?

curl is a versatile tool used for transferring data using a wide range of protocols (HTTP, HTTPS, FTP, SCP, and more). Unlike wget, it is commonly used for API interactions and sending requests beyond just downloading files.

Using curl

Basic File Download with curl

To download a file from a URL:

curl -O https://example.com/file.zip
Enter fullscreen mode Exit fullscreen mode

Saving Files with Custom Names 

curl -o custom_name.zip https://example.com/file.zip
Enter fullscreen mode Exit fullscreen mode

Making API Requests

GET Request 

curl https://api.example.com/data
Enter fullscreen mode Exit fullscreen mode

POST Request with JSON Data 

curl -X POST -H "Content-Type: application/json" -d '{"name":"Brenda"}' https://api.example.com/users
Enter fullscreen mode Exit fullscreen mode

Comparison: wget vs. curl

Feature wget curl
Download files ✅ ✅
Resume downloads ✅ ❌
Recursive downloads ✅ ❌
API requests (POST, PUT) ❌ ✅
Supports multiple protocols ✅ ✅
Works in the background ✅ ❌

Bonus: Installing AWS CLI with curl

Let try a real life example of curl We will install the AWS Command Line Interface (CLI) on a Linux system. Here’s how you can do it:

# Step 1: Download the AWS CLI installation file
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

# Step 2: Unzip the installation file
unzip awscliv2.zip

# Step 3: Run the installation program
sudo ./aws/install
Enter fullscreen mode Exit fullscreen mode

Steps Explained:

  1. Download the AWS CLI installation file using curl. The -o option specifies the file name that the downloaded package is written to (in this case, awscliv2.zip).
  2. Unzip the installation file. When the file is unzipped, a directory named aws is created under the current directory.
  3. Run the installation program. The installation command uses a file named install in the newly unzipped aws directory.

Verifying Installation

After installation, confirm that the AWS CLI is installed correctly by running:

aws --version
Enter fullscreen mode Exit fullscreen mode

This should return the installed AWS CLI version, ensuring that everything is set up properly.

Conclusion

Both wget and curl are essential tools for Linux users. If you need to download files, especially recursively, wget is the way to go. However, if you're working with APIs and need to send HTTP requests, curl is the better choice.

By mastering both tools, you’ll enhance your ability to interact with the web from the command line efficiently. Happy coding! 🚀

Top comments (0)