Build your own curl in Golang

In this tutorial, we'll walk through the process of creating a simple command-line tool similar to curl using Go and Cobra, a CLI library for Go.

Disclaimer: Here we are going to use the net package instead of the http that is available natively in Go. The reason for using net is to get a bit into the basics of creating a HTTP request to a server from scrach. You could easily use http package to enjoy all that stuff that is a pain to make from scratch. For instance, http already handles HTTPS requests.

It is a challenge because handling TCP connections and HTTP requests can be complex, but we'll keep it simple and focus on the basics. But will be a good start to understand how curl works under the hood and how to build a simple HTTP client using Go.

Prerequisites

Before we begin, make sure you have the following installed:

• Go (version 1.22)

Setting Up Your Project

First, let's create a new Go module for our project:

mkdir build-your-own-curl
cd build-your-own-curl
go mod init build-your-own-curl

Next, let's install Cobra:

go get -u github.com/spf13/cobra/cobra

Now, let's initialize Cobra in our project:

cobra-cli init

This will create the structure for our CLI application with the following files:

├── cmd/
│   ├── root.go
├── go.mod
├── go.sum
├── main.go
├── LICENSE

The cmd directory contains the root command file, which is where we'll define our whole application. It is also possible to create subcommands in separate files within the cmd directory. For now, we'll keep it simple and define everything in the root.go file.

Building the Root Command

Now that we have our project set up, let's define our CLI commands using Cobra. We'll create a simple command to make HTTP GET requests.

To achieve this, we need to parse the URL provided as an argument, extract the hostname, port, and path, and then make an HTTP GET request to the specified URL. Firstly, we should parse the URL and extract the necessary information to create our TCP connection.

// cmd/root.go
package cmd

import (
    "net/url"
    "os"

    "github.com/spf13/cobra"
)

var rootCmd = &cobra.Command{
    Use:   "build-your-own-curl",
    Short: "A brief description of your application",
    Long: `A longer description that spans multiple lines and likely contains
examples and usage of using your application. For example:
Cobra is a CLI library for Go that empowers applications.
This application is a tool to generate the needed files
to quickly create a Cobra application.`,
    Args: cobra.ExactArgs(1),

    Run: func(cmd *cobra.Command, args []string) {
        u, err := url.Parse(args[0])

        if err != nil {
            panic(err)
        }

        host := u.Hostname()
        port := u.Port()
        path := u.Path

        println("Host:", host)
        println("Port:", port)
        println("Path:", path)
    },
}

// rest of the code

The Args field specifies the number of arguments the command expects. In this case, we expect exactly one argument, which is the URL we want to make a request to. You may want to add custom validators or other Cobra built-in validators if you want to expand this functionality. If you run the application without an argument, you should see an error message printed to the console.

go run main.go

Error: accepts 1 arg(s), received 0
Usage:
  build-your-own-curl [flags]

Flags:
  -h, --help     help for build-your-own-curl
  -t, --toggle   Help message for toggle

exit status 1

But if you run the application with an URL as an argument, you should see that hostname, port, and path are printed to the console.

go run main.go https://example.com/get

Host: example.com
Port:
Path: /get

As you probably have noticed, the port is not being extracted correctly. This is because the url.Parse function does not return the port if it is not specified in the URL.

Most of us do not specify the port when making a day-to-day HTTP request using curl or a browser. To make our UX better, let's set the default port to 80 if it is not specified in the URL. In this tutorial, I will not handle HTTPS requests, which is why we are going to use only port HTTP (80) for now.

// cmd/root.go
// ...
  Run: func(cmd *cobra.Command, args []string) {
    u, err := url.Parse(args[0])

    if err != nil {
      panic(err)
    }

    host := u.Hostname()
    port := u.Port()

    if port == "" {
      port = "80"
    }

    path := u.Path

    println("Host:", host)
    println("Port:", port)
    println("Path:", path)
  },
}
// ...

Run the application to see the default port bring printed to the console.

go run main.go https://example.com/get

Host: example.com
Port: 80
Path: /get

Now that we have the necessary information to create a TCP connection, let's make an HTTP GET request to the specified URL.

A basic HTTP GET request header consists of the following:

• Request line: GET /path HTTP/1.0
• Host header: Host: hostname

We will send this request to the server using net.Dial and read the response. We will then print the response to the console.

// cmd/root.go
package cmd

import (
    "fmt"
    "net"
    "net/url"
    "os"

    "github.com/spf13/cobra"
)

var rootCmd = &cobra.Command{
    Use:   "build-your-own-curl",
    Short: "A brief description of your application",
    Long: `A longer description that spans multiple lines and likely contains
examples and usage of using your application. For example:
Cobra is a CLI library for Go that empowers applications.
This application is a tool to generate the needed files
to quickly create a Cobra application.`,
    Args: cobra.ExactArgs(1),

    Run: func(cmd *cobra.Command, args []string) {
        u, err := url.Parse(args[0])

        if err != nil {
            panic(err)
        }

        host := u.Hostname()
        port := u.Port()
        path := u.Path

        if port == "" {
            port = "80"
        }

        conn, err := net.Dial("tcp", fmt.Sprintf("%s:%s", host, port))

        if err != nil {
            panic(err)
        }

        defer conn.Close()

        fmt.Fprintf(conn, "GET %s HTTP/1.0\r\nHost: %s\r\n\r\n", path, host)

        buf := make([]byte, 1024)
        n, err := conn.Read(buf)

        if err != nil {
            panic(err)
        }

        fmt.Println(string(buf[:n]))
    },
}

// ...

In the code above, we create a TCP connection to the specified host and port. We then send an HTTP GET request to the server and read the response. Finally, we print the response to the console. If you run the application with a valid URL, you should see the HTTP response printed to the console.

go run main.go http://eu.httpbin.org/get
HTTP/1.1 200 OK
Date: Wed, 27 Mar 2024 22:35:13 GMT
Content-Type: application/json
Content-Length: 203
Connection: close
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

{
  "args": {},
  "headers": {
    "Host": "eu.httpbin.org",
    "X-Amzn-Trace-Id": "Root=1-66049f21-305d0735393fd4ae2bc554a0"
  },
  "url": "http://eu.httpbin.org/get"
}

And you have it! A command-line tool to make GET requests to any URL using Go and Cobra. Additional changes can be made to handle different HTTP methods, headers, and more.

A challenge for you:

• Add support for different HTTP methods (e.g., POST, PUT, DELETE).
• Add support for custom headers.
• Add support for HTTPS requests.

The first and second were made in my project gurl. You can check it out for more inspiration or even to contribute and improve it!

Another project that worth to mention here is go-curling from @chriswiegand

It uses http package instead of net which shows a more simple and fast way to implement a cURL client in Go without making requests from scratch using net.Dial.

Conclusion

Congratulations! You've built a simple command-line tool similar to curl using Go and Cobra. Feel free to expand upon this project by adding more features like handling different HTTP methods, headers, and more.

I have made a project called gurl that is a simple CLI tool that can make HTTP requests using Go. You can check it out for more inspiration.

Comments

[Chris Wiegand]

I actually made a golang version of curl for myself, because Microsoft got rid of it in their .Net 5 docker images, which ended up breaking my images when I upgraded. I didn't notice that they had removed curl, so once I figured that out, I decided to just make my own curl in golang (github.com/cdwiegand/go-curling) so I could pull it in easily via my docker builds. I ended up implementing the most common arguments for the health check, and eventually even got added a few others just for fun, but it is certainly not a complete re-implementation of curl given the complexities that curl actually supports.

I think this article is a really good introduction to the basics of making network connections from golang, although using the http get package would handle some of this for you and add basic HTTPS support as well.

[Eric Santana]

That is awesome! Indeed cURL has a lot of features which would be a pain to replicate in a personal project. The choice of using net.Dial instead of http is to show a bit of how things work under the hood of a HTTP client package.

It is very nice to see that someone also made an implementation of cURL in Go. Thank you for your contribution!

[Kaamkiya]

This is pretty cool :)

What's the difference between net.Dial (with all of your other code) and http.Get?

[Eric Santana]

Hello! Thank you!

The main difference between them is the layer on which you are working.

Using http.Get would work as well and would be WAY more easy to create an HTTP client (http package by itself exposes a HTTP client`. But the purpose of this article is also to show on how we can manually create an HTTP request, inserting all the information we need to get a response from a server.

We developers sometimes are so focused on using libraries and abstractions that the basic stuff stays a bit aside. So the main purpose of using net.Dial is to create indeed a HTTP request from scratch and remember a bit of the basics like using TCP protocol.

Indeed using http.Get would be faster and building a curl using net package may be a bit complex, although it helps us to understand on how HTTP requests work :)

[Enzo Jade]

Thanks for the tutorial! Creating a curl-like tool in Go using Cobra sounds cool. Your guide breaks down the process step by step, making it easy to follow. I like how you explain things clearly, from setting up the project to making HTTP requests. Excited to try out the extra challenges you mentioned. There is another expert which I prefer for assistance with my golang assignments which is available at ProgrammingHomeworkHelp.com