Debugging is just the process of creating hypotheses about what is causing a bug and then testing those hypotheses. One of the biggest differences between junior and senior developers is that senior developers are much better debuggers. They have a better sense of where a bug may be coming from and they have better ideas about how to test their hypotheses
If you are a full-stack engineer or a backend engineer, part of your job is to debug servers. In many cases, you cannot debug an API server unless you send a request to the server. There are many ways to do this: using fetch in a browser, using a third party application, or using cURL. Many great engineers who I have met can use cURL with ease. Being able to use it is not required to be a good engineer, but if you can use it comfortably, it will definitely help you to level up your debugging skills. In this article, I will explain a practical guide on how to get started with cURL.
What’s wrong with fetch or Postman?
Some people might wonder why you need to use a command-line tool in 2020. I have used Postman before, and I agree that it is a great application. In particular, being able to save the history of your requests is a neat feature. However, there is a big limitation. You cannot easily use it inside a Kubernetes cluster, VM, or Docker container. You sometimes have to go inside another machine and send requests from the machine. You cannot easily use fetch or a third-party application in this situation. On the other hand, you can install cURL with one command and start using it right away. cURL also supports many different protocols such as FTP and SMTP along with HTTP/HTTPS protocols. If you need to learn just one approach, then it is better to learn the one that you can use in any situation.
Utilize a browser’s network tab
If you are new to cURL and want to read the documentation, you can just type man curl on your command line. However, if you are not a big fan of reading documentation and prefer to actually start using a tool right away, you can start using cURL quickly by utilizing a browser:
- Go to a page you want to debug
- Open up the browser’s developer tools
- Go to the network tab
- Hover over a network request you want to get
cURLcommand for - Right-click on the network and select “Copy as cURL”
Once you copy the request as cURL, then paste that in your command line. You will see something similar to the following image.
If you hit the Enter, it will send a request. If you look at the command closely, you will find that each line is followed by -H, which represents a header property. When you are debugging, you don’t need most of the header properties except for a few of them that I will talk more about later. In this case, you can simplify your cURL command to curl 'https://www.google.com/' and it works perfectly. Here is a general tip for beginners when you want to use a cURL command and don’t know it exactly: Copy a request as a cURL command from a browser and simplify the command by removing unnecessary header properties.
Basic usage of cURL
Now you know how to get started with cURL, so let’s look at the basics of cURL so that you don’t have to copy a request from a browser every time.
GET request
The first use case is sending a GET request, which is super simple.
POST, PUT, and PATCH request
Now let’s take a look at the case where you send a request with a body massage. There are multiple ways to do it, and how a server is implemented determines which method you can use.
1: When a server accepts application/json
If a server accepts JSON format, you can send JSON data by setting the header property Content-Type to be application/json.
2: When a server accepts application/x-www-form-urlencoded
Another popular way to send a request with a body is to use x-www-form-urlencoded format. Data you want to send will be transferred with the format of key1=value1&key2=value2.
3: When a server accepts multipart/form-data
When you send a request using the above 2 methods, you can only send one data type at once. However, a server can accept multiple data types. This is typically the case when you can upload files to the server. When you use multipart/form-data, you specify the data using the -F flag. In order to upload a file, you need to add @ at the beginning of the file path.
There are more content types you can specify, but knowing these 3 will cover most use cases. Sometimes the body message is way too long to type on the command line. In this situation, you can let a server read data from a file. I will talk more about this later.
Authentication
Another common use case is to send authorization information. Many services have an authentication system that requires you to be authorized to make a certain request. When you make a request from a browser, your authorization information is usually stored as a cookie and attached to requests. However, if you make a request outside of a browser, you need to attach authorization information to a request. What method works depends on how an authentication system is implemented on a server.
Basic Authentication
Basic authentications are one of the most popular authentication systems. They send user credentials encoded in base64. A request has an authorization header that contains encoded values of the username and password. When you use cURL, you don’t need to encode the user credentials on your own because curl will do it for you if you include the -u flag like the following.
Bearer Authentication
Bearer authentications are mainly used for OAuth 2.0, which are industry-standard for applications that send requests on behalf of a user. An example of OAuth 2.0 is a GitHub app that sends requests to the GitHub API server when a user performs a specific action. Bearer tokens are usually generated by a server in response to user authentication. Once you get a bearer token, you can send a cURL request like the following.
If you cannot authenticate correctly using the methods above, this page explains more about authentication using cURL. You might need to pass optional flags as described on the page.
Advanced usage
Now you know how to send requests using cURL. Once you get used to it, you can write up a command to send a request really quickly. However, in reality, you need to know a little bit more about cURL to debug like a pro. Here are some of the advanced usages that are really helpful when I debug a server.
Avoid SSL certificate error
When you send a cURL request using HTTPS, it performs SSL certificate verification. However, if your server does not have a certificate signed by the CA, then your cURL command will fail. This often happens when you send requests to a server with a self-signed SSL. If you want to know the details about SSL, CloudFlare has a good article about it. You can disable certificate verification by passing a -k flag.
Write the content of the body message in a file and pass the file
Your body message can be too big to write all of it in the command line. You might want to save the content for debugging purposes. In these situations, You can write a body message in a file and send the contents of the file as a body message. There are two slightly different ways to do it: The one on the top would strip all newlines in a file, and the one below does not. Choose an appropriate method depending on your use case.
Stress-test a server
cURL can be also used to stress-test your server. I have encountered a bug where a program worked fine when there were few requests to a server, but it broke when there were loads of requests. cURL is really handy in this situation. You can use [] to create dynamic URLs, and mimic the stress test. The following program sends the request 1000 times.
There is a lot more you can do with cURL, but now you probably know enough to explore cURL on your own! If you want to learn more about cURL, just type man curl in your command line. Mastering cURL will help you to debug more quickly.
If you like this post, we'd love it if you "Like" it and/or share it wherever and however you do your social media. You can subscribe here to get an email when we publish new posts. I'm on Twitter at @mkinoshita12. If you have any questions, feel free to reach out at makoto@antcode.dev.
If you're curious what Antcode is, you can check it out here.
Top comments (1)
Nice little tip to get hold of the HTTP headers. Thanks for posting!