Hurl, I've Come to Bargain

Ever since I started the open source journey, I've wanted to contribute to a real Rust project. As much as I adore my own little project Mastermind, it simply doesn't have the complexity for me to practice and refine my Rust coding skills.

That's why I was really glad to find out about hurl. It was recommended by my professor, who also told me that the maintainer of this project (Shout out to you, jcamiel!) was one of his favourite people on GitHub. Plus, they seemed to have a welcoming vibe, so I looked into it more.

Curl and Hurl

As you probably know, curl is a popular command line utility that interacts with various network protocols. Behind curl there's a library libcurl which brings the powerful network communication features to whatever projects that use it. It comes with bindings for virtually every programming language out there, among which Rust is one of them.

Hurl is built upon libcurl with a focus on the HTTP protocol. It runs HTTP requests defined in a simple plain text format, which makes chaining complex requests incredibly easy. You can also use it to evaluate HTTP responses, making sure you always get the valid results.

Attempt #1

Glancing through their GitHub repo, my eyes landed on this issue. Essentially, I needed to add a cli option to disable connection reuse.

Like always, I did a lot of research on this matter. In the issue the maintainer mentioned disabling TCP keepalive, so that's where I started with. Quickly, I found the libcurl option CURLOPT_TCP_KEEPALIVE from their doc, and the corresponding method in the Rust binding library.

Immediately, I noticed that the keepalive option was disabled by default. This was very confusing to me since I knew curl always tries to reuse connections unless configured otherwise.

Determined to figure this out, I tinkered with curl some more and did some experiment on an empty project. I realized that the connections were always left open regardless of the status of tcp keepalive:

With even more confusion in mind, I asked the maintainer. He explained that the CURLOPT_TCP_KEEPALIVE option might not be what we were looking for. Instead, he pointed me to CURLOPT_FRESH_CONNECT and CURLOPT_FORBID_REUSE.

Later I learned that, the --no-keepalive flag in curl, as well as the libcurl option, only controls whether or not the client sends TCP keepalive probes to keep the connection alive. It does not dictate how the client reuses these connections.

I verified this using my trusty networking tool Wireshark as well! I set up an HTTP server that waits a long time before responding and captured the traffic with Wireshark. Without --no-keepalive, I could see the keepalive messages being sent around every 60 seconds. With the option I don't see them at all. Mystery solved!

Attempt #2

Knowing that I'm working with a different option instead of TCP keepalive, I started working on my PR. I carefully examined their PR history and found a similar one, which I used as a reference.

One more thing I really love about this project is that they have included scripts to generate docs. Despite having a ton of documentations, I only had to write in one place and let the script handle the rest.

After I submitted the PR, the maintainer reached back to me and proposed a couple of changes. One involved a workaround in another part of the code, and the other was about docs. I submitted my fix thinking this was ready to merge.

However, later the maintainer replied again. He explained that they had decided to not implement this feature and apologized for not communicating this before assigning me the issue.

I was a bit bummed out by this, to say the least. It was disappointing to see my hard work getting this close and then rejected. Still, I felt accomplished at the same time. After all, I just created a working component for a Rust project as big as hurl. How cool was that?

Later, I asked my professor for advice on this situation. He suggested me to ask for another issue, which was exactly what I did!

Attempt #3

Soon after I asked, the maintainer gave me another issue. It was similar but also different. Once again, the maintainer patiently explained their expectations:

• Implement the -H/--header cli option, as well as in the [Option] field inside the hurl file.
• The cli flag is an existing curl option. It adds additional headers to each request.
• In the future, they will implement a global [Option] field in the hurl file, which this option should also support.
• There should be two PRs on this issue:
  • The first PR implements the cli option.
  • The second PR supports the hurl file.

Perfect! Sounds like a big challenge but not something completely unachievable. This time, however, was not as smooth sailing as I had hoped.

To make sure my code behave the same way as curl, I first looked into their man page, from which I found that curl --header can not only add more headers, but also override internal headers set by curl.

In hurl's header handler function, it handles options from both cli arguments and the hurl file. And here's where I made my first mistake:

I saw the code to apply headers and directed all my attention to it without properly understanding how the hurl file options are processed. I invested in a lot of time and energy to implement the override behavior, and had to use workarounds due to some limitations of libcurl.

After I submitted the PR, I was greeted with a full essay of proposed changes. In the request, the maintainer further explained the wanted behavior, and suggested that I implement some code to aggregate the user specified headers before passing to the handler. He also proposed that I split this PR into two parts: One part implements the aggregation (with unit tests), another part implements the actual cli option.

It was humbling to see feedback as such, but I was also genuinely moved by the thoughts and efforts the maintainer had put into it. It was reassuring and somehow made me feel better about my work going forward.

Attempt #...?

I started with one simple issue and ended up with a totally different issue and 3 requested PRs. Who knows how many more attempts I will have to make?

Despite all that, I'm fully committed (no pun intended) to it! It has been an incredible opportunity to not only learn about Rust, curl, HTTP, but also working with people to solve uncertain problems in a moving project. I'm excited to see how this goes and certainly not gonna stop right here.