Testing HTTP Requests with http.RoundTripper

Last year at work we were working on a B2B SAAS product that other businesses could build upon by calling our APIs. We handle some of the requests and for some we forward the requests to an external provider. Our requirements meant we had to have a very robust testing suite. However, to avoid running out of API credits, rate limit issues, and for our tests to be resilient and fast we have to mock the external API calls. As a recovering JavaScripter, I did the reasonable thing: pull in a library, this time httpmock. This is not a criticism of httpmock, I love and use it extensively.

Recently, I read GoByExample and learned a different way to mock the http calls using the http.RoundTripper interface. The interface is tailor made for this purpose:

RoundTripper is an interface representing the ability to execute a
single HTTP transaction, obtaining the [Response] for a given
[Request].

This tells us we can decide how we want our http.Client to process our requests. Say, we have a function named pingUrl that sends a head request to the provided URL that we need to test:

func pingUrl(ctx context.Context, url string, client *http.Client) error {
    req, err := http.NewRequestWithContext(ctx, http.MethodHead, url, nil)
    if err != nil {
        return fmt.Errorf("setting up request: %w", err)
    }

    res, err := client.Do(req)
    if err != nil {
        return fmt.Errorf("sending request to %s: %w", url, err)
    }
    defer res.Body.Close()

    if res.StatusCode >= http.StatusBadRequest {
        buf := &bytes.Buffer{}
        _, err = io.Copy(buf, res.Body)
        if err != nil {
            return fmt.Errorf("copying response from %s: %w", url, err)
        }
        return fmt.Errorf("%s", buf.String())
    }

    return nil
}

We could also write it without explicitly passing in a http.Client pointer:

func pingUrl(ctx context.Context, url string) error {
    client := http.DefaultClient
    req, err := http.NewRequestWithContext(ctx, http.MethodHead, url, nil)
    if err != nil {
        return fmt.Errorf("setting up request: %w", err)
    }

    res, err := client.Do(req)
    if err != nil {
        return fmt.Errorf("sending request to %s: %w", url, err)
    }
    defer res.Body.Close()

    if res.StatusCode >= http.StatusBadRequest {
        buf := &bytes.Buffer{}
        _, err = io.Copy(buf, res.Body)
        if err != nil {
            return fmt.Errorf("copying response from %s: %w", url, err)
        }
        return fmt.Errorf("%s", buf.String())
    }

    return nil
}

The first function is preferred because it has explicit dependencies and therefore easier to test. We can "inject" different clients with different roundTrippers depending on context. It is also faster because the http client can reuse existing connections to make requests to the same URLs. There are different ways to test this function:

1. With a mocking library

If you are from the Javascript ecosystem, this is probably what you do. We have a mocking library intercept the http request and respond with what we want. This is simple and effective. It has the added benefit of having a built in clean up and ease of setting up responders. We can also reduce the number of arguments we pass to the function (remove the "client", it's cleaner 😁).

func TestPingURLWithMockLibrary(t *testing.T) {
    client := &http.Client{}

    t.Run("mock success", func(t *testing.T) {
        httpmock.Activate(t)
        defer httpmock.DeactivateAndReset()
        // Exact URL match
        httpmock.RegisterResponder(http.MethodHead, url,
            httpmock.NewStringResponder(http.StatusOK, `[{"id": 1, "name": "My Great Article"}]`))
        if err := pingUrl(context.Background(), url, client); err != nil {
            t.Fatalf("expected error to be nil, got %v", err)
        }
    })

    t.Run("mock failure", func(t *testing.T) {
        httpmock.Activate(t)
        defer httpmock.DeactivateAndReset()
        // Exact URL match
        httpmock.RegisterResponder(http.MethodHead, url,
            httpmock.NewStringResponder(http.StatusBadRequest, `[{"id": 1, "name": "My Great Article"}]`))

        if err := pingUrl(context.Background(), url, client); err == nil {
            t.Fatal("expected error to not be nil")
        }
    })
}

1. With a http.RoundTripper

Alternatively, we could dispense with the external dependency and have more control over the request. This is where http.RoundTripper comes in. We pass a client with a custom roundtripper and then test the behaviour we want. As Casey Muratori makes clear every now and again, the effect of dependencies on the reliability of your software cannot be overstated. Each dependency we add makes our software disproportionately flakier. However, this is not just about reducing dependencies, it is also about reducing "magic" and simplicity.

type Tripper func(*http.Request) (*http.Response, error)

func (t Tripper) RoundTrip(request *http.Request) (*http.Response, error) {
    return t(request)
}

var successTripper = Tripper(func(*http.Request) (*http.Response, error) {
    return &http.Response{StatusCode: http.StatusOK}, nil
})

var failureTripper = Tripper(func(*http.Request) (*http.Response, error) {
    body := io.NopCloser(strings.NewReader(`{"error": "something went wrong"}`))
    return &http.Response{
        StatusCode: http.StatusBadRequest,
        Body:       body,
    }, nil
})

var url = "https://api.mybiz.com/articles"

func TestPingURLWithRoundTripper(t *testing.T) {
    client := http.Client{}

    t.Run("roundtrip success", func(t *testing.T) {
        client.Transport = successTripper
        if err := pingUrl(context.Background(), url, &client); err != nil {
            t.Fatalf("expected error to be nil, got %v", err)
        }
    })

    t.Run("roundtrip failure", func(t *testing.T) {
        client.Transport = failureTripper
        if err := pingUrl(context.Background(), url, &client); err == nil {
            t.Fatal("expected error to not be nil")
        }
    })
}

It looks like more code for the same functionality but is it? We are essentially swapping out multi-purpose code written by someone else for our own custom code written for our own particular use case. We can create the helpers if we want.

Anywho, I asked our AI overlords what they thought and here from Claude 4:

Use httpmock when you need:

• Complex URL pattern matching
• Request verification/counting
• Testing multiple endpoints in the same test
• Your team prefers declarative mocking
• You're already using it extensively

Use RoundTripper when you want:

• Zero dependencies
• Maximum control and transparency
• Simple, focused unit tests
• Performance-critical test suites

I'd say that's pretty good advice. Ciao!