How Gost-DOM avoids making HTTP calls

This article is about the implementation of Gost-DOM, the headless browser written in Go.

Web applications written in Go are very easy to test. An web application serves HTTP requests from a single function, ServeHTTP. Test code can just call this function to test the web application behaviour, but still express the test in terms of http requests, responses, headers, body, and status codes; rather than controller method calls.

I wanted Gost-DOM to take advantage of this for two reasons:

1. Performance - The test is just calling Go code, avoiding the overhead of a TCP stack
2. Isolation - By eliminating the need to manage TCP ports, it becomes much easier to run tests in isolation.

Part 2 is really the most important part here. Although there is nothing preventing tests from running in isolation; the management of TCP ports increases complexity of the test setup.

The HTTP client

I wanted to build this in a way that makes the core of the browser unaware of how HTTP requests are handled. Fortunately, Go's excellent standard library has the right tools out of the box.

Outgoing HTTP requests are handled by an http.Client instance; which abstracts the transport layer through the RoundTripper interface.

type Client struct {
    // Transport specifies the mechanism by which individual
    // HTTP requests are made.
    // If nil, DefaultTransport is used.
    Transport RoundTripper

  // ...
}

A great feature of Go is that many essential operations are abstracted by single-method interfaces. The RoundTripper has just one method:

type RoundTripper interface {
    RoundTrip(*Request) (*Response, error)
}

This means, I just need to create a type implementing the RoundTripper interface, that calls ServeHTTP.

Implementing the interface

The http handler function receives an http.ResponseWriter. This is not a concrete type, but an interface.

In addition to promoting a testable design of your production code, Go also supports great test tools in the net/http/httptest package. The ResponseRecorder is a valid ResponseWriter that test code can pass to the implementation, and simplifies dealing with response body streams.

But the ResponseRecorder is also so kind to actually generate a proper *http.Response, that can be retrieved by calling, ResponseRecorder.Result().

The type representing the request is the same in the RoundTripper and the Handler, so I made a copy, to avoid mutation in the tested server code affects the request object generated in the browser. The first version of the TestRoundtripper looked like this:

// A TestRoundTripper is an implementation of the [http.RoundTripper] interface
// that communicates directly with an [http.Handler] instance.
type TestRoundTripper struct{ http.Handler }

func (h TestRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
    rec := httptest.NewRecorder()
    // Make a copy, so the http handler doesn't mutate the outgoing request
    reqCopy := new(http.Request)
    *reqCopy = *req 
    h.ServeHTTP(rec, reqCopy)
    return rec.Result(), nil
}

This worked fine in the beginning, but had some issues.

A slightly odd design decision

In the midst of the very well designed API is what I find to be a somewhat odd design decision.

The *Request passed to the RoundTripper, and the *Request passed to your http Handler is the same type, but they are really two different things. The one is an outgoing request, and the other is an incoming request.

While they share are similarities, they are not the entirely the same, and the type has properties that are only relevant for processing incoming requests, such as decoding form data. And there have different rules determining if a request is valid.

First, the outgoing request is allowed to have a nil request body, where the incoming request always have a body.

Then, the incoming request is a source of a context.Context, which I didn't initialise either.

Fixing the request was easy:

func (h TestRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
    rec := httptest.NewRecorder()
    body := req.Body
    if body == nil {
        body = nullReader{}
    }
    clientReq, err := http.NewRequest(req.Method, req.URL.String(), req.Body)
    if err != nil {
        return nil, err
    }
    clientReq.Header = req.Header
    clientReq.Trailer = req.Trailer
    h.ServeHTTP(rec, clientReq)
    return rec.Result(), nil
}

And now, the browser itself will create HTTP requests using an instance of the http.Client provided. Test code can control the client, replacing default HTTP request behaviour with one that calls your HTTP handler, and an outgoing request from the browser is now just a simple function call.

Adding cookie support

Go also provides a cookie jar. So adding cookie support was so simple that giving it a headling was completely overkill.

The function that constructs an http.Client communicating with a simple handler is as simple as:

import (
    "net/http"
    "net/http/cookiejar"
)

func NewHttpClientFromHandler(handler http.Handler) http.Client {
    cookiejar, err := cookiejar.New(nil)
    if err != nil {
        panic(err)
    }
    return http.Client{
        Transport: TestRoundTripper{Handler: handler},
        Jar:       cookiejar,
    }
}

Testing identity provider integration

While this is not a feature yet, it is intended to extend this functionality to support multiple HTTP handlers, simulating different host names.

This could be valuable in testing an OAuth authentication flow, or sign in using external identity providers. You could create a test HTTP handler simulating the behaviour of an identity provider, and test the flow independently of the external provider.

In my previous experience, this is often accomplished with a real identity provider, configured with some test users. But this approach has some drawbacks:

• The tests may fail due to a temporary outage of an external service (in the worst case, preventing deployeing a critical)
• Tests may fail due to account lockout.
• Using "real test users" makes test code depend on an external context, i.e., there are factors affecting the outcome of the test that is not specified in the test.
• Developers may not have privileges to administer test users, preventing them from writing the right test, waiting for an administrator to add new users.

By mocking an IDP, you gain full control of test environment flow.¹ Then you'd need to setup the round tripper to support two hostnames:

• https://app.example.com/ Calls your application http handler
• https://idp.example.com/ Calls your mock identity provider web app.

And you'd still be able to run everything in parallel.

Intrigued? Check out Gost-DOM. And stay tuned for more nitty-gritty details about how I build a headless browser in Go.

And please, spread the word. 🙏

---

1. I want to emphasise that this tool is meant to support the majority of tests written to support a TDD flow. That doesn't mean there shouldn't be added more tests after the fact to find critical integration issues, and I certainly recommend having ONE automated tests exercise the login flow if your application integrates with an external identity provider. But I don't want that to be part of the normal development flow. ↩