Golang HTTP Handler With Gzip

Introduction

Golang has many kinds of compression technique within its standard library which you can use to compress your data. A compression is needed to reduce the size of the data. Even in a web server, a compression technique would be beneficial to increase the communication speed between client and server. Gzip is one of the compression techniques supported by both Golang and web. This article will cover the creation of golang HTTP handler that (de)compress gzip request/response and how to create an HTTP request that send/receive the gzip-compressed body payload. All the code covered in this article you can find it in this repository.

Directory Structure

$ tree .
.
├── LICENSE
├── Makefile
├── README.md
├── client
│   └── main.go
├── curl.sh
├── go.mod
├── server
│   └── main.go
└── util.go

2 directories, 8 files

server/main.go

package main

import (
    "example"
    "log"
    "net/http"
)

func main() {
    log.Println("server listening at :8000")
    http.ListenAndServe(":8000", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        body := example.MustReadCompressedBody[example.Payload](r.Body)
        body.Number++

        example.MustWriteCompressedResponse(w, body)
    }))
}

The server only run a single endpoint which will read a compressed body, increment the payload content, then respond with new body.

util.go

Payload

You can use the simplest payload just to test the compression, for example:

type Payload struct {
    Number int `json:"number"`
}

Read a Compressed Body Payload

func MustReadCompressedBody[T any](r io.Reader) *T {
    gr, err := gzip.NewReader(r)
    PanicIfErr(err)
    defer gr.Close()

    var t T
    PanicIfErr(json.NewDecoder(gr).Decode(&t))
    return &t
}

To read a gzip-compressed payload, you need to create a gzip.Reader from your response or request. Then decode the JSON as usual using the gzip.Reader instance.

Write a Compressed Response

func MustWriteCompressedResponse(w http.ResponseWriter, body any) {
    w.Header().Set("Content-Type", "application/json")
    w.Header().Set("Content-Encoding", "gzip")

    gw := gzip.NewWriter(w)
    defer gw.Close()
    PanicIfErr(json.NewEncoder(gw).Encode(body))
}

As usual, it is better to inform the client about your Content-Type, and Content-Encoding. But in golang case, informing the client about Content-Encoding will help golang http.Request to decompress the payload automatically. Just like reading the payload, you only need to create a gzip.Writer then encode the content normally. Also, don't forget to call gw.Close() to avoid EOF error.

Test the Server

Now, your server has completely (de)compress the request and response. Let's try to test it using cURL command first.

# run the server on another terminal
# using `go run ./server/` command
$ echo '{"number": 99}' | gzip | \
curl -iXPOST http://localhost:8000/ \
-H "Content-Type: application/json" \
-H "Content-Encoding: gzip" \
--compressed --data-binary @-

Since the server will increase the Payload.Number, you can expect the response number will be 100 by sending a request with 99. The expected result would be similar to this:

HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Type: application/json
Date: Thu, 26 May 2022 12:04:53 GMT
Content-Length: 39

{"number":100}

Now, after you sure that your server works completely fine, let's try to send the HTTP request using golang http.Request.

client/main.go

package main

import (
    "context"
    "encoding/json"
    "example"
    "log"
    "net/http"
    "time"
)

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
    defer cancel()

    body := &example.Payload{
        Number: 100,
    }

    log.Printf("create compressed request with %#v", body)
    req := example.MustCreateCompressedRequest(ctx, http.MethodPost, "http://localhost:8000/", body)
    defer req.Body.Close()

    log.Printf("send compressed request")
    resp, err := http.DefaultClient.Do(req)
    example.PanicIfErr(err)
    defer resp.Body.Close()

    log.Println("resp.Uncompressed?", resp.Uncompressed)
    var responsePayload *example.Payload
    if resp.Uncompressed {
        err = json.NewDecoder(resp.Body).Decode(&responsePayload)
    } else {
        responsePayload = example.MustReadCompressedBody[example.Payload](resp.Body)
    }
    log.Printf("read response %#v", responsePayload)
}

Just like a normal HTTP request, you just need to create an HTTP.Request, send the request, and lastly decode the response. But, before send the request payload, don't forget to compress it first.

Create a Compressed Request

func MustCreateCompressedRequest(ctx context.Context, method, url string, body any) *http.Request {
    pr, pw := io.Pipe()

    go func() {
        gw := gzip.NewWriter(pw)
        err := json.NewEncoder(gw).Encode(body)
        defer PanicIfErr(gw.Close())
        defer pw.CloseWithError(err)
    }()

    r, err := http.NewRequestWithContext(ctx, method, url, pr)
    PanicIfErr(err)

    r.Header.Set("Content-Type", "application/json")
    r.Header.Set("Content-Encoding", "gzip")

    return r
}

Create a new compressed request is quite easy, just like write a compressed response we need to create a new gzip.Writer. Code above utilize io.Pipe to avoid buffer the body into a memory and unecessary allocations. But here's the alternative if you want to buffer the body first:

func MustCreateCompressedRequest(ctx context.Context, method, url string, body any) *http.Request {
    var b bytes.Buffer

    gw := gzip.NewWriter(&b)
    err := json.NewEncoder(gw).Encode(body)
    defer PanicIfErr(gw.Close())

    r, err := http.NewRequestWithContext(ctx, method, url, &b)
    PanicIfErr(err)

    r.Header.Set("Content-Type", "application/json")
    r.Header.Set("Content-Encoding", "gzip")

    return r
}

Again, gw.Close() is also necessary here. If you don't close the gzip.Writer, you'll see an EOF error similar to this:

2022/05/26 19:23:58 http: panic serving [::1]:56436: unexpected EOF

Lastly, after creating the compressed request, you only need to send the request like a normal request. But here's a tricky part.

var responsePayload *example.Payload
if resp.Uncompressed {
    err = json.NewDecoder(resp.Body).Decode(&responsePayload)
} else {
    responsePayload = example.MustReadCompressedBody[example.Payload](resp.Body)
}

After you receive a response which you want to decompress and decode, please be aware with the resp.Uncompressed. If the server returns a header Content-Encoding: gzip, as it said here Golang will try to decompress the payload for you so you don't need to use example.MustReadCompressedBody[example.Payload](resp.Body). But if you add r.Header.Set("Accept-Encoding", "gzip") on your request, it won't be automatically decompressed.

Test the Client

# run the server on another terminal
# using `go run ./server/` command
$ go run ./client
2022/05/26 19:41:34 create compressed request with &example.Payload{Number:100}
2022/05/26 19:41:34 send compressed request
2022/05/26 19:41:34 resp.Uncompressed? true
2022/05/26 19:41:34 read response &example.Payload{Number:101}

Conclusion

Implementation of the gzip compression on Golang HTTP handler and requests might be adding a little complexity on your code, but I believe modern browser / API gateway these days can implement this easily without changing your code. Also this middleware created by NY Times can help you to minimize the effort on implementing this compression. You can find all the code used on in this article here.

Thank you for reading!