How I write HTTP services in Go

Björn V — Fri, 16 Feb 2024 08:28:23 +0000

A couple of days ago Mat Ryer came out with a refreshed version of how he writes HTTP services in Go (2024). The patterns he uses are all fine and dandy, I especially like that he got rid of the server struct from his initial version of "How I write HTTP services in Go" (2018) - where handlers were methods on the server struct. As he states himself, that type of pattern hides dependencies and makes testing more brittle.

I have no major issues with the proposed setup and layout of HTTP services ala Mat Ryer - it is straightforwad and easy to follow. There are some "warts" though: his version of addRoutes has access to all dependencies, access to all configuration values, and an parameter list which is very broad. In this post I'll try to offer an alternative approach, more streamlined and which offers (in my opinion) a smoother experience regarding maintainability.

First of all: I'm a proponent of the CQRS pattern - splitting operations into commands and queries. Most of the time I expose a more classic HTTP API for commands (POST/PUT/DELETE requests) and a GraphQL API for queries (replacing GET requests). I think this gives a solid separation of concerns as well as improving type management - both server side but also client side. My proposed pattern has one flaw: it's not very suitable for generic CRUD APIs! It can easily become cumbersome and over-engineered in that kind of setup. I try to model my domains from operations instead of CRUD since I think that makes the system easier to work and grow with over time.

Example domain

To illustrate how I write HTTP services it would be easiest to have a domain beforehand: a simple banking system. We have a couple of domain objects such as customers and accounts, where customers can transfer funds to and from their accounts.

Example of a Command

Most of the time I create specific types for my commands (and their inputs/outputs):

type (
    WithdrawFundsRequest struct {
        CustomerID      uuid.UUID
        SourceAccountID uuid.UUID
        Amount          uint
    }
    WithdrawFundsResponse struct {
        WithdrawnAt time.Time
    }

    InsertFundsRequest struct {
        CustomerID      uuid.UUID
        TargetAccountID uuid.UUID
        Amount          uint
    }
    InsertFundsResponse struct {
        InsertedAt time.Time
    }

    TransferFundsRequest struct {
        CustomerID      uuid.UUID
        SourceAccountID uuid.UUID
        TargetAccountID uuid.UUID
        Amount          uint
    }
    TransferFundsResponse struct {
        TransferredAt time.Time
        Amount        uint
    }

    WithdrawFundsCommand func(context.Context, WithdrawFundsRequest) (WithdrawFundsResponse, error)
    InsertFundsCommand   func(context.Context, InsertFundsRequest) (InsertFundsResponse, error)
    TransferFundsCommand func(context.Context, TransferFundsRequest) (TransferFundsResponse, error)
)

Since the command signatures doesn't declare any stateful dependencies, implementations are created via maker functions:

type (
    Withdraw func(context.Context, Account, uint) error
    Insert   func(context.Context, Account, uint) error

    AccountService interface {
        Lookup(context.Context, uuid.UUID) (Account, error)
    }
)

func NewWithdrawFundsCommand(withdraw Withdraw, service AccountService) WithdrawFundsCommand {
    return func(ctx context.Context, request WithdrawFundsRequest) (WithdrawFundsResponse, error) {
        sourceAccount, err := service.Lookup(ctx, request.SourceAccountID)
        if err != nil { ... }

        if sourceAccount.Balance() < request.Amount { ... }

        if err := withdraw(ctx, sourceAccount, request.Amount); err != nil { ... }

        return WithdrawFundsResponse{
            WithdrawnAt: time.Now().UTC(),
        }, nil
    }
}

func NewInsertFundsCommand(insert Insert, service AccountService) InsertFundsCommand {
    return func(ctx context.Context, request InsertFundsRequest) (InsertFundsResponse, error) {
        targetAccount, err := service.Lookup(ctx, request.TargetAccountID)
        if err != nil { ... }

        if err := insert(ctx, targetAccount, request.Amount); err != nil { ... }

        return InsertFundsResponse{
            InsertedAt: time.Now().UTC(),
        }, nil
    }
}

func NewTransferFundsCommand(withdraw WithdrawFundsCommand, insert InsertFundsCommand) TransferFundsCommand {
    return func(ctx context.Context, request TransferFundsRequest) (TransferFundsResponse, error) {
        var err error
        _, err = withdraw(ctx, WithdrawFundsRequest{
            CustomerID:      request.CustomerID,
            SourceAccountID: request.SourceAccountID,
            Amount:          request.Amount,
        })
        if err != nil { ... }

        _, err = insert(ctx, InsertFundsRequest{
            CustomerID:      request.CustomerID,
            TargetAccountID: request.TargetAccountID,
            Amount:          request.Amount,
        })
        if err != nil { ... }

        return TransferFundsResponse{
            TransferredAt: time.Now().UTC(),
        }, nil
    }
}

Consumers of the commands doesn't know, neither do they nor should they care, how any transaction works or where data is persisted. They simply want to manage funds! Commands can also make use of other commands, to enable a wider operation (such as withdraw + insert as illustrated above).

Unit testing the commands are super easy as well, using test doubles for the few direct dependencies directly available as function parameters.

Connecting a Command to a HTTP handler

It's easy to map a HTTP handler to a command:

func transferFundsHandler(transfer TransferFundsCommand) http.Handler {
    type (
        transferRequest struct {
            TargetAccountID string `json:"targetAccountId"`
            Amount          uint   `json:"amount"`
        }

        transferResponse struct {
            TransferredAt time.Time `json:"transferredAt"`
        }
    )

    convertAsCommandRequest := func(r *http.Request) (TransferFundsRequest, error) {
        defer r.Body.Close()

        var tr transferRequest
        if err := json.NewDecoder(r.Body).Decode(&tr); err != nil { ... }

        customerID, err := uuid.Parse(r.PathValue("customerId"))
        if err != nil { ... }

        sourceAccountID, err := uuid.Parse(r.PathValue("accountId"))
        if err != nil { ... }

        targetAccountID, err := uuid.Parse(tr.TargetAccountID)
        if err != nil { ... }

        return TransferFundsRequest{
            CustomerID:      customerID,
            SourceAccountID: sourceAccountID,
            TargetAccountID: targetAccountID,
            Amount:          tr.Amount,
        }, nil
    }

    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        request, err := convertAsCommandRequest(r)
        if err != nil { ... }

        response, err := transfer(r.Context(), request)
        if err != nil { ... }

        output := transferResponse{
            TransferredAt: response.TransferredAt,
        }

        if err := json.NewEncoder(w).Encode(output); err != nil { ... }
    })
}

Connecting the dots

func main() {
    ctx := context.Background()
    if err := run(ctx, os.Getenv); err != nil {
        panic(err) // panic() > os.Exit() since os.Exit() doesn't trigger deferred functions
    }
}

func run(ctx context.Context, getenv func(string) string) error {
    ctx, cancel := signal.NotifyContext(ctx, os.Interrupt)
    defer cancel()

    port, err := strconv.Atoi(getenv("HTTP_PORT"))
    if err != nil {
        return err
    }

    db, err := sql.Open("sqlserver", getenv("DB_CONNECTION_STRING"))
    if err != nil {
        return err
    }
    defer db.Close()

    withdraw := func(ctx context.Context, account Account, amount uint) error {
        // withdraw funds logic
    }
    insert := func(ctx context.Context, account Account, amount uint) error {
        // insert funds logic
    }
    accountService := NewAccountService(db)

    withdrawFundsCmd := NewWithdrawFundsCommand(withdraw, accountService)
    insertFundsCmd   := NewInsertFundsCommand(insert, accountService)
    transferFundsCmd := NewTransferFundsCommand(withdrawFundsCmd, insertFundsCmd)

    mux, err := httpCreateMux(
        withdrawFundsCmd,
        insertFundsCmd,
        transferFundsCmd)
    if err != nil {
        return err
    }

    mux = httpWithMiddleware(mux)
    return httpRunServer(cmd.Context(), mux, port)
}

func httpWithMiddleware(mux http.Handler) http.Handler {
    mux = httpEndpointSecurityMiddleware(mux)
    mux = httpLogMiddleware(mux)
    mux = httpOperationIdentityMiddleware(mux)

    return mux
}

func httpCreateMux(
    withdrawFundsCmd WithdrawFundsCommand,
    insertFundsCmd InsertFundsCommand,
    transferFundsCmd TransferFundsCommand,
) (http.Handler, error) {
    mux := http.NewServeMux()

    mux.Handle("POST /customers/{customerId}/accounts/{accountId}/withdraw",
        withdrawFundsHandler(withdrawFundsCmd))

    mux.Handle("POST /customers/{customerId}/accounts/{accountId}/insert",
        insertFundsHandler(insertFundsCmd))

    mux.Handle("POST /customers/{customerId}/accounts/{accountId}/transfer",
        transferFundsHandler(transferFundsCmd))

    return mux, nil
}

type ctxOperationID struct{}
func httpOperationIdentityMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        operationID := r.Header.Get("x-request-id")
        if operationID == "" {
            operationID = uuid.NewString()
        }

        ctx := context.WithValue(r.Context(), ctxOperationID{}, operationID)
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

func httpLogMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        operationID := r.Context().Value(ctxOperationID{}).(string)
        log("incoming request: %s", operationID)
        next.ServeHTTP(w, r)
    })
}

func httpRunServer(ctx context.Context, handler http.Handler, port int) (err error) {
    srv := http.Server{
        Addr:    fmt.Sprintf(":%d", port),
        Handler: handler,
    }

    go func() {
        if serverErr := srv.ListenAndServe(); serverErr != nil && !errors.Is(serverErr, http.ErrServerClosed) {
            err = serverErr
        }
    }()

    var wg sync.WaitGroup
    wg.Add(1)
    go func() {
        defer wg.Done()
        <-ctx.Done()

        if serverErr := srv.Shutdown(ctx); serverErr != nil && err == nil {
            err = serverErr
        }
    }()

    wg.Wait()

    return
}

The httpCreateMux function above covers the entire API surface, specifying all available routes and is where API-global middleware should be applied - such as endpoint security, logging, HTTP telemetry, etc. It has access to the commands available - but not to any direct dependencies nor to any configuration settings. This makes testing a breeze!

Extending a Command

This pattern allows for "function composition" and extendability, as long as the function type signatures is adhered to:

func wrapWithLogger(transfer TransferFundsCommand) TransferFundsCommand {
    return func(ctx context.Context, request TransferFundsRequest) (TransferFundsResponse, error) {
        log("new transfer funds request initiated by %s", request.CustomerID)

        response, err := transfer(ctx, request)
        if err != nil {
            log("transfer funds request failed: %s", err.String())
            return response, err
        }

        log("transfer funds request completed at %s", response.TransferredAt.Format(time.RFC3339))
        return response, err
    }
}

func wrapWithEventWriter(writer EventWriter, transfer TransferFundsCommand) TransferFundsCommand {
    type event struct {
        EventAt time.Time
        Subject string
    }

    convertAsEvent := func(request TransferFundsRequest) ([]byte, error) {
        e := event{
            EventAt: time.Now().UTC(),
            Subject: fmt.Sprintf("%s transfers %d money",
                request.CustomerID.String(),
                request.Amount),
        }

        return json.Marshal(e)
    }

    return func(ctx context.Context, request TransferFundsRequest) (TransferFundsResponse, error) {
        response, err := transfer(ctx, request)
        if err != nil { ... }

        e, err := convertAsEvent(request)
        if err != nil { ... }

        return response, writer.Publish(e)
    }
}

...

    transferFundsCmd := NewTransferFundsCommand(...)
    transferFundsCmd = wrapWithLogger(transferFundsCmd)
    transferFundsCmd = wrapWithEventWriter(eventWriter, transferFundsCmd)

The commands can easily be reused by other entrypoints besides HTTP requests, such as a monthly savings request initiated by a cron job. We can attach middleware (such as telemetry) per command as well, using the composition pattern above.

Using generics one could create helper functions to map commands as HTTP handlers, removing the specific *Handler-functions previously declared:

func mapCommandAsHandler[Req any, Resp any](
    cmd func(context.Context, Req) (Resp, error),
    mapRequest func(*http.Request) (Req, error),
    mapResponse func(Resp) ([]byte, error),
) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        request, err := mapRequest(r)
        if err != nil { ... }

        response, err := cmd(r.Context(), request)
        if err != nil { ... }

        output, err := mapResponse(response)
        if err != nil { ... }

        w.Write(output)
    })
}

func mapInsertFundsRequest(r *http.Request) (InsertFundsRequest, error) {
    defer r.Body.Close()

    var insertRequest struct {
        TargetAccountID string `json:"targetAccountId"`
        Amount          uint   `json:"amount"`
    }

    if err := json.NewDecoder(r.Body).Decode(&insertRequest); err != nil { ... }

    customerID, err := uuid.Parse(r.PathValue("customerId"))
    if err != nil { ... }

    targetAccountID, err := uuid.Parse(insertRequest.TargetAccountID)
    if err != nil { ... }

    return InsertFundsRequest{
        CustomerID:      customerID,
        TargetAccountID: targetAccountID,
        Amount:          insertRequest.Amount,
    }, nil
}

func mapInsertFundsResponse(response InsertFundsResponse) ([]byte, error) {
    return json.Marshal(struct{
        InsertedAt string `json:"insertedAt"`
    }{
        InsertedAt: response.InsertedAt.Format(time.RFC3339),
    })
}

...

    mux.Handle("POST /customers/{customerId}/accounts/{accountId}/insert",
        mapCommandAsHandler(cmd, mapInsertFundsRequest, mapInsertFundsResponse))

Each and every bit of code is (should be) easily unit tested, when using this approach. I often get around mocking service structs when using function types (as with the Withdraw and Insert types in the example code), which makes it ridiculously easy to create a test double.