DEV Community: Jon Calhoun

Crash Course on Go Interfaces

Jon Calhoun — Thu, 21 Jan 2021 16:41:49 +0000

This post was originally written for my mailing list. Feel free to sign up if you want to get Go tutorials like this delivered to your inbox every week or two!

In this article we are going to do a crash course on Go's interfaces, then in the rest of this series we will continue exploring how interfaces work, how to use them effectively, and more.

Let's start by talking a bit about how Go differs from dynamic languages like JavaScript, Ruby, and Python. When creating functions in dynamic languages, you don't define the type you expect to receive. Instead, anything can be passed in. Below is an example of this using JavaScript.

function Greeting(name) {
  return "Hello, " + name
}

// The following is all valid in JS, even if it doesn't make much sense.
Greeting("Jon")
// Output: "Hello, Jon"

Greeting(123)
// Output: "Hello, 123"

Greeting(document)
// Output: "Hello, [object HTMLDocument]"

In Go you have to explicitly state types, which means when you are creating a function you have to state that you want the name argument to be a string.

func Greeting(name string) string {
  return fmt.Sprintf("Hello, %v!", name)
}

For the most part this is a good thing. It makes it clear to anyone reading your code exactly what you expect, and it prevents other developers from accidentally passing in the wrong thing.

If you have ever used a dynamic language, chances are you have experienced errors because someone passed the wrong thing into a function - like passing in the entire document in the JS example! This type of error is far less likely to occur in Go because of the static type system.

Unfortunately, static typing also limits our code quite a bit. For instance, imagine you had some code that wrote our greeting to a file.

func WriteGreeting(name string, f *os.File) error {
  greeting := Greeting(name)
  _, err := f.Write([]byte(greeting))
  if err != nil {
    return err
  }
  return nil
}

This code works perfectly, but it is limited to writing to files. What if we wanted to instead write the greeting to a strings.Builder?

Without interfaces, we would be forced to rewrite the function to accept a strings.Builder!

func WriteGreeting(name string, sb *strings.Builder) error {
  greeting := Greeting(name)
  _, err := sb.Write([]byte(greeting))
  if err != nil {
    return err
  }
  return nil
}

Sidenote: Yes, I know strings.Builder has a WriteString method. It will make a bit more sense why I am opting to instead use Write in a bit.

Looking at the code for both of these functions, you will notice that they are very similar. Ignoring the variable name change, the only real difference is that in the first code sample we are using File.Write and in the second we are using Builder.Write, but in both of these cases the method we are using has the following definition:

Write(p []byte) (int, error)

It turns out, we don't really care if we are dealing with a strings.Builder or an os.File. All we really care is that we are given something with the Write method. With that our code would continue to work.

Luckily, Go has something called interfaces that enable us to express exactly this. Below is an example of an interface that would work for this particular case.

type Writer interface {
  Write([]byte) (int, error)
}

With the interface defined, we could rewrite our WriteGreeting function like so:

type Writer interface {
  Write([]byte) (int, error)
}

func WriteGreeting(name string, w Writer) error {
  greeting := Greeting(name)
  _, err := w.Write([]byte(greeting))
  if err != nil {
    return err
  }
  return nil
}

Now any type with the Write method can be passed into WriteGreeting! In fact, this interface is so common that it is defined in the standard library 👉 io.Writer

One of the neatest parts about Go is that it uses duck typing. To quote Mat (and whoever originally coined the term), "If it looks like a duck, and it quacks like a duck, then it is a duck".

(okay, technically it isn't duck typing, but "structural typing" doesn't have a catchy "quacks like a duck" quote! 😂)

Or in the context of Go, if it looks like a Writer interface, has the methods defined by the Writer interface, then it is a Writer interface implementation.

I mention duck typing because it isn't true in all statically typed languages. For instance, in Java you need to explicitly state that a class implements an interface, otherwise the compiler won't let you use it as an interface implementation. This has a big impact on how interfaces are used in Go compared to languages like Java, and we will discuss that a bit more in a future email.

Getting into more concrete details, any type that has every method defined by an interface will implement that interface. If a type has more methods than the interface defines that is okay too, but those additional methods will NOT be accessible while inside a function that accepts the interface.

func main() {
  var sb strings.Builder
  WriteGreeting("Jon", &sb)
}

func WriteGreeting(name string, w Writer) error {
  greeting := Greeting(name)
  // We can access Write here because it is defined by the Writer interface,
  // but we CANNOT access the Builder.WriteString method here even if we passed
  // in a strings.Builder because WriteString is not defined as part of the
  // Writer interface.
  _, err := w.Write([]byte(greeting))
  if err != nil {
    return err
  }
  return nil
}

type Writer interface {
  Write([]byte) (int, error)
}

Every method defined by the interface must be implemented for a type to implement an interface. The following example will result in a compiler error because our Demo type doesn't define all of the methods defined by BigInterface:

type BigInterface interface {
  A()
  B()
  C()
}

type Demo struct {}

func (Demo) A() {}
func (Demo) B() {}

func Test(bi BigInterface) {}

func main() {
  var demo Demo
  Test(demo) // errors because Demo doesn't implement the C() method.
}

Because of this, it is common to limit interfaces to specifically the methods you need rather than a large set of methods, as this makes it easier for other types to implement the interface.

Hopefully this helped shine some light on what interfaces are and why we might want to use them. In the next few articles I will dive into interfaces a bit more exploring topics like:

The downsides to interfaces, including a story about how the io.Writer and io.Reader interfaces confused me when I first started learning Go.
Common naming patterns for interfaces
How pointer methods affect whether a type implements an interface.
How duck typing changes the way we use interfaces in Go compared to languages like Java
Info about naming arguments for interface methods
Examples of where using interfaces can make your code worse.
How interfaces make testing easier.

That is a lot though, so it will be broken into a few posts and will be gradually released. Until then, give what you learned here a try in your code. Explore a bit and see what you discover 😁

Concurrency Patterns in Go: sync.WaitGroup

Jon Calhoun — Mon, 11 Jan 2021 18:24:45 +0000

This article was originally an email from my Go newsletter. If you are interested in learning a little bit about Go every week please consider subscribing.

When working with concurrent code, the first thing most people notice is that the rest of their code doesn't wait for the concurrent code to finish before moving on. For instance, imagine that we wanted to send a message to a few services before shutting down, and we started with the following code:

package main

import (
    "fmt"
    "math/rand"
    "time"
)

func notify(services ...string) {
    for _, service := range services {
        go func(s string) {
            fmt.Printf("Starting to notifing %s...\n", s)
            time.Sleep(time.Duration(rand.Intn(3)) * time.Second)
            fmt.Printf("Finished notifying %s...\n", s)
        }(service)
    }
    fmt.Println("All services notified!")
}

func main() {
    notify("Service-1", "Service-2", "Service-3")
    // Running this outputs "All services notified!" but we
    // won't see any of the services outputting their finished messages!
}

If we were to run this code with some sleeps in place to similuate latency, we would see an "All services notified!" message output, but none of the "Finished notifying ..." messages would ever print out, suggesting that our applicaiton doesn't wait for these messages to send before shutting down. That is going to be an issue!

One way to solve this problem is to use a sync.WaitGroup. This is a type provided by the standard library that makes it easy to say, "I have N tasks that I need to run concurrently, wait for them to complete, and then resume my code."

To use a sync.WaitGroup we do roughly four things:

Declare the sync.WaitGroup
Add to the WaitGroup queue
Tell our code to wait on the WaitGroup queue to reach zero before proceeding
Inside each goroutine, mark items in the queue as done

The code below shows this, and we will discuss the code after you give it a read.

func notify(services ...string) {
    var wg sync.WaitGroup

    for _, service := range services {
        wg.Add(1)
        go func(s string) {
            fmt.Printf("Starting to notifing %s...\n", s)
            time.Sleep(time.Duration(rand.Intn(3)) * time.Second)
            fmt.Printf("Finished notifying %s...\n", s)
            wg.Done()
        }(service)
    }

    wg.Wait()
    fmt.Println("All services notified!")
}

At the very start of our code we achieve (1) by declaring the sync.WaitGroup. We do this before calling any goroutines so it is available for each goroutine.

Next we need to add items to the WaitGroup queue. We do this by calling Add(n), where n is the number of items we want to add to the queue. This means we can call Add(5) once if we know we want to wait on five tasks, or in this case we opt to call Add(1) for each iteration of the loop. Both approaches work fine, and the code above could easily be replaced with something like:

wg.Add(len(services))
for _, service := range services {
  go func(s string) {
    fmt.Printf("Starting to notifing %s...\n", s)
    time.Sleep(time.Duration(rand.Intn(3)) * time.Second)
    fmt.Printf("Finished notifying %s...\n", s)
    wg.Done()
  }(service)
}

In either case, I recommend calling Add() outside of the concurrent code to ensure it runs immediately. If we were to instead place this inside of the goroutine it is possible that the program will get to the wg.Wait() line before the goroutine has run, in which case wg.Wait() won't have anything to wait on and we will be in the same position we were in before. This is shown on the Go Playground here: https://play.golang.org/p/Yl4f_5We6s7

We need to also mark items in our WaitGroup queue as complete. To do this we call Done(), which unlike Add(), does not take an argument and needs to be called for as many items are in the WaitGroup queue. Because this is dependent on the code running in a goroutine, the call to Done() should be run inside of the goroutine we want to wait on. If we were to instead call Done() inside the for loop but NOT inside the goroutine it would mark every task as complete before the goroutine actually runs. This is shown on the Go Playground here: https://play.golang.org/p/i2vu2vGjYgB

Finally, we need to wait on all the items queued up in the WaitGroup to finish. We do this by calling Wait(), which causes our program to wait there until the WaitGroup's queue is cleared.

It is worth noting that this pattern works best when we don't care about gathering any results from the goroutines. If we find ourselves in a situation where we need data returned from each goroutine, it will likely be easier to use channels to communicate that information. For instance, the following code is very similar to the wait group example, but it uses a channel to receive a message from each goroutine after it completes.

func notify(services ...string) {
    res := make(chan string)
    count := 0

    for _, service := range services {
        count++
        go func(s string) {
            fmt.Printf("Starting to notifing %s...\n", s)
            time.Sleep(time.Duration(rand.Intn(3)) * time.Second)
            res <- fmt.Sprintf("Finished %s", s)
        }(service)
    }

    for i := 0; i < count; i++ {
        fmt.Println(<-res)
    }

    fmt.Println("All services notified!")
}

Why don't we just use channels all the time? Based on this last example, we can do everything a sync.WaitGroup does with a channel, so why the new type?

The short answer is that sync.WaitGroup is a bit clearer when we don't care about the data being returned from the goroutine. It signifies to other developers that we just want to wait for a group of goroutines to complete, whereas a channel communicates that we are interested in results from those goroutines.

That's it for this one. In future articles I'll present a few more concurrency patterns, and we can continue to expand upon the subject. If you happen to have a specific use case and want to share it (or request I write about it), just send me an email.

Testing API Libraries

Jon Calhoun — Wed, 22 Jul 2020 20:01:43 +0000

This article was originally published on my website 👉 calhoun.io. If you enjoy it and want to check out some of my other articles I'd appreciate it 😁

Anymore it is nearly impossible to build something without needing to interact with one API or another. If you are anything like me, chances are you have had to write your own code to interact with one of those APIs, which usually means you also need to figuring out a way to test your code.

Unfortunately, testing API libraries isn't always easy. More often than not, the API you are testing against won't provide a test environment, or worse yet, they may provide a test environment that doesn't work the same as the production environment causing your code to break when you start using the production API.

In this article we are going to explore a few techniques I have found helpful when writing API client libraries in Go. I am also going to provide some advice based on my experience integrating with various APIs.

If a good test server is available, start with integration tests

Unit tests are great at catching errors when they are setup correctly, but when integrating with an API you cannot guarantee that this will be the case. The API docs might be outdated, you might be misinterpreting the docs, or any other number of scenarios might occur where your unit test is testing based on a faulty assumption. I can't begin to tell you how many times I have written code based on outdated or invalid docs.

When a test server is available, I recommend starting out with integration tests that make real API calls to the test server. Yes, I know that this will slow down your tests. I know that these tests will require an internet connection. Despite all of that, it will save you time in the long run because you will know for certain that your code is working with the real API.

When writing these integration tests, you will commonly need an API key of some sort. If that is the case, I suggest letting users provide it via a flag, and then using that flag to determine how to set things up or whether to skip tests.

var (
    apiKey string
)

func init() {
    flag.StringVar(&apiKey, "api_key", "", "Your secret API key for your test account")
}

func TestClient_CreateWidget(t *testing.T) {
    if apiKey == "" {
        t.Skip("skipping integration tests - api_key flag missing")
    }
    // ... run the test
}

// Alternatively, we can setup a test server for unit tests...

// Sets up a client for testing
func client(t *testing.T) *someapi.Client {
    c := someapi.Client{
        Key: apiKey,
    }
    if apiKey == "" {
        // We need a local test server
        handler := func(w http.ResponseWriter, r *http.Request) {
            // ... fill this in with a fake server of some sort
        }
        server := httptest.NewServer(http.HandlerFunc(handler))
        c.BaseURL = server.URL
        t.Cleanup(func() { server.Close() })
    }
    return &c
}

Not all API test servers are good

I added the "good" qualifier in the last section because test servers all fall on a spectrum from "utterly useless" to "amazing".

       |-----------------------|-----------------|------------|
Utterly useless        Sometimes useful        Good        Amazing

On the utterly useless end we have test servers that return responses that differ from what production returns. Or never being available to actually interact with.

Yes, test API servers that return different responses than production APIs actually exists, and I've had to write integrations with APIs that do this more than once. It is a baffling experience, but I mention it here so you know to watch out for it.

On the amazing end of the spectrum we have test servers like Stripe that give you reliable ways to simulate pretty much any behaviors you may need. You can create payment methods that will eventually have valid charges, payment methods that will eventually be rejected due to high fraud risk, and almost any other circumstance you can think of. They even provide a nice testing dashboard in their UI.

My general rule of thumb is to assume a test server is good until proven otherwise. Even if a test server doesn't allow me to test every possible scenario, integration tests will frequently help me discover errors in my understanding of the docs that likely wouldn't be caught with unit tests (because I would put those faulty understandings into my unit tests). I also believe it is easier to get the docs wrong than it is to return completely invalid data from a test server.

If all else fails, you might be able to create a second "live" account and test with it. When I was integrating with ConvertKit they were willing to provide me with an account for this very purpose, and while it did have limitations, it was still useful for sanity checking things.

Integration tests can be recorded and used as unit tests

When creating a Stripe API client in my Test with Go course, one technique I demonstrate there is how to record responses from integration tests to then use as part of your unit tests.

We won't be going in as much detail as the course does here, but I do want to walk through some of main points.

The source code for this is available here: https://github.com/joncalhoun/twg/tree/master/stripe

First, your API client needs to have a customizable BaseURL so we can tell it which server to talk to depending on whether or not we are running a unit or integration test.

type Client struct {
    BaseURL    string
  // ...
}

End users shouldn't need to set this, so you will want to use a default value if it isn't set.

const (
    DefaultBaseURL  = "https://api.stripe.com/v1"
)

// path will be a value like "/customers"
func (c *Client) url(path string) string {
  base := c.BaseURL
  if c.BaseURL == "" {
        base = DefaultBaseURL
    }
    return fmt.Sprintf("%s%s", base, path)
}

All of this allows us to set BaseURL in unit tests. This is often done with a server derived from httptest.NewServer.

server := httptest.NewServer(handler)
defer server.Close()
c := stripe.Client{
  BaseURL: server.URL,
}

If you already have recorded responses, you can setup the http server to return then.

Recording responses is done by replacing the HTTP client used by your API client. For instance, if we started with this Client type:

type Client struct {
  APIKey string
  BaseURL string
  HttpClient interface {
    Do(*http.Request) (*http.Response, error)
  }
}

We can replace HttpClient with the following recorderClient in a test, then when the test is done we can persist the recorded responses however we want. In my specific example I stored them in testdata as a JSON file.

// whatever data your test care about
type response struct {
    StatusCode int    `json:"status_code"`
    Body       []byte `json:"body"`
}

type recorderClient struct {
    t         *testing.T
    responses []response
}

func (rc *recorderClient) Do(req *http.Request) (*http.Response, error) {
    httpClient := &http.Client{}
    res, err := httpClient.Do(req)
    if err != nil {
        rc.t.Fatalf("http request failed. err = %v", err)
    }
    defer res.Body.Close()
    body, err := ioutil.ReadAll(res.Body)
    if err != nil {
        rc.t.Fatalf("failed to read the response body. err = %v", err)
    }
    rc.responses = append(rc.responses, response{
        StatusCode: res.StatusCode,
        Body:       body,
    })
    res.Body = ioutil.NopCloser(bytes.NewReader(body))
    return res, err
}

Finally, helpers used to setup your client combined with global flags can help determine how your client should be setup for testing.

func stripeClient(t *testing.T) (*stripe.Client, func()) {
    teardown := make([]func(), 0)
    c := stripe.Client{
        Key: apiKey,
    }
    if apiKey == "" {
        count := 0
        handler := func(w http.ResponseWriter, r *http.Request) {
            resp := readResponse(t, count)
            w.WriteHeader(resp.StatusCode)
            w.Write(resp.Body)
            count++
        }
        server := httptest.NewServer(http.HandlerFunc(handler))
        c.BaseURL = server.URL
        teardown = append(teardown, server.Close)
    }
    if update {
        rc := &recorderClient{}
        c.HttpClient = rc
        teardown = append(teardown, func() {
            for i, res := range rc.responses {
                recordResponse(t, res, i)
            }
        })
    }
    return &c, func() {
        for _, fn := range teardown {
            fn()
        }
    }
}

Using this setup I find that nearly all of my integration tests can be turned into unit tests and require no code changes whatsoever. I just need to remember to run the integration tests, record the responses, and commit the testdata directory.

While the unit tests won't be as valuable as the integration tests, they can be helpful in specific situations. Just be aware that unit tests should be viewed as helpers, not as a source of complete truth. That is what the integration tests are for.

Use convention over configuration where possible

In many cases you won't have access to a reliable test server, so you need to setup a local server to test with. When doing this, I highly recommend using convention over configuration.

What I mean is that instead of forcing every test case to specify every request/response combination it expects, setup a test server that has a few canned responses that you can use for your tests. When writing the ConvertKit API library I opted to convert a request's HTTP method and path into a string that was then used to determine what JSON and headers were returned.

For instance, if the following API call was made:

GET /forms/213/subscriptions

My test server would convert this into GET_forms_213_subscriptions and then would use two files in testdata to determine how to respond.

GET_forms_213_subscriptions.json → The JSON body to return.
GET_forms_213_subscriptions.headers.json → The headers and status code to return. When a header file was missing I just assumed a status code of 200 and no additional headers.

This made setting up any new endpoint really easy because I could just copy the sample response from the docs into a JSON file and then my local test server would use it in tests.

func TestClient_Account(t *testing.T) {
    c := client(t, "fake-secret-key")
    resp, err := c.Account()
    if err != nil {
        t.Fatalf("Account() err = %v; want %v", err, nil)
    }
    if resp.Name != "Acme Corp." {
        t.Errorf("Name = %v; want %v", resp.Name, "Acme Corp.")
    }
    if resp.PrimaryEmail != "you@example.com" {
        t.Errorf("PrimaryEmail = %v; want %v", resp.PrimaryEmail, "you@example.com")
    }
}

See https://github.com/joncalhoun/convertkit/blob/master/client_test.go#L74 for the code demonstrating how client sets up a test server along with a convertkit.Client. This also uses the new t.Cleanup from the testing package.

This works exceptionally well because it is quick and easy when you don't need anything custom, but if you do want to test something custom you can still do so. For instance, below is a test case where I wanted to verify that one of the request options was correctly being passed to the server when making an API request.

t.Run("page option", func(t *testing.T) {
    c := clientWithHandler(t, func(w http.ResponseWriter, r *http.Request) {
        r.ParseForm()
        got := r.FormValue("page")
        if got != "2" {
            t.Errorf("Server recieved page %v; want 2", got)
        }
        testdataHandler(t, "GET_subscribers_page_2")(w, r)
    })
    resp, err := c.Subscribers(convertkit.SubscribersRequest{
        Page: 2,
    })
    // Most of these checks don't matter much. The point of this test is
    // verifying that the options made it to the server.
    if err != nil {
        t.Fatalf("Subscribers() err = %v; want %v", err, nil)
    }
    if len(resp.Subscribers) != 1 {
        t.Errorf("len(.Subscribers) = %d; want 1", len(resp.Subscribers))
    }
    if resp.Page != 2 {
        t.Errorf("Page = %d; want 2", resp.Page)
    }
})

Write unit tests for complex decoding and encoding

While I do prefer to start with integration tests for API libraries, it is also worth noting that unit tests are still useful tools.

For instance, when I was normalizing variable JSON responses from the ConvertKit API I wrote unit tests for the UnmarshalJSON logic. I didn't necessarily have to do this, as the other tests would cover this case, but this gave me an easy way to verify that this specific code was correct. It also provides me with an easy way to add a new test case if I find out it has a bug in the future without needing to come up with a large integration or unit test that interacts with an API endpoint.

This article was originally published on my website 👉 calhoun.io. If you enjoy it and want to check out some of my other articles I'd appreciate it 😁

More Effective DDD in Go with Interface Test Suites

Jon Calhoun — Wed, 19 Feb 2020 17:17:00 +0000

This article is part of a larger series on Go application structure that was originally posted on calhoun.io, where I write about Go, web dev, testing, and more. I will be porting the entire series over to Dev.to, but I appreciate you checking out on my website and the Go courses that I create 😀

Previously in this series we started to explore domain driven design (DDD) and its benefits. As part of this exploration, we saw that one way to implement DDD in Go is to define interfaces at your domain so that we can write code that is agnostic of implementation details. This allows us to swap out say github for gitlab fairly easily.

Another benefit to having interfaces like these is that we can write tests that run against those interfaces. This not encourages us to write tests focused entirely on behavior, but it also helps ensure that any time we switch one implementation out for another that we won't have unexpected issues. These tests are commonly referred to as interface test suites.

What are interface test suites?

As I stated before, interface test suites are tests that accept an interface and run tests against it. But what does that actually look like?

Imagine that we have an application that needs to verify users. To do this, we might define an AuthService interface that has two methods: Login and Authenticate. The first method is used to log a user in when we don't know who they are and requires their email address and password. After that we generate a token for the user and use that along with the Authenticate method to authenticate them moving forward.

The domain code might end up looking like this:

package app

type Token string

type User struct {
  ID    string
  Email string
}

type AuthService interface {
  Login(email, password string) (Token, error)
  Authenticate(Token) (*User, error)
}

Now we can't test this interface without being provided an implementation, but we can still define some tests that express desired behavior. For instance,we might want to verify that after logging in a user that we can then use the token to authenticate that same user. We might also want to verify that an invalid email address and password combination doesn't somehow result in a token being returned. And there are a number of additional cases we might want to verify.

To do this, we would write a helper function that, given an implementation of AuthService, can run a series of tests to verify that the implementation does what we expect.

package apptest

func AuthService(t *testing.T, as app.AuthService) {
  t.Run("valid login", func(t *testing.T) {
    token, err := as.Login(validEmail, validPw)
    if err != nil {
      t.Errorf("Login() err = %v; want %v", err, nil)
    }
    if len(token) < minTokenLength {
      t.Errorf("len(token) = %v; want >= %v", len(token), minTokenLength)
    }
    // ...
  })
  t.Run("invalid login", func(t *testing.T) {
    // ...
  })
  // ...
}

A common approach is to place this into a subfolder, such as apptest for our app package, so that that is where the package name comes from. We will see later how that is called in an AuthService implementation's test file.

I call this a test suite not because it needs to be some massive thing, but because it isn't actually a test case. It is a helper function used to generate a collection of test cases, which are commonly referred to as a test suite.

Given that this is a test suite, that means these tests won't magically run on their own. We need to call this helper function whenever we implement the authentication service. For instance, if we had a jwt package that implemented the auth service, we might add the following code to it's jwt_test package.

package jwt_test

func TestAuthService(t *testing.T) {
  as := jwt.AuthService(...)
  apptest.AuthService(t, as)
}

In this particular example we are ONLY running the test suite, but you could also write implementation-specific tests if you wish. I am by no means encouraging you to ONLY use interface test suites, but for brevity that is all we see here.

What are the benefits of interface test suites?

When a user signs into our application, we use the Login method to create a token. That token might be a JWT, a remember token, or something else entirely. We don't really care so long as that token works to authenticate the user.

That means we might start with a remember token implementation:

package token

type AuthService struct {
  UserStore interface {
    Authenticate(email, password string) (*app.User, error)
  }
  TokenStore interface {
    User(app.Token) (*app.User, error)
    Create(userID string) (app.Token, error)
  }
}

func (a *AuthService) Login(email, password string) (app.Token, error) {
  user, err := a.UserStore.Authenticate(email, password)
  if err != nil {
    return "", err
  }
  token, err := a.TokenStore.Create(user.ID)
  if err != nil {
    return "", err
  }
  return token, nil
}

func (a *AuthService) Authenticate(t app.Token) (*app.User, error) {
  user, err := a.TokenStore.User(t)
  if err != nil {
    return nil, err
  }
  return user, nil
}

And then we might read a Hacker News article telling us about how cool JWTs are and decide to give them a shot:

package jwt

type AuthService struct {
  UserStore interface {
    Authenticate(email, password string) (*app.User, error)
  }
}

func (a *AuthService) Login(email, password string) (app.Token, error) {
  user, err := a.UserStore.Authenticate(email, password)
  if err != nil {
    return "", err
  }
  token := // build the JWT
  return token, nil
}

func (a *AuthService) Authenticate(t app.Token) (*app.User, error) {
  token, err := Parse(t) // not implemented here
  if err != nil {
    // eg if we couldn't parse the token for any reason - such as an invalid
    // signature
    return nil, err
  }
  // Otherwise pull this data from the JWT. You may need to use fields like
  // "Claims" depending on how you implement this.
  return &app.User{
    ID: token.UserID,
    Email: token.Email,
  }, nil
}

I have intentionally stubbed out or commented out most logic in both of these implementations for brevity, but I think the point still stands; the original AuthService could be implemented in a number of ways.

While we might not care about how each token is generated, there still might be other aspects of the AuthService interface that we care about and want to test across all implementations. For instance, we might expect a token to work for multiple calls to Authenticate rather than expiring after a single use. Or we might expect a specific type of error when a user provides an email address that doesn't exist so that we can inform the user that they do not have an account.

By using interface test suites we are able to express all of these requirements at the domain level, making it clear to anyone who implements the interface what is expected of them.

We are also able to write these tests once, and then with reasonable confidence - that is as long as the test suite is utilized - we can verify that each implementation matches our requirements and is very likely to work in our application.

Additionally, when you do find a bug with one of your implementations you can often write a test at the interface level and ensure that the bug isn't present in any existing or future implementations.

In short, interface test suites pair very nicely with the domain driven design pattern because we often already have interfaces defined as building blocks. They are also a great fit because interface tests can only test behavior, not implementation details, so any tests written this way are less likely to break if our application needs to evolve based on ever-changing requirements.

To reiterate, using interface test suites doesn't mean you can only use interface tests with DDD, but I do find that many unit tests defined at the implementation level can often be replaced with an interface test serving the same purpose which provides a longer lasting benefit.

But what if...

When writing interface test suites, a common problem is that you will realize you need additional data to actually write a useful test. This will usually present itself with a question like, "But what if I don't know what a valid email and password combination is?" or, "But what if I need to reset my auth service between tests and the interface doesn't have a method for that?"

When testing we often need to do things slightly differently from production. I realize that many people will scoff at this remark, but it is true.

When testing anything relating to a database, we will probably need to reset the database between some tests. How often do you reset your database in production? Should there even be a method for that in your production code?

Or when testing an authentication service like we explored here, we might need a way to get some valid email and password combinations. Surely we don't want any way to provide those in a production environment, but in a test we will need access to this information to write a useful test case.

There are a few ways to solve this problem, but my favorite is to just request that additional data in your test suite.

Going back to our authentication service example, we might request functions that will give us valid and invalid credentials for the AuthService implementation.

package apptest

type CredsFn func() (email, pw string)

// valid and invalid are functions we can use to get valid and
// invalid credentials.
func AuthService(t *testing.T, as app.AuthService, valid, invalid CredsFn) {
  t.Run("valid login", func(t *testing.T) {
    validEmail, validPw := valid()
    token, err := as.Login(validEmail, validPw)
    if err != nil {
      t.Errorf("Login() err = %v; want %v", err, nil)
    }
    if len(token) < minTokenLength {
      t.Errorf("len(token) = %v; want >= %v", len(token), minTokenLength)
    }
    // ...
  })
  t.Run("invalid login", func(t *testing.T) {
    invalidEmail, invalidPw := invalid()
    // ...
  })
  // ...
}

Then we can provide this information where we call the interface test suite function.

package jwt_test

func TestAuthService(t *testing.T) {
  validEmail, validPassword := "jon@calhoun.io", "fake-pw"
  us := mockUserStore{
    AuthenticateFn: func(email, password string) (*app.User, error) {
      if email != validEmail || password != validPassword {
        return nil, fmt.Errorf("invalid credentials")
      }
      return &app.User{
        ID: 123,
        Email: validEmail,
      }, nil
    },
  }
  as := jwt.AuthService{
    UserStore: us,
  }
  validFn := func() (email, pw string) {
    return validEmail, validPassword
  }
  invalidFn := func() (email, pw string) {
    return "invalid@email.com", "invalid-pw"
  }
  apptest.AuthService(t, as, validFn, invalidFn)
}

If this starts to become a burden, you can also provide a struct to make it easier to construct these test suites. For example, we might have some optional functions for resetting our auth service before or after each test case:

type AuthServiceSuite struct {
  AuthService *app.AuthService

  // REQUIRED
  Valid CredsFn
  Invalid CredsFn

  // Optional - useful for resetting a DB perhaps.
  BeforeEach func()
  AfterEach func()
}

func AuthService(t *testing.T, as AuthServiceSuite) {
  // ... use the AuthServiceSuite and its AuthService to run tests
}

The bigger point here is that while this data might not be provided as part of the interface definition, it is perfectly acceptable to request it as part of the testing process. Just try to be sure that the information you are requesting isn't locking you into a specific implementation.

One example in the standard library that I like to point people to is the nettest.TestConn test suite. This test suite is designed to test any net.Conn implementation, but interestingly enough it doesn't accept a net.Conn as part of its arguments:

func TestConn(t *testing.T, mp MakePipe)

Instead, TestConn requires a MakePipe type, which is a function that will return two connections such that the test can write to one connection and read whatever was written in the second connection. Additionally, MakePipe also requires a function to stop everything which is used for cleaning up any resources that may need cleaned up.

type MakePipe func() (c1, c2 net.Conn, stop func(), err error)

Note: The stop function here could even be a function that does nothing if your connections don't need cleaned up.

This entire setup allows the TestConn function to create and clean up a brand new pair of net.Conns for each test case. We could even try this pattern with our AuthService test suite.

package apptest

type Credentials struct  {
  Email string
  Password string
}

type MakeAuthService func() (as app.AuthService, teardown func(), valid, invalid Credentials)

// valid and invalid are functions we can use to get valid and
// invalid credentials.
func AuthService(t *testing.T, mas MakeAuthService) {
  t.Run("valid login", func(t *testing.T) {
    as, teardown, valid, invalid := mas()
    defer teardown()
    // run the test using as, valid, and invalid
  })
  t.Run("invalid login", func(t *testing.T) {
    as, teardown, valid, invalid := mas()
    defer teardown()
    // run the test using as, valid, and invalid
  })
}

We could even define a custom type for these subtests to make our tests look more like table driven tests.

type authTest func(t *testing.T, as app.AuthService, valid, invalid Credentials)

// valid and invalid are functions we can use to get valid and
// invalid credentials.
func AuthService(t *testing.T, mas MakeAuthService) {
  for name, test := map[string]authTest{
    // I try to mimic normal testing naming here minus the exporting part
    // You don't have to do that if you don't want to.
    "valid login": testAuthService_validLogin,
    "invalid login": testAuthService_invalidLogin,
    // ...
  }{
    t.Run(name, func(t *testing.T) {
      as, teardown, valid, invalid := mas()
      defer teardown()
      test(t, as, valid, invalid)
    })
  }
}

Wrapping up

While I don't expect everyone to read this and have an immediate use for interface test suites, I do think they are worth thinking about proactively. Interface test suites can be a very powerful tool, especially when paired with DDD.

That said, I am skeptical of suggesting that you use combine interface test suites with TDD. It isn't that it won't work - it definitely can, especially if you already have an implementation of an interface and have already written the interface test suite. Unfortunately, experience has shown me that when people try to practice TDD with interface test suites it can encourage bad behaviors like writing an interface before a use for it naturally evolves from the code. And if you haven't caught on by now, I am a big proponent for letting interfaces surface from your code rather than defining them upfront. After all, we aren't writing Java 😉.

Want to learn Go?

Interested in learning or practicing Go? Check out my FREE courses:

Gophercises - Programming Exercises for Budding Gophers
Algorithms with Go - An introduce to algorithms, data structures, and more using Go

I also have some premium courses that cover Web Dev with Go and Testing with Go that you can check out as well.

Moving Towards Domain Driven Design in Go

Jon Calhoun — Tue, 18 Feb 2020 23:00:55 +0000

This article is part of a larger series on Go application structure that was originally posted on calhoun.io, where I write about Go, web dev, testing, and more. I will be porting the entire series over to Dev.to, but I appreciate you checking out on my website and the Go courses that I create 😀

The goal of this article is to help illustrate how an application might evolve over time so that we can understand some of the problems that a more domain driven design might help with. To that end, we are going to look at a fairly trivial project as it evolves over time. This project won't be complete - the sample code won't compile, isn't tested with a compiler, and doesn't even list imports. It is simply meant to be an example to follow along with. That said, if anything seems wrong feel free to reach out and I'll fix it up or answer your questions (if I can!)

First, let's discuss the project. Imagine that you are at work and your boss asks you to create a way to authenticate users via the GitHub API. More specifically, you are going to be given a user's personal access token, and you need to look up the user as well as all of their organizations. That way you can later restrict their access based on what orgs they are a part of.

Note: We are using access tokens to simplify the examples.

Sounds easy enough, so you fire up your editor and whip up a github package that provides this functionality.

package github

type User struct {
  ID    string
  Email string
  OrgIDs []string
}

type Client struct {
  Key   string
}

func (c *Client) User(token string) (User, error) {
  // ... interact with the github API, and return a user if they are in an org with the c.OrgID
}

Note: I am not really using the GitHub API here - this is a mostly made up example.

Next you to take your github package and write some middleware that can be used to protect some of our HTTP handlers. In this middleware you will retrieve a user's access token from a basic auth header and then use the GitHub code to look up the user, check to see if they are part of the provided org, then grant or deny access accordingly.

package mw

func AuthMiddleware(client *github.Client, reqOrgID string, next http.Handler) http.Handler {
  return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    token, _, ok := r.BasicAuth()
    if !ok {
      http.Error(w, "Unauthorized", http.StatusUnauthorized)
      return
    }
    user, err := client.User(token)
    if err != nil {
      http.Error(w, "Unauthorized", http.StatusUnauthorized)
      return
    }
    permit := false
    for _, orgID := range user.OrgIDs {
      if orgId == reqOrgID {
        permit = true
        break
      }
    }
    if !permit {
      http.Error(w, "Unauthorized", http.StatusUnauthorized)
      return
    }
    // user is authenticated, let them in
    next.ServeHTTP(w, r)
  })
}

You present this to your peers and they are concerned with the lack of tests. More specifically, there doesn't seem to be a way to verify this AuthMiddleware works as advertised. "No problem," you say, "I'll just use an interface so we can test it!"

package mw

type UserService interface {
  User(token string) (github.User, error)
}

func AuthMiddleware(us UserService, reqOrgID string, next http.Handler) http.Handler {
  return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    token, _, ok := r.BasicAuth()
    if !ok {
      http.Error(w, "Unauthorized", http.StatusUnauthorized)
      return
    }
    user, err := us.User(token)
    if err != nil {
      http.Error(w, "Unauthorized", http.StatusUnauthorized)
      return
    }
    permit := false
    for _, orgID := range user.OrgIDs {
      if orgId == reqOrgID {
        permit = true
        break
      }
    }
    if !permit {
      http.Error(w, "Unauthorized", http.StatusUnauthorized)
      return
    }
    // user is authenticated, let them in
    next.ServeHTTP(w, r)
  })
}

Now you can test this code with a mock user service.

package mock

type UserService struct {
  UserFn func(token string) (github.User, error)
}

func (us *UserService) Authenticate(token string) (github.User, error) {
  return us.UserFn(token)
}

There are a bunch of different ways to create this mock user service, but this is a pretty common approach to test both cases where authentication works and cases where it returns an error.

The authentication middleware is then tested, released, and life seems jolly.

And then tragedy strikes. Your CEO hears that Godzilla is on his way to San Francisco and your company just can't keep using GitHub with that kind of uncertainty. What if their entire office is crushed and no engineers are around who know the product?!? Nope, that won't do. Completely unacceptable.

Luckily there is this alternative company named GitLab that seems to do a lot of the same things GitHub does, but they have a remote team. That means Godzilla can never wipe out all of their engineers, right? 🎉

The higher ups at your company seem to agree with this logic and they start to make the transition. Your job? You are tasked with making sure all of that authentication code you wrote works with the new system!

You spend some time looking at the GitLab API docs, and the good news is it looks like the same overall strategy will still work. GitLab has personal access tokens, organizations, and you just need to re-implement the client. The code in the middleware shouldn't need changed at all because you were a smart cookie and you used an INTERFACE! 😁 You get to work creating the GitLab client...

package gitlab

type User struct {
  ID    string
  Email string
  OrgIDs []string
}

type Client struct {
  Key   string
}

func (c *Client) User(token string) (User, error) {
  // ... interact with the gitlab API, and return a user if they are in an org with the c.OrgID
}

Then you go to plug that into the AuthMiddleware, but wait a minute, it won't work!

It turns out even interfaces can fall victim to coupling. In this case it is because your interface expects a github.User to be returned by the User method.

type UserService interface {
  User(token string) (github.User, error)
                      ^^^^^^^^^^^
}

What are you to do? Your boss wants this shipped yesterday!

At this point you have a few options:

Change your middleware so that the UserService interfaces expects a gitlab.User instead of a github.User
Create a new authentication middleware specifically for GitLab.
Create a common user type that will allow both your github and gitlab implementations to be interchangeable in the AuthMiddleware

(1) might make sense if you were confident that your company was going to stick with GitLab. Sure, you will need to change both the user service interface and the mocks, but if it is a one time change isn't so bad.

On the other hand, you don't really know that your org will stick with GitLab. After all, who let's Godzilla attacks dictate their decision process?

This option can also be problematic if many pieces of code throughout your application are reliant on the github.User type returned by this package.

(2) would work, but it seems a bit silly. Why would we want to rewrite ALL of that code and all of those tests when none of the logic is changing? Surely there must be a way to make this interface thing work as you originally intended. After all, that middleware really doesn't care how the user is looked up as long as we have a few critical pieces of information to work with.

So you decide to give (3) a shot. You will create a User type in your mw package and then you will write an adapter to connect it with the GitLab client you created.

package mw

type User struct {
  OrgIDs []string
}

type UserService interface {
  User(token string) (User, error)
}

As you write the code, you come to another realization; because you don't really care about things like a user's ID or email you can drop those fields entirely from your mw.User type. All you need to specify here are fields you actually care about, which should make things easier to maintain and test. Neato!

Next up you need to create an adapter, so you get to work on it.

// Package adapter probably isn't a great package name, but this is a
// demo so deal with it.
package adapter

type GitHubUserService struct {
  Client *github.Client
}

func (us *GitHubUserService) User(token string) (mw.User, error) {
  ghUser, err := us.Client.User(token)
  if err != nil {
    return mw.User{}, err
  }
  return mw.User{
    OrgIDs: ghUser.OrgIDs,
  }, nil
}

type GitLabUserService struct {
  Client *gitlab.Client
}

func (us *GitLabUserservice) User(token string) (mw.User, error) {
  glUser, err := us.Client.User(token)
  if err != nil {
    return mw.User{}, err
  }
  return mw.User{
    OrgIDs: glUser.OrgIDs,
  }, nil
}

You also need to update your mock, but that is a pretty quick change.

package mock

type UserService struct {
  UserFn func(token string) (mw.User, error)
}

func (us *UserService) Authenticate(token string) (mw.User, error) {
  return us.UserFn(token)
}

And now if you want to use our AuthMiddleware with either GitHub or GitLab you can do so with code something like this:

var myHandler http.Handler
var us mw.UserService
us = &GitLabUserservice{
  Client: &gitlab.Client{
    Key: "abc-123",
  },
}
// This protects your handler
myHandler = mw.AuthMiddleware(us, "my-org-id", myHandler)

Alas, we finally have a solution that is completely decoupled. We can easily switch between GitHub and GitLab, and when the new hip source control company kicks off we are prepared to hop on that bandwagon.

Finding a middle ground

In the previous example, we gradually watched the code go from what I consider tightly coupled to completely decoupled. We did this using the adapter package, which handles translating between these decoupled mw and github/gitlab packages.

The primary benefit is what we saw at the very end - we can now decide whether to use a GitHub or GitLab authentication strategy when setting up our handlers and our authentication middleware is entirely agnostic of which we choose.

While these benefits are pretty awesome, it isn't fair to explore these benefits without also exploring the costs. All of these changes presented more and more code, and if you look at the original version of out gitlab and mw packages they were significantly simpler than the final versions that need to make use of the adapter package. This final setup can also lead to more setup, as somewhere in our code we need to instantiate all of these adapters and plug things together.

If we continued down this route, we might quickly find that we need many different User types as well. For example, we might need to associate an internal user type with external user IDs in services like GitHub (or GitLab). This could lead to defining an ExternalUser in our database package and then writing an adapter to convert a github.User into this type so that our database code is agnostic to which service we are using.

I actually tried doing this on one project with my HTTP handlers just to see how it turned out. Specifically, I isolated every endpoint in my web application to its own package with no external dependencies specific to my web application and ended up with packages like this:

// Package enroll provides HTTP handlers for enrolling a user into a new
// course.
// This package is entirely for demonstrative purposes and hasn't been tested,
// but if you do see obvious bugs feel free to let me know and I'll address
// them.
package enroll

import (
  "io"
    "net/http"

    "github.com/gorilla/schema"
)

// Data defines the data that will be provided to the HTML template when it is
// rendered.
type Data struct {
  Form    Form
  // Map of form fields with errors and their error message
  Errors  map[string]string

  User    User
    License License
}

// License is used to show the user more info about what they are enrolling in.
// Eg if they URL query params have a valid key, we might show them:
//
//   "You are about to enroll in Gophercises - FREE using the key `abc-123`"
//                               ^             ^                   ^
//                             Course       Package               Key
//
type License struct {
  Key     string
  Course  string
  Package string
}

// User defines a user that can be enrolled in courses.
type User struct {
  ID string
  // Email is used when rendering a navbar with the user's email address, among
  // other areas of an HTML page.
  Email string
  Avatar string
  // ...
}

// Form defines all of the HTML form fields. It assumes the Form will be
// rendered using struct tags and a form package I created
// (https://github.com/joncalhoun/form), but it isn't really mandatory as
// long as the form field names match the `schema` part here.
type Form struct {
    License  string `form:"name=license;label=License key;footer=You can find this in an email sent over by Gumroad after purchasing a course. Or in the case of Gophercises it will be in an email from me (jon@calhoun.io)." schema:"license"`
}

// Handler provides GET and POST http.Handlers
type Handler struct {
  // Interfaces and function types here serve roughly the same purpose. funcs
  // just tend to be easier to write adapters for since you don't need a
  // struct type with a method.
  UserFn    func(r *http.Request) (User, error)
  LicenseFn func(key string) (License, error)

  // Interface because this one is the least likely to need an adapter
  Enroller interface {
    Enroll(userID, licenseKey string) error
  }

  // Typically satisfied with an HTML template
  Executor interface {
    Execute(wr io.Writer, data interface{}) error
  }
}

// Get handles rendering the Form for a user to enroll in a new course.
func (h *Handler) Get() http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
    user, err := h.UserFn(r)
    if err != nil {
      // redirect or render an error
      return
    }
    var data Data
    data.User = user

    var form Form
    err := r.ParseForm()
    if err != nil {
      // maybe log this? We can still render
    }
    dec := schema.NewDecoder()
    dec.IgnoreUnknownKeys(true)
    err = dec.Decode(&form, r.Form)
    if err != nil {
      // maybe log this? We can still render
    }
    data.Form = form

    if form.License != "" {
      lic, err := h.LicenseFn(form.License)
      data.License = lic
      if err != nil {
        data.Errors = map[string]string{
          "license": "is not valid",
        }
      }
    }

    h.Executor.Execute(r, data)
    }
}

// Post handles processing the form and enrolling a user.
func (h *Handler) Post() http.HandlerFunc {
  return func(w http.ResponseWriter, r *http.Request) {
    user, err := h.UserFn(r)
    if err != nil {
      // redirect or render an error
      return
    }
    var data Data
    data.User = user

    var form Form
    err := r.ParseForm()
    if err != nil {
      // maybe log this? We can still render
    }
    dec := schema.NewDecoder()
    dec.IgnoreUnknownKeys(true)
    err = dec.Decode(&form, r.Form)
    if err != nil {
      // maybe log this? We can still render
    }
    data.Form = form

    err = h.Enroller.Enroll(user.ID, form.License)
    if err != nil {
      data.Errors = map[string]string{
        "license": "is not valid",
      }
      // Re-render the form
      h.View.Execute(r, data)
      return
    }
    http.Redirect(w, r, "/courses", http.StatusFound)
    }
}

In theory this idea sounded pretty cool. I could now define all of my HTTP handlers in isolation without worrying about the rest of my application. Each package could be tested easily, and when I was writing these individual pieces I found myself incredibly productive. I even had an interface named Executor, and who doesn't want an executor in their code?!?

In practice, this idea was awful for my particular use case. Yes, there were benefits, but they weren't outweighing the cost of writing all this code. I was productive when creating the internals of the enroll and similar packages, but I spent so much time writing adapters and connecting pieces together that it crushed my productivity overall. I couldn't find a quick way to plug this into my code without needing to write a custom UserFn, LicenseFn, and I found myself writing a bunch of virtually identical variants of UserFn for every package with http handlers.

This leads to the topic of this section - is there a way to come up with a reasonable middle ground?

I like to decouple my code from third party dependencies. I like writing testable code. But I don't like doubling my coding efforts to make this happen. Surely there must be a middle ground here that gives us most of the benefits without all that extra code, right?

Yes, yes there is a middle ground and the key to finding it isn't to remove all coupling, but to intentionally pick and choose what your code is coupled to.

Let's go back to our original example with the github and gitlab packages. In our first version - the tightly coupled version - we had a github.User type that our mw package had a dependency on. It works well enough to get started, and we can even build interfaces around it, but we are still tightly coupled to the github package.

In our second version - the decoupled version - we had a github.User, a gitlab.User, and mw.User. This allowed us to decouple everything, but we have to create adapters that attach these decoupled pieces together.

The middle ground, and the third version we will explore, is to intentionally define a User type that every package is allowed to be tightly coupled to. By doing this, we are intentionally choosing where that coupling happens and can make that decision in a way that still makes it easy to test, swap implementations, and do everything else we desire from decoupled code.

First up is our User type. This will be created in a domain package that any other packages in our application can import.

package domain

type User struct {
  ID    string
  Email string
  OrgIDs []string
}

Next we will rewrite our github and gitlab packages to leverage this domain.User type. These are basically the same since I stubbed out all the real logic, so I'll only show one.

package gitlab // github is the same basically

type Client struct {
  Key   string
}

// Note the return type is domain.User here - this code is now coupled to our
// domain.
func (c *Client) User(token string) (domain.User, error) {
  // ... interact with the gitlab API, and return a user if they are in an org with the c.OrgID
}

And finally we have the mw package.

package mw

type UserService interface {
  User(token string) (domain.User, error)
}

func AuthMiddleware(us UserService, reqOrgID string, next http.Handler) http.Handler {
  // unchanged
}

We can even write a mock package using this setup.

package mock

type UserService struct {
  UserFn func(token string) (domain.User, error)
}

func (us *UserService) Authenticate(token string) (domain.User, error) {
  return us.UserFn(token)
}

Domain Driven Design

I have tried to avoid any confusing terms up to this point because I find that they often complicate matters rather than simplify them. If you don't believe me, go try to read any articles, books, or other resources on domain driven design (DDD). They will almost always leave you with more questions and less clarity about how to actually implement the ideas in your code.

I'm not suggesting that DDD isn't useful, nor am I suggesting that you shouldn't ever read those books. What I am saying is that many (most?) of my readers are here looking for more practical advice on how to improve their code, not to discuss the theory of software development.

From a practical perspective, the key benefit of domain driven design is writing software that can evolve and change over time. The best way I have discovered to achieve this in Go is to clearly define your domain types, and to then write implementations that depend upon these types. This still results in coupled code, but because your domain is so tightly linked to the problem you are solving this coupling is rarely problematic. In fact I often find that needing a clear definition of domain models to be enlightening rather than troubling.

Note: This idea of defining concrete domain types and coupling code to them isn't unique or new. Ben Johnson wrote about it in 2016 and this is still an incredibly valuable article for any new Gopher.

Going back to the previous example, we saw our domain being defined in the domain package:

package domain

type User struct {
  ID    string
  Email string
  OrgIDs []string
}

Taking that a step further, we could even start to define basic building blocks that the rest of our application can either (a) implement, or (b) make use of without being coupled to implementations details. For instance in the case of our UserService:

package domain

type User struct { ... }

type UserService interface {
  User(token string) (User, error)
}

This is implemented by the github and gitlab packages, and relied upon by the mw package. It could also be relied upon by other packages in our code without worrying about how it gets implemented. And because it is defined at the domain, we don't need to worry about each implementation altering slightly in return types - they all have a common definition to build from.

As an application evolves and changes over time this idea of defining common interfaces to build from becomes even more powerful. For instance, imagine we had a UserService that is a little more complex; perhaps it handles creating users, authenticating users, looking them up via tokens, password reset tokens, changing passwords, and more.

package domain

type UserStore interface {
    Create(NewUser) (*User, RememberToken, error)
    Authenticate(email, pw string) (*User, RememberToken, error)
    ByToken(RememberToken) (*User, error)
    ResetToken(email string) (ResetToken, error)
    UpdatePw(pw string, tok ResetToken) (RememberToken, error)
}

We might start by implementing this with pure SQL code and a local database:

package sql

type UserStore struct {
  DB *sql.DB
}

func (us *UserStore) Create(newUser domain.NewUser) (*domain.User, domain.RememberToken, error) {
  // ...
}

// ... and more methods

This makes complete sense when we have a single application, but perhaps we start to grow into another Google and we decide that we need a centralized user management system that all of our separate applications can utilize.

If we had coupled our code to the sql implementation this might be challenging to achieve, but because most of our code is coupled to the domain.UserService we can just write a new implementation and use it instead.

package userapi


type UserStore struct {
  HTTPClient *http.Client
}

func (us *UserStore) Create(newUser domain.NewUser) (*domain.User, domain.RememberToken, error) {
  // interact with a third party API instead of a local SQL database
}

// ...

More generally speaking, coupling to a domain rather than a specific implementation allows us to stop worrying about details like:

Are we interacting with a microservice or a local database? We can write code with reasonable timeouts regardless of whether our user management system is a local SQL database or a microservice.
Do we communicate with our user API via JSON, GraphQL, gRPC, or something else? While our implementation will need to know how to communicate with the users API, the rest of our code will continue to operate the same regardless of which specific technology we are using.
And much more...

At its crux, this is what I consider to be the primary benefit of domain driven design. It isn't fancy terms, colorful graphics, or looking smart in front of your peers. It is purely about designing software that is capable of evolving to meet your ever-changing requirements.

Why don't we just start here?

The obvious followup question at this point is, "Why didn't we just start with domain driven design if it is so great?"

Anyone with some experience using Model-View-Controller (MVC) can tell you that it is susceptible to the tight coupling. Nearly all of our application will need to depend on our models, and we just explored how that can be problematic. So what gives?

While building from a common domain can be useful, it can also be a nightmare if misused. Domain driven design has a fairly steep learning curve; not because the ideas are particularly hard to grasp, but because you rarely learn where you went wrong in applying them until a project grows to a reasonable size. As a result, it may take a few years until you really start to grasp all the dynamics involved. I have been writing software for quite a while, and I still don't feel like I have a full grasp on all the ways things can go wrong or get complicated.

Note: This is one of the big reasons why I have taken so long to publish this article. I was hesitant to share when, in many ways, I still don't feel like I am an expert on this topic. I ultimately decided to share because I believe others can learn from my limited understanding, and I believe this article can evolve and improve over time as I have discussions with other developers. So feel free to reach out to discuss it - jon@calhoun.io

MVC presents you with a reasonable starting point for organizing your code. Database interacts go here (models), http handlers go here (controllers), and rendering code goes here (views). It might lead to tight coupling, but it allows you to get started pretty quickly.

Unlike MVC, domain driven design doesn't present you with a reasonable starting point for how to organize your code. In fact, starting with DDD is pretty much the exact opposite of starting with MVC - rather than jumping right into building controllers and seeing how your models evolve, you instead have to spend a great deal of time upfront deciding what your domain should be. This likely involves mocking up some ideas and having peers review them, discussing what is/isn't right, a few iteration cycles, and only then can you dive into writing some code. You can see this in Ben Johnson's WTF Dial project where he creates a PR and discusses the domain with Peter Bourgon, Egon Elbre, and Marcus Olsson.

This isn't specifically a bad thing, but it also isn't easy to get right and it requires a great deal of upfront work. As a result, I often find this working best if you have a larger team where everyone needs to agree on some common domain before development can start.

Given that I am often coding in smaller teams (or by myself), I find that my projects evolve much more naturally if I start with something simpler. Maybe that is a flat structure, maybe it is an mvc structure, or maybe it is something else entirely. I don't get too caught up in those details, as long as I am open to my code evolving. This allows it to eventually take the form of something like DDD, but it doesn't require me to start there. As I stated before, this may be harder to do with a larger org where everyone is developing the same application together, so more upfront design discussion is often merited.

In our sample application we did something very similar to this "let it evolve" concept. Every step was taken for a specific purpose; we added a UserService interface because we needed to test our authentication middleware. When we started to migrate from GitHub to GitLab we realized that our interface didn't suffice, so we explored alternative options. It is around that point that I think a more DDD approach starts to make sense and rather than guessing at what the User and UserService should look like, we have real implementations to base it off of.

Another potential issue with starting of with DDD is that types can be defined poorly because we are often defining them before we have concrete use cases. For instance, we might decide that authenticating a user looks like this:

type UserAuthenticator interface {
    Authenticate(email, pw string) (error)
}

Only later we might realize that in practice every single time we authenticate a user we really want to have the user (or maybe a remember token) returned, and that by defining this interface upfront we missed this detail. Now we need to either introduce a second method to retrieve this information, or we need to alter our UserAuthenticator type and refactor any code that implements or utilizes this.

The same thing applies to your models. Before actually implementing a github and gitlab package we might think that the only identifying information we need on a User model is an Email field, but we might later learn through implementing these services that an email address can change, and what we also need is an ID field to uniquely identify users.

Defining a domain model before using it is challenging. We are extremely unlikely to know what information we do and don't need unless we already have a very strong grasp of the domain we are working in. Yes, this might mean that we have to refactor code later, but doing so will be much easier than refactoring your entire codebase because you defined a your domain incorrectly. This is another reason why I don't mind starting with tightly-coupled code and refactoring at a later date.

Finally, not all code needs this sort of decoupling, it doesn't always provide the benefits it promises, and in some circumstances (eg DBs) we rarely take advantage of this decoupling.

For a project that isn't evolving, you likely don't need to spend all the time decoupling your code. If the code isn't evolving, changes are far less likely to occur and the extra effort of preparing for them may just be wasted effort.

Additionally, decoupling doesn't always provide the benefits it promises and we don't always take advantage of that decoupling. As Mat Ryer likes to point out, we very rarely just swap out our database implementation. And even if we do decouple everything, and even if we do happen to be in the very small minority of applications who are transitioning databases, this transition often requires a complete rethinking of how we interact with our data store; after all, a NoSQL database behaves completely differently from a SQL database and to really take advantage of either we have to write code that is specific to the database being used. The final result is that these abstractions don't always provide us with the magical, "the implementation doesn't matter" results that we want.

That doesn't mean DDD can't provide benefits, but it does mean we shouldn't simply drink the Kool-Aid and expect magical results. We need to stop and think for ourselves.

In Summary

In this article we looked firsthand at the problems encountered when code is tightly coupled, and we explored how defining domain types and interfaces can help improve this coupling. We also discussed some of the reasons why it might not be the best idea to start off with this decoupled design, and to instead let our code evolve over time.

In the next article in this series I hope to expand upon the idea of writing Go code using domain driven design. Specifically, I want to discuss:

How interface tests can help ensure implementations can be interchanged without issue.
How subdomains can also stem from different contexts.
Ways you can visualize this all using the traditional DDD hexagon, as well as how code like a third party library might fit into the equation.

I also want to mention that this article is by no means a hard set of rules. It is just my meager attempt at sharing some insights and ideas that have helped me improve my Go software.

I'm also not the first to discuss or explore DDD and design patterns in Go. You should definitely check out some of the following for a more rounded understanding:

GoDDD by Marcus Olsen - This is a GitHub repo along with a writeup and a talk where Marcus Olsen explores porting the traditional sample DDD Java app into Go.
WTF Dial by Ben Johnson - I've already linked to Ben's Standard Pacakage Layout article; this is a series where Ben applies what he discusses in that article. I also recommend checking out the accompanying PRs and reading through the comments.
How Do You Structure Your Go Apps by Kat Zien - In this talk Kat goes through quite a few ways to structure your Go apps. Also check out the repo and slides that accompany this talk.
Design Philosophy On Packaging by Bill Kennedy - While not specifically about structuring your application, this series discusses package design which is tightly linked to structure.

Want to learn Go?

Interested in learning or practicing Go? Check out my FREE courses:

Gophercises - Programming Exercises for Budding Gophers
Algorithms with Go - An introduce to algorithms, data structures, and more using Go

I also have some premium courses that cover Web Dev with Go and Testing with Go that you can check out as well.

Using MVC to Structure Go Web Applications

Jon Calhoun — Tue, 22 Oct 2019 16:59:45 +0000

This article is part of a larger series on Go application structure that was originally posted on calhoun.io, where I write about Go, web dev, testing, and more. I will be porting the entire series over to Dev.to, but I appreciate you checking out on my website anyway 😀

What is MVC?

As you learn to program, you will eventually start to notice that some code bases are easy to navigate while others are confusing and hard to maintain or update. At first it is hard to tell what makes the difference, but over time you will quickly learn that the biggest factor is how well organized and structured the code is.

The reason for this is simple - if you need to jump in and start adding a new feature, fixing a bug, or doing anything else to a program it is much easier to do if you can guess where in the code you need to start looking. When you have no idea where to even start, you are left digging through hundreds of source files, but in a well structured program you can often guess where the code is even if you haven’t read the source code in the past.

Model-View-Controller, commonly referred to as MVC, is a pattern used to help organize and structure code. Specifically, most code will be classified as a model, view, or controller.

Views

Views are responsible for rendering data. That’s it. Given a specific page that we want to render, and data for that page, our view is responsible for generating the correct output. If we are implementing an application using MVC, this will typically be server-rendered HTML that we want to return to the end user’s browser, but it doesn't have to be. Our view could just as easily handle rendering XML, JSON, or other data types.

The important thing to remember is that the code in a view should have as little logic going on as possible; instead, it should focus entirely on displaying data. If logic starts to creep into your views you are likely doing something wrong, and it could become problematic down the road.

In my applications I like to have some global helper functions available in all templates that aren't provided by the html/template package. For instance, it is very common to want to render the current user's avatar or email address in the navbar, but providing this data in every single template render can be tedious. Or I may want helpers for rendering HTML forms via Go structs using something like my form package. None of these extra functions are a requirement of a view package, but I find them helpful when thinking about what my view is providing.

Controllers

Air traffic controllers are the people that inform each plane at an airport where to fly, when to land, and on which runway to land. They don’t actually do any piloting, but instead are in charge of telling everyone what to do so that an airport can operate smoothly.

Controllers in MVC are very similar to air traffic controllers. They won't be directly responsible for writing to a database or creating HTML responses, but instead they will direct incoming data to the appropriate models, views, and other packages that can be used to complete the work requested.

Similar to views, a controller shouldn’t be packed with too much business logic. Instead, they should mostly just parse data and ship it off to other functions, types, etc to handle.

Models

Models are the final part of MVC, and their primary responsibility is interacting with your application's data. This typically means communicating with your database, but it could also mean interacting with data that comes from other services or APIs. It might also mean validating or normalizing data.

A quick example - most web applications will have some notion of a user who signs up and interacts with the web application in some way. In an MVC application we might create a type User struct {...} in the models package that represents a user stored in our database. We might then create some code to help us validate that a user is valid and some additional code to help us create, update, and query for users in our database.

One important distinction that I'd like to make here is that I consider the models package to encompass database interactions. As we dive into some potential problems with MVC, we will look at how not doing this can lead to some potential problems.

MVC can be implemented in a variety of ways

When we talk about MVC it is important to note that MVC doesn't have a single specific implementation. It is a general architectural pattern, not a specific set of rules we must follow. As a result, two developers using MVC might end up with very different structures.

Part of this is because there are really two aspects of an app's structure that we need to consider:

How we organize code into types/funcs.
How we package up those types/funcs.

While MVC definitely gives us guidance on how to organize our code into types/functions, it doesn't necessarily dictate that we package our code into models, views, and controllers directories. As a result, I would consider the following two examples to both use MVC.

Example 1: Packaging our code based on it's layer

app/
  models/
    user.go
    course.go
  controllers/
    user.go
    course.go
  views/
    user.go
    course.go

In this first example we are organizing our code into packages based on its "layer" - or in other words, we package all the views together, all the controllers together, and so on. This is by far the most common layout you will see with MVC.

Example 2: packaging our code based on the resource

app/
  user/
    controller.go
    model.go
    store.go
    view.go
  course/
    controller.go
    model.go
    store.go
    view.go

In the second example we are opting to package our code based on the resource, and then inside of each package we have our model, view, and controller. This is less common for an MVC application, but I would still consider it to be MVC.

Sidenote: I consider both model.go and store.go to be the "models" part of MVC in this example.

Using that same line of reasoning, it is possible to use a flat structure and MVC at the same time.

Why do people like MVC?

People don't like seeing 30+ source files in a single directory/package. It just feels cluttered, regardless of whether or not the files are all separated in a logical way or not. I'm not really sure why this is the case, but developers just want to start adding subdirectories and "organizing" their code.

Without a deep understanding of what you are building, it can be challenging to break your application up into logical packages. While MVC itself doesn't provide a specific solution to this problem, MVC does provide us with an initial set of layers to start with. If we opt to organize our code into packages based on these layers (see Ex1 above) we can start with the models, views, and controllers directories without really having a deep understanding of what we are building. In other words, it allows us to pick a structure right away rather than waiting for one to evolve, and many developers don't like waiting.

MVC is also appealing because it is familiar. Change can often be confusing, so when coming to Go from Django, Rails, ASP.NET, and countless other languages/frameworks that use MVC it feels comfortable to dive into something familiar. We already understand how that code is organized, so doing something similar with our Go application allows us to make one less decision, and decision fatigue is a real thing.

Lastly, MVC is a pretty natural evolution from a flat-structured application where we often break our code into a database, handler, and rendering layer anyway. All three of these map pretty nicely to models, controllers, and views. More on this later.

Using MVC effectively

Contrary to what you may read on Reddit, MVC can be used quite effectively in Go. Frameworks like Buffalo take advantage of the MVC structure, and I have successfully built several applications using the MVC pattern.

Not everything is a model, view, or controller

The first key to using MVC with Go is to remember that not everything has to be a model, view, or controller. Yes, these categorizations are great starting points and will help minimize your decision making, but many parts of your application will not fall into any of these categories and that is okay.

For instance, in my course Web Development with Go we use the MVC pattern because I find it to be a very useful learning tool (more on that later), but it doesn't take very long before we start introducing additional packages. One example is middleware that are related to handling HTTP requests, but don't really fit into the controller package. When this occurs, it is okay to create additional packages in your codebase.

Don't break things up too much

The second key to using MVC is to avoid breaking things up too much for no real reason. I have seen a few people try to implement MVC this way:

app/
  models/
    lesson/
      models.go
      store.go
    course/
      models.go
      store.go
  # and so on...

The problem with this structure is that models tend to be relational; that is, a Course may have many Lessons, so we might want to write queries that return a course along with all of its lessons, and in the structure I just showed this typically requires additional types (like a CourseLesson) to avoid cyclical dependencies.

I have heard of developers having success with this pattern, so I'm not suggesting that it should never be used, but I personally haven't found it to fit my needs very well. As a result, it just isn't what I do.

If you happen to have examples of where this pattern works well feel free to send them to me - jon@calhoun.io

More generally speaking, if you find yourself with cyclical dependencies there is a good chance you either broke things up a bit too much or put a type/interface/func in the wrong place. Take Kat Zien's layered example from her talk, How do you structure your Go apps; in this example the code won't compile because the storage package and the models package have cyclical dependencies. While this seems to point with an issue in the application structure, I would actually argue that the models and storage package in this example should really be one package because they both pertain to storage. The models in this case just happen to be definitions of the types stored in said storage.

To be very clear - I am in no way criticizing Kat's ability to structure applications. I believe she created this cyclical dependency intentionally to make a point in her talk, and it serves as a great example for me as well. Oh and you should totally go check out her talk if you haven't already! 👉 Kat's Talk on YouTube

How should I structure my MVC apps?

Whether I am using MVC, Domain Driven Design (DDD), or another design pattern, I almost always prefer to package my code by the layer that it serves in my architecture. That is basically just a fancy way of saying I like to put all of my database code together in one package, my middleware in one package, my handlers in one package, and so on.

Sidenote: There are definitely times where packaging code in another way makes sense, so don't take this to mean you should always package by layer. This is just what tends to work well for me.

In MVC, that looks basically identical to what you would probably expect in an MVC application:

app/
  models/
    # user.go has all my database logic including:
    #   User - the model definition
    #   UserStore - used to perform user-related CRUD
    #     operations on the DB
    user.go
    course.go
  controllers/
    user.go
    course.go
  views/
    template.go # used to parse gohtml templates
    show_user.gohtml
    new_user.gohtml
    # ...

The only real caveat is that you don't really need to name these folders models, controllers, and views. You can, and that is perfectly fine by me if it makes development easier on you in any way, but you can also name these in a slightly more specific way. For instance, if our models are all stored in a PostgreSQL database, our controllers are all intended to handle HTTP requests, and our views are all used to render HTML, we might end up with packages like this:

app/
  psql/ # models
  http/ #controllers
  html/ #views

Now it is a little more clear that the psql.User represents a user in our Postgres database, and an http.UserHandler is pretty obviously a handler for http requests pertaining to users.

That doesn't mean you always have to do this. There are times when I have felt that a models package makes more sense. For instance, if I know my application is going to be fairly small and one of my models will be a file stored on disk (eg an image), I might opt to name my package models and mix-n-match all of my database and local storage code into that one package.

On the other hand, you could opt to break these two things into separate packages where one is psql specific, and the other is named localdisk or something like that.

I don't see a lot of value in criticizing someone's choice here because which decision is best almost always depends on the size and complexity of the application. What is more important to me is that you understand the tradeoffs you are making and are okay with them.

Lastly, it is worth mentioning that sometimes the reason for a package name is entirely unrelated to development/code. For instance, I prefer keeping the models package name in my web development course because this often makes it easier for people to grasp how we are separating our code, and thus makes it easier for them to pick up Go.

What goes in the `models` / `sql` package?

Models is probably the most important package to get right, so we are going to start with it. For simplicity's sake, let's assume that only means SQL database interactions.

The models package should contain pretty much all of your code related to your data storage. That includes database-specific models, validations, normalizations, etc. It is also worth recognizing that this package shouldn't generally be importing other packages in your application. As with all rules, there are exceptions, but I find that the models package works best if you could literally pull it out of your application and use it in another application (say a CLI) without changing any of the code in the models package. In this sense, all of your models code ends up being isolated and is limited to ONLY interacting with database entities. This will make it basically impossible to introduce cyclical dependencies because your models package doesn't know anything about the rest of your application.

Here are a few examples of logic I might expect to find in a models package:

Code that defines the User type as it is stored in the database.
Logic that will check a user's email address when a new user is being created to verify that it isn't already taken.
If your particular SQL variant is case sensitive, code that will convert a user's email address to lowercase before it is inserted into the DB.
Code for inserting users into the users table in the DB.
Code that retrieves relational data; eg a User along with all of their Reviews.

What I wouldn't expect to find in a models package is code specific to rendering your HTML responses. Eg I wouldn't expect it to return errors with HTTP status codes. You can return an error like NotFound but the controller should be free to interpret and render that as it sees fit. In the context of HTTP that might end up being an http.StatusNotFound status code, but in the context of a CLI it could be entirely different.

What goes in the `views` package?

This one tends to be the least code because it is usaully either:

A glorified wrapper around the html/template.Template package
A wrapper (or direct use of) the encoding/json package

The main way I tend to customize this is by adding a bunch of template.FuncMap functions to the template BEFORE parsing it (so it parses correctly), and then I'll override them using request-specific data if I need to. Eg:

package html

type Template struct {
  template *template.Template
}

func ParseFiles(files ...string) (*Template, error) {
  tpl, err := template.New("").Funcs(template.FuncMap{
    "flash": func() string { return "" },
  }).ParseFiles(files...)
  if err != nil {
    return nil, fmt.Errorf("parsing view files: %w", err)
  }
  return &Template{
    template: tpl,
  }, nil
}

func (t *Template) Execute(w http.ResponseWriter, r *http.Request, data interface{}) {
  // clone the template BEFORE adding user-specifics!
  clone := t.clone()

  // Get the flash cookie
    cookie, err := r.Cookie("flash")
    switch err {
  case nil:
    // override the flash msg
    clone.template = clone.template.Funcs(template.FuncMap{
      "flash": func() string { return cookie.Value },
    })
        // Delete the flash cookie so we don't repeat it.
        expired := http.Cookie{ ... }
        http.SetCookie(w, &expired)
    default:
        // noop
    }

  err = clone.template.Execute(w, data)
    if err != nil { ... }
}

Sidenote: I talk about some additional ideas to try with the view package in an older article Creating the V in MVC. I'm not saying I'd recommend them all at this point, but they are worth looking over as a mental exercise.

What goes in the `controllers` package?

If you get the other two right, this package ends up being pretty boring. You basically just create http.Handlers that parse incoming data, call methods on something like a UserStore or CourseStore provided by your models package, then finally render the results via a view (or redirect the user to an appropriate page in some cases).

// Take note: our UserHandler has a UserStore injected in!
// Global data stores can be problematic long term.
type UserHandler struct {
  Users    models.UserStore
  Sessions models.SessionStore
}

func (uh *UserHandler) Create(w http.ResponseWriter, r *http.Request) {
  err := r.ParseForm()
  if err != nil { ... }

  var form struct {
    Email    string `schema:"email"`
    Password string `schema:"password"`
  }
  schema.NewDecoder().Decode(&form, r.PostForm)
  user := sql.User{
    Email: form.Email,
    Password: form.Password,
  }
  err = uh.Users.Create(&user)
  if err != nil { ... }

  token, err := uh.Session.Create(&user)
  if err != nil { ... }
  // create session cookie using token

  http.Redirect(w, r, "/dashboard", http.StatusFound)
}

You might occasionally find yourself defining custom types to help render views, and to this day I'm still unsure of where the best place is for them. In one sense they are view specific, since your view will rely on a very specific set of data being passed in to render correctly. In that case they should probably be defined inside the views package (or a nested package). On the other hand, your controller could conceivably render data in various formats - HTML, JSON, or XML - so I could also see an argument for placing a creating a custom type that is "returned" from each handler inside the controllers package.

I think both can work, so if you do need custom types pick the one that works for you and roll with it. Even if you end up needing to change this decision, it shouldn't be too hard to refactor.

Don't blindly copy other languages

At this point I hope you are starting to see how MVC can be successfully implemented in Go, but I can't end this article without also discussing some of mistakes you might try to copy from other languages that utilized MVC.

Let's take a second to look at what a model might do in an MVC framework like Ruby on Rails:

While the model usually maps to the DB, it is also expected to serve the entire application's needs (eg UI rendering)
Your application has global access to models and other entities like current_user
Models have methods that trigger additional SQL queries based on relational data

While it is possible to implement many of these "features" in Go, I would never recommend it. So let's look at alternatives.

You will need to define resources more than once

In Rails we might define a model once and then expect it to work across our entire application. There are several ways we do this, but it mostly revolves around:

Creating methods that translate data
Creating wrapper types, like a decorator

While it is possible to create a Go web application where the only time you define your resources is in the models package, it is more likely that you will eventually need to define a second version of each resource. For instance, imagine we were rendering a User for a JSON response - if we were to use the models definition of a user we would need to add all of those json struct tags to the models package even though it has nothing to do with the code there. We would also potentially need to write custom marshalling code there to handle the fact that what we store in the database isn't exactly what we want to render to the user.

You are almost always better off just creating a new type and writing code to translate between the two:

package json // view

type User struct { ... }

package sql // model

type User struct { ... }

package http // controller

func sqlUserToJson(user *sql.User) json.User { ... }

Yes, it is more code, but you (hopefully) didn't pick Go because it was the least verbose language available to you.

Globals are a bad idea

In languages like Ruby or Python your server is most likely handling a single web request at a time. This isn't the fastest way to do things, but it does make development significantly easier because you can create global functions like current_user and know that they only pertain to the current request.

Go is nothing like Ruby or Python in this sense. When a web request comes into your Go server the standard library will spin up a new goroutine to handle it and then start to pass the request along to your handlers. This is incredibly handy from a developers perspective because we don't have to do anything to make our web server work concurrently, but it also means that adding a global function like current_user would lead to all sorts of race conditions.

Even a global database connection, which isn't as error prone, is ill-advised. While you might not have race conditions, a global database will allow SQL queries to be executed from anywhere in your application. If you were to instead just inject your data stores into the rest of your code it is much easier to track where queries are being made from, it is easier to test, and your life will generally just be much better.

func main() {
  userStore := sql.NewUserStore(...)
  // Inject the userStore where it is needed!
  userController := http.NewUserHandler(userStore)

  // ... setup the rest of your app
}

Don't embed DB connections to make relational queries possible

While it is possible to make relational queries work in Go much like they do in dynamic languages, I wouldn't recommend it.

package sql

type OrderStore struct {
  db *sql.DB
}

func (os *OrderStore) ByUserID(id int) ([]Order, error) {
  // ...
}

type User struct {
  ID int
  // ...

  // Don't do this
  db *sql.DB
}

func (u *User) Orders() ([]Order, error) {
  os := OrderStore{u.db}
  return os.ByUserID(u.ID)
}

The reasons are basically the same why we avoid a global database connection; it can lead to all sorts of hidden complexity. I write a bit more about this in my article Subtle issues with ORMs and how to avoid them, but the main takeaway is that when this is possible, a SQL query can literally be triggered from anywhere in your application, not just where you inject the OrderStore, and as a result it can make it very hard to manage your application.

In summary...

We spent the last bit of this article talking about things not to do, but really what I want you to take away from this article are all the things before that. I want you to understand that MVC can, and does, work in Go. That MVC is less about what you name your packages, and more about how you organize your code. If you keep that in mind I suspect you will have no problems using MVC in your next web application.

Want to learn Go?

Interested in learning or practicing Go? Check out my FREE course - Gophercises - Programming Exercises for Budding Gophers.

I also have some premium courses that cover Web Dev with Go and Testing with Go that you can check out as well.

Flat Application Structure in Go

Jon Calhoun — Tue, 22 Oct 2019 16:53:21 +0000

This article is part of a larger series on Go application structure that was originally posted on calhoun.io, where I write about Go, web dev, testing, and more. I will be porting the entire series over to Dev.to, but I appreciate you checking out on my website anyway 😀

Rather than spending time trying to figure out how to break code into packages, an app with a flat structure would just place all of the .go files in a single package.

myapp/
  main.go
  server.go
  user.go
  lesson.go
  course.go

A flat application structure is what almost everyone begins with when diving into Go. Every program in the Go tour, most exercises in Gophercises, and many others early Go programs don't get broken into any packages at all. Instead we just create a few .go files and put all of our code in the same (often main) package.

At first this sounds awful. Won't the code become unwieldy extremely quickly? How will I separate my business logic from my UI rendering code? How will I find the right source files? After all, a big part of why we use packages is to separate concerns while making it easier to navigate to the correct source files quickly.

Using a flat structure effectively

When using a flat structure you should still try to adhere to coding best practices. You will want to separate different parts of your application using different .go files:

myapp/
  main.go # read configs and start your app here
  server.go # overall http handling logic goes here
  user_handler.go # user http handler logic goes here
  user_store.go # user DB logic goes here
  # and so on...

Globals can still become problematic, so you should consider using types with methods to keep them out of your code:

type Server struct {
  apiClient *someapi.Client
  router *some.Router
}

func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
  s.router.ServeHTTP(w, r)
}

And your main() function should probably still be stripped of most logic outside of setting up the application:

// Warning: This example is VERY contrived and may not even compile.

type Config struct {
  SomeAPIKey     string
  Port           string
  EnableTheThing bool
}

func main() {
  var config Config
  config.SomeAPIKey = os.Getenv("SOMEAPI_KEY")
  config.Port = os.Getenv("MYAPP_PORT")
  if config.Port == "" {
    config.Port = "3000"
  }
  config.EnableTheThing = true

  if err := run(config); err != nil {
    log.Fatal(err)
  }
}

func run(config Config) error {
  server := myapp.Server{
    APIClient: someapi.NewClient(config.SomeAPIKey),
  }
  return http.ListenAndServe(":" + config.Port, server)
}

In fact, you could really use what is basically a flat structure with your code all in a single package and a separate main package where you define your command. This would allow you to use the common cmd subdirectory pattern:

myapp/
  cmd/
    web/
      # package main
      main.go
    cli/
      # package main
      main.go
  # package myapp
  server.go
  user_handler.go
  user_store.go
  ...

In this example your application is still basically flat, but you pulled out the main package because you had a need - like perhaps needing to support two commands using the same core application.

Why should I use a flat structure?

The key benefit of a flat structure isn't that we are keeping all of our code in a single directory or anything silly like that. The core benefit of this structure is that you can stop worrying about how to organize things and instead get on with solving the problems you set out to solve with your application.

I absolutely love how reminiscent of my PHP days this application structure feels. When I was first learning to code I started off with random PHP files with logic intermingled with all sorts of HTML and it was a mess. I'm not suggesting that we should build large applications way - it would suck - but I was less worried about where everything should go and more concerned with learning how to write code and solve my particular problems. Using a flat structure just makes it easier to focus on learning and building, whether you are learning about your application's needs, your domain, or how to code in general.

This is true because we can stop worrying about things like, "Where should this logic go?" because it is easy to fix a mistake if we make one. If it is a function, we can move it to any new source file in our package. If it is a method on the wrong type, we can create two new types and split up the logic from the original. And with all of this we don't have to worry about running into weird cyclical dependency issues because we only have one package.

Another big reason to consider a flat structure is that it is much easier for your structure to evolve as your application grows in complexity. When it becomes apparent that you could benefit from breaking code into a separate package, all you often need to do is move a few source files into a subdirectory, change their package, and update any reference to use the new package prefix. Eg if we had SqlUser and decided we would benefit from having a separate sql package to handle all our database related logic, we would update any references to now use sql.User after moving the type to the new package. I have found that structures like MVC are a bit more challenging to refactor, albeit not impossible or as hard as it might be in other programming languages.

A flat structure can be especially useful for beginners who are often too quick to create packages. I can't really say why this phenomenon happens, but newcomers to Go love to create tons of packages and this almost always leads to stuttering (user.User), cyclical dependencies, or some other issue.

Sidenote: In the next article on MVC we will explore how this phenomenon of creating too many packages can make MVC seem impossible in Go, despite that being far from the truth.

By putting off decisions to create new packages until our application grows a bit and we understand it better, budding Gophers are far less likely to make this mistake.

This is also why many people will encourage developers to avoid breaking their code into microservices too early - you often don't have enough knowledge to really know what should and shouldn't be split into a microservice early on and preemptive microservicing (I kinda hope that becomes a saying) will just lead to more work in the future.

A flat structure isn't all sunshine and rainbows

It would be disingenuous of me to pretend like there aren't any downsides to using a flat structure, so we should talk about those as well.

For starters, a flat structure can only get you so far. It will work for a while - probably longer than you think - but at some point your app will become complex enough that you need to start breaking it up. The upside to using a flat structure is that you can put this off and you will probably understand your code better when you do break it up. The downside is that you will need to spend some time refactoring at some point, and you might (maybe - but its a stretch) find yourself refactoring to a structure you wanted to start with anyway.

Naming collisions can also be awkward at times with a flat structure. For instance, let's say you want a Course type in your application, but how you represent a course in the database isn't the same as how you render a course in JSON. A quick solution to this is to create two types, but since they are both in the same package you need different names for each and may end up with something like: SqlCourse and JsonCourse. This isn't really that big of a deal, but it kinda sucks that we ended up with zero types named simply Course.

It also isn't always super simple to refactor code into a new package. Yes, it is usually pretty easy, but because all of your code is in one package you can occasionally run into code that is cyclical by nature. For example, imagine if our courses had an ID that always started with crs_ in the JSON response, and we wanted to return the price in various currencies. We might create a JsonCourse to handle that:

type JsonCourse struct {
  ID       string `json:"id"`
  Price struct {
    USD string `json:"usd"`
  } `json:"price"`
}

Meanwhile the SqlCourse only needs to store an integer ID and a single price in cents that we can format in various currencies.

type SqlCourse struct {
  ID    int
  Price int
}

Now we need a way to convert from SqlCourse to JsonCourse, so we might make this a method on the SqlCourse type:

func (sc SqlCourse) ToJson() (JsonCourse, error) {
  jsonCourse := JsonCourse{
    ID: fmt.Sprintf("crs_%v", sc.ID),
  }
  jsonCourse.Price.USD = Price: fmt.Sprintf("%d.%2d", sc.Price/100, sc.Price%100)
  return jsonCourse, nil
}

And then later we might need a way to parse incoming JSON and convert it into our SQL equivalent, so we add that to the JsonCourse type as another method:

func (jc JsonCourse) ToSql() (SqlCourse, error) {
  var sqlCourse SqlCourse
  // JSON ID is "crs_123" and we convert to "123"
  // for SQL IDs
  id, err := strconv.Atoi(strings.TrimPrefix(jc.ID, "crs_"))
  if err != nil {
    // Note: %w is a Go 1.13 thing that I haven't really
    // tested out, so let me know if I'm using it wrong 😂
    return SqlCourse{}, fmt.Errorf("converting json course to sql: %w", err)
  }
  sqlCourse.ID = id
  // JSON price is 1.00 and we may convert to 100 cents
  sqlCourse.Price = ...
  return sqlCourse, nil
}

Every step we have taken here made sense and felt logical, but we are now left with two types that MUST be in the same package otherwise they will present a cyclical dependency.

I find that issues like this are far less likely to occur when MVC, Domain Driven Design, and other app structures are used ahead of time, but if we are being honest this isn't really that hard to fix. All we really need to do is extract the conversion logic and place it wherever we use both types.

func JsonCourseToSql(jsonCourse json.Course) (sql.Course, error) {
  // move the `ToSql()` functionality here
}

func SqlCourseToJson(sqlCourse sql.Course) (json.Course, error) {
  // Move the `ToJson()` functionality here
}

Lastly, flat structures aren't hip; if you want to rock a sweet mustache and show your buddies at the coffee shop how awesome you are, this might not earn you bonus points. On the other hand, if you just want your code to work, this could be a good fit. 🤷‍♂️

Is a flat structure right for me?

First, let me make a general recommendation: Don't try to skip to the end in an attempt to avoid ever needing to refactor code on your way. It doesn't work, ever, and you will probably just end up doing more work that way. It is nearly impossible to predict future requirements for your software, and this is just one more way we try to do it as developers.

Not only is it unlikely to save you any time, you could also be doing yourself a disservice. Large enterprise organizations use more complicated code structures because they need to. Whether it is because they need to test with a variety of configurations, need rock-solid unit testing, or whatever else, there is pretty much always a reason they use the complicated structure they do. If you are a solo developer learning to code, or a small team trying to move quickly, your needs aren't the same. Trying to pretend like you are a large org without understanding why they chose the structure they did is more likely to slow you down than actually help you.

Sidenote: The caveat here is that if you know what you are doing this isn't always true, but I still find it to be true in more cases than not.

What this all means is that you should pick the structure that best suites your situation. If you are unsure of how complicated your application is going to be or are just learning, a flat structure can be a great starting point. Then you can refactor and/or extract packages once you have a better understanding of what your app needs. This is a point that many developers love to ignore - without building out an application it can often be hard to understand how it should be split up. This problem can also pop up when people jump to microservices too quickly.

On the other hand, if you already know your application is going to be massive - perhaps you are porting a large application from one stack to another - then this might be a bad starting point because you already have a lot of context to work from.

Additional considerations

A few additional things to keep in mind if you do opt to try out a flat structure:

Just because you are only working within one package doesn't mean you should avoid best practices; globals are generally bad, configuration should likely happen in main() (or perhaps a run() if you use that pattern), and init() is almost always a mistake.
Starting with a flat structure doesn't lock you into a single package. Break code into separate packages as soon as it becomes clear this will be beneficial.
You can still benefit from breaking code up into separate source files and using custom types.

Disclaimer - You are welcome to try things like init() and globals when learning. In fact, as a junior dev I think getting your code working and understanding it is more important than perfect structure because you will learn more by coding than by fretting over doing something wrong. Writing the initial, working version is usually far harder than refactoring it using Go best practices, and you are more likely to understand WHY seasoned developers make the recommendations they do after writing it the "bad" way. A similar example is using Redux in React; without ever experiencing the problems redux solves, you can't really appreciate what it does.

Want to learn Go?

Interested in learning or practicing Go? Check out my FREE course - Gophercises - Programming Exercises for Budding Gophers.

I also have some premium courses that cover Web Dev with Go and Testing with Go that you can check out as well.

Why Can't We Settle on a Single Web App Structure in Go?

Jon Calhoun — Tue, 22 Oct 2019 16:47:41 +0000

This article is part of a larger series on Go application structure that was originally posted on calhoun.io, where I write about Go, web dev, testing, and more. I will be porting the entire series over to Dev.to, but I appreciate you checking out on my website anyway 😀

Getting started in Go can be hard. The language itself is pretty easy to pick up, but figuring out how to structure your application can become overwhelming early on. At least it was a big time sink for me coming from a Ruby on Rails background where all of those early decisions were made for me.

As I progressed I kept wondering why I had to make all of these decisions myself. I would second guess myself, considering a framework (but everyone kept telling me not to touch those), and generally just feeling like I wasn't being productive.

Reading tutorials didn't help much either. They either felt so simple and contrived that they ignored hundreds of potential issues and questions, or it felt like I was learning how to draw an owl.

In a way, this seems a bit ridiculous. As I stated before, frameworks like Ruby on Rails take care of all of these early decisions for you, so why can't we settle on some form of web application structure in Go? Why are we instead constantly being told that it depends, or given vague advice that doesn't always feel actionable?

The goal of this article is it explore why exactly there isn't a single app structure that is recommended to new Gophers, and the overall goal of this series is help newcomers to Go understand what choices are available to them so they can get started a little easier.

Can't we just learn from other languages?

First, let's talk about languages like Ruby and Python. Why does it feel like they have this problem solved? Why can't we learn from them?

One of the biggest things working in favor of Ruby, Python, and many other languages is the adoption of frameworks in those languages. Ruby has Ruby on Rails. Python has Django (and a few others). In these languages, the actual programming language itself isn't suggesting any specific app structure for a web application. It is the framework that makes those suggestions, but in languages like Ruby it is so synonymous with "Rails" that people just assume this is how all Ruby web applications should be written.

If you were a Ruby developer and you really wanted to, there is nothing stopping you from using the standard library or something like Sinatra to build a web server without using Rails. Sure, it would take more setup than with Rails, but it would work and you could structure your web application however you wanted. At this point we are in the same position as we are with Go - there isn't a suggested app structure to follow and we need to figure one out on our own.

Most Ruby developers will suggest that you just use Rails; similarly, most Python developers will suggest a framework like Django. So why are Go developers suggesting you use the standard library rather than a framework?

The values of your programming language matter

Every language has a set of values. They might not always be explicitly stated, but they exist and will shape both the language and code that is written in it.

Understanding how a language's values can affect code written in it is easiest to grasp when comparing two languages with very different values. Since I know more about Ruby than most other languages, I will compare Ruby and Go.

Ruby is a dynamic language where development speed and flexibility are given a fairly high priority. Go, on the other hand, tends to prioritize clarity, readability, and explicitness over development speed. What does this look like in practice?

One example that sticks out to me is what type of reflection and metaprogamming is allowed in each language. In Go the reflect package is fairly limited, and developers are encouraged to avoid it. "Reflection is never clear" is even one of the Go Proverbs. As a result, metaprogramming isn't really possible in Go outside of code generation.

Sidenote: Metaprogramming is basically act of writing code that will analyze other code and generate new code. In languages like Go this might be akin to code generation, but in languages like Ruby you can dynamically define methods on types and do a wild array of other things **while your code is running 🤯* which makes it both powerful and insanely complicated to read and understand.*

Ruby has a slightly different take on reflection and metaprogramming. In fact, if you take a tutorial that teaches you Ruby, it is very likely that you will learn how to add new methods to the string type within the first few hours of that tutorial!

In Ruby you can even create a catch-all method (method_missing) that will be triggered anytime you try to call a method that isn't defined and then you can dynamically create that method at runtime if you want. In Go this simply isn't possible, and the reflect library is unlikely to ever allow it as it can lead to incredibly confusing code, so why is this done in Ruby?

Development speed!

Frameworks like Rails use metaprogramming to quickly create a new "language" within ruby that is flexible enough to work with a variety of different use cases, while not needing to be compiled or generated on a case-by-case basis. For instance, if you had an articles table in your Rails application with blog posts like this one, Rails makes it very easy to generate methods that create SQL queries for you:

Article.published.with_author.order(:published_at)

This particular query would probably end up running something like:

SELECT * FROM articles WHERE (published_at != null) AND (author != null) ORDER BY published_at DESC

Metaprogramming can be a little scary sounding at first, but once you get the hang of it, it is actually pretty cool and allows you to do some crazy things very quickly. For example, I once created a ruby admin framework that would dynamically scan all of your Rails database models and relationships and create a complete admin dashboard with links to joined resources, editing ability, and more. The craziest part was all you had to do to add these features to an application was install the library and add a single line of code to your application:

mount Upmin::Engine => '/admin'

That's it. After that it would dynamically look at your Rails code when your application started up create dynamic methods, instantiate all the pages, declare routes, etc. Talk about magic!

Absolutely none of this will ever exist in Go. I can't imagine a PR ever being accepted that allows developers to do anything this crazy in Go. You might be able to use code generation and some SQL database analysis to generate code in a similar way (SQLBoiler does something like this to generate an ORM), but in practice this is very different from metaprogramming in Ruby. For instance, you can actually read generated code and understand what it is doing, meanwhile in Ruby you might not be able to even find where a method gets defined in the source code because it is buried in a metaprogramming jumble.

Metaprogramming allows for fast development, but it does so at the cost of clarity, readability, and explicitness, all of which are core values of Go.

Frameworks aren't always as extreme as this metaprogramming example, but at their core frameworks tend to be about giving up explicitness in exchange for faster development speed. That is, developers using a framework agree to adhere to some restrictions imposed by the framework in exchange for having the framework do some of the work for them. This in turn allows them to create applications faster, but it isn't always as clear how things are happening. It can also lead to giving up control in some situations, as most frameworks don't allow you to customize everything.

Frameworks aren't commonly suggested in Go because these tradeoffs are at odds with Go's core values. I'm not saying there isn't a place for frameworks in Go - there is - but I don't ever see frameworks gaining the adoption you see in languages like Python or Ruby because most Go developers choose Go not for its development speed, but because they are willing to spend extra time writing coding if it means their code is clearer and more explicit. This is exactly why the try proposal was ultimately rejected.

Context is king

Rather than focusing on a single application structure, you are likely to have far more success in Go if you focus on understanding the context of your application.

Context can refer to a wide variety of factors; the size of your team, your experience as a programmer, your deployment environment, etc. Basically anything that might make your application structure decisions different from project to project is part of your context.

There is a great example of this in Go Time episode #94 where we discuss Structuring your Go apps.

Go Time 94: Structuring your Go apps – Listen on Changelog.com

In the episode the panel is talking about how we each start a new Go application, and Johnny Boursiquot and I gave different answers. I said I normally start out with just a main.go and see where it goes, meanwhile Johnny said he almost always starts out with an folder structure similar to this:

appName/
  cmd/
    appName/
        main.go

And then Johnny will try to keep that main.go very thin - it just does initialization, configuration, and calls out to other go packages that contain all of the real application logic.

So why are Johnny and I starting out so differently? Am I doing it wrong?

Let's take a look at our contexts; when I start a new project, I very frequently am creating what I'd call a throwaway program. A quick example to help a student having trouble with a concept. A silly program to press the spacebar in random intervals for a friend who is upset with WoW classic queues 😂. Or even a simple little program that helps my wife clean up duplicate files imported from her camera. The point is, most of these programs are going to be less than a few hundred lines of code and will be a single .go file. Starting off with anything more than that is probably overkill, so in my personal context, this makes a lot of sense.

On the other hand, I suspect that Johnny is frequently building applications in the context of his work at Heroku; other team members are likely to get involved in the project, having consistent app structure across all their apps has real value, and the way he starts his projects makes complete sense.

This is just one example using a really simple aspect, but there are countless others.

Are you a big company that values correctness over moving fast and breaking things? Chances are you will focus on testability a lot more than a startup with your app structure.
Does your team mind testing with a real (not prod) database, or do you prefer more unit testing? This can affect what app structures work out for you.
Are you a relatively new developer, or an seasoned veteran? This would have a drastic effect on what structure you should start with.
Are you deploying with a variety of microservices, or as a single monolithic app? Yep, that changes your app structure too.

This list could literally go on forever, as there are an infinite number of factors that could differentiate applications. And therein lies the problem - there isn't a one size fits all structure because developers all have different needs.

Sidenote: Not shockingly, Peter Bourgon makes a point to bring up context in Go Time #102 - On application design and brings up a few of these example contexts. If you haven't already, I recommend checking that episode out as well.

In summary...

In summary, we can't settle on a single application structure in Go because there are just too many contexts to consider, BUT that doesn't mean we can't give new Gophers better advice. And that is the goal of this series.

I want to spend the next few articles in this series exploring some of the app structures you might use in your next project. I'll try to present them in the order I tend to see developers progress on their own, as this also tends to map directly to their complexity. The current list of structures are:

Flat structure
Model, view, controller (MVC)
Domain driven design (or something resembling it)

Along with exploring how each potential app structure looks, we will also talk about the pros and cons of each. When we get to MVC we will also spend some time discussing mistakes that can make it feel like MVC won't work in Go (eg cyclical dependency mistakes).

As an added bonus, I hope to also dive into things like globals that don't exactly alter how you organize code, but do alter the overall structure of your code in many ways. Globals will still pop up in some of the other articles in the series, but we will really focus on them at the end of this series.

By the end of this series I hope you will have a good grasp on a few app structures you can try out as you venture into building web applications with Go.

Want to learn Go?

Interested in learning or practicing Go? Check out my FREE course - Gophercises - Programming Exercises for Budding Gophers.

I also have some premium courses that cover Web Dev with Go and Testing with Go that you can check out as well.

How do you get the most out of tutorials, courses, and other coding resources?

Jon Calhoun — Wed, 05 Jun 2019 14:43:05 +0000

As someone who teaches in tech, I am always interested in learning ways that educational material can be improved. Part of that is understanding how people are currently using the material at their disposal. So let's talk about how you are using your coding courses, tutorials, blog posts, and everything else out there.

Sidenote: I have written about this a little more here, but I'm positive I am missing all sorts of things others are doing so I'm creating this #discuss post to hear what is working for you!

Here are some ideas to get the conversation starts:

How do you practice what you are learning?

Do you just code along with the tutorial?

Do you create your own app afterwards?

Do you do the entire tutorial first, then create an app, or build an app that is different from the tutorial using it as a guide?

How often do you reference the same resource?

I am possibly at an odd extreme here; not only will I reference some blog posts and tutorials more than once, but I'll even find myself referencing articles that I wrote because I forget things after not using them long enough.

How often do you find yourself reusing resources?

Do you bookmark the great ones? How do you organize your collection of tutorials?

What courses or tutorials do something awesome that allows you to learn more easily?

For instance, a really small detail in the React tutorials is the highlights on changed lines: https://reactjs.org/tutorial/tutorial.html#passing-data-through-props

It is subtle, but goes a long way in making it clearer what changed.

Or another awesome idea is having a mentor. Sites like Exercism.io do this, but it is incredibly hard to scale.

Everything from themed courses (eg Rails for Zombies) or learning through a game - I want to hear what has made your learning experience better and how.

How do you learn to go beyond the tutorial?

One of the biggest complaints against tutorials and courses that hold your hand is that you aren't "learning to code on your own." How do you escape this position and learn to go beyond what is taught in a tutorial?

Do you add new features to whatever you build in the tutorial?

Do you try implementing the same feature using different libraries or approaches?

Getting the most out of coding courses & tutorials

Jon Calhoun — Wed, 05 Jun 2019 14:33:55 +0000

I originally emailed a variation of this to my mailing list and I am sharing it here in case it can help others.

As a teacher of Go, I tend to hear all sorts of feedback about the many different ways of learning to code. For instance, I once had someone tell me that video courses were more harmful than helpful because they lead to developers who couldn't think on their own and were reliant on someone to hold their hand to build anything. On the other hand, I've had students who use my video courses and then go on to obtain jobs as a Go developer.

So what gives?

How are some of the developers using courses and tutorials and becoming reliant on them while others are becoming successful developers?

I think the key difference between these two outcomes is how each developer uses the resources they have at their disposal, and in this post we are going to explore a few ways to really get the most out of the plethora of material available out there to learn programming, Go, and really anything else.

I'm going to go into more detail with each of these, but here is a rough outline of the main points I want to cover:

Don't just copy the code in the tutorial; apply what you learn.
Learning resources should be utilized more than once.
Expand upon tutorials to get more out of them.

Don't just copy the code; apply it

The first point I want to discuss is that practice is a necessity for learning any new skill, and simply copying the code in a tutorial is NOT practicing. At least, it isn't sufficient practice.

Far too many developers pick up a book or a course and expect to read it and "know how to program" after finishing. The truth is, this rarely happens. Why? Because you can't expect to know how to do almost anything after just reading about it. You need to practice!

Sidenote: This is also why titles like "Teach Yourself Go in 24 Hours" are comically deceptive. At best you can expect to familiarize yourself with the language in that amount of time, but to actually learn any programming language you are going to need to actually write some code in that language.

While coding along with a tutorial can be a great starting point, simply coding what someone tells you to code is mostly mindless. You don't have to think about what you are doing or really prove that you understand the material. As a result, only coding along with the tutorial is unlikely to produce the desired results.

Instead, what I highly recommend is some combination of (a) trying to code the project on your own before using the resource, (b) using your resource as an outline or guide, but not relying on it directly, or (c) applying what you learn in other projects/problems after learning it.

What does this look like in the real world? Let's look at a few examples.

First, let's look at a developer who wants to use Gophercises to get some practice writing real Go code. How can they do that?

My recommendation would be to watch the videos where they introduce each exercise, then to stop the videos and try to code each exercise on your own. It is going to be hard; in many cases what might be possible to code in an hour could take you days. But that is okay. The goal here isn't speed, but to force yourself to read the Go docs and to try to piece together how to use the various libraries necessary to solve the problem at hand.

You might get stuck along the way, at which point it is fine to watch the video for ideas. That is why the videos are there - to show you how another developer might approach the problem - but I do think that it is important to give yourself a chance to solve each problem first.

Alternatively, or perhaps if you get stuck and need to watch a video, another approach is to watch the video first to get an idea of what you are doing and then to give yourself some time - perhaps a few hours or a day - and then come back and try to code the same exercise without watching the video. Don't worry about getting it exactly the same as the video, but just try to solve it on your own now that you've seen it in a video.

My bigger point with both of these approaches is that we aren't just letting someone hold our hand and tell us exactly what to code. Sure, watching those videos IS useful in understanding one way to solve the problem and in learning new techniques, but we can't really expect to improve our own development skills by just watching. We need to code some.

This code first approach works well with something like Gophercises which is setup to be exercise-oriented, but what happens when we want to learn from a course or tutorial that teaches us a new set of skills we have no experience with and isn't organized as exercises?

For our second example, let's look at a few ways to get the most out of my Web Development with Go course assuming that you don't actually know how to build a web application.

Sidenote: The examples I give here will be specific to this example, but the techniques we see here are generally applicable to any tutorial or course out there.

Unlike our first example, it isn't really feasible to start writing a web application without first learning how to do it. We might not know how to properly create an authentication system, and that really isn't something we should be winging. So what do we do?

In cases like these I encourage developers to watch the videos (or read the book), and to then use them as an outline. For instance, in my Web Development course we create a photo gallery application, and your first pass through the course videos it probably makes sense to build the same application, but that isn't where I would stop. I would then do a second pass through the course material (or if you prefer, do a second pass through each chapter as you complete it) and try to build something similar, but not quite the same. Maybe a Twitter clone, or a Pinterest clone, or even a mini Reddit.

The goal here is to pick something close enough to the original project that we can use it as a good guide, but to make it different enough that we are forced to think about what we learned in each video and apply it to our unique application. By doing this, we force ourselves to truly understand the material, because if we didn't we wouldn't be able to adapt what we learned for our specific project.

Sidenote: If you are an experienced developer coming to Go from another language you can often do this on your very first pass through of the material.

Lastly, there are going to be tutorials where you can't really use them as an outline and you also can't really code all of the problems without following along and learning the material first. A great example of this are algorithms, where you can't write a bubble sort without understanding what it is, but it is hard to use a tutorial or screencast as an outline without just outright copying their code.

In these cases I highly recommend learning the material and then applying it in other projects or practice problems. Going back to our algorithms example, the best way to REALLY ensure you know how an algorithm works is to find many problems that use that algorithm and to try to solve each of them.

Sidenote: Sites like Topcoder and HackerRank can be great for finding practice problems, but don't get so caught up in those that you don't learn practical skills.

I really cant' emphasize this enough, even if you are the type of person who feels like you "get it" by just watching the video or reading the tutorial. While studying computer science at university I found that a vast majority of my peers could sit through a CS class and afterwards if I were to talk to them about the new algorithm we learned about, they could explain it quite well, but if you actually asked them to code it they would struggle.

With nearly all coding concepts you need to actually put it into practice on your own project before the material really sinks in. So don't just watch a video or read book. Actually go build something or solve a practice problem using what you learned.

Use resources more than once

Another common mistake I will see is the "once and done" mistake. What I mean is that developers will read a tutorial or go through a course once and then be done with it. This is a bad idea for so many reasons.

The first is that you will very often not have all the knowledge you need to really put the entire picture together your first pass through. What I mean by this is that things we learn in programming (and in most areas of life) have these weird cyclical dependencies where we need to understand A to understand B, but we also need to understand B to understand A, so how do we ever get started?

The answer is that we start off with little bits and just ask the developer to come back to the parts they don't understand a little later.

For instance, when a developer is first learning Go we might tell them, "Don't worry about this package main part - we will learn what it does later. For now just understand that our main function is going to run and whatever code we put in there will run as soon as we start our program."

Then once that programmer understands their program we might teach them about packages, and then finally come back and explain what the special main package is for.

When we get into more complex subjects like testing and web development this gets even harder because there are so many moving parts. The truth is, it isn't really possible to understand it all the first time you read about it. So just go through the material, get what you can out of it, but plan to revisit it later once you have a solid foundation to build from.

Expand upon what is taught

Another big difference I see between those successful with courses and those who are not is that the successful ones almost always try to expand upon what is being taught.

A simple example is to take a simple tutorial that teaches you how to build a web server with a few pages and to add new pages, but this can be taken well beyond that.

For instance, my Web Development course is intentionally designed so that you can swap out the DB implementation with pretty much any implementation you want very easily, and I encourage students to do this. Many students have already taken advantage of this and have rewritten that part of the application using a different database library (eg sqlx, an alternative sql library) while others have completely replaced our Postgres database with MongoDB.

It is unrealistic to expect a tutorial for everything you will need to do in your day-to-day job as a developer. Rather than getting bummed out by this, the more successful developers take what they are learning and try to stretch it beyond its limits and then discuss what does and doesn't work with others.

This can be challenging the first few times, but with practice you start to actively think about how you might adapt a tutorial for your needs and it becomes much easier with time and experience.

What works best for you?

I suspect if anyone out there learning to code were to take half of the advice here and apply it when using tutorials, books, or video courses that they would see a drastic improvements in what they learn and retain.

That said, I'm likely missing a lot here and I'd love to know what other tips or advice you have. What has worked well for you? Let me know what works for you in the replies, or check out the #discuss post I created to help start the conversation.

Want to learn Go?

Interested in learning or practicing Go? Check out my FREE course - Gophercises - Programming Exercises for Budding Gophers.

I also have some premium courses that cover Web Development with Go and Testing with Go that you can check out, and I write about Go a lot on Calhoun.io as well.

TDD is Not for Me

Jon Calhoun — Fri, 15 Mar 2019 15:45:05 +0000

This article was original posted on calhoun.io, where I write about Go, web dev, testing, and more.

This will be the second time I've written about an unpopular opinion I have about testing, and I'm sure it will ruffle a few feathers. I don't always practice Test Driven Development (also known as TDD), and I believe there are many situations where practicing TDD is more of a hinderance than a boon.

I'm going to explore why I don't always practice TDD in a moment, but I want to be clear beforehand that this is not a TDD bashing article. In reality, I think learning about and trying TDD is incredibly beneficial for developers. TDD forces developers to think about development and design from another perspective; rather than focusing on, "How am I going to implement this?" they instead need to step back and think, "How would I use this function?" While this change may seem minor, it can result in drastic improvements to a codebase.

So why am I writing this article?

I am writing this article because there are many developers out there who struggle with TDD and feel poorly as a result. Whether it is imposter syndrome or just feeling like they have a dirty secret they need to hide, those feelings exist in far more developers than most people realize because everyone is too worried to admit that they just can't make TDD work for them. This is made even worse when a developer hears others talking about how TDD is this amazing thing that made them so productive. In short, there are many coders out there who feel like crap and they shouldn't; TDD is a great learning tool but I can say firsthand that it isn't always effective, and in many cases it just hinders my ability to produce high quality code quickly.

Alright, now let's dig into why I don't practice TDD all the time.

TDD focuses too much on unit testing

Test driven development is often taught as a process with three steps: Red, Green, Refactor.

The idea here is pretty simple:

The Red step is where you write a test that will fail because it tests something you haven't implemented yet.
The Green step involves writing code to make that test pass
The Refactor step is where you restructuring your code into something more maintainable while using your tests to ensure you don't break anything. You DO NOT add any new functionality during this step.

I tried to follow TDD pretty strictly for a bit, but at times it just felt like a massive pain; it kept slowing me down in every once in a while rather than helping me be more productive.

I couldn't really put my finger on exactly why that was the case until I read this twitter thread by Jeffrey Way:

// Detect dark theme var iframe = document.getElementById('tweet-1102673086799855616-220'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1102673086799855616&theme=dark" }

I only include the first tweet here, but you should go read the entire thread. It is well worth your time.

Jeffrey puts into words what I have been struggling with for quite some time; TDD is hard to learn and grasp when we focus so much on unit tests and completely ignore the more complex scenarios that every developer is bound to run into sooner or later.

In nearly every case where I find using TDD troublesome, it almost always stems from me trying to unit test some code where I need to mock out everything under the sun.

TDD is supposed to help us take a step back from the implementation and instead focus on how the code might be used, but when I am writing unit tests where a bunch of dependencies need mocked out that is no longer true. I am forced to once again start thinking about implementation details like, "Will my code need access to a database?" and "What about encoding? Do we need to support multiple output formats? Should I inject a mock encoder for the test?"

It just doesn't feel natural to me to think about code this way. It isn't beneficial to start thinking about what dependencies I'll need before I even start to use them. Instead, I work better when I take a step back and write a feature test. Something like, "When I POST to the /orders path with the JSON {"price": 123, ...} I expect to get the following JSON back with an ID that matches the ord_[a-zA-Z0-9]{8,}$ pattern."

Not only is this type of test incredibly easy to write - we just spin up an app, hit the endpoint, and check the results - but it is also gets me back into the correct mindset. I'm now thinking about how someone might use actually use the code; I'm thinking about another developer interacting with my API, or a real person filling out a form and submitting it.

There are obviously exceptions to this. For instance, if I'm writing a function to factor numbers TDD could lead to a reasonable solution to the problem. The key components here is that we aren't really focusing on mocks and dependencies; we are instead writing an isolated function and TDD can shine in situations like these assuming we don't fall into the second trap of thinking we have to write the absolute minimum amount of code at all times.

I'm also not saying you shouldn't ever write unit tests where you mock things. These tests can provide value in many environments. I just don't find myself practicing TDD as often when writing my unit tests. Instead, I unit test my feature tests and then write unit tests as I see fit.

Not all code should be written one test case at a time.

TDD is often taught using the following rules:

You are not allowed to write any production code unless it is to make a failing unit test pass.
You are not allowed to write any more of a unit test than is sufficient to fail; and compilation failures are failures.
You are not allowed to write any more production code than is sufficient to pass the one failing unit test.

These rules were authored by Robert C. Martin (Uncle Bob) and can be found on the website: butunclebob.com

While this can work in many situations, I wholly disagree with the idea that all code should be written this way.

What I expect to happen here is for someone to link me a video, a blog, or some other example where the developer uses TDD to derive some complicated algorithm. One popular example is writing a function to determine the factors of a number. I've also seen articles where the author explores whether it is possible to derive something like quicksort via TDD.

Surely if these more complicated algorithms can be derived through TDD then it must work in all cases, right?

While TDD can be used at times to derive a reasonable algorithm, I have also seen countless instances where it has worked in the exact opposite way. By using TDD the developer derived an algorithm that was incredibly slow, inefficient, and sometimes even missed edge cases because it is nearly impossible to think of every edge case, let alone test them all.

The truth is, there are situations where you actually need to sit down and think about a little more than just expected inputs and outputs for a function. The easiest example to grasp is probably a sorting algorithm - while you might derive quicksort from a TDD approach, you could just as easily derive an algorithm that is far slower and less efficient. Even if you did come up with an efficient quicksort, most standard libraries use something a little more complex than this because it isn't as efficient to use quicksort with smaller sized lists.

Going beyond algorithms, there are also times where constantly context switching just hurts overall productivity. While it might work well for many developers to constantly write one test, then one or two lines of production code, test again, refactor, then repeat, I personally find this constant context switching a distraction. I find that I am far more effective when I :

Write a few test cases demonstrating the basic functionality I expect.
Spend time to think about how I might achieve that functionality.
Implement a rough version that gets my tests passing.
Refactor as necessary.

This is pretty similar to TDD, but it isn't quite the same and I'm sure if I taught it as my version of TDD many would tell me I'm "doing it wrong". 🤷‍♂️

Wrapping up

I find TDD to be beneficial at times and I'm not saying we should abandon it. Instead, what I am trying to convey is that getting caught up in this strict set of rules defining what is and isn't TDD is a mistake.

We should instead take the lessons we can learn from TDD and apply them in the way that is most effective for ourselves. If that means we end up breaking a few of the rules, so be it. After all, the goal of TDD, agile, or really any development process is to make us better at our job, and if they aren't doing that then something needs to change.

Want to learn Go?

Interested in learning or practicing Go? Check out my FREE course - Gophercises - Programming Exercises for Budding Gophers.

I also have some premium courses that cover Web Dev with Go and Testing with Go that you can check out as well.

Testing is Not for Beginners

Jon Calhoun — Wed, 13 Mar 2019 14:17:29 +0000

I'm a big fan of testing. I write about it on my blog, I email my mailing list about it, I discuss it with other developers in my spare time, and I have even gone as far as creating a course that teaches testing with Go.

Despite having a fondness for testing, I don't recommend it for beginners.

Crazy, right? In this article I plan to explore why that is in a little more detail, but it basically boils down to two points:

Beginners don't know enough to write anything but the most trivial tests. This inevitably leads to my second point...
Trying to learn the skills required to actually write realistic tests while also trying to learn how to code and build things is overwhelming.

I guess it is kinda one point. Whatever. I'm breaking it up into two because I think it makes it a little easier to consume.

I know many of you may not agree with me at this point, but please give the article an honest read. If after reading it all you don't agree I'm happy to discuss it with you. After all, I'm here to learn as well 🙃

This article was original posted on calhoun.io, where I write about Go, web dev, testing, and more.

Beginners don't know enough to write anything but the most trivial of tests

Whenever a beginner writes code their primary goal isn't to separate concerns, avoid global variables, or to write testable code. Honestly, most beginners probably don't know what half of those things really mean. Instead, their primary goal is simple; it is to get the damn thing to work. That's it.

Confirming this hardly takes any effort. Take some time to review some code written by beginners.

Looking at a web app written in Go? Chances are they are writing SQL queries anywhere and everywhere in that code, and theres a pretty good chance that their DB connection is a global variable.

Looking at a Rails app? Chances are there is business logic in the views and the controllers have tons of logic jam packed into them.

Looking at a PHP web app? It wouldn't shock me if ALL of the logic is in a single php file - parsing a form, interacting with the DB, etc.

Even if we look at something simpler - say a calculator with limited functionality - we are still going to run into issues like this. It isn't that beginners don't care; they simply don't know any better.

Beginners don't know what dependency injection is. They don't understand how global variables make testing hard. They probably don't even know what mocking is, so expecting them to understand how to design code that is easily mocked is quite the stretch.

As a result, the only tests that really make sense to a beginner are the really simple ones like below.

func Add(a, b int) int {
  return a+b
}

// And a test...
func TestAdd(t *testing.T) {
  got := Add(2, 4)
  want := 6
  if got != want {
    t.Errorf("Add() = %d; want %d", got, want)
  }
}

While I don't have any problem with showing this to a beginner and giving them an idea of what testing is, I think it is pretty ridiculous to show them this code and pretend that it is anything like a real test.

So what ultimately ends up happening is we try to teach them more. We try to teach them what dependency injection is, why global variables make it hard to test, how time.Now() can make it hard to verify edge cases, and more. And this is where I start to get worried, because we are no longer teaching a beginner how to program. At this point, we are teaching them how to program, how to build things, AND HOW TO TEST all at the same time. And that brings me to my second point...

Trying to learn the skills required to actually write realistic tests while also trying to learn how to code and build things is overwhelming

Like before, I want you to think about code written by a beginner, but this time I want you to recall some of the first programs you wrote. Think about the first thing you made that was composed of more than one source file. Or maybe think about the first web page you made.

Now if you were anything like me, your first web app might have looked something like this:

<p>
  <?php
  // This may not work. I don't know PHP anymore.
  $name = $_GET['name'];
  echo "Hello, " . $name;
  ?>
</p>

A work of art, isn't it?

Now imagine if you just managed to write that code for the first time and someone told you that you should be testing your code. And you should be using React. And you should be using a framework. Oh and you will want to setup a database, and probably setup GraphQL to query it.

I'm not sure why, but as developers we have this habit of taking what we do now - after years of experience and practice - and expecting others, especially beginners, to be doing the same things right away. It is ridiculous. It is like expecting someone to be able to jump right into calculus just because we learned trig, algebra, and more which eventually lead to us learning and using calculus to solve a particular problem.

Just because something works well for you, doesn't mean it is a good idea for beginners. They might not have the context, the experience, or the practice required to really benefit from what you are using. Or maybe the problems they are tackling are just much simpler, and tossing in all that complexity is overkill.

It's like we all forgot that we gradually learned how HTTP request work. How headers work. How cookies work. How forms work. How POSTing to a web server works - or even that there ARE different HTTP methods. And through all of this we probably learned through some good old fashioned trial-and-error.

I don't actually think testing is at fault here, but the real problem is that we have this stigma that you should learn how to code, testing, web development, and a million other things all at the same time. I don't really know how this came to be, but I suspect part of the issue is that we never qualify things. A beginner asks, "what should I learn?" and we tell them, "learn testing, and react, and graphql, and go, but only use the standard library..."

No, no, no! Just stop.

This is ridiculous to me because of how obvious it is in any other scenario. If you were teaching someone how to play soccer you would start off with the basics like how to pass and how to dribble. You wouldn't start them off with videos of Ronaldo and say, "This is how the professionals do it." So why the hell are we doing this with programming beginners?

We try to make it better by saying, "well of course they should know not to try to learn that all at once", but they don't! And what makes this even worse is that when budding developers fall into this trap it leads to them feeling like shit the minute they get stuck. They feel like they just don't have what it takes to be a developer, and it is a shame because many of them would love programming if they didn't slam into this brick wall.

And that brings me to my real point - most of us learn better if we focus on learning a few things at a time. We want to be challenged, to try new things, but we don't want to be so overwhelmed that we are paralyzed. Trying to learn testing plus everything else - like how to build a web app, how http works, how cookies work, etc - is an easy way to get overwhelmed. As a result, I usually recommend learning other things first, then coming back and learning testing later. You can always revisit your old projects and see how you would redesign it to apply what you are learning about testing, but that is only possible if you don't get overwhelmed, get frustrated, and eventually quit.

But what if I want to learn testing?! (and the million other "what ifs")

Cool, go for it! You are more than welcome to learn testing first, and then learn about web dev or any other topic. I'm sure some people have done this, and you might enjoy it. When I say testing isn't for beginners, I'm not saying it is an awful topic to learn. What I'm saying is that trying to learn testing plus everything else is a mistake.

Most people want to learn how to build web apps or something visual first, but that doesn't mean you have to start there. You can probably learn testing first. Some of it might not make quite as much sense without first experiencing the pains, but by all means don't let me stop you from learning what you want to learn.

This also doesn't mean you can't learn while pair programming or anything like that. When in environments with a mentor you can often learn a lot despite being overwhelmed because you have someone there to guide you through it all. When you get lost you aren't just stuck, and you don't feel like a failure. Someone is around to tell you, "You are doing great, this is just a ton to take in. Try X and Y instead next time!" In short, getting overwhelmed, then stuck, and finally quitting isn't as much of a concern in these scenarios.

Pssstt...

Interested in learning or practicing Go? Check out my FREE course - Gophercises - Programming Exercises for Budding Gophers.

I also have some premium courses that cover Web Dev with Go and Testing with Go that you can check out as well.

DEV Community: Jon Calhoun

Crash Course on Go Interfaces

Concurrency Patterns in Go: sync.WaitGroup

Testing API Libraries

If a good test server is available, start with integration tests

Not all API test servers are good

Integration tests can be recorded and used as unit tests

Use convention over configuration where possible

Write unit tests for complex decoding and encoding

More Effective DDD in Go with Interface Test Suites

What are interface test suites?

What are the benefits of interface test suites?

But what if...

Wrapping up

Want to learn Go?

Moving Towards Domain Driven Design in Go

Finding a middle ground

Domain Driven Design

Why don't we just start here?

In Summary

Want to learn Go?

Using MVC to Structure Go Web Applications

What is MVC?

Views

Controllers

Models

MVC can be implemented in a variety of ways

Example 1: Packaging our code based on it's layer

Example 2: packaging our code based on the resource

Why do people like MVC?

Using MVC effectively

Not everything is a model, view, or controller

Don't break things up too much

How should I structure my MVC apps?

What goes in the models / sql package?

What goes in the views package?

What goes in the controllers package?

Don't blindly copy other languages

You will need to define resources more than once

Globals are a bad idea

Don't embed DB connections to make relational queries possible

In summary...

Want to learn Go?

Flat Application Structure in Go

Using a flat structure effectively

Why should I use a flat structure?

A flat structure isn't all sunshine and rainbows

Is a flat structure right for me?

Additional considerations

Want to learn Go?

Why Can't We Settle on a Single Web App Structure in Go?

Can't we just learn from other languages?

The values of your programming language matter

Context is king

In summary...

Want to learn Go?

How do you get the most out of tutorials, courses, and other coding resources?

Getting the most out of coding courses & tutorials

Don't just copy the code; apply it

Use resources more than once

Expand upon what is taught

What works best for you?

Want to learn Go?

TDD is Not for Me

TDD focuses too much on unit testing

Not all code should be written one test case at a time.

Wrapping up

Want to learn Go?

Testing is Not for Beginners

Beginners don't know enough to write anything but the most trivial of tests

Trying to learn the skills required to actually write realistic tests while also trying to learn how to code and build things is overwhelming

But what if I want to learn testing?! (and the million other "what ifs")

Pssstt...

What goes in the `models` / `sql` package?

What goes in the `views` package?

What goes in the `controllers` package?