DEV Community

loading...
Cover image for Why is GO111MODULE everywhere, and everything about Go Modules (updated with Go 1.16)

Why is GO111MODULE everywhere, and everything about Go Modules (updated with Go 1.16)

maelvls profile image Maël Valais Originally published at maelvls.dev Updated on ・10 min read

You might have noticed that GO111MODULE=on is flourishing everywhere. Many readmes have that:

GO111MODULE=on go get golang.org/x/tools/gopls@latest
Enter fullscreen mode Exit fullscreen mode

In this short post, I will explain why GO111MODULE exists, its caveats and interesting bits that you need to know when dealing with Go Modules.


Table of content:

  1. From GOPATH to GO111MODULE
  2. The GO111MODULE environment variable
    1. GO111MODULE with Go 1.11 and 1.12
    2. GO111MODULE with Go 1.13
    3. GO111MODULE with Go 1.14
    4. GO111MODULE with Go 1.16
    5. So, why is GO111MODULE everywhere?!
    6. The pitfall of go.mod being silently updated
    7. The -u and @version pitfall
  3. Caveats when using Go Modules
    1. Remember that go get also updates your go.mod
    2. Where are the sources of the dependencies with Go Modules
    3. Set GO111MODULE on a per-folder basis with direnv
    4. Private Go Modules and Dockerfile
      1. Solution 1: vendoring
      2. Solution 2: no vendoring

From GOPATH to GO111MODULE

First off, let's talk about GOPATH. When Go was first introduced in 2009, it was not shipped with a package manager. Instead, go get would fetch all the sources by using their import paths and store them in $GOPATH/src. There was no versioning and the 'master' branch would represent a stable version of the package.

Go Modules (previously called vgo -- versioned Go) were introduced with Go 1.11. Instead of using the GOPATH for storing a single git checkout of every package, Go Modules stores tagged versions with go.mod keeping track of each package's version.

Since then, the interaction between the 'GOPATH behavior' and the 'Go Modules behavior' has become one of the biggest gotchas of Go. One environment variable is responsible for 95% of this pain: GO111MODULE.

The GO111MODULE environment variable

GO111MODULE is an environment variable that can be set when using go for changing how Go imports packages. One of the first pain-points is that depending on the Go version, its semantics change.

GO111MODULE with Go 1.11 and 1.12

  • GO111MODULE=on will force using Go modules even if the project is in your GOPATH. Requires go.mod to work.
  • GO111MODULE=off forces Go to behave the GOPATH way, even outside of GOPATH.
  • GO111MODULE=auto is the default mode. In this mode, Go will behave

    • similarly to GO111MODULE=on when you are outside of GOPATH,
    • similarly to GO111MODULE=off when you are inside the GOPATH even if a go.mod is present.

Whenever you are in your GOPATH and you want to do an operation that requires Go modules (e.g., go get a specific version of a binary), you need to do:

GO111MODULE=on go get github.com/golang/mock/tree/master/mockgen@v1.3.1
Enter fullscreen mode Exit fullscreen mode

GO111MODULE with Go 1.13

Using Go 1.13, GO111MODULE's default (auto) changes:

  • behaves like GO111MODULE=on anywhere there is a go.mod OR anywhere outside the GOPATH even if there is no go.mod. So you can keep all your repositories in your GOPATH with Go 1.13.
  • behaves like GO111MODULE=off in the GOPATH with no go.mod.

GO111MODULE with Go 1.14

The GO111MODULE variable has the same behavior as with Go 1.13.

Note that some slight changes in behaviors unrelated to GO111MODULE happened:

  • The vendor/ is picked up automatically. That has the tendency of breaking Gomock (issue) which were unknowingly not using vendor/ before 1.14.
  • You still need to use cd && GO111MODULE=on go get when you don't want to mess up your current project’s go.mod (that's so annoying).

GO111MODULE with Go 1.16

As of Go 1.16, the default behavior is GO111MODULE=on, meaning that if you want to keep using the old GOPATH way, you will have to force Go not to use the Go Modules feature:

export GO111MODULE=off
Enter fullscreen mode Exit fullscreen mode

The best news in Go 1.16 is that we finally get a dedicated command for installing Go tools instead of relying on the does-it-all go get that keeps updating your go.mod. Instead of:

# Old way
(cd && go install golang.org/x/tools/gopls@latest)
Enter fullscreen mode Exit fullscreen mode

you can now run:

go install golang.org/x/tools/gopls@latest
Enter fullscreen mode Exit fullscreen mode

One caveat is that the semantics of go install is slightly different from go get. As detailed on the Go
blog
:

In order to eliminate ambiguity about which versions are used, there are several restrictions on what directives may be present in the program's go.mod file when using this install syntax. In particular, replace and exclude directives are not allowed, at least for now. In the long term, once the new go install program@version is working well for enough use cases, we plan to make go get stop installing command binaries. See issue 43684 for details.

As an example, you cannot (as of April 2021) install the master version of gopls:

% go install golang.org/x/tools/gopls@master
go: downloading golang.org/x/tools v0.1.1-0.20210316190639-9e9211a98eaa
go: downloading golang.org/x/tools/gopls v0.0.0-20210316190639-9e9211a98eaa
go install golang.org/x/tools/gopls@master: golang.org/x/tools/gopls@v0.0.0-20210316190639-9e9211a98eaa
        The go.mod file for the module providing named packages contains one or
        more replace directives. It must not contain directives that would cause
        it to be interpreted differently than if it were the main module.
Enter fullscreen mode Exit fullscreen mode

The replace directive in the go.mod file looks like this:

replace golang.org/x/tools => ../
Enter fullscreen mode Exit fullscreen mode

Fortunately, the gopls project makes sure to remove the replace directive before each release, which means you can use the latest tag:

go install golang.org/x/tools/gopls@latest
#                                   ^^^^^^
Enter fullscreen mode Exit fullscreen mode

So, why is GO111MODULE everywhere?!

Now that we know that GO111MODULE can be very useful for enabling the Go Modules behavior, here is the answer: that's because GO111MODULE=on allows you to select a version. Without Go Modules, go get fetches the latest commit from master. With Go Modules, you can select a specific version based on git tags.

I use GO111MODULE=on very often when I want to switch between the latest version and the HEAD version of gopls (the Go Language Server):

GO111MODULE=on go get golang.org/x/tools/gopls@latest
GO111MODULE=on go get golang.org/x/tools/gopls@master
GO111MODULE=on go get golang.org/x/tools/gopls@v0.1
GO111MODULE=on go get golang.org/x/tools/gopls@v0.1.8
GO111MODULE="on" go get sigs.k8s.io/kind@v0.7.0
Enter fullscreen mode Exit fullscreen mode

The pitfall of go.mod being silently updated

And to make matters even worse, you may have encountered this weird one-liner in READMEs:

(cd && GO111MODULE=on go get golang.org/x/tools/gopls@latest)
Enter fullscreen mode Exit fullscreen mode

This weird line was meant for Go 1.15 and below; in Go 1.15 and below and was fixed in Go 1.16. With Go 1.16, the above line becomes:

go install golang.org/x/tools/gopls@latest
Enter fullscreen mode Exit fullscreen mode

Note: the @latest suffix will use the latest git tag of gopls. Note that -u (which means 'update') is not needed for @v0.1.8 since this is a 'fixed' version, and updating a fixed version does not really make sense. It is also interesting to note that with @v0.1, go get will fetch the latest patch version for that tag.

So, why did we need to use this contrived command that calls a subshell and moves to your HOME? That's yet another Go ideocracy that was fixed with Go 1.16: in Go 1.15 and below, by default (and you can't turn that off), if you are in a folder that has a go.mod, go get will update that go.mod with what you just installed. And in the case of development binaries like gopls or kind, you definitely don't want to have these appearing in the go.mod file!

So the workaround for anyone using Go 1.15 and below was to give a one-liner that makes sure that you won't be in a go.mod-enabled folder: (cd && go get) does exactly that.

I hope that (sooner or later) we will have a clear separation of concerns between go get that is adding a dependency to your go.mod (like npm install) and go install that is meant to install a binary without messing up your go.mod.

The -u and @version pitfall

I have been bitten multiple times by this: when using go get @latest (for a binary, at least), you should avoid using -u so that it uses the dependencies as defined in go.sum. Otherwise, it will update all the dependencies to their latest minor revision... And since a ton of projects choose to have breaking changes between minor versions (e.g. v0.2.0 to v0.3.0), using -u has a large chance of breaking things.

So if you see this:

# Both -u and @latest!
GO111MODULE=on go get -u golang.org/x/tools/gopls@latest
Enter fullscreen mode Exit fullscreen mode

then you will immediately realize that (1) it uses the old preGo .1.16 way of installing a Go binary, and (2) you want to be using the recorded versions given in go.sum when go-getting a binary!

Rebecca Stambler reminds us that we should not use -u in conjunction with a version:

-u should not be used in conjunction with the @latest tag, as it will give you incorrect versions of the dependencies.

But it's kind of hidden in this issue... I guess it is written somewhere in the Go help (btw, what a hideous help compared to git help) but that kind of caveat should be more visible: maybe print a warning when installing a binary with both @version and -u?

Caveats when using Go Modules

Now, let's go through some caveats I encountered when working with Go Modules.

Remember that go get also updates your go.mod

Before Go 1.16 came out, that was one of the weird things with go get: sometimes, it served the purpose of installing binaries or downloading packages. And with Go modules, if you were in a repo with a go.mod, it would silently add the binary that you were go get-ing to your go.mod!

Fortunately, with Go 1.16, go install has learnt about the @version suffix. With go install foo@version, your local go.mod won't be affected!

Where are the sources of the dependencies with Go Modules

When using Go Modules, the packages that are used during go build are stored in $GOPATH/pkg/mod. When trying to inspect an 'import' in vim or VSCode, you might end up in the GOPATH version of the package instead of the pkg/mod one used during compilation.

A second issue that arises is when you want to hack one of your dependencies, for example for testing purposes.

Solution 1: use go mod vendor + go build -mod=vendor. That will force go to use the vendor/ files instead of using the $GOPATH/pkg/mod ones. This option also solves the problem of vim and VSCode not opening the right version of a package’s file.

Solution 2: add a 'replace' line at the end of your go.mod:

replace github.com/maelvls/beers => ../beers
Enter fullscreen mode Exit fullscreen mode

where ../beers is a local copy I made of the dependency I want to inspect and hack.

Set GO111MODULE on a per-folder basis with direnv

On older versions of Go (1.15 and below), while migrating from GOPATH-based projects (mainly using Dep) to Go Modules, I found myself struggling with two different places: inside and outside GOPATH. All Go Modules had to be kept outside of GOPATH, which meant my projects were in different folders. To remediate that, I used GO111MODULE extensively. I would keep all my projects in the GOPATH, and for the Go Modules-enabled projects, I would set export GO111MODULE=on.

Note: since the default behavior in Go 1.16 is now GO111MODULE=on, this trick isn't necessary anymore.

This is where direnv came in handy. Direnv is a lightweight command written in Go that will load a file, .envrc, whenever you enter a directory and .envrc is present. For every Go Module-enabled project, I would have this .envrc:

# .envrc
export GO111MODULE=on
export GOPRIVATE=github.com/mycompany/\*
export GOFLAGS=-mod=vendor
Enter fullscreen mode Exit fullscreen mode

The GOPRIVATE environment variable disables the Go Proxy (introduced in Go 1.13) for certain import paths. I also found useful to set -mod=vendor so that every command uses the vendor folder (go mod vendor).

Private Go Modules and Dockerfile

Many companies choose to use private repositories as import paths. As explained above, we can use GOPRIVATE in order to tell Go (as of Go 1.13) to skip the package proxy and fetch our private packages directly from Github.

But what about building Docker images? How can go get fetch our private repositories from a docker build?

Solution 1: vendoring

With go mod vendor, no need to pass Github credentials to the docker build context. We can just put everything in vendor/ and the problem is solved. In the Dockerfile, -mod=vendor will be required, but developers don't even have to bother with -mod=vendor since they have access to the private Github repositories anyway using their local Git config

  • Pros: faster build on CI (~10 to 30 seconds less)
  • Cons: PRs are bloated with vendor/ changes and the repo's size might be big

Solution 2: no vendoring

If vendor/ is just too big (e.g., for Kubernetes controllers, vendor/ is about 30MB), we can very well do it without vendoring. That would require to pass some form of GITHUB_TOKEN as argument of docker build, and in the Dockerfile, set something like:

git config --global url."https://foo:${GITHUB_TOKEN}@github.com/company".insteadOf "https://github.com/company"
export GOPRIVATE=github.com/company/\*
Enter fullscreen mode Exit fullscreen mode

Illustration by Bailey Beougher, from The Illustrated Children's Guide to Kubernetes.

Update 22 June 2020: it said use replace instead of just replace.
Update 8 April 2021: update with Go 1.16.

Discussion (24)

pic
Editor guide
Collapse
alexshtin profile image
Alex Shtin • Edited

Great post, thanks for sharing. One question. We use GO111MODULE=off to install tools like goimport and not to pollute our go.mod file:

GO111MODULE=off go get -u golang.org/x/tools/cmd/goimports
Enter fullscreen mode Exit fullscreen mode

How is it worse than:

(cd && GO111MODULE=on go get -u golang.org/x/tools/cmd/goimports)
Enter fullscreen mode Exit fullscreen mode

?

Collapse
maelvls profile image
Maël Valais Author

Using GO111MODULE=off, you are falling back on the GOPATH method of installing the binary. In GOPATH mode, the master branch is always going to be the one checked out.

With GO111MODULE=on, you can rely on versions (i.e, git tags). The command

(cd && GO111MODULE=on go get -u golang.org/x/tools/cmd/goimports)
Enter fullscreen mode Exit fullscreen mode

will fetch the latest tag. The great benefit of this method is that you can pin-point the version to a branch or a tag or a commit:

(cd && GO111MODULE=on go get -u golang.org/x/tools/cmd/goimports@2363391)
Enter fullscreen mode Exit fullscreen mode

And since Go 1.16, the command has been greatly simplified (as long as there is no replace directive in goimport's go.mod):

go install golang.org/x/tools/cmd/goimports@2363391
Enter fullscreen mode Exit fullscreen mode
Collapse
daodennis profile image
daodennis

Thank you for this article Maël!
You might find this as a breath of fresh air, regarding "Remember that go get also updates your go.mod"
: groups.google.com/g/golang-tools/c...
go build, go test, and others - stop updating go.mod by default

Seems like there is a flag that certain Go tools have, the "-mod=readonly" flag. Might be worth adding to your article.

Collapse
suntong profile image
suntong • Edited

That's such a helpful post, so many of my questions got answered here. One more:

With go mod vendor no need to pass Github credentials to the docker build context. We can just put everything in vendor/ and the problem is solved.

I haven't fully got this working yet. Could you elaborate more on this a bit more please?

  • what go mod vendor is actually doing, how would it be different without doing it? Would -mod=vendor work without such step? and, -mod=vendor is for go build right?
  • how to put things in vendor/? Can you give a detailed example please -- i.e., Should I put mine in the GOPATH way, or the go mod way (with twisted name and added versions), Can I not put everything, but only a selectful of them that go get is failing to get?

Thanks

Collapse
maelvls profile image
Maël Valais Author

Hi! Thanks for the kind words!!! 🙂

  • go mod vendor will vendor all the .go files that are imported by your project (it is smart and only vendors the files actually used, not the whole repo for each import path). It keeps track of what is being vendored in vendor/modules.txt (not sure but I think this module list is only "for the humans" so that we can see in a diff what import paths have been added to vendor/)
  • To put things into vendor/, you first need to have modules turned on (go.mod at the root of your project) and then you can run go mod vendor once and here you go. Whenever you add import paths in your project, you also want to re-run go mod vendor to add/delete any missing/unused vendored files.
  • After having run go mod vendor you can fall back to using the good old GOPATH way (i.e. GO111MODULE=off) and not care about modules anymore.
  • Before Go 1.14, go build would by default use the modules in $GOPATH/pkg/mod instead of vendor/ and you had to run go build -mod=vendor
  • With Go 1.14 and above, the vendor/ folder is used by default and if you want to force using the $GOPATH/pkg/mod you can use go build -mod=mod.

Should I put my vendor/ in the GOPATH way or the go mod way

I think go mod vendor puts things in the GOPATH way (except it only puts the files that are "actually used"). Not sure I answered the question properly though 😞

Can I not put everything, but only a selectful of them that go get is failing to get?

Ah, right, I see the use case: you want to be able to manually add anything that go get doesn't know how to download (e.g. private networks, corporate firewalls).

I guess you can definitely do that: go mod vendor for anything that works and manually copy-paste the import paths that are failing. I guess you would have to update vendor/modules.txt too but not sure. The important part is that the hash of the files you manually add must match the hashes in the go.sum (I guess). 👍

Collapse
suntong profile image
suntong

Thanks a lot for such detailed explanation.

Collapse
sandeepmeenuga profile image
sameen

Would you share something about govendor(github.com/kardianos/govendor) vs Go111module=on . I've a doubt about Go111module=on which saves one version of packages only in GOPATH but Govendor works as individual project's versions which exactly as node_modules. Any views appreciated.

Collapse
maelvls profile image
Maël Valais Author • Edited

Hi sameen,

Unfortunately, I do not know about govendor. But from what I understand, when building a binary with GO111MODULE=on, a single version of each dependency will be used for the whole build. Russ Cox calls it minimal version selection (see his blog post on Dec 3rd, 2019).

I've a doubt about Go111module=on which saves one version of packages only in GOPATH

What do you mean? GO111MODULE=on will remember (in go.sum) the version of dependencies and save each dependency copy in GOPATH/pkg/mod (i.e. one git clone at a fixed version if you will).

Note that GO111MODULE=on works very differently from node_modules since a single version is used per package, even when the package is used in multiple dependencies across the dependency tree.

Collapse
domq profile image
domq • Edited

Thanks Maël, you single-handedly saved me about 8 hours of RTFM. Now I can get started on a project that enjoys sane, 21st-century style version control of its dependencies. And thanks for the direnv link, which appears to work with Emacs (although I can't vouch for that just yet)

Collapse
murphyke profile image
Kevin Murphy

Hi, your example able says use replace github.com/maelvls/beers => ../beers ... but I think the word use is not supposed to be there. Thanks for the post, btw.

Collapse
maelvls profile image
Maël Valais Author

Good point, thank you!!
I fixed that in the article 👍

Collapse
kelepirci profile image
Ferhat Ozkasgarli • Edited

What a software language. A simple start require great effort and time. It must be great to become GO developer ( ͡° ͜ʖ ͡°).

Wanderful article to sooth my pain while I was trying to figure out the GOPATH and its minions. :D

I thank and salute your sir ...

Collapse
martinflower profile image
Martin Flower • Edited

Thanks Maël

Maybe you can answer the following, related, question:

If I'm using Go 1.14, with go.mod, no vendor folder, should I have a GOPATH environment variable? Does this affect GO111MODULE in any way?

Thanks

/Martin

Collapse
rafalskolasinski profile image
RafalSkolasinski

It could be good to add note on behaviour in Go 1.14. Does it differ from one in Go 1.13?

Collapse
maelvls profile image
Maël Valais Author

Definitely, I’ll try to find time to update to 1.14 🙂

Collapse
rafalskolasinski profile image
RafalSkolasinski

Awesome!
BTW, this is a go to post about understanding GO111MODULES. IMO should be linked / part of official docs.

Collapse
maelvls profile image
Maël Valais Author

Done! I updated with 1.14 and two "new" pitfalls 🙂

Collapse
1oglop1 profile image
Jan Gazda

Reading the discussion I'm glad I'm not a lone who find Go things counter intuitive.
This article has helped me to solve the problem when go get did not create binary and exited with success.

Collapse
gcdcoder profile image
Gustavo Castillo

Great article thanks for sharing.

Collapse
jy226 profile image
Jaafar Yahya

Great Article !

Collapse
themrkumar profile image
themrkumar

Great Article. Please also make a tutorial about Go modules

Collapse
folkengine profile image
Chris Baker

This was really useful. Thanks!

Collapse
benkame profile image
Ben Ka

Very helpful! That spares me lot of pain.

Collapse
xmpace profile image
yema

Helps me a lot, thank you!