DEV Community: Mitali Bisht 🐥

Detecting, Reporting and Mitigating System Vulnerabilities for Go

Mitali Bisht 🐥 — Mon, 01 Jun 2020 21:57:05 +0000

Go Module vulnerabilities frustrate the lives of many Go developers and can turn a simple project into a battle of endurance between the dev and their patience. With the process of CI/CD shifting left more and more, it’s becoming even more pertinent for developers to be able to track and report vulnerabilities as early as possible. JFrog GoCenter can help track and mitigate these vulnerabilities and make the lives of Go developers easier.

Reporting Vulnerabilities

Vulnerabilities are monitored consistently throughout the development lifecycle of any system/application and should be reported by anyone who finds the issue so that it’s remediation can be tracked and shared among common organizations. Known vulnerabilities are tracked and classified using the Common Vulnerability and Exposures (CVE) - a list of publicly disclosed information security vulnerabilities and exposures. Each CVE contains a standard identifier number (CVE ID), a status indicator, a brief description of the vulnerability and references related to vulnerability reports and advisories.

CVE isn't a vulnerability database - instead, CVE is designed to allow vulnerability databases and other tools to be linked together and facilitates comparisons between security tools and services. The National Vulnerability Database (NVD) is a free, public database that uses the CVE list identifiers and includes fix information, scoring and other pertinent information for each vulnerability.

Detected vulnerabilities must be reported to one of the CVE Numbering Authorities (CNA) accompanied by detailed documentation explaining the impact of the vulnerability and at least one codebase that it affects before it can be recognized as a common vulnerability and assigned a CVE ID. For reference, the format for the CVE ID = CVE prefix + Year + Arbitrary Digits. Below is an example of a CVE ID entry:

The Data Complexity of Securing Go Modules

Securing Go modules can be a tricky task, specifically because of the relationship between Go modules and Go packages. Once the security data for a Go module is received, it’s difficult to correlate that data to a specific module version. This is because security vulnerabilities exist on the package level but are reported on the module level. This can leave the impression that the entire module is vulnerable, but that’s often not the case. More than likely, your module will remain secure unless you consume the vulnerable package data.

Let’s use CVE-2020-10660 as an example. Below is an excerpt from the version 1.3.4 changelog detailing the effects of this vulnerability:

gopkg.in/hashicorp/vault.v0 and github.com/hashicorp/vault, have both been affected by CVE-2020-10660 in HashiCorp Vault and Vault Enterprise versions 0.9.0 through 1.3.3. While using these packages they may, under certain circumstances, have an entity's group membership inadvertently include groups that the entity no longer has permissions to.

Another vulnerability was identified in Vault Enterprise such that, under certain circumstances, existing nested-path policies may give access to namespaces created after-the-fact. Luckily, there was a change made to fix these vulnerabilities in version 1.3.4.

Looking above, the fix was made within “github.com/hashicorp/vault/vault”. The module istio.io/istio has a dependency on “github.com/hashicorp/vault/vault” in its go.mod file. Normally, you would assume that the security of istio.io/istio would be compromised, but it’s only using “github.com/hashicorp/vault/api” package, so its code is not affected by the vulnerability. See the source code below.

Mitigating your Software Vulnerabilities

Now that you understand the process of how Go Module vulnerabilities are reported and some details around the security complexities, let’s see how to mitigate these threats in future development.

To start, let’s look at the github/hashicorp/vault

Using CVE data, JFrog Xray is able to scan all the dependencies in a go.mod file held by GoCenter and identify every vulnerability. GoCenter exposes this Xray data on the Dependencies tab to provide you detailed information about vulnerable components at every level of your dependency tree. You’ll see a warning triangle next to each vulnerable module. You can then click on the vulnerable module and be routed to that page. From there, click on the Version tab to look for a secure version of that module so that you can upgrade that in your go.mod file.

Once all components and dependencies have been identified, their information is cross-referenced with other vulnerability feeds and databases to alert you of any potential threats. Basic Xray vulnerability scanning for Go modules is available for free on GoCenter, shown in the Security tab:

Keeping your Software Secure with GoCenter

GoCenter is a public GOPROXY and central repository with over 700,000 Go module versions. When using GoCenter as your GOPROXY, you can be sure that the version of code downloaded is the correct version from the correct source code. GoCenter as your GOPROXY works seamlessly with Go commands and has the advantage of being secure, fast, available and storage efficient.

Many Golang developers can also bring GoCenter’s vulnerability information directly into their IDE, using the free JFrog extension for VS Code.

With the CI/CD cycle shifted to the left more and more, GoCenter’s security features can help you identify whether or not there are vulnerable dependencies in the public Go module versions you are importing.

#Managing Go Module Pseudo-Versions in Go 1.13

Mitali Bisht 🐥 — Sun, 31 May 2020 06:20:15 +0000

Go modules have helped bring order to Go development, but there’s been some disorder lurking. Managing module pseudo-versions can be difficult, especially with some of the latest changes to Go.

Let’s take a look at how pseudo-versions work, and what you can expect from those changes. We also offer some guidance on keeping your Go builds working as you upgrade to Go 1.13 and later.

Go Module Versioning

The ability to version Go modules is a key feature, providing developers a way to make sure their applications use the dependencies they intend. When modules are versioned, an app can specify the use of a module version they know will be compatible with the rest of their runtime.

A Go module version is assigned by tagging its revision in the underlying source repository. The go command uses semantic versioning of the standard form vX.Y.Z to describe the module version. The version number changes based on the changes made in the API :

From this standard format, module versions can be compared to identify which should be considered most or least current.

Using Pseudo-Versions

A versioned Go module is one that has been released for general use and should be preferred by most developers. However, there are some cases where you cannot release the most current version of a module.

For example, a team may need to share an interim version during development. This is especially the case when a dependent project has no released versions yet, so it hasn’t been tagged with a version. Similarly, you may need to develop against a commit that hasn’t yet been tagged.

To use an untagged version of a module as a dependency, it must be referenced by its pseudo-version identifier. A pseudo-version has the following format:

There are 3 acceptable forms of pseudo-version:

vX.0.0-yyyymmddhhmmss-abcdefxyz when no earlier versioned commit with an appropriate major version before target commit
vX.Y.Z-pre.0.yyyymmddhhmmss-abcdefxyz when most recent versioned commit before the target commit is vX.Y.Z-pre
vX.Y.(Z+1)-0.yyyymmddhhmmss-abcdefxyz when most recent versioned commit before the target commit is vX.Y.Z

As a best practice, a pseudo-version string should never be typed by hand. The go command will accept the plain commit hash and translate it into a pseudo-version automatically. This method helps to compare revisions based on the generated timestamp.

For example, a go get command might use just the commit hash for the module query:

-> go get github.com/x/sys@c856192   # records v0.0.0-20180517173623-c85619274f5d

There are problems with not letting the go command automatically generate the pseudo-version:

The pseudo-version participates in minimal version selection. If its version prefix is inaccurate, the pseudo-version may appear to have higher precedence than the releases that follow it, effectively pinning the module to that commit
The commit date within pseudo-version provides a total order among pseudo-versions, so if it gets edited it will mess up the ordering

Despite this recommendation, sometimes a pseudo-version may exist in a go module that has been edited by hand. In other instances, the full pseudo-version string may be generated by a third-party tool.

Stricter Rules

Through release 1.12, Go was forgiving for pseudo-version references. Most operations involving pseudo-versions accepted any arbitrary combination of version string and date, and would resolve to the underlying revision (typically a Git commit hash) as long as that revision existed.

The release of Go 1.13 brought stricter enforcement, in order to address the problems noted above. Go 1.13 restricts the pseudo-versions that ‘go’ command accepts, rendering some previously accepted but not canonical versions invalid.

The go client now performs some validation on different elements of the pseudo-version against the version control metadata:

The tag from which the pseudo-version derives points to the named revision or one of its ancestors as reported by the underlying VCS tool, or the pseudo-version is not derived from any tag (i.e. has a “vX.0.0-” prefix before the date string) and uses the lowest major version appropriate to the module path
The date string within the pseudo-version matches the UTC timestamp of the revision as reported by the underlying VCS tool
The short name of the revision within the pseudo-version is the same as the short name generated by the go command.
The pseudo-version includes a ‘+incompatible’ suffix only if it is needed for the corresponding major version, and only if the underlying module does not have a go.mod file
Even after resolving the module from proxy, the go client will try to fetch the checksum content from the checksum server, which enforces the same pseudo-version validation rules and will refuse to serve the checksum content.

How to Fix Improper Pseudo-versions

In order to move to Go 1.13, a developer must correct all pseudo-version references that don’t align with the above requirements. Otherwise the go client will flag an exception:

go get golang.org/x/sys@v0.0.0-20190726090000-fde4db37ae7a: invalid pseudo-version: does not match version-control timestamp (2019-08-13T06:44:41Z)

Fortunately, this is pretty easy to do through your go.mod file where your pseudo-version references are made.

If the go.mod file’s require directive has an incorrect pseudo-version, this can be corrected by replace the full pseudo-version reference with just the commit hash string

require {
   golang.org/x/sys fde4db37ae7a
}

If one of the transitive dependencies references an invalid pseudo-version, you can use the replace directive in your go.mod file to force the correction:

replace golang.org/x/sys v0.0.0-20190726091711-fde4db37ae7a =>
   golang.org/x/sys fde4db37ae7a

GoCenter Can Help

An important principle of GoCenter, the free repository of versioned Go Modules is being version agnostic. JFrog’s Community Engineering team has made important updates to GoCenter to support all versions of Go through 1.13, and we are in the process of further updates to accommodate Go 1.14 requirements. GoCenter can help you comply with the pseudo-version validation by redirecting to the correct pseudo-version. GoCenter changes the metadata in the .info with the correct version when the module download was requested for incorrect pseudo-version.

-> curl https://gocenter.io/golang.org/x/sys/@v/fde4db37ae7a.info

{ “name” : “v0.0.0-20190813064441-fde4db37ae7a”,
  “shortName” : “v0.0.0-20190813064441-fde4db37ae7a”,
  “version” : “v0.0.0-20190813064441-fde4db37ae7az”,
  “time” : “2019-08-13T08:03:52Z”
 }

To make use of GoCenter, set your GOPROXY

GOPROXY=https://gocenter.io/

Go 1.12 and before

For Go 1.12 and before users, GoCenter will update the go.mod file held in its repository with the correct pseudo-version. GoCenter will still serve the incorrect pseudo-version that was processed in GoCenter before this change.

-> go version
go version go1.12.14 darwin/amd64
-> go get golang.org/x/sys@v0.0.0-20190726090000-fde4db37ae7a
go: finding golang.org/x/sys v0.0.0-20190726090000-fde4db37ae7a
-> cat go.mod
module example

go 1.12

require golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a // indirect

For Go 1.13 and later

Go 1.13 and later users will receive an error message that points to the correct pseudo-version.

-> go version
go version go1.13.5 darwin/amd64

-> go get golang.org/x/sys@v0.0.0-20190726090000-fde4db37ae7a
go: finding golang.org v0.0.0-20190726090000-fde4db37ae7a
go: finding golang.org/x/sys v0.0.0-20190726090000-fde4db37ae7a
go: finding golang.org/x v0.0.0-20190726090000-fde4db37ae7a
go get golang.org/x/sys@v0.0.0-20190726090000-fde4db37ae7a: golang.org/x/sys@v0.0.0-20190726090000-fde4db37ae7a: proxy returned info for version v0.0.0-20190813064441-fde4db37ae7a instead of requested version

-> cat go.mod 
module example.com/mitali

go 1.13

In order to update the correct pseudo-version in the go.mod file, Go 1.13 users need to change the go get to include only the commit hash part of pseudo-version.

-> go get golang.org/x/sys@fde4db37ae7a
go: finding golang.org fde4db37ae7a
go: finding golang.org/x/sys fde4db37ae7a
go: finding golang.org/x fde4db37ae7a

-> cat go.mod 
module example.com/mitali

go 1.13

require golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a // indirect

If you want to override this behavior and have GoCenter serve the incorrect pseudo-version that was processed earlier, then you can set GOSUMDB= off.

Changes in Go 1.14 for Modules

As we noted, JFrog is working on changes to GoCenter to support Go 1.14. Here are some of the changes to Go in that release that affect the operation of modules that you may want to be aware of:

go command flags

go get command no longer accepts the -mod flag
-mod=readonly is set by default if there is no top level vendor directory and go.mod file is read only
-modfile=file new flag introduced which instructs go command to read/write an alternate go.mod file and an alternate go.sum file will also be used. Although a file named go.mod must still be present in order to determine the module root directory

go.mod file changes

go get will not upgrade to an +incompatible major version unless it is requested explicitly or already required
go commands (except go mod tidy) will not remove a require directive that specifies a version of an indirect dependency that is already implied by other dependencies of the main module
When -mod=readonly flag is set then the go command will not fail due to a missing go directive or any error

Module download

go command now supports Subversion repositories in module mode
go command now includes snippets of plain-text error messages from module proxies and other HTTP servers. An error message will only be shown if it is valid UTF-8 and consists of monopoly graphic characters and spaces.

Using Pseudo Versions with Confidence

We hope that the steps outlined help you manage your pseudo versions more easily and effectively. As more versions are released, more will change and JFrog will do all that it can to ensure that we are ahead of the curve! Look forward to more helpful blogs from us in the very near future and If you haven’t explored GoCenter’s free repository of Go modules yet, we invite you to do so! With a rich UI that helps you examine data about all 600,000+ Go modules, it can help you gain powerful command over the GoLang dependencies you use