DEV Community: Dominic Black

Designing a config API for microservices applications built using Go

Dominic Black — Thu, 10 Nov 2022 13:28:58 +0000

We have a saying here at Encore: “Encore says yes”.
For us, this means no matter what an end-user wants to build, it should be possible with Encore. Even if the framework does not yet natively support that. For example, if you want to
use a No-SQL database you should easily be able to wire that into your application.

As Encore builds a standard Go binary and deploys it into your cloud account, you can easily connect to infrastructure
already in your account. However, this raises a question: How do you configure your application to connect to different instances of that resource depending on if you are running locally, in a PR preview environment, or
deployed to production?

In May this year, we released our metadata API. It allowed a running
system to interrogate its environment, thus allowing for the code to switch between infrastructure instances. However,
this was only an MVP to enable this. We knew we wanted a much more robust system, and started thinking about what a
native configuration experience would look like.

What do developers want from a configuration system?

As mentioned above, our primary use case was to allow different environments to have different integration setups. A typical example of this would be in a payment system to have your development and testing environments connect to a testing account with your card processor.

But we also started thinking about other use cases people have for configuration in a backend:

Storing very slow-moving metadata, such as metadata on support currencies and the number of decimal places for them.
Being able to configure library code differently for different services. For instance, having rate limiters installed on a per-service basis with different limits per service.
Providing the ability to switch between code paths, so you can perform A/B tests & slow rollouts of new behaviour without having to commit 100% of your traffic to the new code.
Being able to turn things off during incidents or in tests, such as putting the system into a read-only mode or in tests not sending emails to the test customer accounts.

As with everything we build into Encore, we wanted to ensure there was minimal cognitive overhead when using
configuration. This means it should be easy to identify how the configuration was derived and where the settings came
from. Config should feel natural, be easy to read, and never surprise you at runtime.

For us, it was a prerequisite to ensure whatever we designed had compile-time safety, both in terms of type safety and completeness (i.e. config values couldn’t be missing in production).

We wanted the ability for descriptive comments, to explain what the valid options are, and what the impact of those
options is without having to read the code which uses that config field.

Auditability and controls are essential. Many teams need to know who changed what and why, as well as potentially
have an approval process for making config changes. Luckily as developers, we already have great tools for this! If
we store configuration alongside the code in our source control, we get this for free. This necessitates that Encore reads the configuration from the source code, rather than a database.

Finally, to support incident management, we knew we wanted to support live updating configuration, without having to
commit, rebuild and redeploy your applications. This override would be time-limited, before either being reverted to
the values committed, or the override being committed to the source repository.

Choosing a config language

One of the key, non-reversible decisions to make was how users should write their configuration. There are many options
to pick from; JSON, YAML, Dhall, HOCON, INI, XML etc; the options are endless.

Most of the options do not support conditionals, which leads to the question of how to support different environments.
For instance, if we picked JSON, we’d have to introduce a filename suffix system where config.json is applied
everywhere, but config.env-prod.json is only applied to the environment named “prod”, and config.dev.json is applied
to all development environments. Then a question of the merge order arrives, and how do overrides work:

does a specified environment name override a value defined in the environment type?
what if a field is null or undefined in the override, does that mean unset it or leave the previous value?.

Another disadvantage is IDE warnings around type safety, and auto-completion would not be automatic; we’d have to add it
into the Encore plugin to provide a good development experience.

To be complete, we briefly considered the option of writing the configuration directly into Go using regular Go code, as this would give the ability to easily have expressions to control config. However, we quickly ruled it out as the advantages of type safety and IDE support were far outweighed by the disadvantages. Using Go would mean having to convert static data into Go code, and initializing structs or updating config values would result in a recompile and relink of the app, which would slow down iteration times.

Finally, we settled on the same configuration language that we use internally for our own systems: CUE Lang.
As it has many of the properties we where after:

It’s a strict superset of JSON, which means all valid JSON is valid CUE, so users do not have to learn CUE to get started
CUE has full type safety meaning CUE tooling can report incomplete configuration or invalid types being used
It supports expressions such as if statements and loops
CUE has inline comments
Best of all; it is deterministic no matter which order you process files, the final output will always be the same.

Designing the API

As with most of our features, this is where we spent a lot of time going back and forth. Encore takes backward
compatibility extremely seriously. Similar to the Go compatibility promise, Encore aims to never break working code in future versions of Encore. That means any API we provide must be designed both for what we want to do in the future with that feature, and how the API may interact with other features of Encore.

We needed to be able to statically read all configuration keys and return types, such that we could generate a CUE file
with the required constraints on types, thus gaining our compile-time type safety and completeness checks.

We landed on four main contenders for the design of how to access the configuration from the Go code.

1. Implicit loading

This would be consistent with how secrets are currently defined in Encore. The user would write a package-level variable
with an inline struct type, but not provide an instance of that config. Then at compile time, Encore would inject the
instance against that variable.

package svc

var cfg struct {
    ReadOnlyMode bool

    Currencies map[string]struct{
        Name string
        Code string
    }
}

Documentation of the configuration options would come naturally to Go programmers, as they could just document the struct,
and we would replicate those comments into the generated CUE file.

2. Explicit loading

In this approach, we took the first approach and removed the “magic” by introducing an explicit Load function call.
Under the hood, this is exactly what Encore would have rewritten your code to be in the first option, but due to your
explicitly calling Load, it’s less confusing as to how that variable came to be set in the first place.

package svc

import "encore.dev/config"

type Config struct {
    ReadOnlyMode bool

    Currencies map[string]struct{
        Name string
        Code string
    }
}

var cfg = config.Load[Config]()

3. Reference by Path

We considered an XPath style API, where instead of defining a config struct, you would explicitly read into the
configuration, and we would infer the structure of the configuration from the access pattern.

package svc

import "encore.dev/config"

func SomeCode(currency string) {
    if config.Bool(“ReadOnlyMode”) {
        return
    }

    _ = config.String(“Currencies”, currency, “Name”)
}

However, this API design had some significant disadvantages against the first two;

Typo’s would not be detected (ReadOnlyMode vs ReadOnly) as they would cause two separate required booleans to be created.
The IDE could not autocomplete the path for you.
Refactoring would not be possible inside the IDE (i.e. renaming a field and automatically updating all callsites).

4. Generated Go from CUE

The final approach we considered, was to take the types from the CUE configuration and generate a matching Go struct
type with package-level variables for accessing the config, along the lines of the implicit load and explicit load
options - just with the Go file being fully automatically generated.

This design meant that we’d be guessing the Go types from the CUE types and might use a too-generic type, forcing the
user to typecast the configuration value when it is used. That generic type could then result in an invalid configuration
which wasn’t caught at compile time; for example, we generated an int field, when the app only wanted uint’s.

Updating Config at Runtime & for Tests

After evaluating the options, we decided that the explicit loading made the most sense. However, it left us with a problem: How do we update config values due to live updates and allow tests to override the configuration?

At the end of the day, we only saw two options:

Update the values in the struct or sub-structs directly in memory.
Provide a wrapper type which wraps a value from T into func () T.

While the first option initially feels the best, it leads to some unexpected behavior: Do all call sites that
reference (or have the struct passed in) understand the type is dynamic and might change? If not, there are going to be some really hard-to-debug issues. Especially if the values are passed into a third-party library that isn’t aware it’s running inside an Encore application.

The other impact, which is due to Encore’s unique test tracking & isolation capabilities, is that two parallel running tests could read different values from the same cfg variable. While most people would never even notice this, it could still lead to some WTF moments of confusion.

Going with the second option means people will naturally expect that the returned value could change between calls to
the wrapper function, which leads to easier to pick up bugs during reviews.

Then by providing a function to override the values in these wrapper functions for unit tests, we end up with no
unexpected behaviour, while giving us the ability to have a typesafe override API for the testing framework

-- svc_test.go --
package svc

import (
    "context"
    "fmt"
    "testing"

    "encore.dev/et" // Encore's test helpers
)

func TestA(t *testing.T) {
    t.Parallel()

    et.SetCfg(cfg.SystemName, "Foo")

    // This will always print "TestA: Foo"
    resp, _ := GetName(context.Background())
    fmt.Println("TestA:", resp.Name)
}

func TestB(t *testing.T) {
    t.Parallel()

    et.SetCfg(cfg.SystemName, "Bar")

    // This will always print "TestB: Bar"
    resp, _ := GetName(context.Background())
    fmt.Println("TestB:", resp.Name)
}

func TestC(t *testing.T) {
    t.Parallel()

    // This will always print the original config "TestC: System under test"
    resp, _ := GetName(context.Background())
    fmt.Println("TestC:", resp.Name)
}


-- svc.go --
package svc

import (
    "context"

    "encore.dev/config"
)

type Config struct {
    ReadOnlyMode config.Bool   // Are we in read only mode?
    SystemName   config.String // The name of our system
}

var cfg = config.Load[Config]()

type NameResponse struct {
    Name string
}

//encore:api public
func GetName(ctx context.Context) (*NameResponse, error) {
    return &NameResponse{cfg.SystemName()}, nil
}

-- myconfig.cue --
if #Meta.Environment.Type == "test" {
    SystemName: "System under test"
}

if #Meta.Environment.Type != "test" {
    SystemName: "My System"
}

Writing the Code Generators

Many of Encore’s capabilities come down to generating code based on your intent, removing the boilerplate you normally need to write. For this project, we had two separate languages we needed to generate code in, Go and CUE.

We started with the CUE generator, we took the statically parsed call to config.Load, recursively resolved the types
required, and used CUE’s own AST package to generate an AST of a CUE file, which contained both the CUE definitions of
the CUE types, but also definitions about the metadata, such that the configuration can perform conditions based on
the environment.

// Code generated by encore. DO NOT EDIT.
//
// The contents of this file are generated from the structs used in
// conjunction with Encore's `config.Load[T]()` function. This file
// automatically be regenerated if the data types within the struct
// are changed.
//
// For more information about this file, see:
// https://encore.dev/docs/develop/config
package svc

// #Meta contains metadata about the running Encore application.
// The values in this struct will be injected by Encore upon deployment and can be
// referenced from other config values for example when configuring a callback URL:
//    CallbackURL: "\(#Meta.APIBaseURL)/webhooks.Handle`"
#Meta: {
    APIBaseURL: string @tag(APIBaseURL) // The base URL which can be used to call the API of this running application.
    Environment: {
        Name:  string                                              @tag(EnvName)   // The name of this environment
        Type:  "production" | "development" | "ephemeral" | "test" @tag(EnvType)   // The type of environment that the application is running in
        Cloud: "aws" | "azure" | "gcp" | "encore" | "local"        @tag(CloudType) // The cloud provider that the application is running in
    }
}

// #Config is the top level configuration for the application and is generated
// from the Go types you've passed into `config.Load[T]()`. Encore uses a definition
// of this struct which is closed, such that the CUE tooling can any typos of field names.
// this definition is then immediately inlined, so any fields within it are expected
// as fields at the package level.
#Config: {
    ReadOnlyMode: bool   // Are we in read only mode?
    SystemName:   string // The name of our system
}
#Config

The design of the wrapper functions meant we couldn’t easily unmarshal the CUE value into the wrapper functions.
This meant we needed to generate unmarshalled functions for the config types. We use the excellent Jennifer library by
Dave (no really; github.com/dave/jennifer) for generating Go files.

For each Go type used within the config, we generate a separate unmarshaller function. The unmarshallers use
json-iterator to process the output from CUE, while tracking the path within the
config to the unmarshalled value. This path tracking will allow the function to check if live overrides have been
provided on that path and return the override instead.

The root level unmarshaller is then injected as a parameter into your call to config.Load, such that the Go compiler
is able to statically verify that our generated code will return the exact types that you expect.

Future Enhancements

We launched the initial version of config a couple of weeks ago in
version 1.9.0. However, this is only the start of our journey
of improving the state of the configuration of backend developers. As discussed above one of the key features we want to add next is the ability to allow applications to have parts of their configuration updated in realtime on a
time-limited basis for dealing with incidents. This means adding a new API to give you the ability to get a channel
from each wrapper to watch for changes in the value, allowing your application to react to configuration changes, such as resetting rate limiters to the new rates.

We also want to provide a UI in the Encore platform, listing the current configuration as it is deployed into each environment, as well as the locations in the source CUE files where that configuration is defined.

Following that we want to integrate config with our secrets management system, such that you can load in secrets as part
of your configuration and use CUE’s conditionals to switch between secrets depending on what you need.

Is there something you’d love to see in a configuration system? Come chat with us on Slack, or the Community Forums.

How we used Go 1.18 when designing our Identifiers

Dominic Black — Wed, 01 Jun 2022 11:40:24 +0000

When building any system, distributed or not, you end up dealing with many identifiers for your data. From database rows all the way to identifiers for the version of your system in production. Unsurprisingly Encore's own systems are no different. We need to uniquely identify and track things from individual API endpoints in your applications, traces from runtime calls, to the various
infrastructure resources provisioned in one of our supported clouds, such as an AWS security group.

Deciding how to generate your identifiers can sometimes be very simple. For instance, you might just put an auto-incrementing number as your primary key in your database. Boom, you have your type of identifier and way to generate it, just like magic.

However, it's much harder in a distributed system to just have a number start at 1 and slowly increase. You could build a system that elects a leader, and that leader is in charge of incrementing the number - but this adds a lot of complexity to your system design, it doesn't scale indefinitely as you will still be limited by the throughput of the leader. You could still suffer from a split-brain issue where the same number is generated twice by two different "leaders".

Getting this decision right early on in your project is always important, as it's one of the hardest things to change in a system once you're in production.

Starting from our requirements

When we started to think about how we wanted to represent identifiers in Encore's platform, we wrote down what we wanted as core requirements from an identifier:

Sortable

We want to be able to order the identifiers, such that an A < B when A was created first. However, we don't need total sorting, being k-sortable is acceptable.

Being sortable allows us better indexing performance in the database, will enable us to easily iterate through records in order and improves our ability to debug as the order of events can be determined from the event identifiers.

Scalable

We want a system that will scale with us without bottlenecks. We'll be using this system to generate identifiers for traces and spans in which we'll be creating a vast number of them.

No Collision Risk

Because we run a distributed system, we don't want the risk of two of our processes creating the same identifier.

Zero Configuration

We don't want to have to do any level of configuration when using this system on either a per-machine or per-process level.

Type Safety

We want a system where an identifier for one resource can't accidentally be passed or returned as an identifier for another type of resource.

As a bonus, we also wanted identifiers to be:

Reasonably Small

Both in memory and on the wire.

Human-readable

This is related to type safety. However, we want the string representation to allow a human (us 🤖) to understand what the identifier was created for.

This seems like quite a list of asks, but it's pretty manageable when we break it down.

Existing Options

If we temporarily ignore the type safety and human readability requirements, then what we're asking for has been solved a thousand times before, and we don't need to re-invent the wheel here. Let's look at some existing battled-tested options.

Database powered auto-incrementing keys ✘

The first port of call for most people is an auto-incrementing primary key. This solves being sortable and has no collision risk. However, it can only scale as far as your database server will process writes. It also adds a new requirement that every identifier you generate must have a matching database row. Given we wanted to use this system for things we don't store in our database, we ruled this out pretty quickly.

UUIDs ✘

There are various versions of UUIDs, and they are all 128 bits. At first glance, versions 1, 2, 4 seem perfect from a scalability, zero configuration and collision risk angle. However, when you dig deeper, the different versions of UUIDs start to show warts:

Versions 1 & 2

Even though they can generate approximately 160 billion identifiers per second, they use the machine's MAC address which means two processes on the same machine could generate the same identifier if called simultaneously.

Version 4

Generates completely random identifiers with 2¹²² bits of randomness. This means we're very unlikely to have a collision and can infinitely scale-out without a bottleneck. However, we lose the ability to sort the identifiers chronologically.

Snowflake ✘

Snowflake identifiers were first developed by Twitter and typically are
64 bits (although some variants use 128 bits). This scheme encodes the time the ID was generated as the first 41 bits, then encodes an instance ID for the next 10 bits, and finally a sequence number for the final 12 bits.

This gives us pretty good scalability as the instance set of bits allows us to run 1024 different processes simultaneously while knowing they can not generate the same identifier, as the process's own identifier is encoded within! It gives us our k-sorting, as the upper bits are the time the ID was generated. Finally, we have a sequence number within the process, which allows us to generate 4096 identifiers per second per process.

However, Snowflake misses our zero-configuration requirement because the instance ID must be configured per process.

KSUID ✘

KSUID's are sort of a cross between UUID Version 4 and Snowflake. They are 160 bits where the first 32 bits are the time the identifier was generated (to the second) and then 128 bits of random data. This makes them almost ideal for us. They are k-sortable, require no configuration, and have no collision risk because of the large amount of entropy in the random part of the id.

However, we discovered something interesting during our research of KSUID. The string encoding of KSUID uses Base-62 encoding, and so has both uppercase and lowercase letters. This means depending on your string sorting, you might sort the identifiers differently - i.e. we lose our requirement for sortability depending on the system. For instance, Postgres sorts lowercase before uppercase, whereas most algorithms sort uppercase before lowercase, which could lead to some very nasty & hard-to-identity bugs. (It's worth noting that this impacts any encoding scheme which uses both upper and lower case letters, so it isn't just limited to KSUID)

XID ✓

XID's are 96 bits. The first 32 bits are the time, which means we get our k-sorting immediately. The next 40 bits are a machine identifier and a process identifier. However, unlike the other systems, these are calculated automatically using the library and don't require us to configure anything ourselves. The final 24 bits are a sequence number, which allows a single process to generate 16,777,216 identifiers per second!

XID gives us all our core requirements, and its string encoding uses base 32 (no upper case letters to break our sorting!). This string encoding is always 20 characters, which means we can use that fact for validation purposes in any marshalling code (such as a Postgres CHECK constraint
on the database type).

Our Identifiers

Once we had settled on using XID as the basis for our ID types, we switched focus back to our last two requirements: type safety and human readability.

For the former requirement, we could have solved this as simply as:

import "github.com/rs/xid"

type AppID xid.ID
type TraceID xid.ID

func NewAppID() AppID { return AppID(xid.New()) }
func NewTraceID() TraceID { return TraceID(xid.New()) }

And thanks to Go's type system, both AppID and TraceID would concretely be different types, such that the following
would become a compile error:

var app AppID = NewTraceID() // this won't compile

This would have worked. However, we would have to implement all the marshalling functions (encoding.TextMarshaler,
json.Marshaler, sql.Scanner, etc.) for each concrete type like AppID. To minimise boilerplate writing for our team, this would mean using a code generator to write it for us.

Note: We could have written the marshalling functions once using type aliases (type AppID = xid.ID), but the compiler would have treated the ID types as interchangeable.

Another downside here is our system isn't just Go. We also have a Typescript frontend and a Postgres database. This means once we encode the ID into a wire format, we've lost all guarantees of type safety, and now in another system, it's possible to mistakenly use one form of ID for another.

Before Go 1.18, we could have solved this by adding a wrapper struct containing the type information:

import (
    "fmt"
    "github.com/rs/xid"
)

type EncoreID struct {
    ResourceType string
    ID           xid.ID
}

type AppID *EncoreID
type TraceID *EncoreID

func NewAppID() AppID { return &EncoreID{ "app", xid.New() } }
func NewTraceID() TraceID { return &EncoreID{ "trace", xid.New() } }

Now we can update our marshalling functions to prefix the EncoreID.ResourceType at the beginning of the string. One
slight downside here is that aside from passing around a lovely 20 byte array (which is how xid is encoded in memory),
we would also be passing around a struct with string instances. Not super inefficient, but not as nice!

Enter Go Generics

With Go 1.18, we can create a single abstract ID type and then create different concrete types based on a ResourceType,
but really are just xid's. So conceptually, we started with this:

import (
    "fmt"
    "github.com/rs/xid"
)

type ResourceType struct{}
type App ResourceType
type Trace ResourceType

type ID[T ResourceType] xid.ID

func New[T ResourceType]() ID[T] { return ID[T](xid.New()) }

This gave us an excellent starting point, as the memory format of these identifiers had not changed from XID's underlying [12]byte.
However, the compiler will treat ID[App] and ID[Trace] as two distinct types. We also don't need to use code generation for all the marshalling functions as we can write them once for the generic type.

The one requirement we've not solved with this generic version is our type safety for both the wire formats and the human
readability. (i.e. it's still possible for us to json.Marshal an application ID and json.Unmarshal that into a trace ID). To solve this, we can utilise the ability for generics in Go to create the default instance of a type. Then we can call a method on it. For example, our string method looks a little like this:

func (id ID[T]) String() string {
    var resourceType T // create the default value for the resource type

    return fmt.Sprintf(
        "%s_%s",
        resourceType.Prefix(), // Extract the "prefix" we want from the resource type
        xid.ID(id).String(),   // Use XID's string marshalling
    )
}

However, we've skipped a tiny bit: how do we make our App and Trace resource types have a Prefix method we can call?
Well, we need to change ResourceType into an interface:

type ResourceType interface { Prefix() string }

type App struct{}
func (u App) Prefix() { return "app" }

type Trace struct{}
func (a Trace) Prefix() { return "trace" }

All of our marshalling and unmarshalling code can now include and verify the identifiers type prefix, which means we
can easily validate if due to a typo, we pass trace_0ncid27rirfkbch0hld0 as an argument to an API expecting an application ID.

PS. Postgres is here

As I alluded to earlier, these strongly typed wire formats mean we can create matching types in other systems without losing any of the type safety guarantees we wanted. For instance, we use code generation to automatically create new
Postgres domain types for each of our ResourceTypes in our database so we can strongly type the database columns.

CREATE DOMAIN application_id AS TEXT CHECK (VALUE ~ '^app_[a-z0-9]{20}$');

We then use SQLC to generate type safe Go code to read and write to our database, taking our custom ID types and passing them through all the layers to the database without losing our type safety. We'll talk about how we use SQLC and other code generation in a future blog post to give us type safety into and out of the database. I'll leave you with a little sneak peek of a generated helper:

func GetMembersForApp(q db.Querier, id eid.ID[eid.App]) ([]*db.AppMembers, error)

Final words

While generics have been a long time coming in Go, there has been excellent code generation tooling to fill that gap. The two are not mutually exclusive and can be used simultaneously to reduce the cognitive load and the boilerplate code you have to write. It's easy to overuse and abuse generics, but if you take your time and use them carefully, you'll be thinking with Generics!

You can use all the Go 1.18 features, including Generics, with Encore. We'd love it if you'd try out Encore and join Slack to give us feedback!

This article was originally published on the Encore Blog on 25-03-2022.