DEV Community: Ilya Kaznacheev

Step outside the Happy Path

Ilya Kaznacheev — Wed, 04 Dec 2024 08:17:50 +0000

In software engineering, the term happy path refers to the ideal scenario where everything works as planned. It's natural to begin system design by focusing on these positive scenarios - they’re simple, straightforward, and align with how most people envision their product functioning.

However, stopping at the happy path is a mistake. While it’s often sufficient for stakeholders from business backgrounds who lack technical expertise, it fails to address the majority of a system’s complexity. Robust systems excel not in perfect conditions but in their ability to handle problems and recover from failures.

An architect, technical lead, or principal engineer plays a crucial role in ensuring that all scenarios - not just the happy path - are considered and explained to non-technical stakeholders. Their leadership helps foster a team culture that encourages proactive thinking and the design of resilient systems.

Seeing the Bigger Picture

Experienced engineers - whether architects or senior developers - stand out by having a bird’s-eye view of the systems they build. This means recognizing areas without clear ownership, identifying potential issues, and ensuring product integrity across all teams and components.

In complex systems, particularly distributed ones, overall availability is a product of the availability of each component. Components like microservices, databases, and messaging systems interact with infrastructure elements such as virtual machines, networks, and storage. Failure in one part, such as data consistency or error handling, can lead to cascading issues. Senior engineers are responsible for identifying these risks early and implementing solutions that ensure the system works as a cohesive whole.

Mapping Out Scenarios

A complete understanding of the system is essential to account for every potential scenario. This is especially critical in brownfield projects, where undocumented processes or unaddressed questions are common.

Start by formalizing the architecture. You don’t need heavy enterprise methodologies like TOGAF; use formats that fit the team and product. Frameworks like arc42 are suitable for complex systems, while simpler projects may only need C4 diagrams supplemented with a few additional visualizations.

Identify all data flows within the system. This will help you uncover gaps and overlooked details. At each step, consider what could go wrong. Create a list of possible events, ensuring it is MECE. This practice minimizes blind spots and ensures that every edge case is addressed.

Prioritizing Issues

Not all potential problems are equally likely or impactful. For instance, network errors occur more frequently than hard drive failures, and both are more common than data corruption caused by solar flares.

Similarly, not all problems are equally critical. Losing a log entry is often acceptable, but errors in customer billing are not. Combining the likelihood and importance of each issue helps prioritize which scenarios to address immediately and which can wait. A good system balances proactive preparation with realistic limitations, as it’s impossible to account for every scenario.

Dealing with the Unknown

Even the most thorough analysis won’t uncover every issue. Some problems only become apparent after they occur. For example, throttling in Kubernetes can disrupt Kafka’s heartbeat mechanism, triggering endless rebalancing. Or a misconfigured high-availability setup for virtual machines might cause PostgreSQL transactions to freeze, leading to application crashes. As you can guess, I only discovered these problems when they occurred.

These "unknown unknowns" require systems to be designed for self-healing. For example, managed cloud Kubernetes platforms (e.g., GKE, AKS, EKS) monitor cluster nodes and automatically replace them if they become unresponsive. This ensures the system recovers, regardless of whether the issue was caused by software bugs, network failures, or even server rack fires.

A system capable of self-healing must have a clear understanding of its current and desired states, along with mechanisms to return to the desired state automatically.

Communicating with Stakeholders

Building resilient systems is essential but often expensive. Architects and engineers must communicate with decision-makers - especially those controlling budgets - about the value of investing in solutions for potential problems.

CFOs and other executives may argue, "Our customers only pay for the happy path". While true to an extent, the happy path alone cannot guarantee long-term reliability or customer satisfaction. Effective communication with non-technical stakeholders requires speaking their language: the language of business.

Explaining the Business Impact

Every technical decision has financial implications. Calculate and present these clearly:

How much will it cost if the system fails to handle non-happy-path scenarios?
How likely are these scenarios?
What are the potential losses in revenue, reputation, or customer trust?
How much will the solution cost to develop and maintain?
What additional benefits could the solution provide?

For example, a happy path might assume that a request in a distributed system will complete within 30 seconds. However, as the number of services grows or their workload increases, some requests may exceed the timeout, triggering cascading failures. These might require rollback of distributed transactions, and retrying requests could overload the system, potentially causing hours-long downtime.

One solution could be to shift to asynchronous communication, at least for certain processes. This approach requires implementing and maintaining a message broker, adapting applications, and training engineers to work with a new communication paradigm. While time-consuming and resource-intensive, it would eliminate timeout issues, allow virtually unlimited scaling for critical processes, and enhance fault tolerance through delivery guarantees.

Ultimately, the decision is not solely technical. If a company is under financial pressure, immediate profit from user-facing features might take priority. On the other hand, if the company’s strategy is focused on growth, scalability might align well with its goals.

By speaking the language of money, you can make technical proposals resonate with business decision-makers. Conversely, if you can’t justify a solution financially, it might not be beneficial for the company.

Bring Solutions, Not Just Problems

People tend to dislike hearing about problems without solutions, even if they claim otherwise. When discussing issues with non-technical stakeholders, avoid leaving problems "hanging". Busy individuals already have enough challenges to address without unresolved technical issues added to the mix. Always present a few potential solutions to provide a starting point.

Problem-solving is a highly valued skill in engineering. Often, exploring solutions independently eliminates the need to escalate issues, allowing you to bring fully-formed answers to your manager or client. This initiative demonstrates reliability and builds trust, as long as the solutions are well thought out.

The Sooner, the Better

Architectural decisions are costly and difficult to change later. The same applies to "problematic" scenarios - delays in addressing them can make implementation significantly more expensive.

That’s why it’s critical to discuss and resolve such scenarios as early as possible. Of course, this can only happen once you become aware of them. How can you identify these scenarios sooner?

Building an Engineering Culture

In traditional waterfall methodologies, architects spend significant time analyzing potential risks and solutions before writing any code. While thorough, this approach is often inefficient for projects in dynamic contexts.

With agile methodologies, teams begin building solutions quickly, addressing problems as they arise. While this reduces time spent on unnecessary scenarios, it risks overlooking critical details and under-planning architecture.

Regardless of methodology, early identification of problems depends not only on analytical tools but also on fostering a strong team culture.

Open Discussions About Problems

Unlike business communication, internal team discussions should be open and direct about potential issues. Encourage brainstorming and collaborative problem-solving to explore various options.

Team members must feel safe raising concerns without fear of blame. Unfortunately, in some workplaces, the "no good deed goes unpunished" culture prevails, where the person who identifies an issue is burdened with solving it alone. This discourages initiative and erodes both team morale and system quality over time.

Building a Problem-Solving Culture

While early identification of problematic scenarios is crucial, it’s equally important to cultivate a team culture that values open communication and initiative. Encourage team members to raise concerns without fear of blame. Collaborative brainstorming can often lead to innovative solutions.

Avoid siloed thinking where teams focus solely on their responsibilities. A sense of ownership over the broader product helps engineers identify risks in gray areas that might otherwise be overlooked. This approach prevents costly issues and fosters a more adaptable and resilient organization.

Conclusion

While identifying and addressing the full range of potential events is a key responsibility of architects and senior engineers, it’s a team effort. Early detection and proactive solutions benefit the product and the company. Communicating these issues effectively, especially to non-technical stakeholders, requires a focus on financial impact and actionable solutions.

No system can predict every failure, but by addressing the most critical and likely scenarios, you ensure resilience. For everything else, self-healing mechanisms provide a safety net, allowing the system to recover independently. By fostering a culture of open communication and initiative, you enable teams to build systems that thrive in an unpredictable world.

Product Architecture Cheat Sheet

Ilya Kaznacheev — Tue, 05 Dec 2023 23:00:00 +0000

In this article, I will give you a no-brainer for choosing the right architecture for any project. Right doesn't mean perfect, but it does mean good enough.

You may or may not be a CTO, Lead Architect, or Principal SWE. You may be working on a greenfield or brownfield project. You may work in a startup or a megacorp. But at some point, you will be at the very beginning of a tech product, and you will need to choose a proper architecture for it. At least for MVP. At least before the first paying customers. At least before the first million users. At least...

There will be no deep analysis or theory. Just pure practice based on my own experience and knowledge. If your startup goes to the moon after reading this article, you can buy me a coffee.

Size Matters

One Man Army

Are you building an MVP, PoC, or read-only code to be thrown away after the first round of investment? Say no more - your soul architecture is Monolith. Yeah, it's not swag enough, nothing shiny to share at the smoothie unicorn meetup. There are bad rumors about Monolith, but fear not. It has proven to be very robust and powerful. Just think of SoC, invest in some readable code, use proper tools, and do not over-engineer - and you will be fine. Monolith will save you a lot of time, so much precious at this stage of your product - better spend this time improving your product or finally sleep.

Small Team

No longer MVP, but still small? Well, here we go again. Monolith is your best friend. Don't spend a lot of time communicating between parties, fine-tuning protocols, authentication, zero trust, routing, and service discovery - just put everything you want into a single application! It shouldn't stink, though. Build boundaries, separate logical blocks, and introduce abstractions. Build as if there are microservices, just don't break your application into cans of separate processes. Interfaces instead of RPCs, database transactions instead of SAGAs, imports instead of DNS. Run fast, but don't skimp on quality - there's a recipe for successful teamwork.

Part Of The Ship, Part Of The Crew

Do you have an ambitious team, do you work on an ambitious project, and it is expected to be huge? Does it belong to a single business domain? If so, Monolith probably fits here too. And why is that? Because it is very likely that you will share many of the same business entities and logic across applications. Breaking it up into many separate deployments will just make everything more complex and slower without providing any real benefit. That doesn't mean your service can't scale and handle peak loads. Design the application to be stateless, allow for concurrency, and stay cloud-native - and you will be able to scale it with ease. Read my blog for more tips on scalability, or ask me for a scalability workshop for your company (my own ad in my own blog, why not?).

Rich Man In Poor Man's Shoes

Have multiple domains, but still a small team? Not enough engineers to build a proper (5-7 members) team for each domain? I have news for you - Monolith is your friend. Why? Because without dividing engineers into separate teams, microservice architecture will not work. It only makes sense if each team is independent in terms of organization, only then they will be able to build properly isolated services without a distributed monolith. So why not build a centralized one? You still have all the tools to build a beautiful, well-architected, scalable, resilient service. But without a lot of the limitations that microservices impose.

Why not Lambdas? Well, the answer is maybe Lambdas. But you have to be sure that you know what you are doing. And so does your team. I will make a note about lambdas later.

Higher, Stronger

All The Colors Of The Rainbow

So you have many business domains, and your team is large enough to form independent domain groups? Congratulations, Microservices will work for you. Just don't forget the ground rules - no shared databases, no shared models, no state - only stateless, only APIs, only cloud-native. Microservices are very expensive, not just in cloud costs. They are much harder to build, harder to deploy and run, and harder to monitor and troubleshoot. With microservices, you have to bring a whole layer of supporting tools and roles - proper infrastructure (like Kubernetes), CI/CD, distributed observability, DevOps, DevSecOps, API gateways, and so on. I mean, a mature product will have all that anyway. It's just too expensive for a startup and completely slows down a small team.

Fear Is The Mind-Killer

Want to bring something new to the table? Something like lambdas? But you're afraid that when it's time to go, you'll be stuck with a vendor solution or sacrifice for a so-so vendor-agnostic solution. This fear is real, unfortunately, I could have told you to go for some open source vendor agnostic tools like Knative, but I didn't. Because I don't believe in them myself. While we are still waiting for a Kubernetes-level production-ready WASM orchestrator, Microservices architecture is still better for this case. Hopefully, this will change soon and the whole industry will move towards WASM-based architectures and tools.

The Spirit Is Willing, But The Flesh Is Weak

Still want lambdas, event-driven, reactive programming? Do you want functions, actors, monads? Well, brave one, ask yourself: does your team know all this shiny functional stuff? If the honest answer is no - bye-bye lambdas, hello Microservices. Seriously, choose either business goals or cool concepts, unless your team members are Clojure gurus. Otherwise...

Cherry On Cake

Yes, you deserve it. AWS Lambdas, Azure Functions, GCP Cloud Functions, Alibaba Functions Compute - you name it. You definitely know what you're doing, and your team worships the same god. Let go of the fear, and step into destiny. Let reactive architecture consume you. There is no turning back, and you know it. Enjoy.

Summing Up

For your convenience, I've organized all the ideas into a chart that you can carry with you at all times.

Virtual Machine Scaling

Ilya Kaznacheev — Tue, 26 Jul 2022 10:44:00 +0000

Virtual machines (VMs) are the basic building block of cloud infrastructure - all cloud products like databases, orchestrators, message queues, functions and others are based on them. Many companies also use virtual machines as the foundation of their cloud landscape, running all sorts of services and applications on them.

In this article, I will explain ways and approaches for scaling VMs.

Virtual machines give the user more power and control over the environment and applications, but require greater skill and more atomic configuration. Therefore, scaling a system based on virtual machines is less flexible than the alternatives I will be discussing in other articles. So how do you configure virtual machine scaling to increase resource utilization on the one hand, but on the other hand be prepared for peaks in load?

There are several ways to solve this problem:

manual scaling;
automatic scaling.

Let's go into more detail.

Manual Scaling

When operating virtual machines, a situation may arise when the resources allocated to them (CPU, RAM) become insufficient. To solve the problem, manual scaling is performed - a set of actions that the administrator takes to change the amount of system resources, usually upwards.

But what exactly do you need to do? There are two main ways of scaling in this case:

vertical scaling - changing the resources of a single machine;
horizontal scaling - changing the number of machines without changing their configuration.

Vertical Scaling

When scaling vertically, you can change the main resources of the machine - CPU and RAM. This usually requires stopping and then restarting the virtual machine. You can also change the size of the virtual disk mounted to the machine, but in this case, you will need to make changes to the file system as well.

Vertical scaling is widely used for partition intolerant (CA) systems, which can only run in a single instance. Examples of such systems are classical databases like PostgreSQL and MySQL, as well as applications based on a monolithic architecture.

This is the easiest solution when scaling, since it does not require any changes to the applications. It is the first thing to go to when your system lacks resources.

However, server costs increase non-linearly as resources increase, so at a certain point, this type of scaling becomes too expensive. If you need to further scale your system, horizontal scaling is the way to go.

Horizontal Scaling

This type of scaling assumes that as the load changes, the number of servers handling that load changes proportionally, while the configuration of the machines remains the same. To do this, you can either simply create additional machines manually, or create them as part of an unmanaged instance group.

In the first case, the administrator is responsible for creating and configuring the new virtual machine and putting it into operation. Using instance group, on the other hand, makes the process somewhat easier, as the machine is created from a template with a fixed configuration. However, regardless of the way new machines are created, the development of a horizontally scalable system has a number of complexities:

the application must support multiple replicas running in parallel;
the infrastructure must be able to deliver data to all replicas;
the infrastructure must strive to distribute the load evenly.

Consideration of these complexities deserves a separate article, but today's systems are evolving in this direction.

However, manual changes can lead to untimely responses to load changes. You may miss when the load has increased, and thus lose some requests or data, which can lead to serious business losses. On the other hand, a decrease in load can also go unnoticed, causing purchased servers to sit idle without load.

To make the system more self-sufficient and able to respond quickly to load changes, it is worth taking advantage of automatic scaling.

Automatic Scaling

Automatic scaling involves running virtual machines within a managed instance group. Such a group independently monitors the state of the machines in the group and the distribution of load on them. When load changes, the group decides whether to add or remove machines to the group.

The criteria on the basis of which the group decides to scale are flexible. For example, it can be:

CPU utilization threshold in percentage;
number of http requests per machine;
number of queued messages per machine;
customizable metrics based on monitoring data.

Such options allow for flexible autoscaling depending on the processing scenarios of a particular system. In addition, many providers offer predictive scaling tools powered by analytics or machine learning. Such tools learn from real-world scenarios of machine usage and load changes, and optimize scaling in ways that increase resource utilization.

Managed Instance Group

To control a fleet of machines, a managed instance group is created with a specified setting. Often these settings include:

location (region and zone or zones);
scaling settings (type and parameters);
network settings;
virtual machine template.

The template must start the service at VM startup, so that it starts itself and gets configured if necessary (e.g., with service discovery). This is necessary so that when instances are automatically added to a group, they are started automatically. I usually use systemd for this but the cloud init will also work.

As for the scaling settings, you select the type and parameters, as well as the minimum and maximum number of machines in the group.

When the load is low, the number of instances will automatically decrease until it reaches a minimum number. This number will be kept until the load exceeds the threshold again.

On the other hand, when the load is high, the number of machines can reach a maximum, after which new machines will stop being added no matter how the system handles the load.

When creating an instance group, it is important to verify that your quotas and limits allow you to create the required number of machines and allocate the required number of network addresses to them.

Note that each machine in the group will have a different IP address. For the network communication with the instance group, load balancers are used, which I will talk about in the next article.

How To Set Up An Instance Group In Google Cloud

To set up a managed instance group in GCP, you must first create a virtual machine template with the application. To do this, just create a regular virtual machine in the Google Compute Engine based on the operating system of your choice. In my example I created a virtual machine based on Debian and installed the cpuburn application that I will use to emulate CPU load.

In order for the application to start automatically at the virtual machine startup, the service must be configured, in my example at /etc/systemd/system/cloud-scaling.service.

[Unit]
Description=Cloud Scaling Example

[Service]
ExecStart=+${PWD}/cpuburn-web -p 80

[Install]
WantedBy=multi-user.target

And started.

sudo systemctl daemon-reload
sudo systemctl enable --now cloud-scaling

The virtual machine must then be stopped in order to create an image based on its disk.

Next go to Instance templates and create a new one. You also need to enable HTTP traffic in firewall settings.

Change boot disk to the image you've just created.

Now we have everything we need to create an instance group based on this template.

Specify the minimum and maximum number of machines in the group, add one or more autoscaling criteria.

After the group is created, one virtual machine will be started, matching the minimum number.

Let's burn some CPU! I'll trigger cpuburn application on the node and see what happens next.

The monitoring tab shows the dynamics of adding virtual machines, as well as changes in the total CPU utilization - when cpuburn started, it jumped up dramatically, but after adding new machines ( and therefore new resources) fell down again.

After disabling cpuburn, the instance group removes unused machines after a while...

...going back to the minimum number again.

Looking at the metrics again, we can see the history of changes in the virtual machine count in the group.

With these simple steps, you can configure a fault-tolerant system capable of withstanding large load spikes. The flexibility of the scaling criteria allows you to adjust for different usage scenarios and different types of input data.

However, the low speed of creating and deleting instances makes this tool inconvenient for systems with sudden jumps in load, demanding to process the maximum number of requests. For such tasks, other cloud products are better suited, which I will describe in the following articles.

Where to Place Logger in Golang?

Ilya Kaznacheev — Sat, 25 Jun 2022 09:52:00 +0000

Logging is an integral part of any application. However, the correct location and use of the logger in the project structure raises questions even for experienced developers.

There are several ways of doing this, of which I give preference to one. I will explain why.

When designing an application, the developer chooses from several explicit options:

store the logger in a global variable;
get the logger from the logging library;
add the logger to data structures;
pass the logger explicitly in the function call.

In this article, I will use Zap logger as an example, but you may use any.

Store The Logger in a Global Variable

This is the simplest option that comes to mind.

var l, _ = zap.NewProduction()

It is good in its simplicity, but its implementation can be inefficient:

the use of global variables increases the coupling in the program, worsening the structure of the code;
the global logger is inconvenient to adjust for a particular context, for example - to add common fields for a message set to the logger;
defining a global scope can be tricky for an application with multiple executable files.

Get The Logger from the Logging Library

This option is almost the same as the global variable, only the variable is hidden inside the imported module.

l := zap.L()

This is a little more convenient to use, because you don't have to search for the global variable in the code and import it. On the other hand, it is possible to get a library version conflict when plugging in external dependencies.

Add The Logger to Data Structures

This is a fairly common alternative used by many developers. However, its use adds an element to structures that does not relate to the data structure, but to their instrumentation.

type UserRepository struct {
    l *zap.Logger
    /*
    other fields
    */
}

func (ur *UserRepository) SaveUser(ctx context.Context, u user.User) error {
    ur.l.Info("saving user")
    // ...
}

In other words, logging is not related to data, but to operations, so it makes more sense to have it in functions and methods rather than keeping it in structures.

Pass The Logger Explicitly in the Function Call

It turns out that the most logical place for a logger is in the methods or functions themselves. However, passing a logger into every function obviously leads to a huge code complication:

Adding a parameter to a function will worsen the readability of the signature;
explicitly passing the logger along the call chain adds a lot of copy-paste;
functions with a fixed signature (e.g., HTTP handler) will require you to pass the logger through workarounds.

func CreateUser(ctx context.Context, u user.User, l *zap.Logger) error {
    l = l.With(zap.String("userID", u.ID))

    l.Info("user creation started")

    if err := SaveUser(ctx, u, l); err != nil {
       l.Error("user creation failed", zap.Error(err))
       return err
    }
    l.Info("user creation done")
    return nil
}

This makes the approach inapplicable in practice. But is there a way to combine the pros of the different approaches, avoiding the cons?

The Win-Win Context

As a developer, I would like to pass the logger to a function without adding new parameters to its signature. But these requirements seem to be contradictory.

Fortunately, there is an entity in Golang which is often passed to every function, and performs many utilitarian tasks - it is context.Context.

Context in Go solves very important problems:

gracefully interrupting processing at application shutdown;
limiting the execution time of a function;
managing asynchronous operations;
getting out of infinite loops;
distributed tracing.

As we can see, these tasks are purely utilitarian, that is, they do not relate to the business logic of the application. The same is the task of logging (if we are not talking about special logging kinds, such as an audit log).

This way, passing the log via context will not disrupt the function's signature (since most functions already take the context as a first parameter, and others will benefit from it). This can be done as follows.

package log

import (
    "context"
    "go.uber.org/zap"
)

type ctxLogger struct{}

// ContextWithLogger adds logger to context
func ContextWithLogger(ctx context.Context, l *zap.Logger) context.Context {
    return context.WithValue(ctx, ctxLogger{}, l)
}

// LoggerFromContext returns logger from context
func LoggerFromContext(ctx context.Context) *zap.Logger {
    if l, ok := ctx.Value(ctxLogger{}).(*zap.Logger); ok {
        return l
    }
    return zap.L()
}

This will greatly simplify the use of the logger and provide additional benefits:

add common fields to the log along the call chain, such as adding a userID from a request;
implement middleware for http/grpc, which will add to the log some information about the request;
integrate log with other tools that use context, such as distributed tracing.

In addition, the approach solves the issue of access to global data - now the logger is always local to the called function, and can be used without restrictions.

func CreateUser(ctx context.Context, u user.User) error {
    l := log.LoggerFromContext(ctx).With(zap.String("userID", u.ID))

    l.Info("user creation started")

    ctx = log.ContextWithLogger(ctx, l)
    if err := SaveUser(ctx, u); err != nil {
       l.Error("user creation failed", zap.Error(err))
       return err
    }
    l.Info("user creation done")
    return nil
}

In practice, it is not always convenient to use this approach, but it covers about 90%-95% of cases in a server application. For the rest of the cases, you can use another approach from those listed above.

It also comes in handy in teamwork, where different team members can add call context to the logger at different levels of the application as they work on their changes. This will eventually be combined into informative and useful records that are easy to work with.

Basics of cloud scaling

Ilya Kaznacheev — Sun, 29 May 2022 00:00:00 +0000

In this article, you will learn what application scaling is, and how cloud infrastructure simplifies the process and makes it more useful for business.

Scalability according to Wikipedia is the property of a system to handle a growing amount of work by adding resources to the system. In other words, it is the ability of the system to increase or decrease the amount of used resources (RAM, CPU, network bandwidth, disk size, etc.) depending on the load - the number of requests per second, the amount of stored data, computational load, etc.

When we talk about scaling, it's worth starting with defining the system itself. A system can be either monolithic or distributed, and depending on this, the way it scales is different.

Scaling a Monolithic System

This type of system has existed for quite some time, and is the easiest to implement. Usually the application comes as an executable file or a set of scripts run by the server. It runs on a physical server or a virtual machine, often together with a database and a reverse-proxy like Nginx. Thus, networking problems in the interaction of components are minimized, and the system itself can be delivered as a virtual machine template or even as an installation disk.

The approach to scaling such systems is vertical scaling - when the system runs out of resources, it restarts on a more powerful server with more resources.

This used to work well when manufacturers were releasing computers twice as powerful every six months. In recent decades, however, the rate of releases has decreased as microchip capabilities have approached their physical limit. Therefore, more powerful computers today are much more expensive than their less powerful counterparts. And doubling the capacity can increase the cost by 5 or even 10 times.

So when you reach a certain point, it simply becomes too expensive to buy a more powerful server. It is much more cost-effective to buy a lot of cheap computers, getting the total of the same resources. But to operate a system based on a fleet of individual machines, it must be distributed.

Scaling a Distributed System

The ability to scale the system is divided into three directions:

horisontal duplication;
functional decomposition;
data partitioning.

Let's take a closer look.

Horisontal Duplication

Horisontal duplication is about running multiple copies of the same application in parallel. The copies have the same functionality, and are called replicas. In most cases the load is evenly distributed between replicas. To distribute requests between replicas, a load balancer is used, but that will be discussed in another article.

The more replicas a service has, the more resources are available to it. When resources become insufficient to handle the load on the system, it is sufficient to run several new replicas, and the excess load will be processed.

Horisontal duplication also helps to achieve increased service availability. In case one of the replicas becomes unavailable (application crash due to a bug, network problem, resource shortage and OOM Killer), the load will be distributed among the remaining replicas, and the load balancer will prevent data loss.

This approach is also called as horisontal scaling.

Functional Decomposition

In functional decomposition, system functionality is divided into separate components (e.g., microservices). Each domain of the system or part of the functionality is decoupled into a separate and independent service.

This makes it possible to isolate the logic of individual loosely coupled parts of the system. This simplifies the process of service development and delivery, and reduces the probability of mutual dependencies and blocking between development teams.

Teams become more independent - they can choose their own technologies and development tools, their own team collaboration frameworks (Scrum, Kanban, etc.), best suited to the nature of the application and the skills of team members. In doing so, the various services are no longer dependent on common parts of the application, libraries, or shared memory. They exchange data explicitly through APIs, so all that connects them to each other are synchronous and asynchronous message contracts.

Most importantly, functional decomposition allows each individual service to scale according to its needs. In the case of a monolithic system, the scalability of an individual component is very limited. In the case of a distributed system, one of the components can be run on multiple replicas, while the other will be run on a more powerful machine

Data Partitioning

Data partitioning is the splitting of stored data into several separate parts. For example, data stored in a single database is partitioned into several database instances, and finding a particular record in a particular database is determined by the partitioning key.

This way you can get around the limitation of storing data on a disk - the maximum disk size is physically limited, and when the amount of data exceeds its size, partitioning allows you to divide the data among several disks and thus store more data.

Another possibility of data partitioning is to distribute the query processing load among several databases. Typically, a database is a complex application that consumes a fair amount of CPU and RAM. As the number of queries or their complexity increases, the database consumes more and more resources. Data partitioning allows you to have more database instances, and therefore have more computing resources available to perform queries.

Cloud Benefit of Scaling

Cloud platforms, unlike the bare metal or virtual machines available in conventional hosting, offer a simple infrastructure management option. For example, an administrator can configure the cloud infrastructure through a simple web interface, or through a CLI application, or via an API.

It also provides a wide range of options for automating infrastructure management. From routine tasks automation through simple scripts using the CLI, to implementing the Infrastructure-as-a-Code approach using tools like Terraform.

Finally, clouds also offer built-in infrastructure management solutions, such as automatic scaling (including smart self-learning autoscaling in the largest clouds). These tools allow you to spend less time configuring infrastructure so you can use it to grow your product.

In the following articles, I will talk about specific cloud scaling options and teach you how to set up an autoscalable system by yourself.

Clean Transactions in Golang Hexagon

Ilya Kaznacheev — Mon, 14 Mar 2022 13:55:02 +0000

How to implement transaction management in a hexagonal microservice? How to keep the isolation of the application layer and the database adapter? In this article I will share my experience in solving this problem.

Clean architecture (aka onion architecture) is very popular in modern microservice development. This approach clearly answers a lot of architectural questions, and is also good for services with a small codebase. Another nice feature of clean architecture is that it combines well with Domain Driven Development - they complement each other perfectly.

One of the applications of clean architecture is hexagonal architecture, an approach that explicitly distinguishes layers, adapters, and so on. This approach has gained love among Go developers because it does not require complex abstractions or intricate patterns, and does not contradict complicated language idiom - the so-called Go way.

But there is a problem I often see in many teams adapting hexagons, which I myself have encountered and successfully solved - the implementation of database transactions within DDD and the very hexagon itself. I will tell you what I have done in this post.

The High Abstraction Problem

A hexagonal architecture implies an inversion of dependencies as follows: at the center of everything is the data model, the domain logic is built around it (and depends on it), a layer of application logic is placed on top of it, and then there are adapters hidden behind interfaces called ports. This can vary, but the basic idea that dependencies diverge from the center to the periphery remains.

As an example, let's imagine for a moment that we are doing a microservice that sells used cars.

Let's imagine that one of the adapters is a module for database interaction. But not any random database, but one that supports ACID transactions. Interacting with the database is pretty easy to implement - we wrap the domain models in repositories, hide each repository behind an interface (port), and implement it inside the adapter. Such a port may look something like this:

package port

import ...

// CarRepository car persistence repository
type CarRepository interface {
    UpsertCar(ctx context.Context, car *model.Car) error
    UpsertCarHistoryEvent(ctx context.Context, event *model.CarEvent) error
    GetCar(ctx context.Context, id string) (*model.Car, error)
    GetCarsBatch(ctx context.Context, ids []string) ([]model.Car, error)
    GetCarsByTimePeriod(ctx context.Context, from, to time.Time) ([]model.Car, error)
    GetCarsByModel(ctx context.Context, model string) ([]model.Car, error)
    DeleteCar(ctx context.Context, id string) error
}

On the domain logic side, this adapter will be passed as a DI through the interface:

package domain

import ...

type Car struct {
    carRepo port.CarRepository
}

func NewCar(carRepo port.CarRepository) &Car {
    return &Car{
        carRepo: carRepo,
    }
}

For example, the logic of the car search by year of manufacture would be as follows:

func (c *Car) GetNewCars(ctx context.Context, from time.Time) ([]model.Car, error) {

    if err := c.someValidation(); err := nil {
        return nil, fmt.Errorf("invalid request: %v", err)
    }

    // read latest cars
    cars, err := c.carRepo.GetCarsByTimePeriod(ctx, from, time.Now())
    if err := nil {
        return nil, fmt.Errorf("newest cars read: %v", err)
    }

    return c.filterCars(cars), nil
}

This works pretty well with simple atomic operations like create, delete, or read. But quite often you need to execute complex logic within a single database transaction. I won't describe the examples, you know them well enough.

The issue here is that in terms of architecture a transaction is part of the database adapter - it is opened and closed with certain commands (BEGIN, COMMIT or ROLLBACK in SQL), and has a binding to the generated entity - the transaction object. Transaction object itself is usually not hovering in the clouds of the global program scoop, but explicitly bound to the database connection session over the TCP connection. So we can't (and don't want to) abstractly declare "transaction start" and "transaction end" in business code. When opening a transaction we have some entity - a transaction object - which we need to pass to the adapter to perform database operations exactly within this transaction.

This is where the chicken and egg problem arises. On the one hand, the adapter requires that for each query within a transaction, information about that transaction itself must be passed - usually the library used implements the transaction as some kind of object through which queries can be made. On the other hand, the domain or application layer should not know about the adapter implementation in the hexagon paradigm. You can wrap the transaction in some interface, but this interface will be monstrously huge (with methods like Select, Insert, Delete and others - open your favorite SQL library and see how many methods there). And it will not make any sense to the domain - these methods will be used within the adapter, where there is a direct access to the transaction object without any abstraction.

We could go another way and pass the transaction as interface{} and then cast it to the desired type in the adapter via reflexion, but I consider this approach unserious and unsuitable for productive code. Besides, I don't want to pollute method signature by passing an additional transaction object, because it doesn't directly relate to the method itself. The transaction object a indicates the specifics of the whole process of working with the database as part of the operation. So what to do?

Specific Implementations Solution

Now a couple of words about the context of our solution. In my search for an elegant implementation, I have encountered several times solutions like UnitOfWork, representing a transaction as some business entity (which the business logic core knows about). Indeed, a transaction can be represented as some kind of business entity - after all, business logic can require atomic and non-competitive execution of a transaction. But the problem with elegant ideas is the inelegant implementation - abstract factories, reflexion, and messy handling of the methods of the adapter itself.

Often these complications are dictated by the desire to work with several databases, or to be able to switch from one database to another by changing only the adapter code (and without changing the business logic).

Realizing that this is too "abstract," and in general does not meet the Go Way, I derived a few limitations of our project that were supposed to simplify this task:

The service works with only one database;
We use a particular library to work with the database.

The service works with only one database

And it's PosgreSQL. Well really, how often do you switch between databases? Many people try to write some generic code to work with a generic SQL database, but does it make sense? The experience shows that switching from one SQL database to another will still force you to rework the entire project, and switching from SQL to NoSQL or vice versa is out of the question.

We use a particular library to work with the database

And it is go-pg. I personally really like it as a query builder (rather than an ORM) and it has good performance. It has one feature which I'll tell you about next, without which I would have struggled to implement what I had in mind. But this feature is available in other libraries too, so don't rush to rewrite your code.

What did we end up with

That's why I took transactions in go-pg as the basis with a clear mind. I wanted to keep the repository method signatures clean (only the context and method call parameters), but also make the solution idiomatic from a Go perspective.

Golang has an excellent tool that allows you to pass utility data that doesn't concern a particular method call, but concerns the context of an operation - context.Context. Often telemetry, loggers, idempotent identifiers, and other things go there. From my point of view transaction information perfectly fits the definition of "utility data" - it's some kind of process modifier which doesn't affect the logic directly but has an indirect effect. From words to deeds!

package postgres

import ...

type txKey struct{}

// injectTx injects transaction to context
func injectTx(ctx context.Context, tx *pg.Tx) context.Context {
    return context.WithValue(ctx, txKey{}, tx)
}

// extractTx extracts transaction from context
func extractTx(ctx context.Context) *pg.Tx {
    if tx, ok := ctx.Value(txKey{}).(*pg.Tx); ok {
        return tx
    }
    return nil
}

The first step is to add methods to put the transaction object into the context and retrieve the transaction from the context. The methods are non-exportable, i.e. they can only be called inside the adapter. Note - a transaction from the go-pg package is used here without any wrappers or abstractions. We can afford it inside the adapter!

Next, we need to teach the adapter itself (the repository) how to work with a transaction. And this is where we need a feature that we have in go-pg, but not in some other libraries, such as sqlx. This is a single interface for query methods executed by the library, both in a transaction and without it. They are Select, Insert, Delete and others - they should have the same signature for transaction and non-transaction to be able to be taken out of the interface. If not, you have to write a wrapper. In case of go-pg both database connection object and the transaction object have method ModelContext(c context.Context, model ...interface{}) *Query which we used.

We got a little wrapper that checks if there is a transaction in the context. If there is - it returns Query from the transaction object, and if not - it returns Query from the database connection object.

package postgres

import ...

// model returns query model with context with or without transaction extracted from context
func (db *Database) model(ctx context.Context, model ...interface{}) *orm.Query {
    tx := extractTx(ctx)
    if tx != nil {
        return tx.ModelContext(ctx, model...)
    }
    return db.conn.ModelContext(ctx, model...)
}

Database here is the actual structure that implements CarRepository, whose methods contain SQL queries to PostgreSQL, as well as a connection (or a connection pool) to the database. It can implement more repositories if you have a lot of them.

As a result, the method that reads cars from the database will look like this:

package postgres

import ...

func (db *Database) GetCarsByTimePeriod(ctx context.Context, from, to time.Time) ([]model.Car, error) {
    var m []model.Car

    err := db.model(ctx, &m).
        Where("manufacture_date BETWEEN ? AND ?", from, to).
        Order("model").
        Select()
    if err != nil {
        return nil, err
    }

    return m, nil
}

Moreover, you can use a method with or without a transaction - neither the signature nor the method itself changes. And it is the part of the service with business logic that decides whether to use a transaction or not. Let's see how this is done.

Transactions in Business and Business in Transactions

The only thing left is to implement a method to create a transaction inside the adapter, which would return the context "loaded" with transaction; add this method to the interface, and call it in business logic, obtaining the context with the transaction. And then pass that context to all repository calls, and commit or rollback at the end.

It sounds logical, but it's kind of messy. Maybe Go has a more elegant tool?

And there is one! A closure. Let's create a method that allows us to execute the whole transaction without having to leave our seat:

package postgres

import ...

// WithinTransaction runs function within transaction
//
// The transaction commits when function were finished without error
func (db *Database) WithinTransaction(ctx context.Context, tFunc func(ctx context.Context) error) error {
    // begin transaction
    tx, err := db.conn.Begin()
    if err != nil {
        return fmt.Errorf("begin transaction: %w", err)
    }

    defer func() {
        // finalize transaction on panic, etc.
        if errTx := tx.Close(); errTx != nil {
           log.Printf("close transaction: %v", errTx)
        }
    }()

    // run callback
    err = tFunc(injectTx(ctx, tx))
    if err != nil {
        // if error, rollback
        if errRollback := tx.Rollback(); errRollback != nil {
            log.Printf("rollback transaction: %v", errRollback)
        }
        return err
    }
    // if no error, commit
    if errCommit := tx.Commit(); errCommit != nil {
        log.Printf("commit transaction: %v", errCommit)
    }
    return nil
}

The method takes a context and a function to be executed in a transaction. Based on the context, a new context is created with the transaction, and passed to the function. This also allows to interrupt the execution of the function when the parent context is cancelled - for example, when a graceful shutdown occurs.

Next, if the function is executed without errors, commit is executed, otherwise a rollback is executed and the error is returned from the method.

We'll put this method in a separate port - isolation must be respected!

package port

import ...

// Transactor runs logic inside a single database transaction
type Transactor interface {
    // WithinTransaction runs a function within a database transaction.
    //
    // Transaction is propagated in the context,
    // so it is important to propagate it to underlying repositories.
    // Function commits if error is nil, and rollbacks if not.
    // It returns the same error.
    WithinTransaction(context.Context, func(ctx context.Context) error) error
}

Let's add it via DI to the domain:

package domain

import ...

type Car struct {
    carRepo port.CarRepository
    transactor port.Transactor
}

func NewCar(transactor port.Transactor, carRepo port.CarRepository) &Car {
    return &Car{
        carRepo: carRepo,
        transactor: transactor,
    }
}

This allows us to not worry about transactions within the business logic at all, and simplify transaction operations to the following:

package application

import ...

func (a *Application) BuyCar(ctx context.Context, carId, ownerId string, price int) error {

    if err := a.validateBuyer(ctx, ownerId, price); err != nil {
        return err
    }

    // read owner domain
    car, err := a.readCar(ctx, carId)
    if err != nil {
        return err
    }

    return c.transactor.WithinTransaction(ctx, func(txCtx context.Context) error {
        return car.Sell(txCtx, ownerId, price)
    })

}

Here, transaction management is performed within the application layer.

package domain

import ...

func (c *Car) Sell(ctx context.Context, ownerId string, price int) error {

    if err := c.validateBuyer(ctx, ownerId, price); err != nil {
        return err
    }

    if err := c.validatePurchase(ctx, ownerId); err != nil {
        return err
    }

    event := model.CarEvent{
        CarId:  c.model.Id,
        Seller: c.model.Owner,
        Buyer:  ownerId,
        Price:  price,
        SoldAt: time.Now()
    }

    if err := c.carRepo.UpsertCarHistoryEvent(ctx, event); err != nil {
        return err
    }

    c.model.Owner = ownerId
    c.model.UpdatedAt = time.Now()

    if err := c.carRepo.UpsertCar(ctx, c.model); err != nil {
        return err
    }

    log.Printf("car %s model % sold to %s for %d", 
        car.Id, car.Model, owner.Name, price)

    return nil

}

Note that methods UpsertCarHistoryEvent and UpsertCar are executed in the same transaction, even though the domain layer does not own the it.

Comforting results

In the end, we got:

simple mechanism for transaction execution in terms of the business logic layer;
isolation of levels, abstractions do not leak;
no reflection, all transaction work is typed and fault-tolerant;
clean repository methods, no need to add a transaction to a signature;
query methods are transaction agnostic - if there is a transaction, they are executed within it, if not - directly on the database;
commit and rollback are executed automatically according to the function execution result. No deferring.
in case of panic, rollback will be executed inside tx.Close().

This approach is applicable to any database that supports ACID transactions, as long as there is a common interface for both in-transaction and out-of-transaction queries. If you wish, you can add your own wrapper if your favorite library doesn't have it.

This approach is not applicable when you work with several databases in one service and you need to link two transactions into one. In this case, I do not envy you.

I may have departed from DDD principles or neglected the concepts of hexagonal architecture, but the result came out simple, beautiful and readable.

What would you do? Feel free to comment for ideas and critique!

Practical Use of Finite-State Machines

Ilya Kaznacheev — Mon, 29 Mar 2021 23:18:20 +0000

This is the first article in a series dedicated to FSM usage in distributed system architecture. We will talk about domains, transactions, and sagas. But let's start with the basics.

Finite-State Machine

When we think about the finite-state machine, we probably imagine some computer science-related entities, math, and diagrams like that:

Besides scientific language, a finite-state machine is a final set of states and transitions between them. When it comes to real engineering, states are a set of consistent states in which the model can be. Normally the set is not huge - having an FSM of a hundred states will cause really complex code. In my experience, it is something between three for the simplest models and 20-30 for most complex ones.

Each state represents a real-world state of the model in the current moment. Important note: the state has to describe the state completely - that means you have to rely on the state field only to identify the current model state. If you need to check some extra attributes to identify the state - your FSM isn't granular enough.

Transitions, in turn, are the logic of the model (or class, domain, etc.), which depends on the current state. So now everything your model is supposed to do depends on its current state - well like real-world things do.

FSM in real life

You may wonder why we are talking about something that looks too scientific and forces your code in some boundaries? While you can still implement the same logic without any "weird-looking-diagrams"? That's a great question, and we will answer it.

Let's look at the example. Say, we have a hand-made furniture workshop and store. So the customer orders a tailor-made table for its kitchen. We have to accomplish the following steps:

accept the order;
charge customers money;
request the workshop to produce the table;
request workers to move the table to the warehouse once it is done;
order transport company to deliver the table from the warehouse to customer.

It looks pretty straight-forward, but the devil is in details. You can't just do all these operations in the same API call, because of its duration. While the regular request processing time is under a second, fulfilling the order may take weeks. You can't even calculate the exact time because of the moving part amount involved.

So you have to save the milestones during the order fulfillment. And by "save" I mean "store in a database". Here we are talking about the Order model.



type Order struct {
    ID       string
    ClientID string
    Detais   Details
    State    State
}

The simplest order has a unique ID, customer ID, order details (like delivery address, cost, positions, etc.). We add an attribute State and will discuss it in detail.

A process described using FSM

The process consists of the following steps (more or less):

Now we can prepare a list of states for our order. They will represent each step via a related order state.



type State string

const (
    OrderCreated         State = "order_created"
    PaymentPending       State = "payment_pending"
    ManufacturingPending State = "manufacturing_pending"
    StockMovePending     State = "stock_pending"
    DispatchPending      State = "dispatch_pending"
    DeliveryPending      State = "delivery_pending"
    ConfirmationPending  State = "confirmation_pending"
    OrderConfirmed       State = "order_confirmed"
)

Note that we don't call states like "order_accepted" or "order_delivered", but "payment_pending" and "confirmation_pending" instead. That's because "order_accepted" doesn't represent the state, it represents an event in the past. We can use events to trigger state transitions, but the state has to describe the real-world state.

After successful order placement, we have to charge money from the customer's bank account. And this is the current state of the order - awaiting payment, or "payment_pending". When it's done, we sent the order to the workshop and wait until it is manufactured. In this case, the state "order_paid" will also be irrelevant - it is an event of successful payment, but the state is waiting till the table will be done by carpenters.

So now we can go a bit nerdy and draw a state diagram.

Thus, we got our process explicitly described as a finite-state machine. In fact, it is a pipeline pattern, defining which step will run in which order. But FSM allows us to clearly define validations at each step, and visibly decouple the logic of each step (or state), so you don't have to hold the whole sequence in mind while programming.

Let's now dive deeper into this example.

More than just a pipeline

In real life, things can sometimes go wrong. When you are developing software, there is a myriad of places, where something may go wrong. In other words, you have to handle errors and failures.

It's simple to handle an error when you are processing the synchronous HTTP request. You probably can fit it into a single database transaction and just rollback any changes if an error occurs. You also can simply return an error to the HTTP client without any extra operations like cleanups, compensation transactions, etc.

But when we talk about long or complex operations or actions across several distributed services, a simple approach doesn't work anymore. We need to split into steps, and steps may also include some sub-steps and so on. But how to handle errors in the middle of the pipeline? You can't just return an error message to the user because you now have some mess to clean up. Let's take a look.

In a workshop, many things may go unexpected. Carpenter may take a sick leave or even resign, rare wood may be out of stock, material delivery may delay, and so on. Anyway, there are real cases, when the workshop can't fulfill the order when it is already paid.

So now we have to handle an error in the middle of the process. There may be different kinds of errors or the same error, but the handling will depend on the model state.

If the error occurs when the state is "manufacturing_pending", that means the workshop can't fulfill the order, and you have to refund the money.

There will be new states also.



const (
    RefundPending  State = "refund_pending"
    OrderCancelled State = "order_cancelled"
)

The state diagram will now look like this.

Consistent error handling

The more errors that need to be processed, the more complex the state diagram becomes. But thanks to the FSM, the processing always corresponds to the current state, and the actions taken will always be appropriate.

If an error (or other events) occurs, actions depend primarily on the current state of the model. Next, the transition conditions are checked, and only then the transition logic is executed. Thus, the processing of any events occurring to the entity will always be consistent with its state.

This is what the complete process might look like, with all sorts of errors taken into account.

It may look similar to the saga pattern for those, who have dealt with distributed transactions, and it actually does. But I will tell you about more advanced FSM applications in the following posts.

Practical Advice

Use the FSM to describe states and transitions between them. This will help to clarify and separate the logic of the different states into distinct modules. Try to describe the states in a granular way. Don't be afraid of a large number of states if the data model can actually be in each of them. Have fun drawing diagrams.

Go-Swagger Tricks. Standard HTTP Handler

Ilya Kaznacheev — Wed, 29 Jul 2020 16:15:27 +0000

picture © moise_theodor

Hey, reader! Maybe you had tried go-swagger library so far. If yes, you may notice that sometimes it's not so easy to use. And it may look a bit complicated to start using it.

go-swagger / go-swagger

Swagger 2.0 implementation for go

In this number of small articles, I will share my experience on how to make go-swagger more friendly. Let's start.

Handling requests in go-swagger

By default, go-swagger generates a specific handler type for each endpoint in your Swagger scheme - that is, you will get a code-generated structure with all import parameters kindly parsed for you and a number of responders for each response type (with pre-generated structures as well).

So usually it's ok, because you don't have to parse JSON requests by yourself and can never mind the response data marshaling - the go-swagger framework will handle all that stuff for you. But sometimes you just want to use a standard net/http compatible handler with some endpoint - maybe some library provides that for you (like Prometheus metrics endpoint or Kubernetes probes). Or maybe you have some HTTP libraries written in your organization - e.g. for custom authentication and so on.

So you can't do that in vanilla go-swagger - each generated handler function requires an exact handler to be assigned. Let's say we want to expose a Prometheus metrics endpoint, so for the spec like this

paths:
  /metrics:
    get:
      tags:
        - instruments
      summary: "Prometheus metrics"
      produces:
      - "application/json"
      responses:
        200:
          description: ok
          schema:
            $ref: "#/definitions/Any"

the generated handler function will look like that:

func MetricsHandler(p instruments.GetMetricsParams) middleware.Responder {
    // some logic here
    return instruments.NewGetMetricsOK().
        WithPayload(models.Any(
        // response data here
    ))
}

So is it even possible to use standard handlers? Or should I rewrite all of them in go-swagger-way?

Don't worry, there is no need to do extra work.

Adding a wrapper

There is a way to do that by adding a tiny wrapper. We just need to find a reader and writer and put them to our http.HandlerFunc function. The reader is always in the generated request structure, but the reader is provided to the generated responder object. So we have to implement a middleware.Responder interface to get it. That may look like this:

type CustomResponder func(http.ResponseWriter, runtime.Producer)

func (c CustomResponder) WriteResponse(w http.ResponseWriter, p runtime.Producer) {
    c(w, p)
}

And now we just need to wrap our standard handler into this wrapper and provide reader and writer as usual!

func MetricsHandler(p instruments.GetMetricsParams) middleware.Responder {
    return CustomResponder(func(w http.ResponseWriter, _ runtime.Producer) {
        promhttp.Handler().ServeHTTP(w, p.HTTPRequest)
    })
}

Improvements

We can go deeper and add constructor function, which will save us a couple of lines in future:

func NewCustomResponder(r *http.Request, h http.Handler) middleware.Responder {
    return CustomResponder(func(w http.ResponseWriter, _ runtime.Producer) {
        h.ServeHTTP(w, r)
    })
}

and simplify our handler a bit:

func MetricsHandler(p instruments.GetMetricsParams) middleware.Responder {
    return NewCustomResponder(p.HTTPRequest, promhttp.Handler())
}

Sure it may look not so impressive as a distributed database internals, but it will save you some time and help to integrate a go-swagger with ease.

See you in the next "Go-Swagger Tricks" episode!

How I write my unit tests in Go quickly

Ilya Kaznacheev — Thu, 30 Apr 2020 23:47:40 +0000

Photo by Veri Ivanova

We all love unit tests because they help us to keep our software workable. And we all hate them because they don't appear magically - someone needs to write them. And when it comes to writing, it often takes a huge amount of time to cover the simplest cases.

But I found my way to do that without pain (okay, with less pain). And I will share it with you like a simple illustrated guide.

Separate layers

The principle isn't new, but it is still useful. Nowadays it comes with different names - hexagonal architecture, onion architecture, separation of concerns, etc. The main idea is that different parts (and I mean logically different parts) of your application should be separated into independent parts.

It's very important because you can't just test the whole app you're building. Ok, technically you can. But it will require an enormous amount of time and it will be a nightmare from a long-term perspective.

Instead, make it as independent as possible. But the app or at least the microservice cannot be independent of himself! It can, and we call it dependency injection. And it never was so easy as now in Go because we have...

Duck typing interfaces

That means that if some type has the methods described in the interface, we can use it via that interface. So there is no need to do a lot of paperwork at the beginning of the project and draw a huge UMLs with all possible interactions and relationships - just write an interface for your layer with its minimal requirements and pass the dependencies into it.

Example: you have a business-logic package that has to save some data into a database. No need to create or get a database connection somewhere in this package - just define a Repository or a Storage interface (or any more suitable name) and put there all actions you want to perform with the database - save, update, read, delete, increment a counter, etc. Then you just have to put the database object, that can perform that actions - it will contain the exact database query language code and database-specific logic. You also will handle the database connection and possible connection errors outside of the business-logic layer. The layer will become independent of the database layer.

Inject unmanaged resources

There are many unmanaged resources like random number generators, time, random-based hashes an so on. They don't require outside injection because they have no outside dependencies. But they still can do unpredictable impacts on test results. So instead of trying to work around them in tests, just use the same approach - inject them as an isolated dependency. Bus since they are not provided from the outside, do that from the inside!

Example:

type Example struct {}

func NewExample() *Example {
    ...
    return &Example{}
}

func (e *Example) TimeToGo() string {
    now := time.Now()
    return fmt.Sprintf("its time to go! %s", now.String())
}

Here you can't predict the TimeToGo method response in the test - each time time.Now() will return a new value. But you can take charge of it:

type Example struct {
    now func() time.Time
}

func NewExample() *Example {
    ...
    return &Example{time.Now}
}

func (e *Example) TimeToGo() string {
    return fmt.Sprintf("its time to go! %s", e.now().String())
}

Will work as before, but time.Now is now under your control! You can easily mock it in the test:

nowMock := func() time.Time {
    t, _ := time.Parse(time.Kitchen, time.Kitchen)
    return t
}

e := Example{nowMock}

if e.TimeToGo() != "its time to go! 0000-01-01 15:04:00 +0000 UTC" {
    t.Errorf("test failed")
}

So try to avoid any dependency in your logical entity. Even such a small as a current time or a random value. It's ok to hide them under interfaces or function types.

80/20

100% coverage is a dream of any Open-Source developer. It's so nice to look at a green badge in your project's readme! But within productive projects, things work a different way.

Normally you just don't have enough time or resources to do a 100% test coverage. End even if you'll do so, during the active development phase you will change the logic so many times, so the number of test changes will be enormous.

But in fact, the 80/20 principle works here too: the 20% coverage of the "hottest" or most important code will cover 80% of the app's usage and dataflow. That means, start with the "hottest" code coverage. If you will have time and motivation you will write tests for less important parts and slowly will increase the coverage.

For example, if you're building a web search service, test the search engine first. If it really works as expected, you can cover autocomplete, translation, and live preview next. But without the reliable core feature you will fail.

Don't write your code

Now you have a well isolated and pretty important part of your application ready. You need to fit your test into that frame. And thankfully your app consumes interfaces instead of exact implementations. That means, we can mock them in tests!

But writing mocks is a salt mine job. We don't have to rob a robot's job - let them do what they are designed for!

So I don't write test mocks, I generate them instead. To do that I use Mockery.

vektra / mockery

A mock code autogenerator for Go

Say we have a database interface:

package repo

type Storage interface {
    GetOrder(id string) (*Order, error)
    CreateOrder(order Order) error
    DeleteOrder(id string) error
}

And a code that uses it:

type AccountManager struct {
    storage repo.Storage
}

We can generate the mock simply by calling the following command:

# if the interface is in the internal/repo package
mockery -name=Storage -dir=internal/repo

And it will generate a perfect fitting mock for your interface! It will be stored in the /mocks directory regarding the interface package directory (you can change it by specifying -output parameter). Then you only need to use it in your test:

var testOrder repo.Order
//...
storageMock := new(mocks.Storage)
storageMock.On("GetOrder", "12345").
    Return(&testOrder, nil)

// and then inject it into your code under the test
am := AccountManager{storageMock}

// execute test cases with mocked dependency ...

I normally use either the go-generate to run the mock generation:

//go:generate mockery -name=Storage

type Storage interface { ... }

And then run go generate ./.... Or I just put them as a list in the Makefile with an absolute path. Or you can even generate mocks for all exported interfaces in your directory recursively. See the library details for more information.

Use shortcuts

The test case writing is hard and boring work. You have to prepare a lot of data samples, do a boring work of setting the environment up, preparing infrastructure like HTTP requests and response writers, mock servers, stub data, etc. And it's really tedious to write every got and want data check and corresponding errors.

But you can save some time by using shortcuts. And with shortcuts, I mean test libraries as Testify to wrap repetitive parts of tests in clean and handy helpers!

stretchr / testify

A toolkit with common assertions and mocks that plays nicely with the standard library

Let's say you have a common HTTP response with something like this:

w := httptest.NewRecorder()
//... function under test call ...

// assertion
gotBody, _ := ioutil.ReadAll(rr.Body)
gotStatus := rr.Result().StatusCode

if string(gotBody) != tt.bodyWant {
    t.Errorf("wrong response body %s want %s", string(gotBody), tt.bodyWant)
}

if gotStatus != tt.statusWant {
    t.Errorf("wrong respose status %d want %d", gotStatus, tt.statusWant)
}

Simple, huh? But when you have to do that 100+ times it becomes a little bit boring... You don't have to wait 'till a boreout! Just use shortcuts:

import "github.com/stretchr/testify/assert"

w := httptest.NewRecorder()
//... function under test call ...

// assertion
gotBody, _ := ioutil.ReadAll(rr.Body)
gotStatus := rr.Result().StatusCode

assert.Equal(t, string(gotBody), tt.bodyWant, "wrong response body")
assert.Equal(t, gotStatus, tt.statusWant,. "wrong response status")

It may be used in more interesting ways. Say, you have a method that returns a pointer and an error (a common pattern in Go). So you don't want to assert a returned value in case of error because it will cause a nil pointer to dereference. So you have to build a messy condition with a nil check and error check and so on... You don't have to:

want := "we want to get this exact value"

got, err := GiveMeMyData(input)

// if error is nill, NotNil will return true
// otherwise it will write an error message to testing.T
if assert.NotNil(t, err, "unexpected function error") {
    // here we assert the expected value
    // but only if the error is nil!
    assert.Equal(t, got, want, "unexpected function output)
}

You can make in even shorter by passing testing.T to the assert structure:

assert := assert.New(t)

/// now you don't have to pass t to each function, 500 milliseconds saved!
assert.Equal(123, 123, "they should be equal")

It may look simple, but trust me, it will give you enough saved time during unit testing to drink a cup of coffee. Even to brew a coffee!

Make test meaningful

I like tests, but when it comes to testing output, it getting harder to keep on track what's going on and which test was failed and why. It's ok when you have 10 or 20 test cases, but if you have hundreds of tests (or at least test sets, if you're using table tests for example) it's really hard to understand what's wrong just looking on the test output.

To make it more readable you have to give a proper description. But do you really want your test cases to be looking like this?

if got != want {
    t.Errorf("by calling a function A when there is no data in the DB table a_data and at the same time there is no incoming messages from the MQ and no idle workers in the worker pool, it returns %s but we want %s", got, want)
}

I hope you not. Otherwise please stop reading. Because this is my favorite part.

To achieve readability in test output and make it easy to navigate you can use BDD approach to testing. I use it for many reasons:

it helps to structure the test as a sequence of steps or a story;
it's nice to read in test output because it is a complete story of your test fail;
you can build your test cases as a tree from a root common input to different results;
it's possible to go outside of unit tests and write a multi-step (or not) test for a real business story. So instead of testing abstract parts of your app, you can go ahead and cover a real business-process as works in real life! It's also a good way to focus on really necessary tests first because you can open a spec and write a test for it!

So let's take a closer look. For this kind of testing I use ginkgo library. It is paired with a gomega test matcher library, with a lot of test helpers and wrappers (more shortcuts!).

onsi / ginkgo

A Modern Testing Framework for Go

I'm not a fan of BDD itself, but I really like to use this approach in tests. So let me show you a couple of examples.

You need to set up a test suite. Normally I do that once in a package.

The example above may look like this:

import (
    . "github.com/onsi/ginkgo"
    . "github.com/onsi/gomega"
)

var _ = Describe("Function A", func() {
    When("I call the function", func() {
        Context("and there is no data in the DB table a_data", func() {
            Context("and there is no incoming messages from the MQ", func() {
                Context("and no idle workers in the worker pool", func() {
                    It("should return 123", func() {
                        got, _ := A("123")
                        Expect(got).To(Equal("123"))
                    })
                    It("should work without errors", func() {
                        _, err := A("123")
                        Expect(err).To(BeNil())
                    })
                })
            })
        })
    })
})

Looks more like a real spec, right?

And the error output will look like this:

You can also add some data into any step, and it will be added each time your test will reach this node of the test tree.

A pseudocode example:

Describe some test entity
├─With data X
│ ├─And with data B
│ │ └─And with data C
│ │   ├─It should do this
│ │   └─And this
│ └─But with data D
│     └─It should do this
└─With data Y
  ├─And with data E
  │   └─It should do this
  └─And with data E*
      └─It should error

So here is a whole story, and you can describe it using ginkgo! In this case, for example, the node And with data B will be executed 3 times:

With data X -> And with data B -> And with data C -> It should do this
With data X -> And with data B -> And with data C -> And this
With data X -> And with data B -> But with data D -> It should do this

You can use BeforeEach() to set up some context for each step (set some variables, functions, fill mocks, and so on). And also you can use AftrrEach() to cleanup at the end of the node.

The library has a lot of other useful functions - BeforeSuite and AfterSuite and many variations to help you organize your test better.

So the main idea here is to use it both as a meaningful description of the process under the test, and the test sequence and context of each step.

Putting it all together

So how to do it quickly? Let's summarize:

separate layers by means of DI;
use interfaces to do that;
start with hottest 20% of your code;
mock these interfaces with generated mocks;
use shortcuts to speed-up simple tests;
use the BDD test suite to describe and organize complex tests.

In the end, it may look like this:

import (
    "errors"
    "temp"
    "temp/mocks"

    . "github.com/onsi/ginkgo"
    . "github.com/onsi/gomega"
    "github.com/stretchr/testify/mock"
)

var _ = Describe("Function A", func() {
    var (
        storageMock *mocks.Storage
        testOrder   temp.Order
        testErr     error
    )

    BeforeEach(func() {
        storageMock = new(mocks.Storage)
        testOrder = temp.Order("my test order")
        testErr = errors.New("test error")
    })

    When("we want to get some order", func() {
        Context("and the order exists", func() {
            BeforeEach(func() {
                storageMock.On("GetOrder", "12345").
                    Return(&testOrder, nil)
            })

            It("should return my test order without error", func() {
                sk := temp.NewStorekeeper(storageMock)
                got, err := sk.GetMyOrder("12345")

                Expect(*got).To(BeEquivalentTo("my test order"))
                Expect(err).To(BeNil())
            })
        })

        Context("and there is no order", func() {
            BeforeEach(func() {
                storageMock.On("GetOrder", mock.Anything).
                    Return(nil, testErr)
            })

            It("should return empty order with error", func() {
                sk := temp.NewStorekeeper(storageMock)
                got, err := sk.GetMyOrder("12345")

                Expect(got).To(BeNil())
                Expect(err).To(HaveOccurred())
            })
        })
    })
})

Pretty quick to write. And so easy read in CI logs!

Clean Configuration Management in Golang

Ilya Kaznacheev — Sun, 15 Dec 2019 14:05:48 +0000

image by alexchoffy

There are many good approaches to handle configuration in a modern application. Now we use such things as configuration files, environment variables, command-line parameters, as well as CI configuration patterns and on-fly config file builds, remote config servers, specific mapping, and binding services and even more complex things.

But the target is the same - provide the app with a configuration, which is fast to get and easy to use. But how to do that?

How our applications consume configurations

As a nice developer, I always fight with implicitly. I don’t use global variables, I structure my code the readable way and may my code flow certain and clear. There is something I really don’t like - environment variables because they are the same bad as global variables. But you can handle them the right way too if you will read them once at the beginning.

So the main idea is to get the application settings from where they are (file, env, remote server, etc.) and distribute them through the app, to make them available wherever they are required. Simple task, huh? But there are still so many different ways to do so.

Many approaches, even more tools

So it’s pretty clear how to get configs. Each approach has it’s own best practices. But what to do next? There is no common answer.

Some libraries (like viper store configs as a key-value set in a global area. Pretty useful, but very implicit. Within big projects sometimes you can just lose the way how one or another variable came to this state.

Other libraries put everything in one bucket - environment, files, command-line options. Even there is a nice-looking structure with all config values, that still looks unclear and useless to me. Normally, the app uses either a command-line approach (CLI tools) or config file + environment approach (web services, containerized apps, etc.).

There are a lot of really good libraries for command-line argument handling like go-flags, and I don’t think that’s there is a reason to put everything in one single tool.

But both approaches have the same issue - a lot of magic inside.

Magic

Go isn’t a language for everyone. It has a lot of boilerplate, the type system isn’t so much forgivable, and OOP possibilities are very limited. But there is a nice thing that worth it - it is instantly readable, it has no magic inside. And I like that.

But work with configs is often tricky, in turn. Even if you define a structure instead of using an unsafe map, you still have to declare which environment variables do you use. There is no worst thing as globally used environment variables. But even if the structure contains the explicit mapping of each environment variable, it is still hard to set up the environment outside the app. Someone (it may be you, another developer, DevOps, or just someone forced to do that) needs to find a list of environment variables used, and then use them properly.

Okay, they can be listed in the documentation or in the source code. But both are bad. Documentation can be outdated, and source code needs time and skill to read.

In my view, the only proper way to do that - add a full list of environment variables into a help output, including descriptions, default values and other meaningful information.

I haven’t found anything not overcomplicated but still useful, with explicit configuration setup and clean and informative help output, so I’ve built it on my own.

One way to rule them all the clean way

So, I’ve analyzed many popular configuration management libraries and decided to build my own. Why? There a couple of reasons:

handle config files and environment variables, not command-line parameters;
no magic, explicit way to read and use configuration;
no implicit names, all mapping should be done using tags;
the nice and informative help output;
easy integration with other libraries.

And here we are: cleanenv library is now released and proven in production.

Let’s talk about its major features.

Explicit configuration structure

The main idea behind the cleanenv library is to make everything explicit and clean. No global state, no encapsulated maps, no magical background updates nether any other hidden work.

Everything goes clean and visible. And that’s why a structured format was chosen as a base for configuration. The structure (as deep and complex as you want) being used to parse configuration files and environment variables. It’s also a good place to write config documentation.

Explicit naming and options

The next “no-magic” step is the explicitness of variable names. There are no such things like complex environment name generators based on nested structure names - you have to set names as-is, so you can easily find them using a search.

For file parsing, we are using the same approach as corresponding libraries - JSON, YAML, TOML, etc.

Here is an example of a simple server configuration structure:

type ConfigDatabase struct {
    Port     string `yml:"port" env:"PORT" env-default:"5432"`
    Host     string `yml:"host" env:"HOST" env-default:"localhost"`
    Name     string `yml:"name" env:"NAME" env-default:"postgres"`
    User     string `yml:"user" env:"USER" env-default:"user"`
    Password string `yml:"password" env:"PASSWORD"`
}

This structure can be used to parse YAML configuration file, and then read some data from the environment. First, it will read the file, and then try to find environment variables using names in env tags. If there were no data found in file neither in the environment, the constant from the env-default will be used instead.

Supported data types are:

integers;
floating-point numbers;
strings;
booleans;
arrays (with customizable separator);
maps (with customizable separator);

Readable help output

Modern containerizable applications use environment variables as the main configuration. So, the app can have up to hundreds of variables it depends on. The common problem is that the exact list of variables is often uncertain or outdated (or even worth, they are distributed through the app and being read in some unexpectable places).

To fix that the cleanenv library contains the possibility to add a well-structured list of environment variables with descriptions into help output:

import github.com/ilyakaznacheev/cleanenv

type ConfigServer struct {
    Port     string `env:"PORT" env-description:"server port"`
    Host     string `env:"HOST" env-description:"server host"`
}

var cfg ConfigRemote

help, err := cleanenv.GetDescription(&cfg, nil)
if err != nil {
    ...
}

You will get the following:

Environment variables:
  PORT  server port
  HOST  server host

It will help you to get the documentation synchronized with your app with no need to have extra files.

Integration

Simplicity is everything. But I want not only to give a simple tool but also to make it easy to use with other codes.

Now you can easily combine it with flag library help function:

type config struct {
    Port     string `env:"PORT" env-description:"server port" env-default:"5432"`
    Host     string `env:"HOST" env-description:"server host" env-default:"localhost"`
    Name     string `env:"NAME" env-description:"server name" env-default:"postgres"`
    User     string `env:"USER" env-description:"server username" env-default:"user"`
    Password string `env:"PASSWORD" env-description:"server password"`
}

var (
    cfg     config
    cfgPath string
)

fset := flag.NewFlagSet("My app", flag.ContinueOnError)
fset.StringVar(&cfgPath, "cfg", "", "path to config file")

fset.Usage = cleanenv.FUsage(fset.Output(), &cfg, nil, fset.Usage)

fset.Parse(os.Args[1:])

If you will run go run your_app.go -h, the output will be:

& go run your_app.go -h
Usage of My app:
  -cfg string
        path to config file

Environment variables:
  PORT string
        server port (default "5432")
  HOST string
        server host (default "localhost")
  NAME string
        server name (default "postgres")
  USER string
        server username (default "user")
  PASSWORD string
        server password

Possibility to do more

That’s nice, but you may need more. For example, you may want to get configuration from a remote server, or some other tool, or update them. To do so, you can use the enhancement possibilities of the library. There are some ways to write your own logic to read the data from your own source. Read more in documentation.

Conclustion

So the cleanenv library is not the key for every lock. It is definitely not a tool-for-everything. But it is designed to do a simple, clean and readable, but flexible enough if you need so.

ilyakaznacheev / cleanenv

✨Clean and minimalistic environment configuration reader for Golang

So, I’m happy if it will help you. I’m actively using it in my projects, so it is productive-proved.

Also, feel free to request any features you think may be helpful.

And stay clean!

What's wrong with godoc?

Ilya Kaznacheev — Sun, 21 Jul 2019 05:34:36 +0000

this is a repost from my blog at kaznacheev.me. I welcome new readers and will be happy to discuss what I'm writing about

Like many modern languages, Golang has built-in inline documentation support tool called godoc.

To be honest, it’s awesome. It is a really great tool, that has a real impact on the everyday coding process. At least if you use plugins with function call tips as I do.

But there is the big problem of the majority Go-projects that is tightly related to godoc but lays outside of it.

The godoc is really cool. You can document your methods and packages, paste a code snippet, that will have to highlight on the web page. Some IDE plugins will give you signature tips based on function documentation, e.g. description of parameters.

Also, some code analyzers like go vet will warn you if you miss a package header or global type, function or variable description comment. It disciplines you and makes sources more readable.

Only pros, huh? But there is a big con, that isn’t a fault of godoc itself, but many developers expect it to solve this problem.

The main idea escapes the attention

The major disadvantage of godoc is its declarative nature. You can write excellent documentation with code snippets and parameter description. You can write simple use-cases for your methods, you can give a rich description of each global constant and variable. You can measure your documentation coverage and strive to beat 100%. Finally, you can write a big article in the package header as many open-source projects do, and you will have great documentation.

But it will be a book. An encyclopedia. Not the manual. Not the how-to guide.

There are many examples of encyclopedia-like documentation, even at big and popular projects – take a look at Docker or Kubernetes, for example. They both have really well-documented features. Each aspect has its own page on the documentation portal. They even have some examples with code snippets. But if you really want to use them, if you will try to understand some corner-cases or cross-module interaction, you will rather go to SO, then search through the docks. And that’s why.

You don’t need a book. You need a guide

Last two decades the way of programming has changed radically. Instead of “hackermen” in sweaters, we have now stylish smoothie-boys, which look better than their CEOs on the investment pitch. Instead of memory leakage and Y2K problems, we argue which front-end framework is better, and performance is now a taboo subject (try to ask about performance at some front-end meetup). Modern tools and languages do the majority of the low-level stuff by their own and we can focus on more abstract tasks without the need to understand how each bit flows through the system.

Sure, it’s better to know the low-level stuff. It’s much better to understand, how the compiler processes your code and when it can optimize the runtime. Nice to know the technical limitations of the tool you use, to understand what to expect.

But after all, you need to invest an enormous amount of time to learn all that stuff. So much, that it probably will become obsolete before you will become a master in it. And that even can’t give you a guarantee, that a deep understanding of the tool basics will reward you as much, as much time you have spent to learn it.

As an example, I use Docker in my applications a lot. I understand the basics of containerization and virtualization, but to be honest, I don’t know what exactly goes on in the black box. Will I make better containers if I will meticulously read all the docks and learn each technical aspect, lying under the containerization principle? Sure. Will I make better containers if I will investigate the sources and try to understand or even debug each method. Hmm... Not sure. Probably I will become an expert in the Docker architecture and some interesting solutions, but that will not make you 10x more powerful in the container configuration. You will spend so much time to get almost nothing related to your actual tasks.

That leads us to the following situation: you don’t need to know how the tool functions, but you need to know how to use it in your daily work. That means, that people will be interesting rather in real use-cases and specific problem solutions than a method documentation and option description. For many tools, Stack Overflow is much useful than the official documentation.

In fact, you don’t need book-like documentation. It’s nice to have when you want to check some parameters or unclear behavior. But it shouldn’t be in the first place. What you really need is use-cases. The real examples of the app or tool usage from the first steps.

I’ve seen many great tools, which are really difficult to use because you simply don’t understand where to start. They have a perfect description of each function or class, but you just missing the main idea. It’s like to talk with a person with schizophasia – you understand the words but can’t get the idea of the speech.

The problem of godoc is that there is no place for such use-case-oriented documentation. You can’t just put a ton of examples into a package comment. At least, you can’t structure them. You need something that will give you a documentation hierarchy different from your package content tree. But the solution exists.

How to make your docs useful

When a child learns to read, he doesn’t start with a class book. He starts with fairy tales. He reads funny stories about brave characters, but not definitions of the words. That’s why children can understand us even if they don’t know all the words we use.

To let your users start with ease, don't give them an encyclopedia. Give them a story. Guide them from the beginning to the end. Or at least tell them where to start.

Unfortunately, it isn’t so much possible with the godoc. If your project is small enough, you can do that in your README or in the package descriptions, but usually, it is not the case.

So, what you can do?

Explicit entry point

Often when you find a new package, you don’t want to do research. You ask yourself: “ok, does it fit my requirements, or I just need to search further?”. Don’t force the user to read all the docs. Help him to start using your tool right now.

Can you run the simplest option with only two lines of code? Great, let’s start with that and then add new functions one-by-one.

Can you run a bare module with different parameters? No more words! Let’s do that and then give the user more and more parameters with a description of how they can help to fulfill his needs.

This approach is very useful. In one hand, you will help newcomers to begin using your product with ease. On the other hand, you will gather a bigger and more involved community.

Real use-cases and tips

As I mentioned above, in many cases you don’t need a detailed description of the tool to solve your problem. You need just a solution to your cases. Sometimes you even not interested in the tool or library description at all, you just want it to work.

And that’s what people will be looking for in your documentation. They don’t need a detailed description of every function and every parameter, they need a way to bring it to life from scratch.

So just give them what they need. Describe the main idea of your package and the use-cases. Write from the point of view of usage, not functionality. Let the user read it as a story, chapter by chapter, but avoid formal encyclopedia structure.

To do all of this you need something more flexible, then a README file.

Hierarchical documentation

To navigate resulting “story of tool X usage” with ease you need a flexible structure capable to serve both as a table of contents and as a roadmap. Here you need something like hierarchical navigation/content tree. Maybe there are other good solutions, but I find a hierarchical view most suitable for this task. There are several ways to do that without extra work:

Built-in Wiki on GitHub, Bitbucket or same hosting – they are easy to use, support markdown and requires nothing but a repository that you already have;
Read the Docs - this service is a de-facto standard for documentation. It supports versioning, so you can have documentation for each version in parallel, that can be very helpful if you want to support previous versions;
Directory with documents - you can simply create a directory with a set of markdown documents and add a table of contents in you README.md or same document in the root directory. The same can be done in a single document using anchors – choice of style is up to you. Check this as a nice example.

There are many other options, you can even use your godoc to organize something like this, but I think, better to use each tool for its main purpose.

So, what about godoc?

After all, godoc is still a very cool tool to write functional documentation. I really respect people who write rich and clear code documentation using built-in tools like this.

Don’t forget to use it. Start your documentation with the godoc and then write more complex and more usage-oriented user manuals. As a really nice example, I can give a go-pg library, that has both good inline documentation and also a Wiki with examples and use-cases.

It may be difficult sometimes to write a good manual for a package, so start with simple Q&A page, and fill it from SO and other recourses. There you will see, what causes difficulties for users and what could be misunderstood. Do this work iteratively and listen to your community, and you will be good.

A clean way to pass configs in a Go application

Ilya Kaznacheev — Fri, 12 Jul 2019 22:37:48 +0000

this is a repost from my blog at kaznacheev.me. I welcome new readers and will be happy to discuss what I'm writing about

There are many concurrent approaches to organize application configuration nowadays.

Classic .ini files, .json, .toml, .yaml, configs, modern .env and, of course, container environment. And don’t forget about CLI arguments! Am I missing something?

Let me be honest, I really dislike any implicitness in interfaces. Same for the CLI of course. Any of your interfaces, whether public or internal, API or object interface, a class method or module facade – they have to cooperate fair. The contract between you and the other side should be explicit, rightful and without any notes in small text at the bottom of the page.

That means, that they have to take exactly what they are asking you for and give you exactly what they promise. They shouldn’t take anything from your pocket when you turn away. They shouldn’t send your stuff after a week or two if you agreed on an immediate transaction. They can hold the inner state. Not all of them should be immutable. But you always should be able to interact clearly and fair, that means all interactions have to be visible and legitimate, i.e. you will be able to pass the data from hand to hand. Otherwise, you will lose the contract details and sooner or later you will be fooled.

Any interface also has to have a single entry point, that means you pass the whole amount of data that the interface requires at once, by passing it explicitly. It can be any type of data – values, structured or serialized data, path to file to parse or a socket to connect – the main idea is to put the data at the interface call.

A way to clean configuration

So the modern trend to move any meaningful settings into environment arguments according to twelve-factor app looks ugly to me. No, the ideas of the twelve-factor app are sweet. But the fact that I’m compelled to pass my configuration implicitly with environment variable setup (i.e. with global variables) makes me sad.

So sad, that at the beginning I’ve put all my configuration at the single config file and made a separate config file for each environment (e.g. local, stage, test, prod, etc.). Normally config files are well structured, so I can group parameters depending on their use. But it was not so useful, because I needed to store several configs in the repo for each environment and update each one after configuration structure change.

Then I’ve moved to combine config with CLI arguments, because I have to pass some data directly to the app, e.g. secrets, local paths, and other variable data and the data that I can’t keep in the config file. It was better because now I can pass some values that vary in different environments as a parameter and use the same configuration file for each system. And it worked pretty well, until...

Containers come. And the common way to setup a container environment is to use environment variables. It’s possible, in theory, to store config files in secrets, but it isn’t useful at all.

The main disadvantage of environment variables is their implicit nature. You can’t just look at the application help output, or check a configuration file structure to see a variable set and structure. When your application config is based on environment variables, you should have documentation for them, and you’re getting the burden of its update and synchronization. Otherwise, the user will not be allowed to get config information, and his experience with the app will be really terrible.

It’s also hard to debug such apps. As globals, environment variables act implicitly and can affect the program behavior at any moment. It’s not a big deal when your code is small enough to fit into your memory, but when it grows, you will struggle to try to remember where you get one or another config value.

But after all I have found approach, that looks workable to me and satisfies my requirements. It combines several techniques, so let me make a quick overview.

Configuration file

First, we still use a configuration file. I use it for several purposes:

declare configuration structure;
keep defaults for each variable;
document variables or sections;
provide a config example for the app users;

I prefer to store my configs in YAML files. I don’t want to start a holy war, but just will declare why is it the best for me:

hierarchical structure - I can be as flexible in a variable organization as I want;
clean markup - I don’t need any brackets or a conglomeration of symbols to speak with the parser;
comments - I can provide a detailed documentation, options, limitations, examples, best practices, etc.;
rich semantic - really advanced techniques like anchors, aliases, extensions, embedding, and others. I don’t use them really often, but sometimes they are extremely helpful;

Let’s say we have a simple config file like this:



# Server configurations
server:
  host: "localhost"
  port: 8000

# Database credentials
database:
  user: "admin"
  pass: "super-pedro-1980"

To use data from a .yml file in Go you need to unmarshal it into the structure like you do for JSON.

The mapping looks similar to JSON:



type Config struct {
    Server struct {
        Port string `yaml:"port"`
        Host string `yaml:"host"`
    } `yaml:"server"`
    Database struct {
        Username string `yaml:"user"`
        Password string `yaml:"pass"`
    } `yaml:"database"`
}

I use gopkg.in/yaml.v2 library by Cannonical to parse YAML files.

You can use yml.Unmarshal to parse a byte slice, but In most cases, you will work with some data provided as io.Reader implementation, so I/m using decoder that reads byte stream instead of full data stored in the memory:



f, err := os.Open("config.yml")
if err != nil {
    processError(err)
}
defer f.Close()

var cfg Config
decoder := yaml.NewDecoder(f)
err = decoder.Decode(&cfg)
if err != nil {
    processError(err)
}

And that’s it. Just like any JSON file. Now you can go ahead and write your own well-documented and well-structured config file, that will also serve as an example in your repository.

Environment variables

Now let’s talk about the environment variables. Usually, they scattered through the app. But there is another approach. In Go, you can assign environment variables to structure fields as well you do that for JSON, YAML, and others. It will look like this:



type Config struct {
    Server struct {
        Port string `envconfig:"SERVER_PORT"`
        Host string `envconfig:"SERVER_HOST"`
    }
    Database struct {
        Username string `envconfig:"DB_USERNAME"`
        Password string `envconfig:"DB_PASSWORD"`
    }
}

The magic is in the github.com/kelseyhightower/envconfig library.

In a couple of lines, it retrieves environment variables and assigns them to structure fields you have defined:



var cfg Config
err := envconfig.Process("", &cfg)
if err != nil {
    processError(err)
}

And that’s it. Now you have all your environment variables in the same place. No need to browse through all the repo to find where you use this variable, you can simply track down the config structure, that has a single entry-point.

Also, now you have all your environment variables declared in the same place. You can just open the config structure and see a full list of envs that the app needs. The lib provides a set of Usage functions with wide output possibilities, that will allow you to add the list of env. variables to help output or wherever you want. Your application’s user will appreciate that.

Now you can use your favorite way to provide the environment variables - .env file, container settings, makefile or just a shell script. It’s up to you.

All together

So now let's mix them!

The same structure will look like this:



type Config struct {
    Server struct {
        Port string `yaml:"port", envconfig:"SERVER_PORT"`
        Host string `yaml:"host", envconfig:"SERVER_HOST"`
    } `yaml:"server"`
    Database struct {
        Username string `yaml:"user", envconfig:"DB_USERNAME"`
        Password string `yaml:"pass", envconfig:"DB_PASSWORD"`
    } `yaml:"database"`
}

So first I load the data from the YAML file. It also serves me as a set of default values.

Then I load envs and overwrite filled fields. That is, you don’t need to take care of missing values, they will be filled from the config file.



func main() {
    var cfg Config
    readFile(&cfg)
    readEnv(&cfg)
    fmt.Printf("%+v", cfg)
}

func processError(err error) {
    fmt.Println(err)
    os.Exit(2)
}

func readFile(cfg *Config) {
    f, err := os.Open("config.yml")
    if err != nil {
        processError(err)
    }
    defer f.Close()

    decoder := yaml.NewDecoder(f)
    err = decoder.Decode(cfg)
    if err != nil {
        processError(err)
    }
} 

func readEnv(cfg *Config) { 
    err := envconfig.Process("", cfg) 
    if err != nil { 
        processError(err)
    }
}

So, there are many advantages to this approach:

single entry point - easy to find where each config variable came from;
simple declaration via tags;
structured configuration - you can group config sections depending on their usage;
declare defaults in a config file, overwrite what you need in each environment;
explicit list of environment variables used in the app;

I’m not sure, that this approach covers any possible usage scenario, but it’s pretty useful and most important explicit.

UPD: I've added all discussed technics into a small and simple configuration package called CleanEnv.

ilyakaznacheev / cleanenv

✨Clean and minimalistic environment configuration reader for Golang

Clean Env

Minimalistic configuration reader

Overview

This is a simple configuration reading tool. It just does the following:

reads and parses configuration structure from the file
reads and overwrites configuration structure from environment variables
writes a detailed variable list to help output

Content

Installation

To install the package run

go get -u github.com/ilyakaznacheev/cleanenv

Usage

The package is oriented to be simple in use and explicitness.

The main idea is to use a structured configuration variable instead of any sort of dynamic set of configuration fields like some libraries does, to avoid unnecessary type conversions and move the configuration through the program as a simple structure, not as an object with complex behavior.

There are just several actions you…

View on GitHub