DEV Community: Chen

Understanding Go’s Supercharged Map in v1.24

Chen — Wed, 05 Mar 2025 20:30:31 +0000

Go 1.24 introduces a new map implementation, inspired by Google's Swiss Tables, which brings significant optimizations and performance enhancements to the language's built-in map type. While Go's previous map implementation was already efficient, the new design takes it a step further by introducing a clever approach to data organization and access.

To help digest this complex change, let's use an analogy that illustrates how the new map works and how it differs from the previous implementation.
Let's use a relatable analogy: a library. Just as a library organizes books in a way that makes them easy to find and access, a map organizes data for efficient retrieval.

This analogy will provide a high-level understanding of the key improvements without delving too deeply into technical details. For those interested in a more in-depth exploration, we'll reference additional resources throughout the explanation.

The Library Analogy

Think of Go’s map as a library designed to store books. Here’s how it works:

1. Tables Are Library Sections

The map starts with one table, which is like a section of the library. If this section gets too crowded, the library adds another section. Each table is divided into smaller units called groups.

2. Groups Are Bookshelves

Each table is made up of multiple groups, which are like bookshelves in the library. A group can hold up to 8 books (key-value pairs). These groups are the fundamental storage units in Go maps.

3. Control Word: The Librarian's Cheat Sheet

Each bookshelf has a label that summarizes key information taped to it, called the control word. This cheat sheet contains metadata about the books on that shelf:

It stores tiny "fingerprints" of each book's ID (derived from its hash).
It marks whether slots on the shelf are empty, occupied, or deleted.

This cheat sheet helps librarians quickly locate books without flipping through every slot.

It can be pictured as follows:

+---------------------+        +---------------------+
|         Map         |        |       Library       |
+---------------------+        +---------------------+
|      Table 0        |        |      Section 1      |
+---------------------+        +---------------------+
| Control Word (64b)  | <----> |     Shelf label     |
+---------------------+        +---------------------+
| Key 0  |  Value 0   |        | Book 1 | Location 1 |
| Key 1  |  Value 1   |        | Book 2 | Location 2 |
|        ...          |        |        ...          |
| Key 7  |  Value 7   |        | Book 8 | Location 8 |
+---------------------+        +---------------------+
| Control Word (64b)  | <----> |     Shelf label     |
+---------------------+        +---------------------+
| Key 0  |  Value 0   |        | Book 1 | Location 1 |
| Key 1  |  Value 1   |        | Book 2 | Location 2 |
|        ...          |        |        ...          |
| Key 7  |  Value 7   |        | Book 8 | Location 8 |
+---------------------+        +---------------------+
|        ...          |        |        ...          |
+---------------------+        +---------------------+
|      Table 1        |        |      Section 2      |
+---------------------+        +---------------------+
|        ...          |        |        ...          |

How It Works: Storing and Retrieving Books

Let’s walk through an example of storing and retrieving a book in this library.

Storing a Book

Say you want to store "The Great Gatsby" by F. Scott Fitzgerald in the map.

Generate a Hash
The librarian generates a unique ID for "The Great Gatsby" using a hash function, e.g., 0xf83c6f3a3c.
Find the Section and Bookshelf
The hash is split into two parts:
- H1: Determines which section (table) and bookshelf (group) the book belongs to. (57 bits)
- H2: A small fingerprint stored in the control word for quick identification. (7 bits)

For example:

H1 says: "Go to Section 1, Bookshelf 3."
H2 says: "Fingerprint is 3c."

Place the Book The librarian places "The Great Gatsby" into an available slot on Bookshelf 3 and updates the control word with 3c.

Retrieving a Book

Now you want to retrieve the book from the map.

Find the Section and Bookshelf
The librarian uses H1 from the hash of "The Great Gatsby" to go directly to Section 1, Bookshelf 3.
Check the Cheat Sheet (Control Word)
The librarian looks at the control word (3c) to see if any slots match "The Great Gatsby"'s fingerprint. If the fingerprint does not match, they know that slot doesn’t hold the desired book, saving time.
Confirm and Return
If there’s a match, they compare keys directly to confirm it’s "The Great Gatsby". Once confirmed, they return its value (10101..).

This process avoids unnecessary checks and minimizes memory lookups, making retrieval lightning-fast.

Handling Collisions

What happens if multiple books generate the same H1 (i.e., they hash to the same group)? This scenario is known as a collision.

In our library analogy:

If a bookshelf is full, the librarian moves to the next available shelf in the same section.
This is called linear probing, where nearby groups are checked for free slots.

To ensure efficiency, if all shelves in a section are full, a new section (table) is added, and some books are redistributed between sections based on updated hash calculations to maintain optimal access.

Why This Design is Brilliant

The new map implementation in Go 1.24 introduces several optimizations inspired by Swiss Table design:

1. Cache-Friendly Layout

Keys and values are stored together in groups, improving cache locality (storing related items close together takes advantage of how memory access works).
When looking for an item, both its key and value are likely loaded into memory at once.

2. Fast Probing with Metadata

The control word allows fast rejection of irrelevant slots using SIMD (Single Instruction, Multiple Data allows processing multiple data points with a single instruction, thus boosting performance during lookups) operations.
This means multiple slots can be checked simultaneously, speeding up lookups significantly.

While pre 1.24 use tophash which reminds this strategy, it was still required to pointer chasing when overflow buckets were involved, reducing cache efficiency.

Memory Layout Example

Here’s a simplified version of how this might look in memory for a single group:

Control Word	Key0	Value0	Key1	Value1	...	Key7	Value7
`3c`	`"The Great Gatsby"`	`10101..`	...	...	...	...	...

The control word (3c) stores fingerprints for all 8 slots. (We have only one element in the group)
Keys ("The Great Gatsby") and values (10101..) are stored adjacently within each group for better performance.

Performance Gains

The redesign brings significant improvements over older implementations:

Faster lookups: Metadata allows skipping irrelevant slots quickly.
Reduced memory overhead: Group storage eliminates extra pointers, and has better load-factor.
Better scalability: Incremental resizing avoids performance bottlenecks during growth.

For example:

With the optimizations, lookups are up to ~30% faster compared to previous versions.
Memory usage is reduced by as much as ~28% compared to older versions of Go maps.

Conclusion

Go 1.24’s map is like a librarian who gets smarter with every update—finding books faster, using space more efficiently where:

Sections (tables) expand as needed.
Bookshelves (groups) keep related items close together.
Cheat sheets (control words) help librarians find books faster without flipping through every slot.

This design balances speed, memory efficiency, and scalability beautifully—making Go maps one of the most optimized hash table implementations out there!

Whether you're building high-performance systems or just curious about how things work under the hood, understanding these concepts can help you appreciate Go's thoughtful engineering even more.

For more detailed post about this implementation, check the official Go's blog post Faster Go maps with Swiss Tables
or ByteSizeGo Swiss Table Maps.

References to very good explanations of maps prior 1.24

Why I Switched from Makefile to Taskfile

Chen — Tue, 05 Nov 2024 12:45:45 +0000

Photo by Kelly Sikkema on Unsplash

Introduction

Software projects involve several phases, including building, testing, and deploying code.
For instance, compiling Go source code results in an executable, while frontend frameworks compile into HTML, CSS, and JavaScript files.
Testing is crucial before merging changes or releasing new versions. Deployment scripts often ship software to production.
Each phase requires different tools, typically command-line utilities with various flags and parameters.
Automation tools simplify these processes, enhancing efficiency in daily workflows.

Makefile

Makefiles are powerful tools that automate software project workflows. Initially developed for C programs, they now support diverse tasks like website generation and data processing.

A Makefile contains directives for the make utility to build or maintain programs and files. It defines tasks and their dependencies, ensuring efficient and reproducible builds.

I won’t dive into Makefiles in this blog post as I’m assuming the reader is familiar with the concept. If not, there is plenty of information over the internet (like this tutorial for example or it’s wikipedia page).

Advantages of Makefile:

Integrated with the make utility, available on most Linux/MacOS systems.
A well-established tool with nearly 50 years of history.

These are the main advantages I think Makefile have. However, Makefiles have limitations, particularly their syntax, which can be cumbersome for complex tasks.

Why I Switched

In one of my projects, I used a Makefile for tasks like running frontend/backend services and database migrations. Here's an example of a migration task:

migrate-up:
    GOOSE_DRIVER=postgres GOOSE_DBSTRING="user=app host=localhost port=5432 dbname=my-app sslmode=disable user=app" \
    goose -dir database/migrations up

I wanted to load environment variables from a .env file by default but allow overrides with ENV_FILE=.env.production. After struggling with Makefile syntax and solutions that didn't work, I sought alternatives.

Introducing Taskfile

Taskfile is a Go-based task runner using YAML syntax for defining tasks. It simplifies project workflows by automating repetitive tasks like building, testing, and deploying code.

Benefits of Taskfile:

Readable YAML Syntax: Easier to understand than Makefiles.
Single Binary: No dependencies beyond the Go runtime.
Cross-Platform Support: Works on Linux, macOS, and Windows.

Here's how I solved my problem using Taskfile:

version: '3'
dotenv:
  - '.env'

tasks:
  migrate-up:
    cmds:
      - goose -dir database/migrations up

  migrate-up-prod:
    dotenv:
      - .env.production
    cmds:
      - echo executing DB migration on PRODUCTION ..
      - sleep 2 # allow time to cancel
      - goose -dir database/migrations up

Taskfile's intuitive API allowed me to quickly implement a solution that was both functional and readable.

Summary

Choosing the right tool can significantly impact productivity. While Makefile served its purpose initially, Taskfile offered a more elegant solution for my needs. Transitioning took less than 30 minutes and simplified my build process considerably.

If you're seeking an easy-to-use build tool, consider giving Taskfile a try.

The value of API-First design on side-projects

Chen — Fri, 12 Jul 2024 14:02:07 +0000

Cover Photo by Douglas Lopes on Unsplash

Intro

Lately, I had a chance to try out the API-First design approach. I had never written an OpenAPI document before, so I had no real knowledge of its benefits. It always seemed like too much prep work.

As developers, we often prefer writing code to writing documentation. We dive straight into coding, eager to see our project in action. However, I recently discovered a game-changing approach that has transformed my development process: API-First design. In this post, I'll share my experience implementing this method in a full-stack hobby project, highlighting how it streamlined my workflow and why it's worth considering for your next side project.

tl;dr: It will force you to think about your users and how they use your API before writing any code.

I’ve been working on a full-stack hobby project where my backend and frontend use different languages (Go and SvelteKit). I decided to give this approach a try and had my “aha” moment. I wish I had done it before.

Prioritizing Your Application's Foundation

The API is how we are going to expose our app functionality. An API-first design approach prioritizes the development of APIs before implementing other parts of a software system (or writing code). This method focuses on creating a well-designed, consistent, and user-friendly API that is the foundation for the entire application.

This methodology places the API at the center of the development process, treating it as a first-class citizen rather than an afterthought. Your API comes first, then the implementation.

With a written API specification, we can leverage code generation tools to create some boilerplate code. By defining objects in the specification, code-gen tools can generate the relevant structs, for both the frontend and backend (yes, even when the language used is different). This is a big time saver and it helps us to be consistent.

What is Open API?

“The OpenAPI Specification (OAS) defines a standard, language-agnostic interface to RESTful APIs which allows both humans and computers to discover and understand the capabilities of the service without access to source code, documentation, or through network traffic inspection.”

Simply put, it’s a contract that describes your API types and endpoints. You list all your API endpoints, their HTTP methods, what they possibly return, and some description of what they do.

Now, what if I told you, you can use this document to improve and accelerate your dev experience?

Once I had this document, that describes the contract between my API server and its clients, these are the things I could do:

Generate my backend types (Go)
Generate my frontend types (Typescript)
Generate a client code for my server (also Typescript)
Generate a testing client with Insomnia or Postman

This is a lot of boilerplate code I could save myself from writing. It ensures the frontend and backend types are synchronized since both are generated.

Grab a 🍺, and let’s walk through an example.

The Project Structure

We will be using Go for the backend and some JS framework for the frontend, and a simple structure would look like:

app/
├─ api/ -- the place for the Swagger OpenAPI document
├─ client/ -- the client-side code
├─ cmd/
│  ├─ app.go -- thin main func that runs our API server
├─ internal/
│  ├─ api/
│  │  ├─ main.go -- for code-gen
│  ├─ users/
│  │  ├─ handlers.go -- implements the API contract

Generate The APIs

Let's create our API specification. It includes two endpoints and two structs: User and Error.
Place this file under your /api directory

openapi: 3.0.3
info:
  title: Devopsian OpenAPI Example
  version: 0.1.0
  contact:
    name: dev
    url: https://devopsian.net
servers:
  - url: "http://localhost/v1"
components:
  schemas:
    Error:
      type: object
      required:
        - code
        - message
      properties:
        code:
          type: integer
          format: int32
        message:
          type: string
    User:
      type: object
      required:
        - id
        - name
        - email
      properties:
        id:
          type: string
        name:
          type: string
        email:
          type: string
          format: email

paths:
  /user:
    get:
      description: Get the current logged-in user
      responses:
        200:
          description: user response
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/User"
        default:
          description: error
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
  /signup:
    post:
      description: Creates a new user
      responses:
        200:
          description: Creates a user
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/User"
        default:
          description: error
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                name:
                  type: string
                email:
                  type: string
                  format: email

Generate Server-Side Code

To generate the server-side code, we need some library. I found oapi-codegen for that. It supports many popular HTTP libraries (echo, gin, etc.) At the time of writing, I used oapi-codegen@v2.3.0

Add the following files to your /internal/api directory

// /internal/api/main.go
//go:generate oapi-codegen --config cfg.yaml ../api/openapi3.yaml
package api

// make sure to install: 
// go install github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen@v2.3.0

# /internal/api/cfg.yaml
package: api
output: server.gen.go
generate:
  models: true
  echo-server: true

I’m using the echo web framework, you can browse the library documentation to use other frameworks. Now run go generate ./... and it will generate the interfaces (handlers) your web server has to implement to fulfill this contract, including the types.

Interface Implementation

Now it’s time to write the implementation. We create a users package where all the user's API handlers, business logic, storage, etc. are defined. We will keep it simple and implement the handlers with static content.

type UsersHandler struct {
    DB *sql.DB
}

func (u *UsersHandler) GetUser(ctx echo.Context) error {
    // load the user from the database and return it to the caller
    return ctx.JSON(http.StatusOK, api.User{
       Email: types.Email("demo@devopsian.net"),
       Name:  "DemoUser",
       Id:    "1",
    })
}

func (u *UsersHandler) PostSignup(ctx echo.Context) error {
    var body api.PostSignupJSONBody
    if err := ctx.Bind(&body); err != nil {
       return ctx.JSON(http.StatusBadRequest, api.Error{Code: http.StatusBadRequest, Message: "invalid request"})
    }
    // save user in database

    return ctx.NoContent(http.StatusOK)
}

func New(db *sql.DB) *UsersHandler {
    return &UsersHandler{DB: db}
}

Next, we need to create our web server entry point, we define that at cmd/server.go

type Server struct {
    users.UsersHandler
}

func main() {
    e := echo.New()
    s := Server{UsersHandler: *users.New(nil)}

    api.RegisterHandlers(e, &s)
    e.Logger.Fatal(e.Start(":8080"))
}

Note I explicitly pass in nil as DB implementation for this example, because we don’t use it.

That’s it. If the code compiles, our server implements the API contract. All the API endpoints are handled by the spec. If I had missed something, it would have broken at compile time.

How nice is that?

Generate a Typescript Client

It’s time to generate a client for our API. We use the same openapi schema file to generate a JS client. I won’t include a full frontend project in this example, but rather show how you can generate a client to an existing one.

In the client/ directory, install the code-generation tool for JS:

npm install openapi-typescript-codegen --save-dev. (This post was tested with v0.29.0)

Create a client/api/ directory, and let’s run the tool:

npx openapi-typescript-codegen --input ../api/openapi3.yaml --output api/ --name ApiClient.

This will generate a bunch of typescript files. To use our client we need to create an instance of it.

// api.ts
import { ApiClient } from './api/ApiClient'

const client = new ApiClient().default

// client has all the methods of our API:
// - getUser()
// - postSignup(requestBody: {name?: string, email?: string})

That's it.

Summary

API-First design isn't just another development buzzword—it's a powerful approach that can significantly enhance your side projects.

By prioritizing your API design before implementation, you gain clarity, consistency, and efficiency.

The OpenAPI specification is a contract between your frontend and backend, enabling automatic code generation for types, clients, and even testing tools.

This approach not only saves time but also ensures better synchronization between different parts of your application.

While it may seem like extra work upfront, the long-term benefits—including improved development speed, reduced errors, and better API documentation—make it a valuable investment for any side project.

If you haven't tried it yet, now might be the perfect time to give it a shot and experience these benefits firsthand.

Inside EKS Networking: Decoding the Service IP Journey

Chen — Mon, 25 Mar 2024 14:00:00 +0000

Cover Photo by Taylor Vick on Unsplash

Intro

Have you ever found yourself deep in the trenches of Kubernetes networking, only to be surprised by a hidden quirk that challenges your understanding? In this blog post, I unravel the mysteries behind Kubernetes networking in Amazon EKS, shedding light on the intricate journey of a packet from the client through the NLB to an ingress controller pod.

This blog is a story about a change in perception. While I was debugging a problem the other day, it got me to question what I was certain I knew. When you’re sure you understand how Kubernetes works, you encounter another road bump that challenges your understanding and makes you do some research, this article is the output of this research.

The Setup

The minimal setup required to examine what’s presented here. You would need an EKS cluster with NGINX ingress-controller deployed with NLB as the entry to your cluster.

The Problem

I was debugging a service with an ingress resource. I had incoming traffic from NLB to my ingress controller. I was surprised to see my ingress controller wasn’t listening on that port. The AWS console shows those endpoints as ‘healthy’, meaning they respond to health check samples. But how is that even possible if there is no process listening on that port?

I gotta say this drove me nuts. I had looked online for similar issues, but nobody mentioned this problem. So I started doing some research, looking for an answer to the question “How does Kubernetes handle Service IP under the hood, on EKS?”

Exploring Kubernetes Networking Magic

Before I present the network flow of a packet, here are some assumptions I take. Kubernetes is a modular platform; this post is relevant for EKS with the VPC-CNI add-on running on v1.27. It was checked with NLB, other load-balancer types might behave differently.

The next part is low-level. You should be familiar with iptables. Two blog posts that cover this topic greatly are — Laymans iptables 101 and the iptables — a comprehensive guide

When a packet arrives, it first goes through the iptables PREROUTING chain. (That’s a builtin one):

[root@...] # iptables -t nat -nvL PREROUTING
Chain PREROUTING (policy ACCEPT 1987 packets, 119K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 756M   5G KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

This rule captures every incoming packet and forwards it to the “service portals", the KUBE-SERVICES, (that’s a custom chain) where it’s being matched and designated to the relevant Kubernetes Service IP (nginx, in our case).

Incoming packets on a matching port are DNAT’ed to a Service IP. There’s a matching rule for our nginx instance. Incoming packets on its listening port 32443 are sent to its Service IP:

iptables -t nat -nvL KUBE-SERVICES | grep nginx
0     0 KUBE-SVC-I66WCJWOLI45ORGK  tcp  --  *      *       0.0.0.0/0            172.20.152.36        /* ingress-nginx/ingress-nginx-controller:https-32443 cluster IP */ tcp dpt:32443

Now that our packet has a destination Service IP in the cluster. But service objects in Kubernetes are just a layer of abstraction; there are no actual Pods with such IP. It’s synthetic. Kubernetes networking layer needs to translate this IP to the relevant Pods. This process is done by transforming Service IP to Endpoints. This is also done by iptables.

We have a specific KUBE-SVC-* chain, which is constructed for every service object we have. The purpose of this chain is to translate Service IP to Endpoints (the actual pods behind it). This chain has an entry for every pod alive. This is where the kernel performs ’load balancing’ between the pods.

iptables -t nat -nvL KUBE-SVC-I66WCJWOLI45ORGK
Chain KUBE-SVC-I66WCJWOLI45ORGK (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-TQZ3GYVANOOYIJMO  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:https-32443 -> 10.1.21.61:443 */ statistic mode random probability 0.50000000000
    0     0 KUBE-SEP-NEOTPBRCKXF3UWGX  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:https-32443 -> 10.1.36.6:443 */

We’re not done with iptables just yet. As you can see from the output, there’s another chain our packet has to go through. We’re getting close. Each chain corresponds to a Pod, with a KUBE-SEP-* chain. If we check the rules of this chain:

iptables -t nat -nvL KUBE-SEP-TQZ3GYVANOOYIJMO
Chain KUBE-SEP-TQZ3GYVANOOYIJMO (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-MARK-MASQ  all  --  *      *       10.1.21.61         0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:https-32443 */
   64  3840 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:https-32443 */ tcp to:10.1.21.61:443

The first rule is for outgoing packets (SNAT). Incoming traffic matches the second rule. This rule has a DNAT target, which is Destination Network Address Translation. This rule rewrite the TCP packet destination IP to the Pod’s IP: 10.1.21.61:443 Now routing continues as normal, reaching the Pod on the relevant port.

Summary

When a packet arrives, it first goes through iptables PREROUTING, where it’s forwarded to the KUBE-SERVICES chain. This chain directs packets to the relevant Kubernetes Service IP. From there, iptables translates the Service IP to Endpoints using the KUBE-SVC- chains, ultimately reaching the correct pod.

In conclusion, understanding Kubernetes networking in EKS requires delving into iptables manipulations and Kubernetes Service abstractions. I hope this post sheds light on these concepts.

How to Structure a Go Project: Start Simple, Refactor Later

Chen — Sun, 18 Feb 2024 20:08:48 +0000

Once upon a codebase, in a kingdom of endless debates, a lone developer pondered the age-old question: “What's the perfect project structure?” Spoiler alert: there isn't one. Let's embark on a quest to discover the charm of simplicity and the art of Go project structuring.

The perennial question about what directory structure should I use echoes across various social platforms now and then. This subject was discussed many times. I’ve been programming in Go for a couple of years now and asked myself this question every time I had to start a new project. If you’re asking it yourself, here are my 2 cents for you.

Why is project structure important?

"A well-organized directory structure is the scaffolding upon which a robust codebase stands tall."

A project structure is important because it affects readability & maintainability. Think about your future self, six months from now reading your code when it’s not fresh in your memory. If you have structured it right (and there are many ways to do it) it would be easier to jump in. It would also be easier for collaboration as other developers can navigate through it more easily. Most of the time we read and maintain existing code, not writing it from scratch.

What are the common pitfalls of project structure?

There is no such thing as a perfect, one-size-fits-all project structure, so stop looking. It’s like asking for the perfect car. It depends on who you ask, and what are their standards and requirements. You won’t find a single answer, because like many things in computer science — it depends. While there’s no one-size-fits-all solution, some general guidelines can be followed.

Some of the common pitfalls:

Over-engineering: Trying to anticipate all the possible scenarios and use-cases before writing any code. This leads to unnecessary complexity and abstractions, which makes your code harder to read and maintain
Under-engineering: Writing all the code in a single package, without any structure or organization. This leads to spaghetti code base, which makes it harder to test and debug
Copy-pasta: Blindly following the structure of another project, without putting any thoughts or understanding the trade-offs behind it.

How to start simple and refactor later?

My advice is to start simple and let the packages grow organically as you’re writing the code. Don’t worry about creating the perfect packages and abstractions at this point. Just start by writing your code in a single package — main and see how it works. A simple directory tree is easier to read, which reflects that it is easier to maintain and collaborate with a team or individuals.

As you write more code, you’ll notice some patterns emerge and repetitions that you can extract to a different package. This will make your code more readable, maintainable, and testable.

I find it much harder to do the right abstraction before writing code. How do you know which packages do you need? Start writing your business logic. It is easier to refactor code when you see it in front of you. It’s usually easier to abstract some logic from a package, rather than refactoring multiple packages because you got it wrong the first time.

The key is to write code and refactor it later.

The cmd package pattern

A powerful pattern but make sure to ask yourself if you need it

This is a great pattern when necessary. Do you write a library? Or an app with a single binary? Then you probably don’t need to use the cmd package. Ask yourself what this abstraction allows you. Remember — keep it simple.
This pattern is useful when your code base is compiled to multiple binaries, for example, if you have a Server and a CLI.

Inside this package, you put the sub-directories that contain your main.go files. Each sub-directory is compiled into its own binary. This is the entry point of your app. You should look into spf13/cobra: A Commander for modern Go CLI interactions project, which is a widely used library for creating powerful modern CLI applications. It blends well with spf13/viper: Go configuration with fangs project, which handles loading configurations nicely.

The internal package

The internal package has a special meaning in Go. Packages that reside under internal/ may not be imported by packages outside the source subtree in which they reside. Therefore, these are said to be internal packages. The code placed here is internal to the project, and can’t be used outside of it.

There used to be times people grouped external packages under pkg/, but I find it meaningless. A directory that exists only to hold other packages is a potential code smell.
Instead, give your packages descriptive names. Try to describe what they do, while avoiding ambiguous names like utils or common.

The config package

The config package is responsible for creating and managing the configuration object that your app depends on. It simplifies the process of importing configurations from a single source and accessing them from anywhere in your code base. It also avoids the circular dependency problem that can arise from importing configurations from multiple places.

Let the config package handle your app's settings, harmonizing inputs from flags, environment variables, or files into a cohesive configuration object.

The API package

The API package defines the interface of your app with the outside world. It contains the schema definition (openapi) and the models (which can be generated from schema) that represent the data types and endpoints of your app. Unless a struct is internal to a specific package and is not used outside of it, it should be placed here. The API package should include any struct that is used by more than one package, or that is exposed to the client or the storage.

The API package is the protocol that enables communication between different components of your app.

Controllers, handlers, and storage

Now, let's delve into structuring controllers, handlers, and storage. Two approaches stand out:

Approach 1: Storage and Handlers (or controllers) packages

With the first approach, you write your storage and service/controller layers in separate packages where the service imports the storage. This allows you to decouple your business logic from your data access layer and use different storage implementations (postgres, sqlite, redis, etc.) without changing your service code. Here’s a simple view of this approach:

.
└── project/
    ├── api/
    │   ├── models.go
    │   └── openapi3.yaml
    ├── cmd/
    │   ├── server/
    │   │   └── main.go
    │   └── cli/
    │       └── main.go
    └── internal/
        ├── config/
        │   └── config.go
        ├── storage/
        │   └── postgres/
        │       ├── users.go
        │       └── ...
        ├── handlers/
        │   ├── users.go
        │   ├── users_test.go
        │   ├── storage.go
        │   └── ..
        └── server/
            ├── server.go
            ├── server_test.go
            └── ..

Approach 2: Domain Entity Package

With this approach, you encapsulate logic by functionality so that both your service and storage code reside in the same package. This follows the principle of domain-driven design, where you model your code around the business domain and its entities. Each package represents a domain entity (such as user, post, comment, etc.) and contains all the code related to it (such as internal structs, methods, handlers, queries, etc.)

This is a topic for a separate blog post I might do later on, but in the meantime for more details about this type of architecture check out introduction to DDD or watch Kat Zien great talk from GopherCon 2018.. Here’s a simple view of this approach:

.
└── project/
    ├── api/
    │   ├── models.go
    │   └── openapi3.yaml
    ├── cmd/
    │   ├── server/
    │   │   └── main.go
    │   └── cli/
    │       └── main.go
    └── internal/
        ├── config/
        │   └── config.go
        ├── users/
        │   ├── postgres/
        │   │   └── db.go
        │   ├── handlers.go
        │   ├── handlers_test.go
        │   ├── storage.go
        │   └── ..
        └── server/
            ├── server.go
            ├── server_test.go
            └── ..

Conclusion

In conclusion, there is no single ‘perfect’ way to structure your Go project. There isn’t a one-size-fits-all. There are multiple ways, and it depends on your use-case, preferences, and the trade-offs you’re willing to take. Experiment based on your needs and requirements.

Embrace simplicity initially. Write code in a single package, observe emerging patterns, and refactor as needed. The key here is to write code first, and ‘perfect’ structure later.

I hope this post gave you some insights and ideas on how to structure your next project. Happy coding!

This is how you want to manage your Terraform modules

Chen — Sun, 08 Jan 2023 07:21:57 +0000

What if I told you there is a way to manage all your private terraform modules, in a mono-repo, with independent versioning, without using git tags? After researching for a proper open-source tool, I found the right one for the job.

Intro

A private registry is needed.

I looked for a way to manage private terraform modules like public ones.
In my company, we write tailor-maid modules that describe our infrastructure. These modules have to be private.
Terraform recommends each module has its own git repository, yet, this has the burden of managing and syncing multiple repositories.

The second option suggested is to use a mono-repo, and reference the module's version using git tags. At first, this seemed alright,
but there was a drawback — you had to give up the terraform version syntax. You no longer can use the convenient version = ~> 1.0.0 module parameter.

It seemed to me that there had to be a better way.

The problem

When it comes down to managing my infrastructure, I prefer the mono-repo approach. Since our terraform modules are small configuration blocks,
it makes sense. It is much easier to find a module in a single repository rather than searching multiple repositories for a module.
Our module release workflow looked something like this

Our repository was structured with sub-directories per module
We used Git tags for module versioning in the form of moduleName-vX.Y.Z
Updates to module versions were hand-delivered to clients (even minor patches)

Our root modules (where we execute terraform plan and apply) reference modules using git tags:

terraform {
  source = “https://github.com/my-org/my-repo?ref=s3-v1.0.0"

  .. module inputs ..
}

This has worked well for a while. But once we had released a new module version and wanted to use it, there was no convenient way to apply that.

We had to go through our root modules using this module and update their references. (We could write a simple script, but we decided not to. More on that later)

The problem we had with this approach is, you can’t use the version argument in the terraform block.
This argument lets you specify a range of acceptable versions instead of a hard-coded one.

For example, you can provide version constraints such as version = “~> 1.2.0, < 2.0” which allows incrementing the “patch” automatically every time you run terraform init.
No need to manually update a patch release, and no need to write a bash script. Terraform can manage that reliably for us,
with a better API, if we can only discover a way to use this feature.

Manually managing our tags was chaotic. It is not an easy task to manage independent module versions in a mono-repo using the Git tags approach.

A simpler approach is to release a version for the whole repository. That means every new release includes all the modules together.
This couples the modules into a single artifact, which is easier to manage, but at the cost of development velocity.
You cannot release minor patches just for your module, no matter how small the code change is.

Towards a better future

The structure and workflows we had in place worked; just as not as well or easily as we wanted to.
What we needed was to be able to use terraform versioning syntax, while maintaining our mono-repo.
To achieve that, we need to treat our terraform modules as artifacts; something we can archive, version, and release independently.

After doing some research, we found an open-source project called Terralist.

Terralist is a private Terraform registry for providers and modules following the published HashiCorp protocols.
It provides a secure way to distribute your confidential modules and providers. That looked like a project that might help us to solve our problem, so we decided to try it out.

So we maintain our mono-repo as it is. We created a job in our CI system that archives a single terraform module and uploads it to Terralist.
Since it's a private registry, clients need to authenticate to be able to download the modules. After authentication (using terraform login <terralist-url>)
we could use our private modules just as we use the public modules. Yes, it means we can use the versioning syntax I was talking about.

Here is how we use our modules nowadays

 terraform {
     source = “https://my-terralist.com/my-org/s3/aws”
     version = “~> 1.0.0”

     .. module inputs ..
 }

This client code will retrieve automatic patch updates on every execution from our CI system (if you execute terraform locally, you would need to re-run terraform init).
If we make larger code changes to one of our modules, for example, something that might break the API
we would increment the major or minor version, so it wouldn't impact our clients.

Terralist

Terralist has an API that lets you upload a module from a git repository. It means you don't need to archive the module yourself,
just point Terralist to its location.

Terralist will clone the repository and create the artifact for you. Let's walk through an example.

I have a demo module, which resides in my terraform.git repository under the modules/demo directory. In the module itself, we keep a version.tf
with the major and minor versions of the module. These values change only when we introduce a change that breaks our existing API
(e.g, adding a new mandatory parameter without defaults). In such case we update the major or minor manually.

A patch number in the semver represents a safe change, such as a bug fix or non-breaking changes to the module.
This value is calculated based on the build number of our CI. We don't really care about its value, because the version constraint
applied is in the form of version = "~> 1.0.0". This automatically updates to the latest patch on every execution.

As part of our module build process, we read the values of version.tf and append the patch. Then we upload the module to Terralist using this API call:

 curl -X POST registry.example.com/v1/api/modules/demo/aws/1.0.10/upload \
      -H "Authorization: Bearer x-api-key:$TERRALIST_API_KEY" \
      -d '{ "download_url": "https://github.com/example-org/terraform/archive/refs/heads/master.zip//modules/demo" }'

The module version is composed of major.minor coming from the version.tf file, and the .patch is the build ID, which is incremented with every run and guaranteed to be unique.

Pay close attention to the double // -- this instruct Terralist to:

Download an archived repository
Extract it locally
Make an archive only from the path modules/demo
Upload it to the registry (basically upload it to S3 and update the registry database)

It's an elegant solution: I tell Terralist where my module is, and what version it is tagged with, and it takes care for everything else.

Summary

There are various ways to manage your infrastructure code.

It depends on multiple factors, such as team size, how much you are willing to spend on 3rd party tools, and your company policy, to name a few.
As of today, I've been using Terraform for more than 3 years, relying solely on the open-source ecosystem. That means my team and I manage everything related to our infrastructure (it might change soon, as our infrastructure size has grown).

Terralist was a helpful addition to our stack; it solved a problem we had in our existing workflow, with minimal effort and no code changes to our existing modules.
We just had to upload and version them to the new registry.
Now, we can release each module independently and decide if we want our clients to automatically upgrade their module version.

Happy Terraforming.

For further reading, HashiCorp documentation contains a lot of good information. Check out the links below

Practical unit-testing web client in Go

Chen — Fri, 28 May 2021 11:24:29 +0000

I've started a Go project, a lightweight Hashicorp Vault¹ client with no dependencies, and a simple API (for the user). Part of the reason I use no 3rd party modules is, I want to better understand Go internals, structure, and improve my skills. In this post, I'll conduct a practical example of how I ended up testing it.

The the project's name is libvault, and it's my first open-source project.

Vault is a secret manager service with web API and CLI. Applications can communicate with it through HTTP. I find the CLI easy to use, however, the official Vault library is a bit more complicated; it felt like a swiss-army knife when all I needed was just a simple kitchen knife. I decided to implement a light version, that covers basic functionality while maintaining a simple API.

I'm not going to cover the basics in this post. The intention of it is to be practical and provide real-life, working examples. The opinions here are mine. It worked for me, still, it doesn't mean it would work for you.

I removed ALL the error handling from the code for brevity. Please make sure you handle your errors.

The Go standard library provides a really good testing package. You can manage without external frameworks or 3rd party packages. In my scenario, I need to test a web client that I wrote. The technique I found useful is mock testing.

Mock testing is an approach to unit testing that lets you make assertions about how the code under test is interacting with other system modules. In mock testing, the dependencies are replaced with objects that simulate the behaviour of the real ones. ... Such a service can be replaced with a mock object. ~ Wikipedia

I had to choose which component to mock:

Client
Webserver

At a high level, mocking the client means creating a new struct that implements the interface you are testing (mocking the interface). Then, provide your mocking client to the code under test. A good library with examples is testify.

I didn't find it useful for my use case, as I need to mock the server-side. I prefer not to modify any code on the client if I can. This would result in more reliable tests for my package. So, I've chosen to go with the second option. Read on for how.

The httptest package

Go is very friendly to web services; it has a utility package (httptest) for testing an http server. You can simply start a webserver in your testing code.

First I had to fetch an exact response from a real webserver (Vault server), then I could easily mock it using this package. For example, when querying the /v1/auth/approle/login endpoint, the response looks like this:

{
  "request_id": "de7c8097-1a38-50a6-b971-fe1836840e45",
  "lease_id": "",
  "renewable": false,
  ...
  ...
}

Now that I have the content, I find it easier to save it to a file (rather than put it inside the code). Go testing package has another cool feature:

The go tool will ignore a directory named "testdata", making it available
to hold ancillary data needed by the tests.²

I created a directory named testdata inside my project and placed the JSON content in a file - approleExample.json.

Start the mocking server

I'll start with code, followed by an explanation:

package main

import (
    "fmt"
    "io"
    "net/http"
    "net/http/httptest"
)

func TestClientLogin(t *testing.T*) {
    mux := http.NewServeMux()
    ts := httptest.NewServer(mux)
    defer ts.Close()

    mux.HandleFunc("/v1/auth/approle/login", func(w http.ResponseWriter, r *http.Request) {
        // request validation logic
        ...
        // read json response
        jsonPayload, _ := ioutil.ReadFile("testdata/approleExample.json")
        w.Header().Set("Content-Type", "application/json")
        w.WriteHeader(200)
        w.Write(jsonPayload)
    })

    // initalize new client pointing to the testing server
    client, err := NewClient(ts.URL)
    if err != nil {
        // error handling
    }

    // test logic
}

If you're familiar with the Go http package, this code is pretty self-explanatory. This is another advantage of using the standard library - you have one less thing to learn when you want to contribute.

mux is of ServeMux type. Which is an HTTP request multiplexer. It matches the URL of each incoming request (/v1/auth/approle/login) against a list of registered patterns and calls the handler for the pattern that most closely matches the URL (our mux.HandleFunc function body).
ts is of Server type. A Server is an HTTP server listening on a system-chosen port on the local loopback interface, for use in end-to-end HTTP tests. Note that I use httptest.NewServer to initialize it.
mux.HandleFunc(..) defines a path and an handler (a function) to call. The content inside describes our server's response.
client, err := NewClient(ts.URL) creates a new Client (my Vault client), providing it the test server URL to work with.

The elegance of this pattern is, every test case has its test webserver with all the relevant configurations. The mocked content, the test logic, etc. are all implemented inside the test itself.
This really makes life easier when debugging, reviewing a test case logic or coverage.

We can improve it further, make the code more clear and concise. This code includes some boilerplate: creating the mux, the server, and reading the json content. We just need to refactor these elements out (to a setup() function, and readJson(path string), for example). Then call this setup() function for every test case. I'll leave that to the reader to decide.

Summary

There are numerous articles about the importance of software testing. Go makes it easier. You should always write tests for your packages; it has so many advantages to just skip it. However, many people do that and I can assume the reason, which is it doesn't provide any additional functionality to your code. Don't be one of these people.

Go standard library provides great tools, and you should use them. Personally, I prefer it over other dependencies.

In this post, I gave a practical example of how you can unit-test a web client by mocking it.

The takeaways:

Always write tests. They are too valuable to give up and very easy to do with Go.
Mock the part your code connects with, don't mock your code. If you write a server, mock the client, and vice-versa. It makes your tests much more reliable.
1. If you mock a server, get a real server response and save it to a file
2. Do that for every API you would like to mock
Use testdata directory to hold your ancillary data needed by your tests

In the next article, I will cover how to test with a TLS (HTTPS) web server and self-signed certificates, adding more examples. You can also find more examples in the source code.

Feedback and comments are welcome.

How to share persistent storage volumes in Swarm

Chen — Tue, 23 Feb 2021 18:07:22 +0000

Docker swarm is an orchestration tool, similar to Kubernetes, but simpler to set up and manage.
A swarm consists of multiple Docker hosts which run in swarm mode and act as managers (to manage membership and delegation) and workers (which run swarm services). A given Docker host can be a manager, a worker, or perform both roles.

The docker swarm feature is embedded in the Docker Engine (using swarmkit). This means you don't need to install extra packages to use it. You just docker. If you decided to put it in place, one of the open problems to address is the persistent storage.

In the old days, many processes could share a local disk on the host. It was a common pattern to have a process that writes to files, and another that consumes them. The two processes were living happily on the same host, having no problems working together. But it's another story in the containerized world. When containers are managed by an orchestrator, you can't really tell where your container would be scheduled to run. It won't have a permanent host, neither you want to lock it to a specific host, as you lose many of the orchestrator benefits (resiliency, fail-over, etc.).

The Problem: Data Persistency on the Swarm cluster

Suppose you have two services that need to share a disk, or a service that requires data persistency such as a Redis. What are your options?

One easy option is to use docker volumes. It's a good option if you run on a single node because when you create a new docker volume, it resides on the host it was created on.

But what happens when it runs on a cluster? good question. Let's walk through an example. I'm going to one from the use the Docker Documentation (Get Started with Docker Compose)

Here are the files I'm using for this example:

devopsian: ~/demo
➜ tree
.
 |-app.py
 |-Dockerfile
 |-docker-compose.yml

# app.py
import time

import redis
from flask import Flask

app = Flask(__name__)
cache = redis.Redis(host='redis', port=6379)

def get_hit_count():
    retries = 5
    while True:
        try:
            return cache.incr('hits')
        except redis.exceptions.ConnectionError as exc:
            if retries == 0:
                raise exc
            retries -= 1
            time.sleep(0.5)

@app.route('/')
def hello():
    count = get_hit_count()
    return 'Hello World! I have been seen {} times.\n'.format(count)

# Dockerfile
FROM python:3.7-alpine
WORKDIR /code
ENV FLASK_APP=app.py
ENV FLASK_RUN_HOST=0.0.0.0
RUN apk add --no-cache gcc musl-dev linux-headers
RUN pip install -r flask redis
EXPOSE 5000
COPY . .
CMD ["flask", "run"]

# docker-compose.yml
version: "3.7"
services:
  app:
    build: .
    ports:
      - "5000:5000"
  redis:
    image: "redis:alpine"
    volumes:
      - "redisdata:/data"

volumes:
  redisdata:
    driver: "local"

Suppose you have a small cluster of 3 nodes. (Their roles in the cluster don't matter for example). You run your docker-compose file with your app and a Redis instance, and define a volume for it. The first time you run your docker-compose, docker creates the volume and mounts it to your service on startup.

Now you want to deploy a new version of your service. The cluster decides to schedule the Redis instance on a different node than before. What will happen to your volume?

Since the volume doesn't exist on the node the service runs at, docker would create a new volume with the same name on the new node. The previous volume still exists with the data but, it resides on the old node. The Redis service has no access to it. Redis now actually has a new, empty volume for use. You end up in an inconsistent state. This solution doesn't work for us.

AWS Elastic File System (EFS)

Amazon Elastic File System (Amazon EFS) provides a simple, scalable, fully managed elastic NFS file system for use .. It is built to scale on-demand to petabytes without disrupting applications, growing and shrinking automatically as you add and remove files, eliminating the need to provision and manage capacity to accommodate growth.

EFS provides a NFS volume you can mount at runtime. The swarm cluster schedules to run the service on one of its nodes and mounts the NFS volume to the container.

With these configurations, if the cluster decides to move our Redis service to a different node (due to a failure, deployment, etc.), the mount (and hence the data) will move to the new node too. I won't go through how to create an EFS volume, you can find the steps here on aws tutorial.

To achieve that, we need to install the nfs-common package on the swarm nodes. On Ubuntu, you can install it with: sudo apt-get install nfs-common

Next, we will update the volumes definition of our docker-compose, with the new driver type and the address. It looks as follows:

# docker-compose-with-efs.yml
version: "3.7"
services:
  app:
    build: .
    ports:
      - "5000:5000"
  redis:
    image: "redis:alpine"
    volumes:
      - "redis-efs:/data"

volumes:
  redis-efs:
    driver: local
    driver_opts:
      type: nfs
      o: addr=fs-1224ea45.efs.us-east-1.amazonaws.com,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2
      device: "fs-1224ea45.efs.us-east-1.amazonaws.com:/"

When we run our app now, the Redis data volume is persistent. It will move with it to whatever node Redis runs on.

This solves our original problem. We now have a way to persist data across our cluster.

Costs

This blog post uses AWS as the cloud infrastructure. When comparing EFS, the closest service AWS I could find is EBS volumes. I assume you're familiar with it.

You should use these pricing calculators from AWS to tailor the price for your use case.

These are not apples to apples, but let's do a quick price comparison between the two. We will compare a 100GB volume in the EU region.

EBS

Type	Charged for	Price
Storage	0.088$/GB * 730 * 100	$64.24
IOPS	3K - included	$0
Throughput	125MB/s - included	$0

You can increase the throughput or IOPS, with additional costs.

EFS

With EFS, the charges are different (shocked ah?). You pay for "Standard Storage Class" which is designed for active file system workloads.

You pay a different price for "Infrequent Access Storage Class" (IA) which is cost-optimized for files accessed less frequently. For this comparison, let's estimate 50% of the data is frequently accessed.

Type	Charged for	Price
Standard Storage	0.33$/GB monthly * 50	$16.5
IA Storage	0.025$/GB monthly * 50	$1.25
Throughput	2.5MB/s included, 6.60$ per additional MB/s	$0

The throughput part is kinda tricky. You get 50KB/s per Stand Storage GB (50GB * 50KB = 2.5MB/s) for write operations and 150KB/s (50GB * 150KB = 7.5MB/s) for read operations.

As you see, there is a big performance impact when comparing EBS with EFS. If you need high throughput, EFS might get very expensive.

Conclusion

The problem I wrote about in this blog is a known problem when running microservices on a managed cluster using an orchestration tool. As with other problems, it has more than one solution where each has its trade-offs.

I showed that EFS is a convenient, easy solution you can use to solve the shared storage problem. Yet, it is not the right solution for everything. The performance is limited, and it can get very expensive if your app requires high throughput. In those cases, you would want to use something else.

If you can compromise on the throughput, EFS becomes an attractive option to manage your cluster shared storage.

When you define your NFS volumes inside your docker-compose file, it is part of the service definition. It takes care of the infrastructure it uses. I find this pattern a good one when dealing with microservices.
If you need to move this service to another cluster, anytime in the future, the NFS mount is one thing less you need to remember to do.

Docker healthcheck experiments with Go web app

Chen — Fri, 12 Feb 2021 12:33:32 +0000

One good way to monitor your container status is to use Docker's HEALTHCHECK feature.

As part of testing the followed actions upon an unhealthy container, I had to experiment how this works. I wrote a small Go app that replies to /health requests, and the status it responds is configurable.

The webserver listens on $PORT (defaults to 8080). To change its status, simply make an api call to one of the supported actions. You can either connect the container to execute it,
or if you have mapped a port on the host, you can use it.

Supported actions are:

/sabotage will make it respond with 500.
/timeout will make it respond after 20s.
/recover will return it back to healthy state, with 200 response code.

You can find the source code here.

From the Docker docs -

The HEALTHCHECK instruction has two forms:

    HEALTHCHECK [OPTIONS] CMD command (check container health by running a command inside the container)
    HEALTHCHECK NONE (disable any healthcheck inherited from the base image)

The HEALTHCHECK instruction tells Docker how to test a container to check that it is still working. 
This can detect cases such as a web server that is stuck in an infinite loop and unable 
to handle new connections, even though the server process is still running.

When a container has a healthcheck specified, it has a health status in addition to its normal status. 
This status is initially starting. Whenever a health check passes, it becomes healthy (whatever state 
it was previously in). After a certain number of consecutive failures, it becomes unhealthy.

The options that can appear before CMD are:

--interval=DURATION (default: 30s)
--timeout=DURATION (default: 30s)
--start-period=DURATION (default: 0s)
--retries=N (default: 3)

You can configure the HEALTHCHECK settings in the Dockerfile or the docker-compose.yml. In my example,
I use it in the Dockerfile.

Demo

Clone the repo and build the image: docker build . -t go-healthchecker
Bring up the container, wait until it gets healthy.

To change the healthcheck response, connect to the container and update its status

In a future blog post, I will share how you can use the HEALTHCHECK feature to control docker-compose services startup with a more fine-grained option, e.g "How to make service X start before service Y".

How can Stackoverflow make you a better developer

Chen — Fri, 05 Feb 2021 14:53:52 +0000

"Time is the most valuable resource because you cannot get more of it."

There isn't a single developer who doesn't use Stackoverflow. Most of us use it daily. How many times a day have you googled an error, and reached Stackoverflow? It is such a valuable resource that improves productivity every single day. Consider the amount of debugging time it saved you. Yet, the majority of the developers I know don't even own an account, nevertheless trying to answer other people's questions.

How easy and convenient it is to find a solution to the problem you faced. In this post, I'll share with you the benefits I found of being a contributor on Stackoverflow.

Most of the experienced developers will probably stop reading right now. Individuals look to improve their coding and problem-solving skills, I hope you will own an account (and contribute) after reading this post.

If you're not familiar with where it started, the site was introduced by Jeff Atwood on his blog codinghorror:

"Stackoverflow is sort of like the anti-experts-exchange (minus the nausea-inducing sleaze and quasi-legal search engine gaming) meets Wikipedia meets programming Reddit. It is by programmers, for programmers, with the ultimate intent of collectively increasing the sum total of good programming knowledge in the world. No matter what programming language you use, or what operating system you call home. Better programming is our goal."

Learn something new

When I started learning Python about five years ago, I built my curriculum that was made of few books and online courses. I put the time and effort (about four months) to practice. As most of us, when I had faced challenges I couldn't solve by myself, I used Google and found an answer on .. you guessed it right, Stackoverflow. After a couple of iterations of this scenario, I thought to myself,

"If I find answers to my questions on Stackoverflow, let's take a peek at the most upvoted questions tagged with Python and see what it reveals".

It was like a goldmine for me. Clear, refined questions with detailed answers and examples, for free. No subscription, no money-back guarantee, plain simple. I started checking all of them, reading thoroughly all the answers to nail the subject in question. These are practical questions with high-quality answers from experienced professionals.

Next time you plan to learn something new, give it a try. Go to Stackoverflow and look for the most upvoted questions and answers. If you haven't ever done it before, you're gonna be amazed.

I then challenged myself. I pulled my sleeves and decided I will try to answer questions.

Why would I spend my precious time solving other people's problems?

That's a damn good question. When I took the challenge, I was selfish. I wanted to gain confidence in my knowledge. I liked to solve problems using the things I have learned. I looked for new questions that I had a solution for and tried to help using the gained knowledge. I did this to improve myself as a programmer.

I feel the time I spent, was paid back double. It made me a better programmer, problem-solver, and contributed to my professional career.

"Knowledge is like money: to be of value it must circulate, and in circulating it can increase in quality and, hopefully, in value" -- Louis L'Amour

Why YOU should contribute

Improve your coding and problem-solving skills: I can't stress enough the value of spending time on this site, reading through popular questions, and try to solve them yourself. Answering other people's questions strengthens your knowledge and confidence in the topic
Improve your debugging capabilities: It's one of the most valuable skills of a software engineer. By helping other people, you practice debugging their code, which is harder than debugging your own
If you don't find the question you're looking for - ask: Let the community help, and help other people in the future who encounters the same problem as you did
Learn something new, solve a new problem, dig deeper into a topic you're familiar with, and expand your knowledge
Use your Stackoverflow profile to promote yourself. By accurately answering questions, you demonstrate your experience in a topic

Where to start?

This blog post aimed to get you started. First and foremost, if you don't have an account, sign up.

I recommend checking these two threads, “How do I contribute to Stackoverflow” and “How does a new user get started”. Spend the time reading to understand how this platform works and how to use it.

Either you're learning something new, or want to improve any of your coding skills, try to participate and answer questions:

Search for the tags of interest to see only these kinds of questions (e.g, python, go, linux)
Spend ~30-60min a day for 14 days. It doesn't need to be contiguous, you can check it while you're waiting for the compilation or deployment to finish
Make it a habit

That's all it takes.

Conclusion

I found that actively participating on Stackoverflow of great value, for my development skills and career. Every beginner and experienced developers alike will benefit from it if used correctly. I tried to share my own experience, what works for me, and I encourage you to share your knowledge. And have fun.

Stay humble, be kind.

If you don't use a secret management tool, you're doing it wrong

Chen — Wed, 27 Jan 2021 20:19:17 +0000

Secrets management refers to the tools and methods for managing digital authentication credentials (secrets), including passwords, keys, APIs, and tokens for use in applications, services, privileged accounts, and other sensitive parts of the IT ecosystem.

While secrets management is applicable across an entire enterprise, the terms secrets and secrets management are referred to more commonly in IT about DevOps environments, tools, and processes.

“Three may keep a secret, if two of them are dead.”
― Benjamin Franklin, Poor Richard's Almanack

Managing and sharing secrets is a complicated task. There are various environments, with many services, where each needs to authenticate itself.

If you're working in the Operations team, you clearly faced a secret committed to a git repository at one point. Even if the repository is private, this is a big no-no. If someone accidentally changes the repository to be public, or push a file with secrets to a public repository, there's no turning back. The secret will be out there in the wild.

Before I proceed, I want to ask you few questions:

Where does your organization keep its sensitive data? Is it in an encrypted file? A database? A shared tool such as 1Password?
Who can access it?
How easy is it to add a new secret, or update an existing one?
Do you manage your secrets or they manage you?

Hashicorp Vault

Hashicorp Vault is an open-source robust secret management. It serves as a secret repository with access control lists, auditing, and TTL access to the secrets. It also supports a variety of authentication mechanisms and storage backends.

Vault keeps your secrets encrypted on disk and on transit. It has a simple API to communicate with and great documentation. The website presents two main use-cases:

Secrets Management: centrally store, access, and distribute dynamic secrets such as tokens, passwords, certificates, encryption keys.

Data Protection: keep application data secure with centralized key management and simple APIs for data encryption.

It gives you a central place to safely store secrets. Define who can access what, with an audit on the operations. Use short-lived tokens to reduce the impact in case of secrets gets exposed. Has great APIs you can leverage to automate processes.

Vault takes their business field seriously. Its architecture is a little complex, so let's quickly overview the components and explain them in simple terms.

Secrets Paths

Inside Vault, secrets kept on a file-system like path. Every secret is structured as a JSON object, with key-value pairs. These files are versioned and keep their history (refer to KV secrets engine)

Policies

Vault uses policies to define fine-grained access-control. A policy is made of a list of paths (with regex support). A policy is then attached to the token and determines what secrets a token can access and what actions it can perform (create, read, update, delete).

Authentication and authorization

We use the term authentication to refer to the first phase of the login process. For example, the user provides the correct username and password to authenticate. If the credentials match, the user is authenticated.

The authorization part comes next. It defines what an authenticated user has access to.

To perform operations on the Vault, a user needs to authenticate first.
Vault refers to this as authentication backends.
It is a list of authentication methods a user can use to authenticate.

The simplest option is a token, where you perform the login operation using a pre-defined token to gain access. This is a bad idea. There are better options.

Vault has SSO (OIDC) integration. People in the organization can access using their AzureAD or Google accounts. Once you set it up, everyone that has an account can access Vault! You don't need to manage credentials for every person individually.

Once the user logs in, the policies applied to his account or groups in the organization are applied to his token. This defines what he can do inside Vault.

The Flow

Except for token-based authentication, all other methods work the same way. You first provide your credentials (username/password, SSO, etc.) to perform a login action. In exchange, Vault returns a temporary token for you to use in subsequent API calls. The temporary token is defined by its attributes - lifetime, is it renewable, etc.

When using the UI or the CLI, these things happen behind the scene. Yet, if you're using the HTTP API, you'll need to perform login action and grab the token returned by Vault.

This is a new era. Developers can now read and add secrets by themselves and share these with Operations seamlessly. This increases the velocity of both teams and improving the security as a side-effect. For example, when an employee leaves the company, his access is revoked easily. Or the usage of short lived tokens instead of static passwords.

CI/CD pipelines

The deployment platform should also access Vault. During the build and deployment processes, the platform can get the secrets using the approle authentication method. The approle method gives you the option to configure applications access type. This type should have a short time-to-live so it's useless after the deployment process.

We can take it a step further. Now that we have a centralized secrets store with an API, we can build a Jenkins job that generates secrets and push them to the Vault. This formalizes the procedure and prevents humble human mistakes, such as weak passwords.

Increase teams productivity

As the developers' team grows, without an appropriate tool time is going to be wasted here. Nobody likes to wait to test something just because they don't have the password yet. How much time is spent here? This can be solved by.. using the right tool for the job.

Having an interface to read and add secrets would remove the toil of secrets maintenance.

Summary

Using a secrets management tool has numerous benefits, but I presume most of them are unseen. On the surface, it eases the process of adding or updating a secret and managing a unified access to them, by both Dev and Ops. Replacing a secret becomes an easy task for Operations.

It provides all this while keeping high-security standards and auditing. It presents an API to automate the daunting toil of managing them yourself using a less sophisticated mechanism such as files or a database.

It takes your secrets to the next level, and I haven't touched the more advanced capabilities this tool has (dynamic secrets and AWS authentication method, for example). If you are intrigued, go check the documentation.

The new capabilities introduced to automate processes shouldn't be considered lightly.

How do you manage your secrets?

SRE in layman’s terms (4 core concepts)

Chen — Mon, 02 Mar 2020 18:50:03 +0000

There are job titles in the industry that requires prior knowledge in order to understand them. What are their responsibilities are.

I oftentimes find myself try to explain what do I do for living to foreign people to tech industry.

How do you explain SRE then? In this post, I’ll try to describe it in simple terms.

SRE are developers with operations responsibilities. They are in-charge of production environment, to keep it up and running.

Service

A business, by definition, sells a product or a service.

Many of them these days has their business online. You can order almost anything through the internet. Very popular world scale services are Google and Amazon. They are available for you no matter where you are (almost).

You, the client, consumes a service. You use Google to search for interesting stuff or things you need (a nice restaurant). You read the news at your favourite news site or shop online at Amazon.

These companies serves you through the Internet. It seems to be they are online 24/7, 365 days a year. Pause for a second, and think about it. Isn’t that magical? it’s like a store that is always open, but easier to access - to enter the store you don’t need to leave your house.

Now that we have defined what a (online) service is, we can cover the 4 core concepts SRE’s are usually accounted for. I say usually, because responsibilities may be different between companies.

1. Design a reliable and resilient system

What does this mean anyway?

as SRE, we design the infrastructure for the product. We decide which hardware to use, do the capacity planning with room to grow as needed, etc.

One requirement is to make it reliable and resilient so service downtime is minimized as much as possible.
We try to eliminate any single-point of failures a long the road (from hardware to application). Always have redundancy for your infrastructure, so if something fails - be it hardware, network or software - the system can quickly recover from it.

SRE’s knows things break down. They are the ones who gets called when something critical is not working.
It is our job to recognize possible failures along the way and mitigate them ahead of time when that's possible.

2. Monitor and alert

Online services are composed from multiple applications or features. Today, many applications run on distributed systems. We need visibility to what’s going on.

In order to meet this, we use monitoring systems that expose the service’s health. These usually looks like dashboards from control room in the movies.

Image by SpaceX-Imagery from Pixabay

Using these systems we define alerts - for example, we know how to recognize unhealthy patterns in the application, or hardware failures. The alert system sends us notification when things break (by email, sms, phone, etc.), instead of having someone to watch the dashboards all day long and yell :)

3. Pursue product features velocity

Once in a while, these services release new features and security updates. Like a change to the UI or an addition of new buttons. These changes require a software update that happens behind the scene, most of the time without user interruptions.

Changes to the system introduce some risk, but they also introduce new features and bug fixes clients are waiting for. This leads us to -> deployment strategy.

We define a software deployment strategy so software updates (installation of a new software) are to be successful, and when they are not to recover quickly.
Of every change we make to the system, we always keep in mind “how do we recover from this if something goes wrong?”
Then combining the two (software deployment and recovery) procedures into a “playbook”, which can be taught of as a task list to execute. Last, we
automate this to ease the process.

4. Automate everything

This concept is, in my opinion the most important one. Once we formalize a procedure in our daily work, if we repeat it we want to automate it. This allows us to spend our time on more important domains (research, development) rather than doing the task repeatedly. Let’s abstract that.

When we have a “problem” or a “task” on our desk, we prefer to solve it one time only. This is made possible by coding it. So, as SRE’s we always prefer to code things rather than performing them manually, even tough this requires more of our time when solving the problem the first time.

For example, using our monitoring and alert systems, we get notified when things do not work as expected. We can use these to trigger some code that handle the issue. Simple example is, if the application crash (becomes unavailable) automatically start it. It will bring the service back up for customers, while we can debug what happened later.

Sum up - in layman’s terms

The systems reliability - simply make sure the service is online and serving customers. Grow the infrastructure as needed, while keeping things on the budget. A lot of things happen behind the scene to make it like that.
Monitoring - we design, develop and integrate tools that gives us visibility of what’s going on in the system. Multiple graphs and counters that help us to know the status.
Automate everything - this goes without saying. If you have automated a task, you would only spend 'thinking' time on the 'problem' once.

If I had needed to describe SRE role in a short paragraph, it would probably be this:

SRE is responsible for keeping the service up and provide the ability to release software faster while reducing the risk involved with it using tools and deployment strategies. In order to achieve that, we write code 🧞

I hope now on the next occasion you meet someone with SRE title, you would know a little better what their role is all about.

Cover Image by GregoryButler from Pixabay