DEV Community: Nikhil Dev Chunchu

Building with Go in a monorepo using Bazel, Gazelle and bzlmod

Nikhil Dev Chunchu — Sat, 26 Oct 2024 09:02:02 +0000

This post is relevant to any engineer who joined in a small-ish company and stuck around long enough to watch it grow, scale, hire more engineers, form more specialised teams. These are good problems to deal with. By the time I quit Google in 2016, it had mastered the infrastructure of being able to support a very large number of engineering population being able to derive benefits from everything historic to still have the capability to grow perhaps 10x or more. One part of the art that Google had mastered is developing in a monorepo. Google internally uses a repo called "google3" which is where all code (at least the one I was aware of resided). This was the mother of everything tech that Google runs. Right from the ability to commit code, getting reviews, cross team/functions/org dependencies, etc. All this was possible by this amazing build system that Google developed called "blaze" and in March 2015 they made it accessible to everyone outside Google under the name "bazel" which is just an anagram for blaze. So, why all this psycho-babble you ask?

When I joined Wolt in 2020 we still called ourselves a startup who still had full autonomy into how, where and what software we wrong. This meant that we had the freedom to pick the language of our choice, the framework, build system and how we deployed. As engineers we had to deal with a lot of boiler plate by ourself over and over, all the way from managing infrastructure to how the binaries are built. At times we spent more time setting things up for a new service than writing the actual useful code.

Then as we started scaling up came the painful aspect of working with a massive crowd.

How do we share common code between teams?
How do we ensure that service although distributed are testable in one go?
How do we orchestrate releases of feature where there are more than one services involved?
How do we reduce the friction for engineers to work on cross ownership code?
Any many more

So the question is how do we build and environment where adopt all the good aspects of a monorepo but also retain the freedom, safety and speed that microservices provide.

In this post I will try to do my best to document who some of my relearning and hopefully provide a scaffold for anyone else who might be interested.

The following are the versions this post will be working with.

bazel v7.3.1
gazelle v0.39.0
rules_go v0.50.1
go v1.23.1

This is purely for the purpose of the latest with best compatibility. Please feel free to comment if I might be mistaken

There are some simple simple goals we will try to achieve here

Code is well structured despite being beyond team boundaries
Reusability is a key. Anything that could be helpful to other teams should be placed in a common places and relevant definition need to be there to make it usable like a library
You should be able to run any part of the monorepo locally without much knowledge of setting up SDKs, compilers, etc.

This post will not cover how to deploy the binaries.

The name of the repo that I have created for the sake of this post os called"basil". You can call it whatever you want.

Step 1: Starting Bazel 6.3 you do not need to have a WORKSPACE file anymore. So, we will start with creating the following at the root of the repo

MODULE.bazel

module(name = "basil", version = "0.0.0")
http_file = use_repo_rule(
    "@bazel_tools//tools/build_defs/repo:http.bzl", "http_file"
)
http_archive = use_repo_rule(
    "@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive"
)

# https://github.com/bazelbuild/rules_go/blob/master/docs/go/core/bzlmod.md
bazel_dep(name = "rules_go", version = "0.41.0")
bazel_dep(name = "gazelle", version = "0.37.0")

GO_VERSION = "1.23.1"

go_sdk = use_extension("@rules_go//go:extensions.bzl", "go_sdk")
go_sdk.download(version = GO_VERSION)

go_deps = use_extension("@gazelle//:extensions.bzl", "go_deps")
go_deps.from_file(go_mod = "//:go.mod")

use_repo(go_deps, "com_github_google_go_cmp")

The module file does the following

Define the report for rules_go and gazelle
Define the version of go that we will be using for the SDK and the gazelle extensions for the same

Step 2: Then some more boilerplate in the BUILD file

BUILD

load("@gazelle//:def.bzl", "gazelle")

gazelle(name = "gazelle")

gazelle(
    name = "gazelle-update-repos",
    args = [
        "-from_file=go.mod",
        "-to_macro=deps.bzl%go_dependencies",
        "-prune",
    ],
    command = "update-repos",
)

This also ensure that dependencies from go.mod from the role are provided

Now that this is out of way we need to decide what kind of a code organisation we would like to have. For the sake of this post we will assume that all of the code will be in a directory called packages , we have two team who work adjacently called ledgering and invoicing who can share code between themselves as well as from common libs package. This sort of hierarchy is fairly common and extensible.

Step 3: So let's create a directory called packages Next, run the following at the root

bazel run //:gazelle

Gazelle is a build file generator for bazel. Writing these files with dependencies, naming, etc can get quite cumbersome. Gazelle can inspect your code and add the necessary library dependencies it the build file.

Step 4: Next, in the root of the monorepo, run the following to initialise the go.mod file

go mod init basil
go mody tidy

Step 5: Alright, now we are ready to write to actual code. I will start by creating the following directories in the packages directory

libs to hold all common libraries which could be nested if needed as well
invoicing for the first team which can serve as a provider for invoices
ledgering for the second team that can serve as a consumer for invoices

Step 6: Let's start with the common libs directory which can potential hold some formatting functions. Let's create another directory called common in libs.

Step 7: Next in the formatters directory create a file called invoice_id_generator.go and place the following code in that

package libs

import (
    "fmt"
    "math/rand"
)

func GetInvoiceId() string {
    return fmt.Sprint(int64(rand.Int()))
}

Now when you run bazel run //:gazelle on the root, you can see that Gazelle creates a build for you automatically.

load("@rules_go//go:def.bzl", "go_library")

go_library(
    name = "formatters",
    srcs = ["concatinator.go"],
    importpath = "basil/packages/libs/formatters",
    visibility = ["//visibility:public"],
)

Gazelle by default creates a go_library but we will see what else we can create manually later

Step 8: Now that we have something useful let's imagine that ledgering team is requesting the invoicing team to create an invoices which in turn uses a common library from libs to generate the id.

We start by creating a file called invoice.go in the team's root with the following content

package invoicer

import (
    "math/rand"
    "time"
)

type Invoice struct {
    ID        int64
    Amount    float64
    CreatedAt time.Time
}

func GetNewInvoice() *Invoice {
    return &Invoice{
        ID:        int64(rand.Int()),
        Amount:    rand.Float64(),
        CreatedAt: time.Now(),
    }
}

And then another file called booker.go in ledgering with the following contents

package main

import (
    lib "basil/packages/fintech/invoicing"
    "fmt"
)

func main() {
    i := lib.GetNewInvoice()
    fmt.Println("Generate new Invoice with ID:", i.ID)
}

Now when you run blaze run //:gazelle you will notice that gazelle has create the libaries along with the dependencies that the go code uses as well.

Step 9: Now, we want the make the booker.go an executable since it might be invoked as a standalone. Since gazelle has create a binary for us, let's create one with the following contents in the BUILD.bazel. So, your final build file should look like

load("@rules_go//go:def.bzl", "go_binary", "go_library")

go_binary(
    name = "app",
    embed = [":lib"],
    importpath = "basil/packages/fintech/ledgering_app",
    visibility = ["//visibility:public"],
)

go_library(
    name = "lib",
    srcs = ["booker.go"],
    importpath = "basil/packages/fintech/ledgering",
    visibility = ["//visibility:private"],
    deps = ["//packages/fintech/invoicing:lib"],
)

Step 10: Now its time to take this whole thing for a spin. You can build and run the binary as shows below

bazel build //packages/ledgering:app
bazel run //packages/ledgering:app

If all went fine, you should see something like below

INFO: Analyzed target //packages/ledgering:app (1 packages loaded, 5 targets configured).
INFO: Found 1 target...
Target //packages/ledgering:app up-to-date:
  bazel-bin/packages/ledgering/app_/app
INFO: Elapsed time: 0.747s, Critical Path: 0.52s
INFO: 4 processes: 1 internal, 3 darwin-sandbox.
INFO: Build completed successfully, 4 total actions
INFO: Running command line: bazel-bin/packages/ledgering/app_/app
Generate new Invoice with ID: 7213007727604049262

And there you are. You no longer have to create individual repositories for each project that you might be working on but instead use same repo and libraries across projects. I will probably write a follow up post to this on how you can build binaries through CI/CD for this setup.

Please do leave comments in case I have mispoken or misunderstand anything. You can find all the code I have demoed above in my repo

Self hosting ghost blog on your home server

Nikhil Dev Chunchu — Thu, 24 Oct 2024 12:16:09 +0000

If you are reading this post, then you need to be aware that this post is at the moment running on my own server at home.

Ghost has one of most beautiful interface for both writing and reading posts. The only downside that I found was that to leverage the full benefit I had to pay up. Of course Wordpress is an option to self host but only recently did I become aware that Ghost can be self hosted as well and has docker images that I could use.

In this post I will try my best to explaining how I got mine up and working

Prerequisites

I will assume that you have the following already set up

A linux server
Docker installed within it
Cloudflare tunnel or some way of exposing your service to the outside world

Let's start with the docker-compose.yaml

services:
  ghost:
    image: ghost:latest
    container_name: ghost-server
    environment:
      - url=https://blog.nikhildev.com
      - mail__transport=SMTP
      - mail__options__host=smtp.eu.mailgun.org
      - mail__options__port=587
      - mail__options__auth__user=<your_auth_user_email>
      - mail__options__auth__pass=<the_password_from_mailgun>
      - mail__from=blog@example.com #This probably will need verification
      - database__client=mysql
      - database__connection__host=db # Name of the service from below
      - database__connection__user=ghost
      - database__connection__password=ghost # You can choose a better password
      - database__connection__database=ghost
    volumes:
      - /docker_volumes/ghost/content:/var/lib/ghost/content # I prefer to map a different location for the safe keep of the data. The /docker_volumes is mirrored on zpool for redundancy
    restart: unless-stopped
    depends_on:
      - db
    ports:
      - "2368:2368"
  db:
    image: mysql:9
    container_name: ghost-db
    restart: always
    networks:
      - default
    volumes:
      - /docker_volumes/ghost/db/data:/var/lib/mysql
    environment:
      - MYSQL_DATABASE=ghost
      - MYSQL_USER=ghost
      - MYSQL_PASSWORD=ghost
      - MYSQL_ROOT_PASSWORD=ghost
      - MYSQL_ROOT_HOST=172.*.*.* ## Ensures the Docker network has access

In the above docker compose files we are defining two service. ghost-server is the main application and db is a mysql instance that it would be using.

Now, if you want the newsletter part of ghost be available to you, I would highly recommend signing up with something like Mailgun which has pretty generous free tier. Once you have signed up, use the generated credentials in ghost. Here is a great article on it https://brightthemes.com/blog/ghost-mailgun-config

I'm sticking to exposing the default port of the service as in my entire stack there are no conflicting ones.

The db service is vanilla mysql without any bells and whistles. The official guide suggests using mysql 8 but I've tried this with mysql 9 and it works just fine. We simply are setting the credentials before hand to be made accessible to the ghost instance. Even here I prefer to mount a redundant local directory just for safe keeps.

And that's it. Go ahead and run the container with docker compose up -d and run through the setup process. If you run into any issues, please leave a comment and I can probably try and help you out

A practical example of shared libraries in a monorepo

Nikhil Dev Chunchu — Thu, 24 Oct 2024 08:48:44 +0000

One of the most powerful aspects of working in a monorepo is the ability to share code between packages/teams/hierarchies. In this post I will try to explain a very simple real world scenario

Example Scenario

Imagine you want to develop a library to show the file sizes in megabytes which you feel might be useful to other parts of your monorepo. The library accepts the size as Integer (ex: 2048 bytes) and can return a humanized string (ex: 2 MB). To add some quality assurance, we will also write a test for the same.

How does Bazel enable sharing code?

From the above scenario we are aware that we need to develop this function as a shared library which then be imported by another package for usage. Bazel makes this extremely simple by allowing us to define the function in a library and export it other services that will need it. As explained in my earlier post linked at the bottom of this post, we can also control which other libraries can be allowed to import it for usage as well.

Let's get coding

For code organization purpose, we will have a libraries directory at the root of our workspace with a child directory called humanize_filesize which is where we will write our library code.

Let's write some very elementary Go code in humanize_filesize.go

package humanize_filesize

import "fmt"

// GetHumanizedFilesize takes size_in_bytes as an int32 pointer and returns the size in megabytes.
func GetHumanizedFilesize(size_in_bytes *int32) string {
    if size_in_bytes != nil {
        size_in_megabytes := float64(*size_in_bytes) / (1024 * 1024)
        return fmt.Sprintf("%.4f MB", size_in_megabytes)
    }
    return "0 MB"
}

This code simply takes an int32 as an input and returns a computed readable megabyte string to 4 decimal precision

This function is definitely not comprehensive and can definitely be improved, but that's not the point of this exercise.

Also assert that our logic is working as intended, we will add a very elementary test alongside our go code in a file called humanize_filesize_test.go

package humanize_filesize

import (
    "testing"
)

func TestHumanizeFilesize(t *testing.T) {
    tests := []struct {
        name          string
        size_in_bytes *int32
        expected      string
    }{
        {
            name:          "nil bytes",
            size_in_bytes: nil,
            expected:      "0 MB",
        },
        {
            name:          "2048 bytes",
            size_in_bytes: int32Ptr(2048),
            expected:      "0.0020 MB",
        },
        {
            name:          "0 bytes",
            size_in_bytes: int32Ptr(0),
            expected:      "0.0000 MB",
        },
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            result := GetHumanizedFilesize(tt.size_in_bytes)
            if result != tt.expected {
                t.Errorf("expected %s, got %s", tt.expected, result)
            }
        })
    }
}

func int32Ptr(n int32) *int32 {
    return &n
}

A very simple test with basic tests for nil, int32 and 0 as inputs

Now comes the juicy part of how to export this function so that this can be imported within other packages or services. This is where we have to define the BUILD.bazel file.

load("@rules_go//go:def.bzl", "go_library", "go_test")

go_library(
    name = "humanize_filesize",
    srcs = ["humanize_filesize.go"],
    importpath = "basil/libraries/humanize_filesize",
    visibility = ["//visibility:public"],
)

go_test(
    name = "humanize_filesize_test",
    srcs = ["humanize_filesize_test.go"],
    embed = [":humanize_filesize"],
)

In here we are defining two main rules. One for the actual library and one for the test file that we wrote.

The go_library defines that the target humanize_filesize uses humanize_filesize.go as one of its sources which can be imported by the path specified in importpath and it is visible publicly within the workspace for other packages to import. We will learn how to control the visibility in a future post.

The go_test defines a test target which embeds the code from the output of go_library.

At this point we should be able to test the library by running our test suite as following

bazel build //... && bazel run //libraries/humanize_filesize:humanize_filesize_test

You should be able to see the test output as following indicating that all the tests have passed.

INFO: Analyzed target //libraries/humanize_filesize:humanize_filesize_test (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //libraries/humanize_filesize:humanize_filesize_test up-to-date:
  bazel-bin/libraries/humanize_filesize/humanize_filesize_test_/humanize_filesize_test
INFO: Elapsed time: 0.392s, Critical Path: 0.24s
INFO: 5 processes: 1 internal, 4 darwin-sandbox.
INFO: Build completed successfully, 5 total actions
INFO: Running command line: external/bazel_tools/tools/test/test-setup.sh libraries/humanize_filesize/humanize_filesize_test_/humanize_filesize_test
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //libraries/humanize_filesize:humanize_filesize_test
-----------------------------------------------------------------------------
PASS

🎉 Wohoo!!! 🥳 Now we know that our library is working as intended.

Now let's use this library in a service service1 within a services directory that we will create at the root of the workspace with the following go code and BUILD.bazel file.

service1.go

package main

import (
    "basil/libraries/humanize_filesize"
    "fmt"
    "math/rand"
)

func main() {
    v := rand.Int31n(1000000)
    fmt.Printf(`%d bytes = %s\n`, v, humanize_filesize.GetHumanizedFilesize(&v))
}

BUILD.bazel

load("@rules_go//go:def.bzl", "go_binary", "go_library")

go_library(
    name = "service1_lib",
    srcs = ["service1.go"],
    importpath = "basil/services/service1",
    visibility = ["//visibility:private"],
    deps = ["//libraries/humanize_filesize"],
)

go_binary(
    name = "service1",
    embed = [":service1_lib"],
    visibility = ["//visibility:public"],
)

The go code is pretty simple which imports our library that we declared earlier and uses the GetHumanizedFilesize function from our library and passes a random integer value and prints the output.

Now when execute bazel build //services/service1 , bazel will resolve all dependencies for our target including the library that we developed and build them.

service1 can now be executed using bazel run //services/service1 since we have only one binary target defined. If you have more than one binary targets, ex: serviceX, you can execute that using bazel run //services/service1:serviceX. By default when not specifying a target, bazel will always try to find a binary target with the same name as the directory and run that.

So... there you go. We have made your first shared library that can be used by other parts of our monorepo.

All code for this example can be found at https://github.com/nikhildev/basil/pull/3/commits/61c673b8757860bd5e60eb2ab6c35f3f4da78c87

If you like the content of this post feel free to share it. Also, please subscribe and leave comments on what you think about this post and if there are things that you would like to see me improving on.

Simple hello world program using Bazel and Go lang

Nikhil Dev Chunchu — Thu, 24 Oct 2024 08:39:43 +0000

After writing the post on "Building with Go in a monorepo using Bazel, Gazelle and bzlmod" and sharing with some colleagues, I saw some growing interest in monorepo development. I learnt that not many people were still experienced enough to understand the benefits it could bring. So I decided to convert this to a series starting with this post on Simple hello world program using Bazel and Go lang

In this post, I will try to cover some extremely basic concepts of Bazel along with code snippets to get someone up and going in no time.

What is Bazel?

According to the official documentation

Bazel is an open-source build and test tool similar to Make, Maven, and Gradle. It uses a human-readable, high-level build language. Bazel supports projects in multiple languages and builds outputs for multiple platforms. Bazel supports large codebases across multiple repositories, and large numbers of users.

It has been in use at Google for decades and is thoroughly battle tested. You can read more about how I became aware of this on the post linked above.

Step 1: Setting up repository

For the purpose of this series I have created a repository at github.com/nikhildev/basil which will evolve over time. At the time of writing this post, the latest commit is https://github.com/nikhildev/basil/tree/d8af2aea6fb8b692f735f9959429df9fcd28422b

So, go ahead an create a new git repo on any provider you want to choose

For the .gitignore I highly recommend adding the following so as to not include unnecessary files in your commits

# If you prefer the allow list template instead of the deny list, see community template:
# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
#
bazel-*
/.ijwb
/.clwb
/.idea
/.project
*.swp
/.bazelrc.user

# macOS-specific excludes
.DS_Store

# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib

# Test binary, built with `go test -c`
*.test

# Output of the go coverage tool, specifically when used with LiteIDE
*.out

# Dependency directories (remove the comment below to include it)
# vendor/

# Go workspace file
go.work
go.work.sum

# env file
.env

Step 2: Adding some Bazel boilerplate

Starting Bazel 6.3 you do not need to have a WORKSPACE file anymore. So, we will start with creating the following at the root of the repo

MODULE.bazel

"""Building go applications in a monorepo environment"""

module(name = "basil", version = "0.0.0")
http_file = use_repo_rule(
    "@bazel_tools//tools/build_defs/repo:http.bzl", "http_file"
)
http_archive = use_repo_rule(
    "@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive"
)

# https://github.com/bazelbuild/rules_go/blob/master/docs/go/core/bzlmod.md
bazel_dep(name = "rules_go", version = "0.50.1")
bazel_dep(name = "gazelle", version = "0.39.1")

GO_VERSION = "1.23.2"

go_sdk = use_extension("@rules_go//go:extensions.bzl", "go_sdk")
go_sdk.download(version = GO_VERSION)

go_deps = use_extension("@gazelle//:extensions.bzl", "go_deps")
go_deps.from_file(go_mod = "//:go.mod")

Here is where we set the go version we want to use as well as the gazelle and rules_go version.

We will be using Gazelle for BUILD file management. Gazelle makes generating build file quite convenient. You can read more about it here

BUILD.bazel

load("@gazelle//:def.bzl", "gazelle")

gazelle(name = "gazelle")

gazelle(
    name = "gazelle-update-repos",
    args = [
        "-from_file=go.mod",
        "-to_macro=deps.bzl%go_dependencies",
        "-prune",
    ],
    command = "update-repos",
)

This should be at the root of the repo. This instructs gazelle on the source of the go mod file and other processes to run

Next we create 3 more files with the respective contents. Don't worry about what these do for now.

.bazelignore

runfiles/
bzlmod/

.bazelrc

# Enable Bzlmod for every Bazel command
common --enable_bzlmod

.bazelversion

7.3.1

Alright, by this point you should have everything that's needed to get some basics working. Now if you run bazel build //... at the root, bazel should be able traverse through the repo and build any packages that it discovers. Bazel caches package outputs from earlier builds, so any subsequent build from here on should be extremely fast.

Next up, let's get cracking on that actual hello world program.

Step 3: Writing a helloworld package

For the basic organisation of code we will be writing all the Go code in a directory called /packages. This way any references in other parts of the code can refer to it as //packages/...

Let's create a directory in the packages directory called helloworld and add the following

helloworld.go

package main

import (
    "fmt"
)

func main() {
    fmt.Println("Hello World!")
}

BUILD.bazel

load("@rules_go//go:def.bzl", "go_binary", "go_library")

go_binary(
    name = "helloworld",
    embed = [":helloworld_lib"],
    importpath = "basil/packages/helloworld",
    visibility = ["//visibility:private"],
)

go_library(
    name = "helloworld_lib",
    srcs = ["helloworld.go"],
    importpath = "basil/packages/helloworld_lib",
    visibility = ["//visibility:private"],
)

The go program is fairly simple. We have a simple main function that simple prints a hello world message.

The bit we are interested in is the BUILD.bazel file. Let's go over this block by block and try to understand what it means.

The first line loads the go_binary and go_library macros from rules_go . You will find these being used later in the code.

In line 10 we are defining a library called helloworld_lib and specify the sources of the library as helloworld.go. If this package needs to be me made importable, then you can also specify the path it will be available at which is basil/packages/helloworld_lib. Then comes the visibility and here we are specifying that the library is private only visible within this package. In the future posts we might look into how to change these parameters to use dependencies from other packages.

In line 3 we then use the go_binary macro from rules_go to generate a binary for our program. Here we specify the library that we have defined earlier in the embed parameter. The package visibility applies to binaries as well.

Step 4: Build and Run

And that's it. Voila! You can run the binary by first building it using bazel build //packages/helloworld followed by bazel run //packages/helloworld

If you like this post and would like to get future posts that build upon this as a part of the series, please subscribe and share this post.