DEV Community: Clavin June

Update on My Mistake on Converting Slice to Slice of Ptr in Golang

Clavin June — Sat, 17 Feb 2024 07:00:00 +0000

Photo by @iavnt on Unsplash

Three years ago I wrote on how I wrongly implementing slice to slice pointer converter. It was not trivial for me, not really sure whether it is related to how Go implement their for loop.

I'm trying to patch the previous article since Go just released their latest version (1.22.0) and have changes on how the language itself implementing for loop, you can check it out here.

Now, if we try to implement our code from the last article using go1.22.0, we can see that the result is correct since they now no longer mutating the variable. The variable is not shared between each loop.

package main

import "fmt"

func Slice2SliceOfPtr(slice []int) []*int {
    n := len(slice)
    r := make([]*int, n, n)

    for i, s := range slice {
        r[i] = &s
    }

    return r
}

func main() {
    x := []int{1, 2, 3}
    y := Slice2SliceOfPtr(x)

    fmt.Println(x)

    for _, yy := range y {
        fmt.Printf("%d ", *yy)
    }
}

Before go1.22.0:

$ go run main.go 
[1 2 3]
3 3 3

In go1.22.0:

$ go run main.go
[1 2 3]
1 2 3

Thank you for reading!

Golang Structured Logging Introduction

Clavin June — Sun, 28 Jan 2024 05:00:00 +0000

I was doing sharing session this week related to structured logging and how we can utilize it in my team at works. My team has already used Zerolog, but I guess it would be no harm if we update the knowledge regarding a new library more over, because it is Golang standard library which we don't need to depend to a 3rd party library. Here's the presentation I created.

Introduction

Golang has introduced their first structured logging library when they released go 1.20.x as an experimental library. Then they included it as their standard library in the release of 1.21.x. The API of the structured logging is quite easy to understand it is similar to the popular structured logging library out there like Zerolog or Logrus.

Code Example

Golang log/slog is integrated with log library as its default logger, but you can also create a new text/JSON logging handler or create your own handler by just implementing its interfaces. Below is the code example on how to use log/slog:

var (
    textLogger *slog.Logger = slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{}))
    jsonLogger *slog.Logger = slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{}))
)

func main() {
    slog.Info("this is info using default logger")
    textLogger.Warn("this is warning", slog.Int("response_code", http.StatusNotFound))
    jsonLogger.Error("this is error",
        slog.String("status", http.StatusText(http.StatusInternalServerError)),
        slog.Bool("is_json", true),
    )
}

slog.Info/Warn/Error means that you will send an info/warn/error level logs, followed with the message and the attributes (slog.String/Bool/Int). The output of code above will be like:

2009/11/10 23:00:00 INFO this is info using default logger
time=2009-11-10T23:00:00.000Z level=WARN msg="this is warning" response_code=404
{"time":"2009-11-10T23:00:00Z","level":"ERROR","msg":"this is error","status":"Internal Server Error","is_json":true}

Allocations to be Concerned

The log/slog API is very flexible that you can put your log attributes as a key/pair variadic arguments, but please be aware that it might introduce extra allocations:

func main() {
    slog.Info("❌ have more allocations since we don't define the type",
        "key", "value",
        "key2", time.Minute,
    )
    slog.LogAttrs(context.Background(), slog.LevelInfo,
        "✅ avoid extra allocations",
        slog.String("key", "value"),
        slog.Duration("key2", time.Minute),
    )
}

Output:

2009/11/10 23:00:00 INFO ❌ have more allocations since we don't define the type key=value key2=1m0s
2009/11/10 23:00:00 INFO ✅ avoid extra allocations key=value key2=1m0s

Both code above will output the same thing, the only difference is you don't create the attributes using slog.LogAttrs. Here's the benchmark result:

benchstat keyval.txt attr.txt
name     old time/op    new time/op    delta
Slog-10    1.39µs ± 1%    1.45µs ± 2%   +4.37%  (p=0.000 n=10+9)

name     old alloc/op   new alloc/op   delta
Slog-10      734B ± 2%      676B ± 1%   -7.90%  (p=0.000 n=10+9)

name     old allocs/op  new allocs/op  delta
Slog-10      7.00 ± 0%      5.00 ± 0%  -28.57%  (p=0.000 n=10+10)

Derive Logger from another Logger

You can also derive a logger from one logger and keep the atrributes, let's say you have a complex nested struct where you want to keep atrributes from parent struct inside their child struct.

func main() {
    logger2 := slog.With(slog.Int("num", 3))

    logger2.Info("whenever logger2 is called, you can see the num attribute derived")
}

The output will be similar to this:

2009/11/10 23:00:00 INFO whenever logger2 is called, you can see the num attribute derived num=3

Logger Group

You can also group logger attributes so you could know the scope of those attributes

func main() {
    foobarLogger := slog.
        With(slog.String("non_grouped", "value")).
        WithGroup("grouped")

    foobarLogger.Info("from foboar",
        slog.Any("args", []string{"a", "b"}),
        slog.Bool("ok", true),
    )
}

Output:

2009/11/10 23:00:00 INFO from foboar non_grouped=value grouped.args="[a b]" grouped.ok=true

The Most Efficient Way to Use Slog

here's the recommended way on how to use slog:

func main() {
    slog.LogAttrs(context.Background(), // context
        slog.LevelInfo,          // level
        "your message",          // message
        slog.Float64("key", 10), // keys
        slog.Group("latency", // grouping keys
            slog.Duration("duration", time.Millisecond),
            slog.Time("start", time.Now()),
        ),
    )
}

Use LogAttrs to avoid extra allocations. ref
Send context in case your handler handles the context (tracing, metrics, etc)
Define the slog.Level
Define the message
Put contextual attributes
Group attributes together if needed

Use Case

Dynamic Log Level

Dynamic log level means you can change your log level at runtime level without needing any re-deployment

var (
    level  = new(slog.LevelVar)
    logger = slog.New(slog.NewTextHandler(
        os.Stdout, &slog.HandlerOptions{Level: level}))
)

func main() {
    logger.Debug("won't be shown") // by default log level is INFO
    level.Set(slog.LevelDebug)     // change it to DEBUG
    logger.Debug("shown")
}

Output:

time=2009-11-10T23:00:00.000Z level=DEBUG msg=shown

Redact Data

You can manipulate the atrributes value by utilizing ReplaceAttr inside the slog.HandlerOptions struct.

var (
    logger = slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{
        ReplaceAttr: func(g []string, a slog.Attr) slog.Attr {
            if "password" == a.Key {
                a.Value = slog.StringValue("[PII DATA]")
            }
            return a
        }
    }))
)

func main() {
    logger.Info("password will be redacted",
        slog.String("password", "P@$$w0rd?!"))
}

Output:

time=2009-11-10T23:00:00.000Z level=INFO msg="password will be redacted" password="[PII DATA]"

Custom Handler

By default, log/slog provides you with three handler, the default, text, and JSON. You can create your own custom handler by just implementing their interfaces.

type CustomHandler struct {
    slog.Handler
}

func (c *CustomHandler) Handle(ctx context.Context, r slog.Record) error {
    // handle context here for tracing, monitoring, logging, etc
    return c.Handler.Handle(ctx, r)
}

Here's the reference on how to do it. Perhaps I'll cover this in another article.

Thank you for reading!

Golang Testing HTTP Handler

Clavin June — Sun, 31 Dec 2023 12:52:21 +0000

FaFiFu

This section is a bullshittery, you can skip it to the main section

It's been a year since my last blog post, 2023 is quite a year that makes me hard to spend time updating this blog. The thing is, going back working from the office take hours of productive hour. Going to the office preparation, commuting, chit-chat, snack time makes me less productive. Nah, that's maybe the me issue, just trying to blame others because it's easier :D. Just hoping that I will have enough time to update this blog more frequent in 2024. Enough of the chit-chat, let's go to the main section.

Introduction

When you start working with Golang as a web server and developing some HTTP endpoints, you might want to test it as well. In Golang, in order to test the HTTP endpoints, it has a specific library called httptest. I also have created one blog post that highlight one specific use case of httptest related to mocking external HTTP call, you can read it here.

Components

To create a unit test for testing HTTP endpoints, we can utilize httptest.NewRecorder and httptest.NewRequest. If you remember, golang basic http.Handler looks like below:

func FooHandler(w http.ResponseWriter, r *http.Request) {}

It needs both http.ResponseWriter and http.Request, which the value of both of them will be provided by http.Server whenever a new request is coming to the handler. In testing, instead of running an http.Server that will provide the value, you can fill the argument using httptest.NewRecorder and httptest.NewRequest functions return value since both of the function consecutively return http.ResponseWriter and http.Request.

Code

Now let's go to the code. Let's say you have this handler that you want to test:

package handler

import (
    "fmt"
    "net/http"
)

func RootHandler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "[%s] %s OK!", r.Method, r.URL.String())
}

It's quite a simple handler, let's create a test for RootHandler:

package handler_test

import (
    "fmt"
    "io"
    "net/http"
    "net/http/httptest"
    "testing"

    "github.com/clavinjune/example/handler"
)

func TestRootHandler(t *testing.T) {
    method := http.MethodPost
    endpoint := "/foo"

    w := httptest.NewRecorder()
    r := httptest.NewRequest(method, endpoint, http.NoBody)
    defer r.Body.Close()

    handler.RootHandler(w, r)
    defer w.Result().Body.Close()

    if w.Result().StatusCode != http.StatusOK {
        t.Fatalf("want: %03d, got: %03d", http.StatusOK, w.Result().StatusCode)
    }

    b, err := io.ReadAll(w.Result().Body)
    if err != nil {
        t.Fatalf("want: nil, got: %+q", err)
    }

    want := fmt.Sprintf(`[%s] %s OK!`, method, endpoint)
    got := string(b)

    if want != got {
        t.Fatalf("want: %+q, got: %+q", want, got)
    }
}

As you see the example above, we called handler.RootHandler(w, r) where the argument is filled by the httptest.NewRecorder and http.NewRequest. The recorder, will record whatever the response of the handler, you can read the body, status code, header, etc by only using w.Result(). Meanwhile the request will create a similar request like http.NewRequest with some hard-coded fields since httptest.NewRequest creates an incoming server request not like http.NewRequest that creates a client HTTP request which are usually used in http.Client{}.Do function, and that's it. Now you have the unit test for your HTTP handler.

Thank you for reading!

Fetch Current Script Tag Attributes in Javascript

Clavin June — Fri, 25 Nov 2022 12:23:00 +0000

Introduction

Javascript is a language with thousands of users and modules. Every single thing that you think you may not want, it's there. Starts from a simple module like is-odd, jquery, until svelte. Javascript is a native way to do scripting in browser. Sometimes, you may need to create simple javascript module for your day to day works or maybe just for fun. To import your plain old javascript inside your HTML project, you can simply use the OG <script> tag. But, do you know that you can put custom attributes and access it inside your script?

Example

For example, let's take a look at this giscuss.app script tag:

<script src="https://giscus.app/client.js"
        data-repo="[ENTER REPO HERE]"
        data-repo-id="[ENTER REPO ID HERE]"
        data-category="[ENTER CATEGORY NAME HERE]"
        data-category-id="[ENTER CATEGORY ID HERE]"
        data-mapping="pathname"
        data-strict="0"
        data-reactions-enabled="1"
        data-emit-metadata="0"
        data-input-position="bottom"
        data-theme="preferred_color_scheme"
        data-lang="en"
        crossorigin="anonymous"
        async>
</script>

Line 2 until 12 (data-repo until data-lang) are called dataset. It's a custom attributes that you can put to HTML elements. As long as you use data- as a prefix, it's considered as dataset. You can also put your custom elements/attributes without data- prefixed, but it's just an attribute, not dataset. Now, how do you fetch the attributes value?

Code

Let's say you have index.js script that you import to your code like below:

<script
    src="./index.js"
    data-attr1="val1"
    data-attr2="val2"
    attr3="val3"
    attr4="val4">
</script>

To access data-attr1, data-attr2, attr3, and attr4 you can do this inside index.js:

// fetch current script tag that import this script
const script = document.currentScript

const dataset = script.dataset
const attr1 = dataset.attr1
const attr2 = dataset.attr2

const attributes = script.attributes
const attr3 = attributes.getNamedItem('attr3').value
const attr4 = attributes.getNamedItem('attr4').value

As for the dataset, the data- prefix will be removed. But for the rests, it is considered as common attributes just like src.

Conclusion

Now you can get your user input from attributes, just think of it like function arguments. It's simple and straightforward. Although you can choose between 2 methods above, I recommend you to use the dataset since it is more semantic than just a plain attribute.

Thank you for reading!

Learning PromQL by Exercising

Clavin June — Thu, 20 Oct 2022 15:30:55 +0000

Hi, I just want to ask is there any web / resources to learn about promQL by use cases / tasks?

e.g. "create a query to fetch the max requests number from the defined timerange"

thank you in advance

Working with Multiple Gitconfig

Clavin June — Sat, 08 Oct 2022 04:57:59 +0000

Introduction

This article will be a quick tips if you're using multiple git config inside one local machine. For example if you're working on both Gitlab/Bitbucket/Github with different email/username/gpgsign, or you're working on your personal and work git account on the same machine.

Setup Gitconfig Directory

This directory name would be anything, but for the sake of naming convention let's call it ~/.gitconfig.d/.

$ mkdir -p "$HOME/.gitconfig.d/" && cd "$_"
$ pwd
/Users/clavianus.juneardo/.gitconfig.d

Setup Each Gitconfig

Now you have create the directory, let's say you want to set your personal and work account:

Setup Personal Gitconfig

$ git config --file=personal user.name "foo"
$ git config --file=personal user.email "foo@gmail.com"
$ git config --file=personal core.editor "vim"
...

By using --file=personal flag, the gitconfig will be configured inside personal file.

$ cat personal
[user]
    name = foo
    email = foo@gmail.com
[core]
    editor = vim

Now let's do the same with your work account.

Setup Work Gitconfig

$ git config --file=work user.name "foo bar"
$ git config --file=work user.email "foo@company.com"
$ git config --file=work user.signingKey "ABCDEF012345"
$ git config --file=work commit.gpgsign true
$ git config --file=work core.editor "vim"
...

$ cat work
[user]
    name = foo bar
    email = foo@company.com
    signingKey = ABCDEF012345
[commit]
    gpgsign = true
[core]
    editor = vim

Now you have set both your account. Then, how to switch between each gitconfig?
Let's say you put all your company's git directory at ~/Works and yours at ~/Personals. You can switch easily by configure the global gitconfig using includeIf.

cat <<EOF > ~/.gitconfig
[includeIf "gitdir:~/Works/"]
  path = ~/.gitconfig.d/work
[includeIf "gitdir:~/Personals/"]"
  path = ~/.gitconfig.d/personal
EOF

Conclusion

Now everytime you're inside ~/Works/ you are using the ~/.gitconfig.d/work, and when you're inside ~/Personals/ you are using the ~/.gitconfig.d/personal.

You can check whether the gitconfig load properly or not by simply executing git config user.email command and see what email is showing up.

Thank you for reading!

Ways to Define Custom Command-Line Flags in Golang

Clavin June — Sat, 03 Sep 2022 18:18:05 +0000

Introduction

If you're familiar with command-line interfaces, you might have seen flags like -h, -v, --name foo, etc. Some of the flags use primitive data types like string, int, bool, etc. The rest of the flags are custom flags that using specific patterns or data structure like IP, URL, Array, Map, etc. Golang provides a built-in library called flag to define command-line flags. Some of the provided functions to define flags are straight forward like flag.String, flag.Int, etc. But, how about the custom flags? In this article, you will learn some ways to define custom command-line flags in Golang.

Example

Let's say you want to create a custom flags that accepts a list of string values. The usage of the flag should be like this:

$ go run main.go --list a --list b,c,d
[a b c d]

Below are some ways how to define the custom flag.

Flag Var

flag.Var is a way to define a custom flag. It accepts a custom struct that implements flag.Value interface.

type ListFlag []string

func (f *ListFlag) String() string {
    b, _ := json.Marshal(*f)
    return string(b)
}

func (f *ListFlag) Set(value string) error {
    for _, str := range strings.Split(value, ",") {
        *f = append(*f, str)
    }
    return nil
}

func main() {
    // foo and bar as a default value
    list := ListFlag([]string{"foo", "bar"})
    flag.Var(&list, "list", "your flag usage")

    flag.Parse()
    fmt.Println(list) // [foo bar a b c d]
}

The important thing to note is the Set method. Everytime your flag is called, the Set method will be called.

Flag Func

flag.Func is a straight forward way to define a custom flag. It uses flag.Var under the hood so you don't need to create a custom struct.

func main() {
    list := ListFlag([]string{"foo", "bar"})
    flag.Func("list", "your flag usage", func(s string) error {
        for _, str := range strings.Split(s, ",") {
            list = append(list, str)
        }
        return nil
    })

    flag.Parse()
    fmt.Println(list) // [foo bar a b c d]
}

As you see, the last parameter of flag.Func is a Set method you implemented in the previous example.

Flag TextVar

flag.TextVar is a new way introduced in go1.19. It uses flag.Var under the hood like flag.Func, but it accepts a encoding.TextUnmarshaler and encoding.TextMarshaler interface instead of flag.Value interface. It means you can use built-in struct like big.Int, netip.Addr, and time.Time as flags without needed to implement a custom struct.

type ListFlag []string

func (f *ListFlag) MarshalText() ([]byte, error) {
    return json.Marshal(*f)
}

func (f *ListFlag) UnmarshalText(b []byte) error {
    for _, str := range strings.Split(string(b), ",") {
        *f = append(*f, str)
    }
    return nil
}

func main() {
    list := ListFlag([]string{"foo", "bar"})
    flag.TextVar(&list, "list", &list, "your flag usage")

    flag.Parse()
    fmt.Println(list)
}

As you see, it's similar to flag.Var but it uses MarshalText and UnmarshalText instead of String and Set method.

Conclusion

Now you know some ways to define custom command-line flags in Golang. If you want to learn more about flag package, you can read the official documentation here.

Thank you for reading!

Handle Database Credential Rotation in Golang

Clavin June — Mon, 29 Aug 2022 09:30:58 +0000

Introduction

Many of you might have heard of the database credential rotation. It is a common solution for securing your database. Many providers have implemented this solution such as Hashicorp Vault, AWS Secret Manager, etc. By automatically rotating your database credential, no one can really know how to connect to the database. Indeed it is secure, but it leads to another problem. By rotating your database credential, your application might not be able to connect to the database after some times. Especially when you declare your database maximum connection lifetime. In this article, you will learn how to rotate your database credential programmatically in golang.

Custom Driver

To implement this rotator as a generic solution, you will need to create a customDriver struct which holds your base database driver (e.g postgres, sqlite3, mysql, etc) and a Fetcher interface which will be used to fetch the database credential.

type Fetcher interface {
    Fetch() (string, error)
}

type FetcherFunc func() (string, error)
func (f FetcherFunc) Fetch() (string, error) {
    return f()
}

// customDriver implements the `sql.Driver` interface.
type customDriver struct {
    // base is the base database driver
    base      driver.Driver

    // fetcherFn is the function that will be used to fetch the database credential
    fetcherFn FetcherFunc
}

func (d *customDriver) Open(_ string) (driver.Conn, error) {
    // fetch the database credential
    dsn, err := d.fetcherFn()
    if err != nil {
        return nil, err
    }

    // open the database connection using the fetched credential and base driver
    return d.base.Open(dsn)
}

Register the Driver and Open the Connection

Now to be able to use the driver, you need to register the driver and open the connection.

func OpenWithRotator(name string, base driver.Driver, fetcher Fetcher) (*sql.DB, error) {
    sql.Register(name, &customDriver{
        base:      base,
        fetcherFn: fetcher.Fetch,
    })

    // you don't need to fill the dsn, it will be fetched from the fetcher.
    return sql.Open(name, "")
}

Implement Fetcher Interface

Let's implement the Fetcher interface by using a simple function. Let's assume that you want to connect to a different database for every time your database lifetime is up.

var counter int

func simpleFetcher() (string, error) {
    log.Println("fetcher called")

    // add your custom logic
    // e.g. fetching from vault / config / etc.

    counter++
    return fmt.Sprintf("file:foobar-%d.sqlite", counter), nil
}

Open the Connection with Rotator

Open and set the maximum lifetime of the connection to 2 seconds. simpleFetcher will be called every 2 seconds.

If your database credential is rotated faster than your connection lifetime, Golang still can use the old connection.

// you can adjust the `&sqlite3.SQLiteDriver{}` accordingly (e.g. &pq.Driver{}, etc.)
db, err := OpenWithRotator("foobar", &sqlite3.SQLiteDriver{}, FetcherFunc(simpleFetcher))
if err != nil {
    log.Fatal(err)
}
defer db.Close()
db.SetConnMaxLifetime(2 * time.Second)

Test the Connection

To simply test whether the connection is working, you can use the Ping method every second.

for range time.Tick(time.Second) {
    if err := db.Ping(); err != nil {
        log.Fatal(err)
    }
}

Now when you run the program, you will see the following output:

$ go run .
2022/08/29 16:56:16 fetcher called
2022/08/29 16:56:19 fetcher called
2022/08/29 16:56:22 fetcher called
2022/08/29 16:56:25 fetcher called
2022/08/29 16:56:28 fetcher called
2022/08/29 16:56:31 fetcher called

Here's the sqlite3 database files:

$ ls | grep foobar-
foobar-1.sqlite
foobar-2.sqlite
foobar-3.sqlite
foobar-4.sqlite
foobar-5.sqlite
foobar-6.sqlite

Here's the full code:

package main

import (
    "database/sql"
    "database/sql/driver"
    "fmt"
    "log"
    "time"

    "github.com/mattn/go-sqlite3"
)

type Fetcher interface {
    Fetch() (string, error)
}

type FetcherFunc func() (string, error)
func (f FetcherFunc) Fetch() (string, error) {
    return f()
}

// customDriver implements the `sql.Driver` interface.
type customDriver struct {
    // base is the base database driver
    base      driver.Driver

    // fetcherFn is the function that will be used to fetch the database credential
    fetcherFn FetcherFunc
}

func (d *customDriver) Open(_ string) (driver.Conn, error) {
    // fetch the database credential
    dsn, err := d.fetcherFn()
    if err != nil {
        return nil, err
    }

    // open the database connection using the fetched credential and base driver
    return d.base.Open(dsn)
}

func OpenWithRotator(name string, base driver.Driver, fetcher Fetcher) (*sql.DB, error) {
    sql.Register(name, &customDriver{
        base:      base,
        fetcherFn: fetcher.Fetch,
    })

    // you don't need to fill the dsn, it will be fetched from the fetcher.
    return sql.Open(name, "")
}

var counter int
func simpleFetcher() (string, error) {
    log.Println("fetcher called")

    // add your custom logic
    // e.g. fetching from vault / config / etc.

    counter++
    return fmt.Sprintf("file:foobar-%d.sqlite", counter), nil
}

func main() {
    // you can adjust the `&sqlite3.SQLiteDriver{}` accordingly (e.g. &pq.Driver{}, etc.)
    db, err := OpenWithRotator("foobar", &sqlite3.SQLiteDriver{}, FetcherFunc(simpleFetcher))
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()
    db.SetConnMaxLifetime(2 * time.Second)

    for range time.Tick(time.Second) {
        if err := db.Ping(); err != nil {
            log.Fatal(err)
        }
    }
}

Conclusion

This is the simple implementation of a database rotator. You might want to implement your fetcher accordingly depends on your use case.

Thank you for reading!

Create Flag Using Bitmask and Bitwise Operator in Go

Clavin June — Fri, 15 Jul 2022 04:23:00 +0000

Introduction

I revisit some of my college notes, and find that in my first year of college, in Algorithm and Programming Course, Bitwise operator was mentioned but wasn't highlighted. It's either I didn't write the notes on that specific topic (I stopped writing notes in my second semester) or the Lecturer didn't explain anything about the usage. But then I found it is quite useful for Bitmasking. For example, in Unix' file access control it uses bitmask to represent the access:

000 => 0 => can't access anything
001 => 1 => can only execute
010 => 2 => can only read
011 => 3 => can execute+read
100 => 4 => can only write
101 => 5 => can execute+write
110 => 6 => can read+write
111 => 7 => can execute+read+write

All of those combination can be construct with just only 3 bits. As you can see, Bitmask uses sets of bit to show specific access is given or not. If it is 0, means that specific access is not given, otherwise it is given. In this article, you will learn how to implement Bitmasking and its operation in Golang.

Code

package main

import "fmt"

type Permission uint8

const (
    // 1
    PermissionExecute Permission = 1 << iota

    // 2
    PermissionRead

    // 4
    PermissionWrite
)

const (
    PermissionAll = PermissionExecute | PermissionRead | PermissionWrite
)

func Set(p, flag Permission) Permission {
    return p | flag
}

func Clear(p, flag Permission) Permission {
    return p &^ flag
}

func HasAll(p, flag Permission) bool {
    return p&flag == flag
}

func HasOneOf(p, flag Permission) bool {
    return p&flag != 0
}

func main() {
    var p Permission
    // you can use Set/Clear(p, PermissionExecute|PermissionRead|PermissionWrite) to set multiple bits
    p = Set(p, PermissionAll)
    p = Clear(p, PermissionRead)

    // false, because PermissionRead is cleared
    hasExecuteAndRead := HasAll(p, PermissionExecute|PermissionRead)

    // true, because PermissionExecute and PermissionWrite are set
    hasExecuteAndWrite := HasAll(p, PermissionExecute|PermissionWrite)

    // true, because PermissionExecute is set even though PermissionRead is cleared
    hasExecuteOrRead := HasOneOf(p, PermissionExecute|PermissionRead)

    fmt.Println(hasExecuteAndRead, hasExecuteAndWrite, hasExecuteOrRead)
}

Let's go to the explanation line per line.

Set

var p Permission // has no permission 000
p = Set(p, PermissionAll) // set all permission 111

// 000 (current permission)
// 111 (PermissionAll)
// --- OR
// 111

Clear

// clear the bit of PermissionRead (010)
p = Clear(p, PermissionRead)

// reverse the PermissionRead first
// 010
// ---- NOT
// 101 (NOT result)

// 111 (current permission)
// 101 (NOT result)
// --- AND
// 101

It is important to use BITCLEAR (AND NOT) operator instead of NOT (in this case, XOR) operator.
BITCLEAR is always set the bit to 0 because it reverse the flag first using NOT, then using AND.
Meanwhile, NOT is reverse the bit, so if you doing NOT operation twice, it will toggle back.
See the snippet below for example.

Toggle vs Clear

// toggling using NOT/XOR
foo := PermissionAll
fmt.Printf("Set All %03b\n", foo) // 111
foo ^= PermissionRead
fmt.Printf("Toggle %03b\n", foo) // 101
foo ^= PermissionRead
fmt.Printf("Toggle %03b\n", foo) // 111 it toggles back!

// clearing using BITCLEAR
bar := PermissionAll
fmt.Printf("Set All %03b\n", bar) // 111
bar &^= PermissionRead
fmt.Printf("Clear %03b\n", bar) // 101
bar &^= PermissionRead
fmt.Printf("Clear %03b\n", bar) // 101

HasAll

// check whether has both Execute AND Read permission
hasExecuteAndRead := HasAll(p, PermissionExecute|PermissionRead) // false

// PermissionExecute|PermissionRead
// 001 (PermissionExecute)
// 010 (PermissionRead)
// --- OR
// 011 (PermissionRead+PermissionExecute)

// check whether has both Execute AND Read permission
// 101 (current permission)
// 011 (PermissionRead+PermissionExecute)
// --- AND
// 001 (PermissionExecute)

// 001 != 011 so it doesn't have both Execute AND Read permission

// check whether has both Execute AND Write permission
hasExecuteAndWrite := HasAll(p, PermissionExecute|PermissionWrite)

// PermissionExecute|PermissionWrite
// 001 (PermissionExecute)
// 100 (PermissionWrite)
// --- OR
// 101 (Write+Execute)

// check whether has both Execute AND Write permission
// 101 (current permission)
// 101 (PermissionWrite+PermissionExecute)
// --- AND
// 101 (PermissionWrite+PermissionExecute)

// 101 == 101 so it has both Execute AND Write permission

HasOneOf

// check whether has either Execute OR Read permission
hasExecuteOrRead := HasOneOf(p, PermissionExecute|PermissionRead)

// PermissionExecute|PermissionRead
// 001 (PermissionExecute)
// 010 (PermissionRead)
// --- OR
// 011 (PermissionRead+PermissionExecute)

// check whether has either Execute OR Read permission
// 101 (current permission)
// 011 (PermissionRead+PermissionExecute)
// --- AND
// 001 (PermissionExecute)

// 001 != 0 so it has either Execute OR Read permission

Things To Be Concerned

You can store the bitmasks on your databases also, but it might be not a good idea to use iota on your logic. From the code above, let's say you want to remove PermissionRead. Then PermissionWrite will be represented by 2 instead of 4. To avoid this, you may want to set a Deprecated flag to your permission instead of remove it completely from your code. Or, as an alternative, you can put the number manually but it just another pain to handle if you have lots of permission.

Conclusion

In this case, Bitmasking is a useful tool to implement access control. You have implemented the access control in only 1 byte because you only need to use uint8. For example, if you use sets of boolean variables to represent the access control, you need more than 1 variable and it will take more than 1 byte to store.

Thank you for reading!

Find Git Branching Point Between Branch

Clavin June — Sat, 04 Jun 2022 17:00:17 +0000

➜  pet git:(master) git --no-pager log --oneline -1
2422e70 (HEAD -> master, origin/master) update README.md
➜  pet git:(master) git checkout -b testing
Switched to a new branch 'testing'
➜  pet git:(testing) git commit --allow-empty -sam "first"
[testing 8210c2d] first
➜  pet git:(testing) git commit --allow-empty -sam "second"
[testing feb9b4f] second
➜  pet git:(testing) git commit --allow-empty -sam "third"
[testing 7120047] third
➜  pet git:(testing) git --no-pager log --oneline -4
7120047 (HEAD -> testing) third
feb9b4f second
8210c2d first
2422e70 (origin/master, master) update README.md
# how to get this branching point (2422e70) from `testing` branch?
# git rev-list --boundary branch1...branch2 | grep "^-" | cut -c2-
➜  pet git:(testing) git rev-list --boundary testing...master  | grep "^-" | cut -c2-
2422e70001b80d186fdd1df1a9227dc3d8e0061e

Podman Support Multi Arch

Clavin June — Sat, 28 May 2022 17:00:01 +0000

$ podman machine ssh
# inside podman machine
$ sudo rpm-ostree install qemu-user-static
$ sudo systemctl reboot
# outside podman machine, wait for podman machine done restarting
$ podman run -it --rm --arch=amd64 alpine:3.14 uname -m
x86_64
$ podman run -it --rm --arch=arm64 alpine:3.14 uname -m
aarch64

Golang HTTP Handler With Gzip

Clavin June — Thu, 26 May 2022 09:52:21 +0000

Introduction

Golang has many kinds of compression technique within its standard library which you can use to compress your data. A compression is needed to reduce the size of the data. Even in a web server, a compression technique would be beneficial to increase the communication speed between client and server. Gzip is one of the compression techniques supported by both Golang and web. This article will cover the creation of golang HTTP handler that (de)compress gzip request/response and how to create an HTTP request that send/receive the gzip-compressed body payload. All the code covered in this article you can find it in this repository.

Directory Structure

$ tree .
.
├── LICENSE
├── Makefile
├── README.md
├── client
│   └── main.go
├── curl.sh
├── go.mod
├── server
│   └── main.go
└── util.go

2 directories, 8 files

server/main.go

package main

import (
    "example"
    "log"
    "net/http"
)

func main() {
    log.Println("server listening at :8000")
    http.ListenAndServe(":8000", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        body := example.MustReadCompressedBody[example.Payload](r.Body)
        body.Number++

        example.MustWriteCompressedResponse(w, body)
    }))
}

The server only run a single endpoint which will read a compressed body, increment the payload content, then respond with new body.

util.go

Payload

You can use the simplest payload just to test the compression, for example:

type Payload struct {
    Number int `json:"number"`
}

Read a Compressed Body Payload

func MustReadCompressedBody[T any](r io.Reader) *T {
    gr, err := gzip.NewReader(r)
    PanicIfErr(err)
    defer gr.Close()

    var t T
    PanicIfErr(json.NewDecoder(gr).Decode(&t))
    return &t
}

To read a gzip-compressed payload, you need to create a gzip.Reader from your response or request. Then decode the JSON as usual using the gzip.Reader instance.

Write a Compressed Response

func MustWriteCompressedResponse(w http.ResponseWriter, body any) {
    w.Header().Set("Content-Type", "application/json")
    w.Header().Set("Content-Encoding", "gzip")

    gw := gzip.NewWriter(w)
    defer gw.Close()
    PanicIfErr(json.NewEncoder(gw).Encode(body))
}

As usual, it is better to inform the client about your Content-Type, and Content-Encoding. But in golang case, informing the client about Content-Encoding will help golang http.Request to decompress the payload automatically. Just like reading the payload, you only need to create a gzip.Writer then encode the content normally. Also, don't forget to call gw.Close() to avoid EOF error.

Test the Server

Now, your server has completely (de)compress the request and response. Let's try to test it using cURL command first.

# run the server on another terminal
# using `go run ./server/` command
$ echo '{"number": 99}' | gzip | \
curl -iXPOST http://localhost:8000/ \
-H "Content-Type: application/json" \
-H "Content-Encoding: gzip" \
--compressed --data-binary @-

Since the server will increase the Payload.Number, you can expect the response number will be 100 by sending a request with 99. The expected result would be similar to this:

HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Type: application/json
Date: Thu, 26 May 2022 12:04:53 GMT
Content-Length: 39

{"number":100}

Now, after you sure that your server works completely fine, let's try to send the HTTP request using golang http.Request.

client/main.go

package main

import (
    "context"
    "encoding/json"
    "example"
    "log"
    "net/http"
    "time"
)

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
    defer cancel()

    body := &example.Payload{
        Number: 100,
    }

    log.Printf("create compressed request with %#v", body)
    req := example.MustCreateCompressedRequest(ctx, http.MethodPost, "http://localhost:8000/", body)
    defer req.Body.Close()

    log.Printf("send compressed request")
    resp, err := http.DefaultClient.Do(req)
    example.PanicIfErr(err)
    defer resp.Body.Close()

    log.Println("resp.Uncompressed?", resp.Uncompressed)
    var responsePayload *example.Payload
    if resp.Uncompressed {
        err = json.NewDecoder(resp.Body).Decode(&responsePayload)
    } else {
        responsePayload = example.MustReadCompressedBody[example.Payload](resp.Body)
    }
    log.Printf("read response %#v", responsePayload)
}

Just like a normal HTTP request, you just need to create an HTTP.Request, send the request, and lastly decode the response. But, before send the request payload, don't forget to compress it first.

Create a Compressed Request

func MustCreateCompressedRequest(ctx context.Context, method, url string, body any) *http.Request {
    pr, pw := io.Pipe()

    go func() {
        gw := gzip.NewWriter(pw)
        err := json.NewEncoder(gw).Encode(body)
        defer PanicIfErr(gw.Close())
        defer pw.CloseWithError(err)
    }()

    r, err := http.NewRequestWithContext(ctx, method, url, pr)
    PanicIfErr(err)

    r.Header.Set("Content-Type", "application/json")
    r.Header.Set("Content-Encoding", "gzip")

    return r
}

Create a new compressed request is quite easy, just like write a compressed response we need to create a new gzip.Writer. Code above utilize io.Pipe to avoid buffer the body into a memory and unecessary allocations. But here's the alternative if you want to buffer the body first:

func MustCreateCompressedRequest(ctx context.Context, method, url string, body any) *http.Request {
    var b bytes.Buffer

    gw := gzip.NewWriter(&b)
    err := json.NewEncoder(gw).Encode(body)
    defer PanicIfErr(gw.Close())

    r, err := http.NewRequestWithContext(ctx, method, url, &b)
    PanicIfErr(err)

    r.Header.Set("Content-Type", "application/json")
    r.Header.Set("Content-Encoding", "gzip")

    return r
}

Again, gw.Close() is also necessary here. If you don't close the gzip.Writer, you'll see an EOF error similar to this:

2022/05/26 19:23:58 http: panic serving [::1]:56436: unexpected EOF

Lastly, after creating the compressed request, you only need to send the request like a normal request. But here's a tricky part.

var responsePayload *example.Payload
if resp.Uncompressed {
    err = json.NewDecoder(resp.Body).Decode(&responsePayload)
} else {
    responsePayload = example.MustReadCompressedBody[example.Payload](resp.Body)
}

After you receive a response which you want to decompress and decode, please be aware with the resp.Uncompressed. If the server returns a header Content-Encoding: gzip, as it said here Golang will try to decompress the payload for you so you don't need to use example.MustReadCompressedBody[example.Payload](resp.Body). But if you add r.Header.Set("Accept-Encoding", "gzip") on your request, it won't be automatically decompressed.

Test the Client

# run the server on another terminal
# using `go run ./server/` command
$ go run ./client
2022/05/26 19:41:34 create compressed request with &example.Payload{Number:100}
2022/05/26 19:41:34 send compressed request
2022/05/26 19:41:34 resp.Uncompressed? true
2022/05/26 19:41:34 read response &example.Payload{Number:101}

Conclusion

Implementation of the gzip compression on Golang HTTP handler and requests might be adding a little complexity on your code, but I believe modern browser / API gateway these days can implement this easily without changing your code. Also this middleware created by NY Times can help you to minimize the effort on implementing this compression. You can find all the code used on in this article here.

Thank you for reading!