DEV Community: Viktoras

Redis is fast - I'll cache in Postgres

Viktoras — Wed, 24 Sep 2025 13:31:43 +0000

There are books & many articles online, like this one arguing for using Postgres for everything. I thought I’d take a look at one use case - using Postgres instead of Redis for caching. I work with APIs quite a bit, so I’d build a super simple HTTP server that responds with data from that cache. I’d start from Redis as this is something I frequently encounter at work, switch it out to Postgres using unlogged tables and see if there’s a difference.

The setup

I’ll run the experiment on my homelab’s k8s cluster. The idea is to run Postgres or Redis on one node, limiting it to 2CPUs via k8s limits, as well as 8GiB of memory. On another node, I’ll run the web server itself and then spin a pod for the benchmark executed via k6 on the third.

Both postgres and redis are used with the out of the box settings for the following images:

Postgres - postgres:17.6
Redis - redis:8.2

I wrote a simple webserver, with 2 endpoints, a cache and a “Session” struct which we’ll store in the cache:

var ErrCacheMiss = errors.New("cache miss")

type Cache interface {
    Get(ctx context.Context, key string) (string, error)
    Set(ctx context.Context, key string, value string) error
}

type Session struct {
    ID string
}

func serveHTTP(c Cache) {
    http.HandleFunc("/get", getHandler(c))
    http.HandleFunc("/set", setHandler(c))

    port := os.Getenv("PORT")
    if port == "" {
        port = "8080"
    }

    fmt.Println("Server starting on http://0.0.0.0:" + port)

    server := &http.Server{Addr: "0.0.0.0:" + port}

    go func() {
        if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            fmt.Println("Error starting server:", err)
        }
    }()

    quit := make(chan os.Signal, 1)
    signal.Notify(quit, os.Interrupt)
    <-quit

    fmt.Println("Shutting down server...")

    if err := server.Close(); err != nil {
        fmt.Println("Error shutting down server:", err)
    }
}

For redis, I’ve implemented the cache using github.com/redis/go-redis/v9 as follows:

type RedisCache struct {
    client *redis.Client
}

func NewRedisCache() *RedisCache {
    redisURL := os.Getenv("REDIS_URL")
    if redisURL == "" {
        redisURL = "localhost:6379"
    }

    fmt.Println("Connecting to Redis at", redisURL)

    client := redis.NewClient(&redis.Options{
        Addr: redisURL,
        Password: "",
        DB: 0,
    })

    return &RedisCache{
        client: client,
    }
}

func (r *RedisCache) Get(ctx context.Context, key string) (string, error) {
    val, err := r.client.Get(ctx, key).Result()
    if err == redis.Nil {
        return "", ErrCacheMiss
    }
    if err != nil {
        return "", err
    }
    return val, nil
}

func (r *RedisCache) Set(ctx context.Context, key string, value string) error {
    return r.client.Set(ctx, key, value, 0).Err()
}

The postgres cache is implemented using the github.com/jackc/pgx/v5 library:

type PostgresCache struct {
    db *pgxpool.Pool
}

func NewPostgresCache() (*PostgresCache, error) {
    pgDSN := os.Getenv("POSTGRES_DSN")
    if pgDSN == "" {
        pgDSN = "postgres://user:password@localhost:5432/mydb"
    }

    cfg, err := pgxpool.ParseConfig(pgDSN)
    if err != nil {
        return nil, err
    }

    cfg.MaxConns = 50
    cfg.MinConns = 10

    pool, err := pgxpool.NewWithConfig(context.Background(), cfg)
    if err != nil {
        return nil, err
    }

    _, err = pool.Exec(context.Background(), `
        CREATE UNLOGGED TABLE IF NOT EXISTS cache (
            key VARCHAR(255) PRIMARY KEY,
            value TEXT
        );
    `)
    if err != nil {
        return nil, err
    }

    return &PostgresCache{
        db: pool,
    }, nil
}

func (p *PostgresCache) Get(ctx context.Context, key string) (string, error) {
    var content string
    err := p.db.QueryRow(ctx, `SELECT value FROM cache WHERE key = $1`, key).Scan(&content)
    if err == pgx.ErrNoRows {
        return "", ErrCacheMiss
    }
    if err != nil {
        return "", err
    }
    return content, nil
}

func (p *PostgresCache) Set(ctx context.Context, key string, value string) error {
    _, err := p.db.Exec(ctx, `INSERT INTO cache (key, value) VALUES ($1, $2) ON CONFLICT (key) DO UPDATE SET value = $2`, key, value)
    return err
}

I’ll seed the redis and postgres with 30 million entries each, keeping record of the inserted uuids. From there, I’ll generate a subset of existing uuids to use while benchmarking. This allows for simulating both hits and misses.

I’ll do a few runs of benchmarks for gets first, then sets and then a mixed run. Each run will execute for 2 minutes. I’ll look at the number of operations per second, latencies as well as memory and CPU usage during those times.

To simulate a somewhat real scenario where only a subset of keys exist in the cache the set benchmark will have a 10% chance to update an existing key, whereas the get will have an 80% chance of picking an existing key. The mixed workload will have a 20% chance to execute a set scenario and 80% for the get scenario.

The results

Getting values from cache

Redis performed better than Postgres, which did not surprise me at all. The bottleneck was actually the HTTP server. The machine running the http server maxed out on CPU, with redis running comfortably with ~1280mCPU - short of the 2000mCPU limit imposed. Redis used ~3800MiB of RAM, which stayed flat across the runs.

For postgres, the bottleneck was the CPU on postgres side. It consistently maxed out the 2 cores dedicated to it, while also using ~5000MiB of RAM.

Redis also did better when it comes to latencies of the HTTP responses:

Setting values in cache

Once again Redis performed better. The CPU usage stayed roughly the same as in the case of the GET experiment, with the RAM usage growing to ~4300MiB due to the new keys being inserted. The bottleneck stayed on the HTTP server side, with Redis using ~1280mCPU once again.

Postgres once again was bottlenecked by the CPU, constantly using 100% of the 2 cores it was limited to. During the course of the run, the memory usage grew to ~5500MiB.

During the test, the endpoints with the Redis cache implementation also had better latencies:

Read/write performance

The mixed benchmark also returned the predictable result of Redis reigning superior. As has been the story so far, the CPU stayed put at ~1280mCPU, RAM usage grew a bit due to the new keys being inserted.

Postgres maxed out the two cores and reached around 6GiB of memory used.

Latencies once again were better when using redis:

Unlogged tables

In the benchmark, I’ve used an unlogged table for postgres but this has not seemed to help, or has it? If I rerun the same benchmark with a normal(logged) table we can look at the numbers.

The unlogged table makes a huge difference for the write benchmark and a somewhat smaller but still significant one for the mixed workload. This is because the unlogged tables skip the write ahead log making them a lot faster for writes. There’s very little difference for the read performance though and I expect more runs would show the two test cases converging.

Conclusion

Redis is faster than postgres when it comes to caching, there’s no doubt about it. It conveniently comes with a bunch of other useful functionality that one would expect from a cache, such as TTLs. It was also bottlenecked by the hardware, my service or a combination of both and could definitely show better numbers. Surely, we should all use Redis for our caching needs then, right? Well, I think I’ll still use postgres. Almost always, my projects need a database. Not having to add another dependency comes with its own benefits. If I need my keys to expire, I’ll add a column for it, and a cron job to remove those keys from the table. As far as speed goes - 7425 requests per second is still a lot. That’s more than half a billion requests per day. All on hardware that’s 10 years old and using laptop CPUs. Not many projects will reach this scale and if they do I can just upgrade the postgres instance or if need be spin up a redis then. Having an interface for your cache so you can easily switch out the underlying store is definitely something I’ll keep doing exactly for this purpose.

Thanks for reading!

Wrapping Go errors with caller info

Viktoras — Thu, 10 Jul 2025 17:35:59 +0000

I find Go’s error handling refreshingly simple & clear. Nowadays, all I do is wrap my errors using fmt.Errorf with some additional context. The pattern I tend to use most of the time just includes a function/method name, like so:

func someFunction() error {
    err := someStruct{}.someMethod()
    return fmt.Errorf("someFunction: %w", err)
}

type someStruct struct {
}

func (ss someStruct) someMethod() error {
    return fmt.Errorf("someMethod: %w", errors.New("boom"))
}

If I now call someFunction, the error - when logged or printed - will look like this: someFunction: someMethod: boom. This gives just enough context to figure out what happened, and where.

I thought to myself - this can be wrapped in a helper function, to automatically take the caller name and do this wrapping for me.

Here’s an implementation:

func WrapWithCaller(err error) error {
    pc, _, _, ok := runtime.Caller(1)
    if !ok {
        return fmt.Errorf("%v: %w", "unknown", err)
    }
    fn := runtime.FuncForPC(pc).Name()
    pkgFunc := path.Base(fn)
    return fmt.Errorf("%v: %w", pkgFunc, err)
}

We can now use it like so:

func someFunction() error {
    err := someStruct{}.someMethod()
    return WrapWithCaller(err)
}

type someStruct struct {
}

func (ss someStruct) someMethod() error {
    return WrapWithCaller(errors.New("boom"))
}

And it will return the following: main.someFunction: main.someStruct.someMethod: boom. Keep in mind that this will have performance overhead:

func Benchmark_WrapWithCaller(b *testing.B) {
    err := errors.New("boom")
    for i := 0; i < b.N; i++ {
        _ = WrapWithCaller(err)
    }
}
// Result:
// Benchmark_WrapWithCaller-10 2801312 427.0 ns/op 344 B/op 5 allocs/op

func Benchmark_FmtErrorf(b *testing.B) {
    err := errors.New("boom")
    for i := 0; i < b.N; i++ {
        _ = fmt.Errorf("FmtErrorf: %w", err)
    }
}

// Result:
// Benchmark_FmtErrorf-10 13475013 87.19 ns/op 48 B/op 2 allocs/op

So it’s 4-5 times slower than a simple wrap using fmt.Errorf.

One benefit of using WrapWithCaller is that refactoring is easier - you change the function/method name and don’t need to bother yourself with changing the string in the fmt.Errorf. The performance hit is likely not an issue for most programs, unless you’re wrapping tons of errors in hot loops or something like that.

I’m a bit torn if I want to add this helper to my codebases and use it everywhere - seems like an extra dependency with little benefit, but I’ll have to give it a spin before I decide if this is something I’m ready to accept. Time will tell. Thanks for reading!

BLAKE2b performance on Apple Silicon

Viktoras — Thu, 27 Mar 2025 12:02:13 +0000

For work, I was going to store some hashed tokens in a database. I was going to keep it simple and go with HMAC-SHA256 for it but having recently read Jean-Philippe Aumasson’s book “Serious Cryptography” I remembered that BLAKE2 should be quicker:

BLAKE2 was designed with the following ideas in mind:

…

It should be faster than all previous hash standards

Cool, I thought, let’s consider BLAKE2 then. First, let’s write a simple benchmark to see just how much faster BLAKE2 would be than HMAC-SHA256. Performance is not important for my use case as the hashing will almost certainly not be a bottleneck but I was curious. So I write a benchmark:

import (
    "crypto/hmac"
    "crypto/rand"
    "crypto/sha256"
    "log"
    "testing"

    "golang.org/x/crypto/blake2b"
)

func BenchmarkHashes(b *testing.B) {
    token := []byte("some-api-token")
    secretKey := generateSecretKey()

    b.ResetTimer()

    _ = b.Run("HMACSHA256", func(b *testing.B) {
        for b.Loop() {
            _ = HMACSHA256(token, secretKey)
        }
    })

    _ = b.Run("BLAKE2b", func(b *testing.B) {
        for b.Loop() {
            _ = BLAKE2b(token, secretKey)
        }
    })
}

func generateSecretKey() []byte {
    key := make([]byte, 32)
    _, err := rand.Read(key)
    if err != nil {
        panic(err)
    }
    return key
}

func HMACSHA256(token []byte, secretKey []byte) []byte {
    h := hmac.New(sha256.New, secretKey)
    h.Write(token)
    return h.Sum(nil)
}

func BLAKE2b(token []byte, secretKey []byte) []byte {
    hasher, err := blake2b.New256(secretKey)
    if err != nil {
        log.Fatal(err)
    }
    hasher.Write(token)
    return hasher.Sum(nil)
}

Run it and the results are:

cpu: Apple M1 Max
BenchmarkHashes/HMACSHA256-10 3442680 341.0 ns/op 512 B/op 6 allocs/op
BenchmarkHashes/BLAKE2b-10 1966382 584.0 ns/op 416 B/op 2 allocs/op

OK… So BLAKE2 is slower than HMAC-SHA256. Yes, we have less allocations which is nice, but it does take quite a few more CPU cycles. My first thought is that it might indeed be faster but only if the input is like way, way longer. So I switch to the following token:

    token := []byte(strings.Repeat("some-api-token", 10000))

cpu: Apple M1 Max
BenchmarkHashes/HMACSHA256-10 19536 59946 ns/op 512 B/op 6 allocs/op
BenchmarkHashes/BLAKE2b-10 6452 190926 ns/op 416 B/op 2 allocs/op

Hmmmm… This is even worse. Now, it could be that we’re dealing with a non optimal implementation. SHA256 is implemented in the stdlib in Go and very likely well optimized, whereas the BLAKE2 implementation I’m using comes from the golang.org/x/crypto package. Perhaps we can find a better one. The first search result recommends github.com/minio/blake2b-simd which has been archived since 2018. Not promising, but let’s give it a shot.

import (
    blakeMinio "github.com/minio/blake2b-simd"
)

func BLAKE2bMinio(token []byte, secretKey []byte) []byte {
    hasher := blakeMinio.NewMAC(32, secretKey)
    hasher.Write(token)
    return hasher.Sum(nil)
}

These are the results with the original token.

cpu: Apple M1 Max
BenchmarkHashes/HMACSHA256-10 3622322 316.7 ns/op 512 B/op 6 allocs/op
BenchmarkHashes/BLAKE2b-10 2881012 415.6 ns/op 416 B/op 2 allocs/op
BenchmarkHashes/BLAKE2b-minio-10 2138151 566.4 ns/op 480 B/op 2 allocs/op

So this is even slower… I quickly glance over the codebase of github.com/minio/blake2b-simd and notice all the architecture specific files but there aren’t any for ARM architecture. Let’s try on a machine with an AMD64 processor.

cpu: Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz
BenchmarkHashes/HMACSHA256-4 761613 1488 ns/op 512 B/op 6 allocs/op
BenchmarkHashes/BLAKE2b-4 2287569 566.9 ns/op 416 B/op 2 allocs/op
BenchmarkHashes/BLAKE2b-minio-4 1541990 765.4 ns/op 480 B/op 2 allocs/op

Right, so it seems that the implementations I’m using are only optimized for AMD64. Or are they? One final test, I spin up an ARM based VPS on Hetzner to test it out. Note, that since the benchmark was not able to determine the exact CPU model the CPU line was omitted.

BenchmarkHashes/HMACSHA256-2 1000000 1007 ns/op 512 B/op 6 allocs/op
BenchmarkHashes/BLAKE2b-2 1367085 879.6 ns/op 416 B/op 2 allocs/op
BenchmarkHashes/BLAKE2b-minio-2 1123965 1080 ns/op 480 B/op 2 allocs/op

So with this, it seems that the issue only occurs on Apple Silicon processors. In the benchmarks with other processors, the BLAKE2b does win. I’m not entirely sure what causes this as CPU architectures are not my strong suite. If you do - please let me know by posting a comment.

State of my Homelab 2025

Viktoras — Mon, 10 Mar 2025 16:48:21 +0000

For many years now I’ve had at least one machine at home which would work as a server to host some apps. In the last couple of years I’ve been getting more into it which has led me to purchase additional hardware. I’ve decided it would be nice to document my journey so I’ll try to make a post like this - detailing the setup that I currently have with both the hardware and the software once a year.

Since this is (almost) the first post on my homelab setup let me start with some background. I am a software developer who likes to dabble in infrastructure. I like hosting things that I use on my own servers. However, as much as I’d like to run them on racks due to the electricity costs in Europe this is prohibitively expensive. What is more - this requires space, which I do not want to sacrifice at home. Therefore, my homelab is on the small side - both physically and from the resource perspective. I also tend to try not to spend too much on hardware so I end up running old and/or second hand hardware.

Hardware

Compute

The homelab consists of 4 machines. These are:

MSI Cubi 3 Silent NUC


CPU	Dual-core i5-7200U
RAM	2 × 16GB
SSD	240GB KINGSTON SA400S37240G

This NUC has been running for close to 6 years now. I originally bought it in 2019 with 16GB of RAM, but since upgraded it to 32GB. It still has the original SSD in it and has run faultlessly.

I’ve an old blog post about this machine specifically, if you’re interested.

Currently, this machine acts as k3s server.

2x Lenovo ThinkCentre M910 Tiny


CPU	Quad-core i5-6500T
RAM	2 × 8GB
SSD	Apacer AS350 256GB

These I bought post lease for around 130 Euros each in 2023. I’ve no clue how much use they saw before becoming part of my homelab, but they have been running flawlessly for the last two years here. These things are tiny and powerful. They are also quiet. This was very important when I was living in an apartment. If I ever need to expand my homelab again, I’m sure to be looking for more post lease ThinkCentres.

These machines are the agents on my k3s cluster.

Mostly second-hand NAS


CPU	Quad-core i5-4590T
RAM	2 × 8GB
Motherboard	MSI H81I-S01
PSU	Silverstone ST30SF 300W
Boot Drive	A random 120GB SSD
Storage	4 × 2TB SSD (Mixed: Samsung EVO 870 & Crucial MX500)
SATA Expansion	4-port PCIe card
Case	Jonsbo N2

Originally assembled at the end of 2023. The new parts on the NAS are the case and the storage drives. The rest of the components I scavenged off of local bulletin boards & local recycling companies working with electronic waste. The whole system(minus the storage) set me back around 250 Euros, with the case being half that. I would have gone for a bigger case(a used one) but living in an apartment space was at a premium and I could not find anything second-hand that would fit the specific place I had in mind for this machine. I would also have gone for HDDs were it not for the noise.

This machine now runs TrueNAS SCALE and serves as durable storage for the k3s cluster.

Networking

I currently live in a location where no wired internet is available. Therefore, I’ve a 4/5G CPE (H352-381) providing the main connectivity. Since the quality of the internet depends heavily on the load of the nearby cellular towers, I see quite dramatic slowdowns during busy hours, such as Sunday evenings. On working days, I’ll get 200-300Mbps down, 50mbps up. On weekends and busy evenings this can drop down to 50 down, 10 up.

Knowing that wireless internet can be a bit of pain, I’ve opted to for a router with dual WAN support. I went for TP-Link’s ER605. The plan is to get a starlink at some point and plug that in.

From there, I’ve 2x TP-Link Deco XE75 wireless access points - one per floor of the house. These have far exceeded my expectations. They provide excellent coverage even in a concrete house and far beyond it.

There are also a couple of unmanaged switches somewhere along the way here as well.

Auxiliary devices

I also own a few shelly smart plugs, those are mostly used for measuring the power consumption of the homelab and for whenever I’m curious about a random device’s power consumption. Besides that, I have 2x Shelly H&T Gen 3 humidity and temperature sensors for monitoring these around the house.

Power consumption

Over the years, I’ve grown quite conscious about how much electricity the devices around the house use. When choosing the hardware I always try to consider energy consumption, to a reasonable degree that is. The homelab uses approximately 49 watts of power in total with the usual load it’s seeing. The NAS is consuming ~22w of that, the Lenovo ThinkCentres take around ~8W each and the Cubi draws ~10W. At the current electricity prices here this comes to right around 7.5 euros a month.

I’ve written in more detail about the power consumption of these machines in my previous blog post if you’re interested.

The “rack”

The homelab neatly fits in a custom built rack with NAS sitting on the bottom shelf, and compute units above that. There’s a power strip attached to the side of it, as well as a small switch. Each of the machines is plugged into a shelly smart plug, which continuously emits updates of the usage to an MQTT server inside the k3s cluster. The rack sits in the utility room of my house. Here’s what it looks like:

The idea here is to put a monitor on top in the future, as well as a keyboard to allow for easy access if the servers would start acting up. This, however, is still in the works. I’ve purposefully left some space to be able to shove more one litre computers if needed.

Software

The NAS

The NAS machine runs TrueNAS Scale. It has a single 4-wide RAIDZ2. 2TB drives each give me a usable capacity of 3.4TB. While this might not sound like a lot, I don’t really store much media here. I store data for applications that I run in k8s and in my case these are usually tiny. I expose the dataset as an SMB share.

The only additional application I end up running on the NAS is netdata for metric collection. This netdata instance only collects the metrics and forwards the data to a netdata parent running on the k3s cluster. I’ve written a blog post on how to do that previously.

Periodically, a job on the k3s cluster will backup the important bits from NAS into an off-site s3 bucket. I use kopia for backups.

The k3s cluster

The cluster consists of a single k3s server and two agents. With the exception of my NAS, all the other hosts run debian. On top of that, I have k3s installed. From there, I use helm with the helm secrets plugin to deploy a whole bunch of things on the kubernetes cluster. The helm charts are stored in a private git repo on github.

Some of the first helm charts I installed on the cluster are the kubernetes csi driver to access the SMB share on the NAS and MetalLB for any services that I need exposed on the local network. I also deploy a PostgreSQL instance to serve as my main database.

For each of the domains I own and use (including the one you’re currently on) I deploy three things - a cloudflared instance for the cloudflare tunnel which forwards traffic to an nginx instance. This then forwards requests further to either a specific kubernetes service or authelia for authentication.

For monitoring I use Netdata. I use their helm chart in the k3s cluster, with a single parent and children running on each node. It will automatically pick up prometheus metrics from pods with the appropriate annotations present.

The actual applications

I think it’s time to move away from the infrastructure and talk about the actual things I have running there:

Audiobookshelf - I don’t listen to ebooks that often, but when I do I prefer the self hosted option.
Cyberchef - comes in handy for the work that I do.
Excalidraw - extensively used for work, self hosting gives me peace of mind.
Harbor - all the applications I build immediately get Dockerized, and CI builds & pushes them here.
Jellyfin - this one does not need an introduction, I think.
Miniflux - let’s me keep up to date with a few blogs I like to read.
Mouthful - is how I handle comments on this blog.
Owntracks - a work in progress, helps with home automation.
Paperless-ngx - I’d say this is my second most popular app on my homelab. An absolute lifesaver. I’ve moved away from paper documentation completely thanks to it.
EMQX - my MQTT broker of choice. Shellies report their energy consumption here, owntracks uses it as well. I also have some applications hooking into it and emitting events.
Pi-hole - helps with ad blocking.
Tandoor - a library of food recipes I enjoy.
Github action runner controller - allows the cluster to run github actions CI workloads on my own servers.
Ripe atlas probe - my meager contribution to the RIPE Atlas project. At some point I might even find a way to spend all the credits I’ve accumulated…
Silverbullet - probably the app that I use the most out of all. I’ve tried many different notes apps, for some reason this one stuck.
13ft - for those few times a year when I actually have to read the Medium article.
Unbound - self hosted DNS resolver, my pi-hole uses it, and my network uses pi-hole as the dns server.
Umami - self hosted, privacy respecting analytics. There are many like it, but this one uses few resources and is generally pretty good. See my full post on it if you’d like.
Vaultwarden - password manager of choice.
Yopass - for when I need to share a secret.
Custom prometheus exporters. These export things such as shelly statuses, power consumption and various other metrics.
A few discord bots I’ve written.
A few web apps & internal APIs I’ve developed over the years, like the embedding api for post recommendations
And a few cronjobs to back up all the things that need backing up.

The whole cluster uses around 1.1 cores under normal load and around 19GB of RAM.

The future

While I’m generally happy with my homelab, there are a few points that I got wrong or would like to improve.

First, I’d like to run postgres on a completely separate host, or more ideally on two separate hosts with automatic failover. For this, I’d like to use the two M910 Tiny’s I already have. This would require me to remove them from the k3s cluster. I’d like to replace them 2 small machines, possibly with the Intel’s N series processors. However, I’ve struggled to find second hand options for them. I’d use them as k3s servers with the whole cluster consisting of 3 servers. This would allow a failure of a single node with no downtime.

Second, I wish to switch my NAS to an HDD array for extra storage. This will come in handy for future projects. I would also like to move away from SMB to NFS shares, as I’d expect those to play nicer with linux and mac machines at home. I chose SMB for the theoretical use-case of having a windows machine at home, but I’ve never gotten one.

Third, I intend to move away from cloudflare tunnels. I’ll use a VPS and some self hosted tunnel alternative, I’m yet to decide which one. I’ll need to write an ansible playbook to set the server up to my liking so that I can freely move between VPS providers if/when needed.

Fourth, there’s a few applications that are on my to self-host list:

Changedetection for bargain hunting.
Frigate to replace the dedicated NVR I currently have running at home.
A dashboard for the entirety of my home. Ideally it would have links to all the listed applications and allow for custom pages to help with home automation.

Fifth, integrate all the utility appliances at home, such as the recuperator and the heat pump with the dashboard mentioned above to allow for easy control in a single panel, instead of having to use multiple applications.

Sixth, add a UPS to the homelab. This is a nice to have and might not happen any time soon but it’s something I could grab if the right deal comes along.

Seventh, add an outdoor thermometer or a whole weather station to the setup and a prometheus exporter for it so it can be visualized in Netdata.

Eight, actually mount a monitor on top of the rack. A keyboard would help there as well in case I need to directly hook into one of the machines.

And probably a few other things as well…

That’s what my homelab setup looks like. If you made it this far - thank you for reading. If you have any questions, do drop a comment below!

My homelabs power consumption

Viktoras — Sat, 01 Mar 2025 13:00:55 +0000

My homelab consists of 4 machines currently. When choosing them I tried to be energy conscious - using hardware which would not consume too much electrical power, while still trying to maintain the up-front cost low. These are mostly older systems and I was unable to find decent power consumption numbers for them.

The specs

1x MSI Cubi 3 Silent NUC


CPU	Dual-core i5-7200U
RAM	2×16GB

2x Lenovo ThinkCentre M910 Tiny


CPU	Quad-core i5-6500T
RAM	2×8GB

1x Mostly second-hand NAS


CPU	Quad-core i5-4590T
RAM	2×8GB
Motherboard	MSI H81I-S01
PSU	Silverstone ST30SF 300W
Storage	4x2TB SSD - mixed, Samsung EVO 870 & Crucial MX500

The setup

Since these are all currently plugged into shelly smart plugs, I’m able to tell the power usage of each of them separately. The plug measures the power consumption and reports that to EMQX via MQTT. From there, I have a custom prometheus collector implemented. It subscribes to the relevant topics on EMQX, and exposes a /metrics endpoint in the prometheus exposition format. This is periodically crawled by netdata, which is running in the same cluster.

I created a dashboard in netdata which allows me to visualize the CPU usage of each machine and its power consumption. It looks something like this:

I’m not going to measure the consumption of the systems when they are idle as there are quite a few things running on my cluster and I wouldn’t like to interrupt them. However, I’ll show the current CPU usage and the power draw from the plug. The CPU usage is what I’d call steady in my homelab, it rarely spikes and seems to be pretty consistent across the board. We’ll call this the baseline.

I’ll then do a 10 minute stress test using the stress tool on each machine, to see what the power consumption looks like at that moment.

The results

Note that the CPU % shows the % of cpu used across all cores - 100% indicates that all cores are working at their max. The cost estimates assume a price of 0,211479 €/kWh - coming directly from my last bill and I’ll call 30 days a month.

Here are the results for the baseline:

System	CPU %	Power Draw	Estimated Monthly Cost
Cubi	27%	12.4W	€1.89
Lenovo M910 Tiny	8%	7.3W	€1.11
NAS	4.5%	21W	€3.20

And for the full load test:

System	CPU %	Power Draw	Estimated Monthly Cost
Cubi	100%	17.7W	€2.70
Lenovo M910 Tiny	100%	29.4W	€4.48
NAS	100%	39.9W	€6.08

Thanks for reading!

On Umami

Viktoras — Tue, 18 Feb 2025 19:30:23 +0000

I’ve been using Umami analytics on this blog for quite some time now. I self host an instance on my homelab. I spent a bit of time researching self-hosted analytics and generally they had a few issues.

First, I’d like the analytics platform to be privacy focused. No cookies, GDPR compliant, no PII. Umami checks this mark.

Second, many of the alternatives had quite a bit of hardware resource overhead once hosted. They would either consume a ton of memory, the cpu usage would be high or require me to host something like ClickHouse to run them. Since this blog is not the new york times I feel like hosting a specific database for it would be overkill. My homelab is rather small, so keeping things minimal is how I manage to stretch it. Umami consumes around 1% of a CPU core and ~240MiB of RAM on my homelab.

Third, postgres as the datastore. Postgres is my go to database, and I host tons of small tools that use it as the backend. I like having a single instance that can then be easily backed up & restored, without having to resort to a ton of different databases. Therefore, any analytics tool would have to use it.

Fourth, the tracking script should be minimal in size, not to affect load times too badly. The blog itself is pretty lightweight, so any tracking script bloat would defeat that. The Umami script has a content lenght of 1482 bytes once gzipped, not too shabby.

Generally, I’ve been happy with the choice. However, there is one thing in Umami that annoys me more than it probably should: the visit timer. Apparently, the visit time is only updated once a user navigates to another page on the blog. If they simply leave, there’s no visit duration stored whatsoever. This makes the visit time tracker completely useless. I’m not the first one to notice this but the issue has since been moved to a discussion which has seen no progress.

Good news is there’s a few things one could do - perhaps add a custom event to track this? Or fork Umami, since it’s open source and fix it. Both of these fall strictly into my “can’t be arsed to do” category, so I guess it’s not that important.

Thanks for reading! Perhaps there are other analytics tools that tick the boxes above? Let me know in the comments below.

ML for related posts on Hugo

Viktoras — Mon, 02 Dec 2024 11:46:55 +0000

After reading the technicalwriting.dev post on embeddings I thought to myself - this seems like something that I can implement quite quickly and would serve as a good starting point for some hands on experience with machine learning. I want to start small, so something as simple as filling the related posts section of this blog seems like an ideal candidate to get my hands dirty.

This blog is made with Hugo and uses the related content feature which provides a list of related posts based on the tags & keywords you use. While I have no quarrels with the mechanism, I thought this would be a good place to try and experiment with embeddings.

The thought process

Since this blog is static, and I’d like it to keep it that way wherever possible I figured I’ll need to pre-compute the related posts for each post whenever a Hugo build is issued. However, there seems to be no way to natively hook into the Hugo’s build pipeline, so I’ll have to handle this via a script. The script would build a matrix of posts and their embeddings, compare each post to the rest of the posts and rank them according to their cosine similarity. I’d then keep the top N of those. I would then need to get Hugo to render these as links to those posts. To achieve this, I’ll have to save the recommended posts as part of the original post, probably in the frontmatter, so that these are available when rendering the related posts section via Hugo.

This pretty much lead me to the following architecture:

A CLI(written in Go) to compare posts via their embeddings and manipulate the .md files.
An API(written in Python) with a model to calculate an embedding for a given text.

I could write everything in Python and have a single CLI for this, but my intent is to have this as a part of a generic blog cli helper tool which I’d like to keep in Go. Ideally, I’d stick with Go for the API part as well but using Python here makes life easier due to the excellent ML support and plentiful examples. I might re-visit this choice at some point though.

Keep in mind that any code you find here is just a script intended for personal use, it will not be the prettiest or most optimized code you’ve ever seen, but that’s not my goal here.

Choosing the model

ML models can be huge. I’m not aiming for a world-class solution here, and I’d like something that is reasonably small as I intend to run it on my homelab. My homelab runs on very modest hardware - second hand 1L computers with no dedicated GPUs, old CPUs and limited amounts of RAM, so I’d like something relatively small. Since I’m going to feed in full blog posts, I might choose something with a larger token limit. Other than that, I have very limited knowledge of what I should pay attention to as far as models for this specific task.

After typing in a few search queries, I’ve stumbled upon the Massive Text Embedding Benchmark (MTEB) Leaderboard. I then glanced through the list, choosing jina-embeddings-v3 due to its small size(2.13GB listed memory usage) and good accuracy - that’s the hope, at least - it’s rank 26 on the leaderboard at the time of writing after all.

The API

From there, I’ve opted to quickly build a super small python HTTP API with a single endpoint. It takes a POST at /embedding with a payload of:

{
  "text": "Lorem ipsum dolor sit amet."
}

And returns an embedding:

[
  0.052685488015413284,
  // omitted 1022 entries for brevity
  // ...
  0.017726296558976173
]

I tested this locally and found that the embedding calculation can take some time. For longer texts, this might take as long as 10 seconds. While I don’t expect to run the process for all the posts in my blog frequently, I figured I’d need to add some persistence here. I intend to run this on my homelab, which consists of 1L post lease machines with no dedicated GPUs and old generations of CPUs so it would be even slower there. To keep things simple, I opted to add support for a postgres table that stores the hash of the input text as well as the resulting embedding. Upon receiving a request I’ll calculate the hash of the text and check for its existence in the database first allowing for quick responses if a text has already been processed. In reality this means that unless I edit a blog post, an embedding response for it should be rapid if I had it calculated in the past.

I also added an optional API token to add some very crude auth support.

I then wrote a Dockerfile for it, and proceeded to host it in my homelab. With it now accessible by my CLI, I had the API ready.

Oh, and it’s currently using right around ~3.6GiB of memory:

To calculate embeddings for a single blog post usually takes around 7 seconds on my hardware. The time scales with the length of the post. It also seems to consume a full CPU core when doing so:

The full code for the API is available here.

The CLI

The folder structure of my blog posts looks something like this:

content/
├── posts/
│ ├── 2024/
│ │ ├── post-title/
│ │ | └── index.md

The cli accepts a base path, ./content/ in my case. It will walk through the directory, parsing each index.md file as a post. It will store it in memory, in the following structure:

type Frontmatter struct {
    Title string `yaml:"title"`
    Date time.Time `yaml:"date"`
    Draft bool `yaml:"draft"`
    Author string `yaml:"author"`
    Tags []string `yaml:"tags"`
    Description string `yaml:"description"`
    SimilarPosts []string `yaml:"similar_posts"`
}

type Content []byte

type MarkdownPost struct {
    RawContent []byte // frontmatter + separators + content
    Path string
    Frontmatter Frontmatter // frontmatter
    Content Content // does not include frontmatter
    Embedding []float64
}

Once a markdown file is read, I do a few operations on it. First, I strip the hugo shortcodes. Then I render the markdown document to HTML by using the blackfriday markdown processor. Afterwards, I strip out the HTML tags. This leaves me with a plain text, which is ready to be passed into the model for embedding calculation. I did this because I worried that the raw contents of the file with shortcodes might yield results that are not representative. If that’s true or not - I do not know.

Now I need to calculate the embeddings. This is done in a following pass through, where I iterate over all the posts and fetch their embeddings from the api.

The third and final pass over the posts calculates the cosine similarity between each of the posts. It then takes the top 3 most similar posts, and marks their paths in the frontmatter struct’s similar posts property. The markdown documents are then saved.

This is all quite inefficient and I could optimize this, but I’m really not worried about the performance here.

There are also a few caveats - first, there’s a script that updates the markdown files that I’m working on. I’m not very fond of this, as it could lead to loss of data. If I were to run the script and while it’s executing change something, the change would get overwritten by the script once its job is complete. However, since I’m the only person working on this blog, and I have it version controlled I deem this an acceptable risk. I’ve had to do it this way as I could not figure out an alternative to hook into the hugo’s build chain.

If you’d like a deeper look at the code - the full code for the CLI is available here.

The Hugo template

With all my posts now containing a similar_posts property in their frontmatter, all that’s left to do is to adjust the template I use to render the related posts section. The template will render the similar_posts property if it’s present. If it’s not, it will fall back to the Hugo’s built-in related feature.

Here’s what it looks like:

{{ $similarPosts := .Params.similar_posts }}
{{ if and (isset .Params "similar_posts") (gt (len $similarPosts) 0) }}
  <h4>Related posts</h4>
  <ul>
   {{ range $index, $simpost := $similarPosts }}
    {{ with $.Site.GetPage $simpost }}
      <li><a href="{{ .RelPermalink }}">{{ .LinkTitle }}</a></li>
    {{ end }}
   {{ end }}
  </ul>
{{ else }}
  {{ $related := .Site.RegularPages.Related . | first 3 }}
  {{ if gt (len $related) 0 }}
    <h4>Related posts</h4>
    <ul>
      {{ range $related }}
        <li><a href="{{ .RelPermalink }}">{{ .LinkTitle }}</a></li>
      {{ end }}
    </ul>
  {{ end }}
{{ end }}

With this, the blog is armed to provide ML related posts instead of relying on keywords and tags.

The results

Let’s take my previous post - Probabilistic Early Expiration in Go - and compare the tag based related posts vs the ML related posts.

The old:

The new:

For this particular case, I feel like the tag based recommendations are semantically closer to what I would expect. Nevertheless, I’m planning on sticking with the ML generated ones at least for a short while to get to know them better!

So, is it all worth it? Well, probably not. I wasn’t really doing this for any sort of tangible result. I don’t even have the metrics to verify that this indeed has any effect. However, this was a fun little experiment and a good entry to getting to know ML a little. I’ll probably do more and deeper dives into ML at some point and maybe re-visit this feature for a second iteration in the future.

Thanks for reading! If you’ve got any comments or suggestions, do leave a comment below.

Probabilistic Early Expiration in Go

Viktoras — Mon, 23 Sep 2024 11:14:06 +0000

About cache stampedes

I often end up in situations where I need to cache this or that. Often, these values are cached for a period of time. You’re probably familiar with the pattern. You try to get a value from cache, if you succeed, you return it to the caller and call it a day. If the value is not there, you fetch it(most likely from the database) or compute it and the put it in the cache. In most cases, this works great. However, if the key you’re using for your cache entry gets accessed frequently and the operation to compute the data takes a while you’ll end up in a situation where multiple parallel requests will simultaneously get a cache miss. All of these requests will independently load the from source and store the value in cache. This results in wasted resources and can even lead to a denial of service.

Let me illustrate with an example. I’ll use redis for cache and a simple Go http server on top. Here’s the full code:

package main

import (
    "errors"
    "log"
    "net/http"
    "time"

    "github.com/redis/go-redis/v9"
)

type handler struct {
    rdb *redis.Client
    cacheTTL time.Duration
}

func (ch *handler) simple(w http.ResponseWriter, r *http.Request) {
    cacheKey := "my_cache_key"
    // we'll use 200 to signify a cache hit & 201 to signify a miss
    responseCode := http.StatusOK
    cachedData, err := ch.rdb.Get(r.Context(), cacheKey).Result()
    if err != nil {
        if !errors.Is(err, redis.Nil) {
            log.Println("could not reach redis", err.Error())
            http.Error(w, "could not reach redis", http.StatusInternalServerError)
            return
        }

        // cache miss - fetch & store
        res := longRunningOperation()
        responseCode = http.StatusCreated

        err = ch.rdb.Set(r.Context(), cacheKey, res, ch.cacheTTL).Err()
        if err != nil {
            log.Println("failed to set cache value", err.Error())
            http.Error(w, "failed to set cache value", http.StatusInternalServerError)
            return
        }
        cachedData = res
    }
    w.WriteHeader(responseCode)
    _, _ = w.Write([]byte(cachedData))
}

func longRunningOperation() string {
    time.Sleep(time.Millisecond * 500)
    return "hello"
}

func main() {
    ttl := time.Second * 3
    rdb := redis.NewClient(&redis.Options{
        Addr: "localhost:6379",
    })

    handler := &handler{
        rdb: rdb,
        cacheTTL: ttl,
    }

    http.HandleFunc("/simple", handler.simple)
    if err := http.ListenAndServe(":8080", nil); err != nil {
        log.Fatalf("Could not start server: %s\n", err.Error())
    }
}

Let’s put some load on the /simple endpoint and see what happens. I’ll use vegeta for this.

I run vegeta attack -duration=30s -rate=500 -targets=./targets_simple.txt > res_simple.bin. Vegeta ends up making 500 requests every second for 30 seconds. I graph them as a histogram of HTTP result codes with buckets that span 100ms each. The result is the following graph.

When we start the experiment the cache is empty - we have no value stored there. We get the initial stampede as a bunch of requests reach our server. All of them check the cache find nothing there, call the longRunningOperation and store it in cache. Since the longRunningOperation takes ~500ms to complete any requests made in the first 500ms end up calling longRunningOperation. Once one of the requests manages to store the value in the cache all the following requests fetch it from cache and we start seeing responses with the status code of 200. The pattern then repeats every 3 seconds as the expiry mechanism on redis kicks in.

In this toy example this doesn’t cause any issues but in a production environment this can lead to unnecessary load on your systems, degraded user experience or even a self induced denial of service. So how can we prevent this? Well, there’s a few ways. We could introduce a lock - any cache miss would result in code trying to achieve a lock. Distributed locking is not a trivial thing to do and often these have subtle edge cases that require delicate handling. We could also periodically re-compute the value using a background job but this requires an extra process to be running introducing yet another cog that needs to be maintained and monitored in our code. This approach might also not be feasible to do if you have dynamic cache keys. There is another approach, called probabilistic early expiration and this is something I’d like to explore further.

Probabilistic early expiration

This technique allows one to recompute the value based on a probability. When fetching the value from cache, you also compute if you need to regenerate the cache value based on a probability. The closer you are to the expiry of the existing value, the higher the probability.

I’m basing the specific implementation on XFetch by A. Vattani, F.Chierichetti & K. Lowenstein in Optimal Probabilistic Cache Stampede Prevention.

I’ll introduce a new endpoint on the HTTP server which will also perform the expensive calculation but this time use XFetch when caching. For XFetch to work, we need to store how long the expensive operation took(the delta) and when the cache key expires. To achieve that, I’ll introduce a struct that will hold these values as well as the message itself:

type probabilisticValue struct {
    Message string
    Expiry time.Time
    Delta time.Duration
}

I add a function to wrap the original message with these attributes & serialize it for storing in redis:

func wrapMessage(message string, delta, cacheTTL time.Duration) (string, error) {
    bts, err := json.Marshal(probabilisticValue{
        Message: message,
        Delta: delta,
        Expiry: time.Now().Add(cacheTTL),
    })
    if err != nil {
        return "", fmt.Errorf("could not marshal message: %w", err)
    }

    return string(bts), nil
}

Let’s also write a method to recompute and store the value in redis:

func (ch *handler) recomputeValue(ctx context.Context, cacheKey string) (string, error) {
    start := time.Now()
    message := longRunningOperation()
    delta := time.Since(start)

    wrapped, err := wrapMessage(message, delta, ch.cacheTTL)
    if err != nil {
        return "", fmt.Errorf("could not wrap message: %w", err)
    }
    err = ch.rdb.Set(ctx, cacheKey, wrapped, ch.cacheTTL).Err()
    if err != nil {
        return "", fmt.Errorf("could not save value: %w", err)
    }
    return message, nil
}

To determine if we need to update the value based on the probability, we can add a method to probabilisticValue:

func (pv probabilisticValue) shouldUpdate() bool {
    // suggested default param in XFetch implementation
    // if increased - results in earlier expirations
    beta := 1.0
    now := time.Now()
    scaledGap := pv.Delta.Seconds() * beta * math.Log(rand.Float64())
    return now.Sub(pv.Expiry).Seconds() >= scaledGap
}

If we hook it all up we end up with the following handler:

func (ch *handler) probabilistic(w http.ResponseWriter, r *http.Request) {
    cacheKey := "probabilistic_cache_key"
    // we'll use 200 to signify a cache hit & 201 to signify a miss
    responseCode := http.StatusOK
    cachedData, err := ch.rdb.Get(r.Context(), cacheKey).Result()
    if err != nil {
        if !errors.Is(err, redis.Nil) {
            log.Println("could not reach redis", err.Error())
            http.Error(w, "could not reach redis", http.StatusInternalServerError)
            return
        }

        res, err := ch.recomputeValue(r.Context(), cacheKey)
        if err != nil {
            log.Println("could not recompute value", err.Error())
            http.Error(w, "could not recompute value", http.StatusInternalServerError)
            return
        }
        responseCode = http.StatusCreated
        cachedData = res

        w.WriteHeader(responseCode)
        _, _ = w.Write([]byte(cachedData))
        return
    }

    pv := probabilisticValue{}
    err = json.Unmarshal([]byte(cachedData), &pv)
    if err != nil {
        log.Println("could not unmarshal probabilistic value", err.Error())
        http.Error(w, "could not unmarshal probabilistic value", http.StatusInternalServerError)
        return
    }

    if pv.shouldUpdate() {
        _, err := ch.recomputeValue(r.Context(), cacheKey)
        if err != nil {
            log.Println("could not recompute value", err.Error())
            http.Error(w, "could not recompute value", http.StatusInternalServerError)
            return
        }
        responseCode = http.StatusAccepted
    }

    w.WriteHeader(responseCode)
    _, _ = w.Write([]byte(cachedData))
}

The handler works much like the first one, however, upon getting a cache hit we roll the dice. Depending on the outcome we either just return the value we just fetched, or update the value early.

We’ll use the HTTP status codes to determine between the 3 cases:

200 - we returned the value from cache
201 - cache miss, no value present
202 - cache hit, triggered probabilistic update

I start up vegeta once again this time running against the new endpoint and here’s the result:

The tiny blue blobs there indicate when we actually ended up updating the cache value early. We no longer see cache misses after the initial warm up period. To avoid the initial spike you could pre-store the cached value if this is important for your use case.

If you’d like to be more aggressive with your caching and refresh the value more frequently, you can play with the beta parameter. Here’s what the same experiment looks like with the beta param set to 2:

We’re now seeing probabilistic updates way more frequently.

All in all this is a neat little technique that can help with avoiding cache stampedes. Keep in mind though, this only works if you are periodically fetching the same key from the cache - otherwise you won’t see much benefit.

Got another way of dealing with cache stampedes? Noticed a mistake? Let me know in the comments below!

SQLC & dynamic queries

Viktoras — Wed, 03 Jul 2024 13:54:19 +0000

SQLC has become my go-to tool for interacting with databases in Go. It gives you full control over your queries since you end up writing SQL yourself. It then generates models and type safe code to interact with those queries.

I won’t go over the basics here, if you feel like it you can try their interactive playground.

Dynamic queries

Frequently I end up needing to filter the data by a set of fields in the database. This set of fields is often determined by the caller, be it via REST API or other means. This means that the code I’m writing has to support dynamic queries, where we query by a subset of fields.

Let’s see an example. Assume my API returns some car data and I store it in the following table:

CREATE TABLE cars(
  id SERIAL PRIMARY KEY, 
  brand VARCHAR(255) NOT NULL,
  model VARCHAR(255) NOT NULL,
  year INT NOT NULL,
  state VARCHAR(255) NOT NULL,
  color VARCHAR(255) NOT NULL,
  fuel_type VARCHAR(50) NOT NULL,
  body_type VARCHAR(50) NOT NULL
);

The user might want to filter by brand, or by model. Or by brand and model. Or by brand, color, model, state and body type. You get the point, there’s a whole bunch of permutations here and SQLC is not great at handling this.

I usually approach it with the following SQLC query:

SELECT * FROM cars
  WHERE brand = @brand -- mandatory fields go in like this 
  AND (NOT @has_model::boolean or model = @model) -- optional fields follow this pattern
  AND (NOT @has_year::boolean or year = @year)
  AND (NOT @has_state::boolean or state = @state)
  AND (NOT @has_color::boolean or color = @color)
  AND (NOT @has_fuel_type::boolean or fuel_type = @fuel_type)
  AND (NOT @has_body_type::boolean or body_type = @body_type);

It might not be the prettiest solution, but it has worked for me quite well. There are a couple of downsides though. First, the param struct that the SQLC generates contains quite a bunch of fields:

type GetCarsParams struct {
    Brand string
    HasModel bool
    Model string
    HasYear bool
    Year int32
    HasState bool
    State string
    HasColor bool
    Color string
    HasFuelType bool
    FuelType string
    HasBodyType bool
    BodyType string
}

You’ll have to handle this in your code, to set the proper ones if the user provides the relevant input. Second, people always ask me if there is a cost associated with having such a query versus using something like a query builder to build the query with specific params. To this, I always answer: “Probably, but it’s unlikely you’ll notice”.

I have decided to try and benchmark this, to see if there is a meaningful cost associated with it.

Benchmarking setup

I’ll use postgres & pgbench, since I’m only really interested in the query performance ignoring any code overhead. For schema, we’ll use the table above.

We’ll seed the data with the following query:

INSERT INTO cars(brand, model, YEAR, state, color, fuel_type, body_type)
SELECT (CASE FLOOR(RANDOM() * 5)::INT
            WHEN 0 THEN 'Toyota'
            WHEN 1 THEN 'Ford'
            WHEN 2 THEN 'Honda'
            WHEN 3 THEN 'BMW'
            WHEN 4 THEN 'Tesla'
        END) AS brand,
       (CASE FLOOR(RANDOM() * 5)::INT
            WHEN 0 THEN 'Camry'
            WHEN 1 THEN 'F-150'
            WHEN 2 THEN 'Civic'
            WHEN 3 THEN '3 Series'
            WHEN 4 THEN 'Model S'
        END) AS model,
       (CASE FLOOR(RANDOM() * 5)::INT
            WHEN 0 THEN 2024
            WHEN 1 THEN 2023
            WHEN 2 THEN 2022
            WHEN 3 THEN 2021
            WHEN 4 THEN 2020
        END) AS YEAR,
       (CASE FLOOR(RANDOM() * 3)::INT
            WHEN 0 THEN 'Operational'
            WHEN 1 THEN 'Under maintenance'
            WHEN 2 THEN 'Totalled'
        END) AS state,
       (CASE FLOOR(RANDOM() * 5)::INT
            WHEN 0 THEN 'Red'
            WHEN 1 THEN 'Green'
            WHEN 2 THEN 'Blue'
            WHEN 3 THEN 'Black'
            WHEN 4 THEN 'White'
        END) AS color,
       (CASE FLOOR(RANDOM() * 3)::INT
            WHEN 0 THEN 'Diesel'
            WHEN 1 THEN 'Petrol'
            WHEN 2 THEN 'Electric'
        END) AS fuel_type,
       (CASE FLOOR(RANDOM() * 3)::INT
            WHEN 0 THEN 'Sedan'
            WHEN 1 THEN 'SUV'
            WHEN 2 THEN 'Hatchback'
        END) AS body_type
FROM GENERATE_SERIES(1, 10000000) seq;

We’ll then have two different queries - one that we’d build with a query builder:

SELECT * FROM cars
WHERE brand = 'Ford'
and model = 'Model S'   
and year = 2021
and color = 'Green'
and fuel_type = 'Diesel';

And one where we’d have some extra overhead from our SQLC approach:

SELECT * FROM cars
  WHERE brand = 'Ford' 
  AND (NOT true::boolean or model = 'Model S')
  AND (NOT true::boolean or year = 2021)
  AND (NOT false::boolean or state = '')
  AND (NOT true::boolean or color = 'Green')
  AND (NOT true::boolean or fuel_type = 'Diesel')
  AND (NOT false::boolean or body_type = '');

I’ve also added a composite index for these specific queries:

CREATE INDEX idx_cars_brand_model_year_color_fuel 
ON cars (brand, model, year, color, fuel_type);

Armed with this, I’ve spun up an instance of postgres in docker on my machine, created the schema and generated the dataset. I then pre-warmed the cache by executing a couple of pgbench runs but not logging any results.

From there, I ran pgbench 4 times:

pgbench -f builderq.sql --log --log-prefix=builder --transactions=10000 -j 5 -c 5
pgbench -f sqlcq.sql --log --log-prefix=sqlc --transactions=10000 -j 5 -c 5
pgbench -f builderq.sql --log --log-prefix=builder --transactions=1000 -j 5 -c 5
pgbench -f sqlcq.sql --log --log-prefix=sqlc --transactions=10000 -j 5 -c 5

And these are the latencies for both queries after being run through plotly:

There is very little difference between the two, with SQLC approach having a larger number of outliers. Despite this, I would state that there is no significant difference between the two.

I then re-ran the experiment, only this time introduced a limit of 1 on both queries. My thought process was that this would perhaps allow the faster(in theory) query to shine, since we would not have to spend so much time transferring data. Here is the box plot:

The SQLC approach does seem a tad slower here, but not by a lot.

In conclusion, there is very little difference in terms of performance for these queries, meaning that using the a bunch of ANDs to implement dynamic querying in SQLC is something you can do without fear of massive performance repercussions. Your mileage may vary and in extreme cases this might not be these - always benchmark your own use cases.

The only price you pay for manually constructing the dynamic queries is the manual work involved in writing the queries & mapping of values from the request parameters to the query parameters.

Thanks for reading! If you’ve enjoyed my babbling you might also like my post on SQL string constant gotcha.

Enums in Go

Viktoras — Fri, 26 Jan 2024 10:11:20 +0000

I’ve seen many discussions about whether Go should add enum support to the language. I’m not going to bother arguing for or against but instead show how to make due with what we have in the language now.

A very short enum intro

Enumerated types, or enums, represent finite sets of named values. They are usually introduced to signal that variables can only take one of the predefined values for the enum. For example, we could have an enum called Colors with members Red, Green, Blue. Usually, the members are represented as integer values, starting from zero. In this case, Red would correspond to 0, Green to 1 and Blue to 2 with Red, Green and Blue being the names of corresponding members. They help simplify the code as they are self-documenting and explicitly list all possible values for the given type. In many languages enums will also return compile errors if you’ll try to assign to an invalid value. However, since enums do not exist in Go, we do not have such guarantees.

Defining a custom type

Usually, one of the first steps of defining an enum in Go is to define a custom type for it. There are 2 commonly used types for this purpose, string and int. Let’s start with strings:

type Color string
const (
    Red Color = "red"
    Green Color = "green"
    Blue Color = "blue"
)

I actively avoid defining my enums in this style, since using strings increases the likelihood of errors. You don’t really know if the enum members are defined in uppercase, lowercase, title case or something else entirely. Besides, there is a high chance of miss-spelling the strings both in the definition and subsequent use and blue becomes bleu. I also often use bitmasks so that might influence my judgement, but I’ll talk about bitmasks in a separate post at some point.

For these reasons I prefer an int declaration:

type Color int
const (
    Red Color = 0
    Green Color = 1
    Blue Color = 2
)

Keep in mind that this is highly subjective and people will have their own preferences. Also, the int definition does not read as nicely when displayed, but we will fix that later.

Iota in go

In the colors example, I’m only using three colors, but what if we had 5, 10 or 20 of them? It would be quite tedious to assign values to each and every single one of them. Luckily, we can simplify this by using the iota keyword that Go provides:

type Color int
const (
    Red Color = iota
    Green
    Blue
)
fmt.Println(Red, Green, Blue) // 0 1 2

Iota acts as syntactic sugar, automatically incrementing the value for each successive integer constant in a constant declaration.

If we’d like to start at another number, we can achieve this with the following:

type Color int

const (
    Red Color = iota + 42
    Green
    Blue
)
fmt.Println(Red, Green, Blue) // 42 43 44

You can also use a variety of expressions on iota but I hardly recommend that except for the most trivial of cases as this leads to code that is hard to read and comprehend. One of the common use cases of such expressions which is still readable is defining bitmasks:

type Color int

const (
    Red Color = 1 << iota
    Green
    Blue
    _ // this skips one value
    Yellow
)
fmt.Println(Red, Green, Blue, Yellow) // 1 2 4 16

For more on iota, please refer to the Go spec.

One thing to note is that you should be very careful when making changes to already established constant declarations with iota. It’s easy to cause headaches if you remove or change the order of members as these could have already been saved to a database or stored in some other way. Once you ingest those, what was once blue might become red so keep that in mind.

While such declarations might suffice as an enum in some circumstances you usually will expect more from your enum. For starters, you’d like to be able to return the name of the member. Right now, a fmt.Print(Red) will print 0, but how would we print the name? How would I determine if 4 is valid color or not? I’m also able to define a custom color by simply defining a variable var Brown Color = 7. What if I’d like to marshal my enum to their string representation when returning this via an API? Let’s see how we can address some of these concerns.

Getting the member name

Since we’ve defined Color as a custom type we can implement the stringer interface on it to get the member names.

func (c Color) String() string {
    switch c {
    case 0:
        return "Red"
    case 1:
        return "Green"
    case 2:
        return "Blue"
    }
    return fmt.Sprintf("Color(%q)", int(c))
}

We can now print the name by calling the .String() method on any of the Color and get the names out. There are many ways one could implement this method but all of them have the same caveat - whenever I add a new color in my constant declaration, I will also need to modify the .String() method. Should I forget to do so, I’ll have a bug on my hands.

Luckily, we can leverage code generation with the stringer tool can help us. It can generate the code required for our Color enum to implement the stringer interface. You’ll need to have the stringer tool installed so run go install golang.org/x/tools/cmd/stringer@latest to do that. Afterwards, include the following directive, I usually plop it right above my enum type declaration:

//go:generate stringer -type=Color
type Color int

If you run go generate ./... you’ll see a colors_string.go file appear, with the stringer interface implemented, allowing you to access the names of the members like so:

fmt.Println(Red.String(), Green.String(), Blue.String()) // Red Green Blue

Marshalling and unmarshalling

If we use our Color enum in a struct and marshal it it will be represented as int:

type MyResponse struct {
    Color Color
}

bts, _ := json.Marshal(MyResponse{Color: Blue})
fmt.Println(string(bts)) // {"Color":4}

Sometimes, this behavior might suit you. For instance, you might be OK if the value is stored as an integer however if you’re exposing this information to the end user it might make sense to display the color name instead. To achieve this, we can implement the MarshalText() ([]byte, error) method for our Color enum. I’m specifically implementing the MarshalText over MarshalJSON as the latter falls back to using MarshalText internally in the std libs json library. This means that by implementing it we will get the color represented as string in the marshalled form for both JSON, XML and text representations and, depending on the implementation, perhaps other formats and libraries.

func (c Color) MarshalText() ([]byte, error) {
    return []byte(c.String()), nil
}
// ...
bts, _ := json.Marshal(MyResponse{Color: Blue})
fmt.Println(string(bts)) // {"Color":"Blue"}

If we’d like to accept string colors as input, we’ll have to do a bit more work. First, we’ll need to be able to determine if a given string is a valid color for our enum or not. To achieve this, let’s implement a ParseColor function

var ErrInvalidColor = errors.New("invalid color")

func ParseColor(in string) (Color, error) {
    switch in {
    case Red.String():
        return Red, nil
    case Green.String():
        return Green, nil
    case Blue.String():
        return Blue, nil
    }

    return Red, fmt.Errorf("%q is not a valid color: %w", in, ErrInvalidColor)
}

Once again, we could implement this in many different ways, but they will have the downside that if we’re ever expanding our Color enum, we’ll have to go into the ParseColor and extend it to support our new members. There are tools that can generate this for us, and I’ll talk about them later.

With this, we can implement the UnmarshalText method and unmarshal an input with colors as strings like so:

func (c *Color) UnmarshalText(text []byte) error {
    parsed, err := ParseColor(string(text))
    if err != nil {
        return err
    }

    *c = parsed
    return nil
}

type MyRequest struct {
    Color Color
}

dest := MyRequest{}
_ = json.Unmarshal([]byte(`{"Color": "Blue"}`), &dest)
fmt.Println(dest.Color.String()) // Blue

If an invalid color is provided, the unmarshalling will result in an ErrInvalidColor error.

Similarly, we could implement the Valuer and Scanner interfaces for database interactions:

func (c *Color) Scan(v interface{}) error {
    col, ok := v.(string)
    if !ok {
        return fmt.Errorf("could not convert %T to color", v)
    }

    color, err := ParseColor(col)
    if err != nil {
        return err
    }

    *c = color
    return nil
}

func (c Color) Value() (driver.Value, error) {
    return c.String(), nil
}

Reducing boilerplate

If you’re working with quite a few enums and need the custom marshalling and stringer/valuer/scanner interface implementations it can become quite tedious having to do all these steps for each of your enums. Everything that I’ve discussed so far can be generated with the go-enum library. With it, the enum definition becomes a bit different:

//go:generate go-enum --marshal --sql

// ENUM(Red, Green, Blue)
type Color int

If you run go generate ./... a file will be generated including all the custom marshalling, parsing and stringer/valuer/scanner implementations. This is a great tool if you work with multiple enums and have to do this often.

Another alternative that leverages generics and avoids generation is the enum library.

Both of these are valid options and it is up to the reader to choose one that suits your needs. I will go over my preferences at the end of this blog post.

Improving type safety by using structs

There’s one caveat with these enums and that’s the fact that one can just construct a new enum member by hand. There’s nothing preventing me from defining a var Yellow = Color(3) and passing that to a function expecting a color:

func UseColors(c Color) {
    // would actually do something useful
    fmt.Println(c.String())
}

var Yellow = Color(3)
UseColors(Yellow) // completely valid code

Firstly, I would like to say that there is no bulletproof way to protect from this, but there are some things you can do.

If we define our enum as a struct in a separate package:

package colors

type Color struct {
    id int
    name string
}

var (
    Red = Color{id: 0, name: "Red"}
    Green = Color{id: 1, name: "Green"}
    Blue = Color{id: 2, name: "Blue"}
)

You would obviously include ways to construct valid enums from outside the package by either the ID or name, the methods for serializing, stringifying and any other needs you have. However, this only provides an illusion of safety, since you can do any of the following:

Reassign one color as another: colors.Red = colors.Blue.
Create an uninitialized color: var myColor = colors.Color{}.

For 2) we could shift our enum by 1, and include an unknown value with the id of 0.

var (
    Unknown = Color{id: 0, name: ""}
    Red = Color{id: 1, name: "Red"}
    Green = Color{id: 2, name: "Green"}
    Blue = Color{id: 3, name: "Blue"}
)

We’d still have to handle this unknown value in any code dealing with colors:

func UseColors(c colors.Color) error {
    if c == colors.Unknown {
        return errors.New("received unknown color")
    }
    // do something with the valid colors
    return nil
}

Since structs can not be declared const, we have to become inventive to cover ourselves from 1). We can define the colors as funcs:

func Unknown() Color {
    return Color{}
}

func Red() Color {
    return Color{id: 1, name: "Red"}
}

func Green() Color {
    return Color{id: 2, name: "Green"}
}

func Blue() Color {
    return Color{id: 3, name: "Blue"}
}

With this in place, you can no longer assign a color as another.

An alternative approach would be to make the color type an unexported int and only export its members:

type color int

const (
    Red color = iota
    Green
    Blue
)

To make this type even remotely useful, we could export and implement a Colors interface:

type Color interface {
    ID() int
    Name() string
    Equals(Color) bool
}

func (c color) ID() int {
    return int(c)
}

func (c color) Name() string {
    switch c {
    case 0:
        return "Red"
    case 1:
        return "Green"
    case 2:
        return "Blue"
    }
    return fmt.Sprintf("Color(%q)", int(c))
}

func (c color) Equals(in Color) bool {
    return c.ID() == in.ID()
}

You could then use it like this:

func UseColors(c colors.Color) {
    if c.Equals(colors.Red) {
        // do something with red
    }
}

In theory, you could still write a custom implementation of this interface and create a custom color like that but I think that is highly unlikely to happen.

However, I’m not a big fan of these approaches as they seem a tad cumbersome while providing little in return.

My personal enum preferences

With all this said, I’d like to point out a few things that have been working quite well for me in practice and my general experience:

Enums are not as widespread as one might think. Even in large code bases, you’re very likely to have only a handful of enums. Using libraries for this might be overkill.
I value consistency very highly when it comes to code so I usually follow whatever pattern is already established for enum definitions in the existing code base.
I only ever reference existing enum members and try to never type cast to an enum.
For fresh projects, I use an int based type with iota with any additional interfaces implemented by hand. If the time comes where this becomes tedious to maintain, I switch to code generation.
Unless enums become part of the language spec, I’ll stick to using iota based enums. Any fancy tricks to add more type safety to them just add more boilerplate. I trust the people I work with, our review processes and don’t feel the need for these extra safety measures.

This is just my opinion and it might not match the situation you’re in. Always choose what works best for you!

If you have any suggestions or alternatives, I’d be glad to hear them in the comments below.

Streaming Netdata metrics from TrueNAS SCALE

Viktoras — Sat, 20 Jan 2024 16:41:22 +0000

When you install TrueNAS SCALE your NAS runs a Netdata. You can verify that by executing systemctl status netdata in the TrueNAS shell. I use Netdata to monitor my homelab and have a parent set up for long term metric storage. I’d love to configure the NAS to also push metrics to a parent, so that I can access them all from a single place.

Normally, if you want to set up streaming in Netdata it’s enough to edit /etc/netdata/stream.conf. However, I wouldn’t recommend doing this on a TrueNAS install for a couple of reasons. Firstly, this is not a recommended way of adjusting configuration and that is clearly evident when you open up the TrueNAS shell:

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI, CLI, and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

Secondly, if you ignore these warnings and use the shell to adjust the configuration, you’ll still end up in a pickle. Trust me, I’ve tried. The Netdata version that comes preinstalled with TrueNAS SCALE(23.10.1.1) is v1.37.1. You can verify this by running curl localhost:6999/api/v1/info | grep version in the TrueNAS shell. This version is quite old, over a year old at the time of writing and seems to have bug in the streaming protocol. When I set it up to stream to my Netdata parent, the Netdata daemon in TrueNAS started crashlooping. This meant that no metrics were visible on the TrueNAS Web UI or being streamed to the parent.

At that point I realized I’d probably need to use the Netdata app on True NAS.

Configuring streaming to Netdata parent via the app

The good thing about this app is that it uses the Netdata helm chart under the hood, meaning new versions of Netdata are available. Simply install it from the available applications on the TrueNAS Web UI. As far as configuration goes, there is no need to configure anything.

Once the app is installed and running, open up a shell by hitting the button on the app menu:

We’re now inside the Netdata container running inside of the kubernetes cluster running on your TrueNAS SCALE. Using your favorite text editor, open /etc/netdata/stream.conf and paste the following:

[stream]
  enabled = yes
  destination = 192.168.1.1:19999
  api key = 11111111-2222-3333-4444-555555555555
  timeout seconds = 60
  buffer size bytes = 1048576
  reconnect delay seconds = 5
  initial clock resync iterations = 60

Replace 192.168.1.1:19999 with the host and port of your Netdata parent and 11111111-2222-3333-4444-555555555555 with your Netdata streaming API key.

The next thing you’ll probably want to do is assign this node a proper hostname. By default, Netdata uses the hostname of machine it’s running on and since we’re on a pod in kubernetes you’ll get something catchy like netdata-5f9684f696-mdrhf. Luckily, we can fix that. I will also make a few general adjustments to the Netdata configuration, switching it to in memory mode as well as disable some auxiliary features as that will get handled by the parent. This makes the installation lightweight, consuming less resources of our NAS.

Open /etc/netdata/netdata.conf and paste the following:

[global]
  hostname = my-catchy-hostname
[db]
  mode = ram
[health]
  enabled = no
[ml]
  enabled = no

Exit the shell, stop and start the app again. You should now be able to see the new node on your Netdata parent.

Claiming to Netdata Cloud

If you don’t have a parent but would like to be able to access the metrics from Netdata Cloud it is possible to claim it using the same TrueNAS app. To do so, first grab a few things from the cloud. You’ll need the claim token, and the room ID. You can get them by hitting the Add nodes button in the Netdata Cloud. You’ll see something like this:

wget -O /tmp/netdata-kickstart.sh https://get.netdata.cloud/kickstart.sh && sh /tmp/netdata-kickstart.sh --nightly-channel --claim-token {your_token} --claim-rooms 89f01a07-697e-40ce-b90c-3ee0091a02b5 --claim-url https://app.netdata.cloud

Next, head to the TrueNAS SCALE Web UI apps section, find and hit install on Netdata. However, this time some extra configuration is required. Under the Netdata Configuration you’ll want to add 3 environment variables:

NETDATA_CLAIM_TOKEN=your_token
NETDATA_CLAIM_URL=https://app.netdata.cloud
NETDATA_CLAIM_ROOMS=89f01a07-697e-40ce-b90c-3ee0091a02b5

After the installation is complete, you should see your node appear on Netdata cloud.

I hope this helps! If you’ve got any questions shoot them in the comments below.

SQL string constant gotcha

Viktoras — Tue, 02 Jan 2024 10:11:20 +0000

I was working on a project that uses PostgreSQL and as part of my task I needed to write a migration. The migrations are written as plain SQL and applied using the migrate library. The migration itself was not that complex, some rows needed to be updated where a column matched one of the values in a list.

In my rush, I’ve opted for a rather simple query that went something like this:

UPDATE some_table SET some_column = 'some_value'
WHERE some_other_column IN (
    'value_1',
    'value_2',
    'value_3',
    'value_4',
    -- ...
    'value_26',
    'value_27'
    'value_28',
    'value_29'
);

When the migration got executed on the testing environment, no errors were reported and all looked well. However, after some time I noticed that the migration failed to update some rows that I had anticipated.

The observant readers probably noticed that there’s a missing comma after value_27 in the previous snippet and this is indeed what caused the issue. What surprised me is that the query did not result in an error. Why is that? Let’s explore.

I’ll create a simple table and insert a few entries.

CREATE TABLE sessions(id SERIAL, browser varchar(50));

INSERT INTO sessions(browser)
VALUES ('chrome'), ('brave'), ('firefox');

Running the following query returns 3 rows, as expected.

SELECT * FROM sessions 
WHERE browser in (
    'chrome',
    'brave',
    'firefox'
);

-------------
-- id|browser|
-- --+-------+
-- 1|chrome |
-- 2|brave |
-- 3|firefox|

However, if I fail to include a comma after brave the resulting query returns a single row:

SELECT * FROM sessions 
WHERE browser in (
    'chrome',
    'brave'
    'firefox'
);

-------------
-- id|browser|
-- --+-------+
-- 1|chrome |

The query does not result in an error and still yields some results. To better understand what is happening we can further simplify this example to a simple SELECT:

SELECT 'brave'
'firefox';

-- -------------
-- ?column? |
-- ------------+
-- bravefirefox|

The original query failed to include the sessions with brave and firefox browsers because brave and firefox got concatenated resulting in the following query:

SELECT * FROM sessions 
WHERE browser in (
    'chrome',
    'bravefirefox'
);

What I find odd is that this behavior only exists if you include at least one newline between the two string constants. This means that

SELECT 'brave' 'firefox';

results in a syntax error.

This behavior is documented in the PostgreSQL documentation:

This slightly bizarre behavior is specified by SQL; PostgreSQL is following the standard.

After scratching my head for a bit, I then went on to try and figure out a way to minimize the chance of this happening to me in the future. If you’re working with strings using ANY and providing an array as a parameter to it should keep you safe:

SELECT * FROM sessions 
WHERE browser = ANY('{
  "chrome",
  "brave",
  "firefox"
}');

Should you omit a comma it will let you know that you’ve got a malformed array literal.