DEV Community: Siddarth

Multi-tenant Loki on Kubernetes

Siddarth — Wed, 17 Dec 2025 16:44:41 +0000

I'm known as the "monitoring guy" in the company I work at, and this post is about how I deployed Grafana Loki in SimpleScalable mode on Kubernetes and used Promtail to ingest logs per tenant.

What's Loki ?

Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus.

If you are using Loki for the first time, I highly recommend you to checkout their docs. Loki Docs

What we need

A Loki deployment (SimpleScalable via Helm)
Promtail
S3 bucket

Loki setup (SimpleScalable + S3 storage)

The deployment relies on object storage, so you need a s3 compatible bucket and its credentials to store the logs.

Our Helm chart is pretty simple.

loki-helm-chart/
  ├── templates/
  │   └── loki-secrets.yaml
  ├── Chart.yaml
  └── values.yaml

Now, the loki-secrets.yaml would have our s3 credentials secret

apiVersion: v1
kind: Secret
metadata:
  name: loki-s3-credentials
  namespace: monitoring
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: "<id>"
  AWS_SECRET_ACCESS_KEY: "<access-key>"

Loki values.yaml

deploymentMode: SimpleScalable

loki:
  auth_enabled: true

  extraEnvFrom:
    - secretRef:
        name: loki-s3-credentials

  schemaConfig:
    configs:
      - from: "2024-04-01"
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h

  storage_config:
    tsdb_shipper:
      index_gateway_client:
        server_address: '{{ include "loki.indexGatewayAddress" . }}'


  storage:
    type: s3
    s3:
      endpoint: rook-ceph-rgw-store.rook-ceph.svc.cluster.local:80
      s3ForcePathStyle: true
      insecure: true


    bucketNames:
      chunks: loki-logs
      ruler: loki-ruler
      admin: loki-admin


  commonConfig:
    replication_factor: 3

  ingester:
    chunk_encoding: snappy

  querier:
    multi_tenant_queries_enabled: true
    max_concurrent: 4

  pattern_ingester:
    enabled: true

  limits_config:
    allow_structured_metadata: true
    volume_enabled: true

A couple of important notes:

auth_enabled: true means Loki expects a tenant identifier (via X-Scope-OrgID) for multi-tenancy.
Retention in Loki is implemented by the Compactor. If retention isn’t enabled there, logs can live forever even if retention_period is set. (Log Retention)

Rest of the values

Loki components still need local disk for things like WAL / cache / working state so they can perform well. This is why you’ll see PVCs on the read/write/backend pods.

ceph-block is our expandable storage class by the way

backend:
  replicas: 2
  persistence:
    enabled: true
    size: 2Gi
    storageClass: ceph-block
    accessModes:
      - ReadWriteOnce
  extraEnvFrom:
    - secretRef:
        name: loki-s3-credentials

write:
  replicas: 3
  persistence:
    enabled: true
    size: 2Gi
    storageClass: ceph-block
    accessModes:
      - ReadWriteOnce
  extraEnvFrom:
    - secretRef:
        name: loki-s3-credentials

read:
  replicas: 2
  extraEnvFrom:
    - secretRef:
        name: loki-s3-credentials

Gateway config (what it does and does not do)

gateway:
  nginx:
    customHeaders:
      - name: X-Scope-OrgID
        value: $http_x_scope_orgid

This does not enforce isolation by itself, it just forwards whatever tenant header the caller provides.

Grafana’s docs recommend that X-Scope-OrgID be set by an authenticating reverse proxy (so users can’t spoof other tenants). (Grafana Labs)

In my setup, clients do not directly access Loki. Only Promtail (inside the cluster) pushes logs, and I control the tenant mapping, so forwarding this header is fine for now.

Great ! Now moving on to Promtail

Promtail setup

Promtail runs on each node and ships container logs to Loki

daemonset:
  enabled: true
deployment:
  enabled: false

config:
  clients:
    - url: http://loki-gateway.monitoring.svc.cluster.local:80/loki/api/v1/push
      timeout: 60s
      batchwait: 1s
      batchsize: 1048576

  serverPort: 3101

The important part: setting the tenant correctly

I used the Kubernetes namespace into a namespace label:

snippets:
  common:
    - action: replace
      source_labels: [__meta_kubernetes_namespace]
      target_label: namespace

Now in the pipeline, set tenant from the label:

snippets:
  pipelineStages:
    - cri: {}
    - tenant:
        label: namespace

And voila ! we have a multi-tenant Loki setup done.

Now, we can add the Loki data source in Grafana

Setup X-Scope-OrgID, I put the value as monitoring

And now we will get to only see logs from monitoring namespace

A HUGEE shoutout to Ben Ye for his help ! Github Profile Link

Hope you found this useful. I'm always up for a quick chat, connect with me on Linkedin or Twitter

My Linux Kernel Mentorship experience

Siddarth — Tue, 03 Jun 2025 15:59:19 +0000

I always loved tweaking around with my computer, trying various Linux distros and window managers. Contributing to the Linux kernel had always been on my bucket list.

I first heard about the LFX Mentorship Program at KubeDay India, and the idea of Linux kernel bug fixing immediately caught my eye.

I applied last year but didn’t get in. Fast forward to this year, and when I received the acceptance email, I was literally yelling! It was an unforgettable moment.

My experience from last year in setting up the kernel build environment helped me finish my tasks a bit quicker this time. It felt like those earlier efforts weren't wasted.

Mentorship and Community

The program began in March 2025. The initial days involved a lot of learning, diving into videos, blogs, and experiencing a little bit of panic that I hadn't submitted any patches yet.

Office hours to the rescue! Our mentor, Shuah, taught us a lot about the kernel ecosystem, cscope, Git tips, and how to interpret syzkaller reports. This was incredibly helpful.

Being able to talk with fellow mentees and share our learnings in the discord server made a huge difference.

The greatest part of the Linux community is its vast amount of resources. You truly learn a lot from studying other people’s patches. The kernel mailing list archives (lore) became my bread and butter. Over three months, I went through countless patches from many contributors, which taught me a ton.

A shout out to Nathan Chancellor and Kuan-Wei Chiu for answering all my questions and for guidance.

Key Learnings

Over these 12 weeks, I experimented with various tools and learned many aspects of the kernel. I submitted 13 patches, experiencing both successes and rejections.

Mindset: When my first patch was merged, I hesitated to send similar fixes because I was afraid of harsh feedback. Ironically, those same changes were later sent in by others were accepted. This experience changed my mindset and I no longer fear making mistakes. Every patch has been a learning experience.

I once sent a patch to the wrong maintainer (thanks, Zsh autocomplete). After making a silly mistake like that, I guess there's no more room to be afraid! :)

Writing Good Patches: Reading the documentation for the subsystem you’re working on is so important. Always check how patches for that subsystem are being sent in the archives.
Avoiding Churn: Avoid sending patches that adds unnecessary noise. When working on a bug it's important to understand where a particular error is coming from and how your fix solves it, and most importantly to test thoroughly!

While my mentorship comes to an end, I still have a lot of things to do and hope to make some significant contributions!

Introduction to Message Brokers: Kafka essentials

Siddarth — Mon, 10 Feb 2025 20:59:22 +0000

Let's say we have a blog website like Dev.to! So, every time a user publishes a blog, the server needs to update the user's blog count in the database.

With millions of users publishing blogs, API server has to compute and update the DB.

Isn't that too much of load for the API server ?
Why not let someone else do this job.

This is where Message Brokers come in. Who knows dev.to might also be using these

Message Brokers

Instead of the API server directly updating the database, it sends a message to the broker. The broker stores the message in a queue or stream, and worker components process it later.

So our worker component have two jobs in our scenerio

Updating Count in DB
Indexing

How Message Brokers Work

Message brokers can be implemented in two main ways:

1. Message Queues (e.g., RabbitMQ, SQS)

In a message queue, messages are stored in a queue and pulled by consumers (workers). For example, a worker might update the blog count in the database and perform indexing.

Once a message goes to one consumer it cannot go to another one.

Great! Whats the problem then ?

What if the worker crashes after updating the database but before indexing? This can lead to inconsistencies and this is why we go for message streams

2. Message Streams (e.g., Kafka, Kinesis)

In message streams like Kafka or Kinesis, messages are loaded into a stream, and consumers process them at their own pace.

Unlike queues, messages are not pulled by consumers. Instead, consumers independently go through the stream and process data.

This approach ensures better fault tolerance and consistency, as each consumer processes messages independently.

Conclusion

Well this is high level architecture of things, I hope you found this helpful. Thanks for reading :)

Faster Terminal Navigation with Tmux and Fuzzy finder

Siddarth — Tue, 21 Jan 2025 20:19:23 +0000

Using the terminal is one of my favourite things, its just so fasst and its the jam of us linux nerds.

Buut I've faced with one problem..

The Problem: Too Many Terminal Windows

Yep, In this blog, we'll tackle exactly this. Though there are many workarounds and tools for this but here is what worked for me

Introducing Tmux: The Terminal Multiplexer

Gosh I just love this thing! So Tmux is a powerful tool that creates sessions on top of your terminal. Let's just see this in practice

Installing Tmux

To get started with Tmux, install it using your favourite package manager. For our lovely linux community just do this:

sudo apt-get install tmux

Well windows users can open up WSL i guess..

Basic Tmux Usage

Let's walk through the basics of using Tmux:

Open a terminal and type
```
tmux
```
The green bar at the bottom indicates that you're in a Tmux session.
Run the below Python command
```
python -c "import time; print('Running...'); 
time.sleep(300)"
```
This command just shows Running... for 300 seconds
Now close the Terminal

Everything seems to be gone, right? Now type the below command
```
tmux attach
```
And Voila! We're back in the previous session, exactly where we left off.

What the heck happened?

Tmux runs as a server, so even after you close the terminal, your sessions remain active.

Some basic Tmux commands:

List active sessions
```
tmux list-sessions
```
Create a new detached session
```
tmux new-session -d
```

Running tmux list-sessions again: You'll now see two sessions listed.

Once you're in a session, try tmux new-window. Notice how the asterisk moves to the new window number.

You can create multiple windows this way.

Let's Dive into Tmux Navigation—the Real Power of This Tool!

The default Tmux prefix key is Ctrl+B. Here's how to use it:

Press Ctrl+B
RELEASE BOTH KEYS and then press either p (previous) or n (next) to switch windows

You have no idea how many times i have failed to do this, releasing the prefix key is so, so important.

I have spent a lot of time cursing tmux and figuring out why in the world it was not working because of this dumb mistake.

Let's Take It to the Next Level

#!/bin/bash
session=$(find ~/.config ~/personal ~/courses -mindepth 1 -maxdepth 1 -type d | fzf)
session_name=$(basename "$session" | tr . _)
if ! tmux has-session -t "$session_name" 2>/dev/null; then 
    tmux new-session -s "$session_name" -c "$session" -d
fi 
tmux switch-client -t "$session_name"

For my zsh folks, the first line might be

#!/usr/bin/env bash

So are my three frequently used folders – personal, courses, and .config – our bash script utilizes fzf (fuzzy finder) to quickly find the desired folder and then creates a new Tmux session if one doesn't already exist.

Integrate with Your Terminal

Add a shortcut in your shell configuration (e.g., .zshrc or .bashrc) so that you can run the script quickly:

bindkey -s ^f "/home/<username>/session.sh\n"

Replace <username> with your actual username. This example binds the script to Ctrl+F, giving you instant access to your Tmux sessions.

Wrapping Up

I personally found this very helpful. I have like a super fast workflow now and really love using this.

Thank you so much for reading. You're Awesome!

You can check out my Tmux config here.

Sources

The Primeagen's Dev Productivity Shell