DEV Community: Shehroz Ali

Achieving 99.99% uptime on single Pod rollout using NEG-based Network Load Balancing

Shehroz Ali — Tue, 29 Sep 2026 15:03:52 +0000

If you’re using an external passthrough Network Load Balancer sitting in front of your Kuberentes cluster, facts are most of the times its not aware of your Service Pod’s container health which results in users receiving Connection Reset (RST) or invalid response.

This becomes increasingly more critical during peak hours of your traffic or especially during times when RPS is extremely high and you can’t afford to drop any network packet or cause connection reset as it can long-term hurt you business or revenue.

In our company, we’ve been noticing for a long time that some of our uptime monitoring checks starts failing randomly especially during node upgrades, rollouts, pod restarts or any upgrade/maintainance work going on. At first, we thought this could be a random network blip over the wire, but sooner it became clear that this was happnening specifically whenever a Kubernetes node goes upgrading or workloads migrate to existing/new nodes.

We started digging down and after a while it got clear to us that this was happening in our single-replica deployment meaning 1 proxy (NGNIX) pod on each node sitting behind 1 global external Network Load Balancer.

Let me get into straight

In a traditional NLB setup, the Load Balancer picks any random VM from the list (target-pool based) and forwards the TCP packet to that VM. Once the packet is forwarded to the VM, LB did its job and moves on to forwarding next packet. That’s all. So in short, the LB is not aware of the running Service Pod health, if Pod is Not Ready/Terminating LB will still end up forwarding traffic to that VM.

That’s what causing connection lost and reset.

Google Cloud offers another type of more modern Network Load Balancer and its called Regional service-based Network Load Balancer. So instead of maintaing a fixed target-pool of instances, it maintains NEGs (Network Endpoint Groups) which contains the VM IP inside. The interesting thing is the NEG list gets updated by a dedicated NEG-controller running inside GKE control plane which watches Kubernetes EndpointSlice specifically for the app defined in Service selector label for Load Balancer.

To enable regional external NLB, add this in your manifest and apply (immutable):

apiVersion: v1
kind: Service
metadata:
  labels:
    app: proxy
  name: tls-proxy
spec:
  externalTrafficPolicy: Local
  loadBalancerClass: networking.gke.io/l4-regional-external # add this

Now let me walk you down through a scenario:

Pod goes in Terminating state

EndpointSlice gets updated marking that Pod Not Ready and Cilium reponds with 503 to mark node as unhealthy

NEG controller watches EndpointSlice and updates the NEGs removing the VM IP of the terminating Pod

LB send new connections to other nodes running health replica

You might will think we’re done here and no more connection reset right? The answer is no!

You just solved the problem for new connections routing to other nodes running healthy replica but what about existing connections with that VM with terminating Pod inside?

Now either could be your situation:

Pod goes in Terminating state or shutting down during node upgrade, rollout, restart, etc. (we can fix)
Pod dies immidiately, crashed, killed, server error (your luck!)

If a Pod is in terminating state and you want to gracefully handle all existing connections without dropping any packets you need to tweak and playaround your proxy settings and LB connection draining timeout.

In our case since we’re using NGNIX as our proxy layer for TLS and backend routing, we tweaked these settings and observed close to 99.99% uptime without connection loss.

Added sleep 40s in preStop to let NGNIX stay alive and keep serving exisiting connections
Set Connection Draining duration of regional LB to 30s to allow existing connections to finish before cutting off and removing from NEGs
Added keepalive_time to 15s (or your choice) to gracefully close existing connections after 15s and let user/client open fresh TCP connection with other VMs running healthy replica

During 40s NGNIX is still alive and serving so readiness probe keeps passing which means any packet entererd in VM until draining Cilium will forward it to NGNIX pod as EndpointSlice is still serving: true for terminating pod. Also we’ve externalTrafficPolicy: Local set which means no internal network hop allowed so the terminating NGNIX Pod stays serving and Cilium uses it as a fallback
Make sure to increase terminationGracePeriodSeconds to sleep + N seconds so you allow enough time for shutdown before SIGKILL
All existing connections gracefully closes within 40s sleep duration and no packet lost!

New connections gets routed to other nodes running healthy replica of your LoadBalancer Service Pod

Once the new Pod is up and Ready, NEG controller will attach the same VM to NEGs again and new traffic starts coming.

Why the AI Era Demands a Shift from Product to Platform Engineering

Shehroz Ali — Sat, 16 May 2026 14:33:01 +0000

Right now, while a lot of software engineering is moving up the abstraction layer (with LLMs writing routine product code), the physical and architectural realities of where and how that code runs have never been more critical.

My Backstory — The Product Era (The Abstraction)

I started where most people do — focusing on user value, feature delivery, and rapid iteration. But eventually, the magic wears off when you realize you’re constantly building on top of systems you don’t fully control or understand. Everything is abstracted under the hood — application frameworks, libraries, methods, databases, memory management, etc. It kind of felt repetitive to me just calling functions/utils/methods to process data for input/output and return it back to customer.

Especially after AI era, it didn’t feel like I’m actually solving any hardcore engineering problem, I sort of became more like a business/product person rather than calling myself an “engineer”. There was no engineering left, it was just prompting, gathering requirements, doing CRUD, reviewing, testing, merging code and that’s all. And all this because we programmers were already working at the most abstracted part of a software — very far from hardware.

AI made me realize, the job of a programmer was essentially not the hardest part in the past let’s say 10 years because everything was already solved and abstracted. Our job was to call functions/methods, wait for data to get fetched, perform some math on it and return it back. There was no more thinking left to dig down hardware and question why and how to optimize it better. Everything felt repetitive to me, same APIs, same business requirements, same code, same principles, microservices patterns, I became more like a machine programmed to write code rather than a human to do some critical thinking and being curious.

And to be honest, this is exactly what AI is so good at — writing application code, but miserable at building large-scale distributed systems that requires combination of human + systems thinking.

The Revenge of the Hardware

In the early days of SaaS, the mantra was “hardware is a commodity; code is king.” Cloud abstractions made us forget about bytes, disks, and network packets.

The twist is AI and massive scale changed that. When you are provisioning clusters, managing GPUs, or optimizing Kubernetes nodes for high-throughput workloads, you suddenly care deeply about memory bandwidth, network latency, and compute density. Every single line of code matters alot.

Platform engineering demands system thinking, for e.g:

How does a failure in the database pool ripple through the ingress controller?
How do we scale a cluster dynamically without cascading timeouts of millions of real-time users?
If we suddenly spin up 20 more instances, can the underlying database handle 20x more concurrent connection pools?
Will our NAT gateway handle the massive surge in outbound traffic?

The AI Era

AI boom shattered the “infrastructure is invisible” paradigm.

Today running large-scale systems and cloud infrastructure today requires a deep understanding of low-level constraints. If you don’t respect the machine, the scale will break you.

For the last decade, engineering glory was found at the top of the stack. It was about building slick UIs, optimizing user conversion funnels, and shipping features at lightning speed. But the AI era has completely inverted this dynamic.

The Death of Routine Product Code (Upstairs)

At the top of the stack, abstractions have become so high that code is becoming heavily commoditized.

With LLMs, generating a React component, scaffolding a CRUD API, or writing glue code for a product feature is fast, cheap, and increasingly automated.
The “problems” at the top are becoming less about deep technical complexity and more about product design, prompt engineering, and stitching together pre-existing services. It’s a space where complexity is being managed away.

The Explosion of Physical Reality (Downstairs)

Meanwhile, at the bottom of the stack, the problems have become intensely complex, fascinating, and deeply tied to physical constraints. You can’t “prompt engineer” your way out of a networking bottleneck, a noisy neighbor saturating the CPU cache, or a memory leak under massive concurrent loads.

When a product app breaks, you check the logs and fix a null pointer. When a platform breaks at scale, you might be dealing with Linux kernel OOM (Out Of Memory) killers renegading through your pods, network packet drops at the NAT gateway, or storage IOPS throttling. It requires genuine detective work.

It’s Now More Fun

AI models, LLMs, and massive data pipelines are the ultimate test of systems thinking and I’m really enjoying it learning, studying, building and practicing around.

How fast can data move from Cloud Storage to GPU memory (VRAM)?
Are we bottlenecked by PCIe bandwidth?
How do we orchestrate distributed training across multiple nodes without the network switches becoming a massive choking point?

Training or running inference on large models requires moving terrifying amounts of data. The bottle-neck isn’t how fast the code executes; it’s how fast data can cross the PCIe bus or the network switch.

How do you dynamically provision Kubernetes nodes with massive GPU attachments right when a burst of heavy compute hits?

I didn’t move to platform engineering to escape the AI revolution; I moved because that was always my core foundation — love for hardware and code, it’s a beautiful blend of systems, physics and humans.

A Weekend Project: Building a Sidecarless Observability Engine for K8s

Shehroz Ali — Sun, 19 Apr 2026 12:28:04 +0000

The best observability is invisible. It shouldn’t require SDKs, it shouldn’t require sidecars, and it definitely shouldn’t eat up half your cluster’s CPU.

Why build another tool?

In the world of Kubernetes, observability is often a choice between two extremes: too little or too much.

On one hand, you have standard application logs. They tell you what the app thinks happened, but they don’t tell you what actually happened on the wire. On the other hand, you have massive Service Meshes like Istio or Linkerd. While they give you incredible insights, they come with a heavy “sidecar tax” — injecting a proxy into every pod, adding latency, and making your YAML files look like a phone book.

I realized that all the information I needed every HTTP header, every JSON body, every latency spike was already floating through the air (or rather, the virtual wires) of my Kubernetes nodes. It was just a matter of reaching out and grabbing it.

The Vision for KubeSocket

I wanted to build something lean and simple, something which can just make the job done without any extra overhead or infrastructure configuration. The goal was:

No extra sidecar container (zero maintenance)
No SDKs and dependencies
No dependency on any programming framework (zero configuration)
Should be platform agnostic

Then, I asked myself; “How about we capture the internet traffic entering in the host machine before it touches anything?”

This meant that something which can talk directly to host NIC (Network Interface Card), copy raw bytes (0s and 1s), parse ethernet headers and extract HTTP payload. Since HTTP payload starts at Layer 7, it means that we need to capture the entire raw Ethernet packet entering the host machine through NIC mounted on main motherboard before anyone touches it or it gets modified by any application/webserver, etc. Keeping it raw and simple. Pretty interesting.

To visualise it, this is something I wanted:

Ethernet packet flying from wire directly to RAM (OS kernel)

Network Interface Card (NIC)

A Network Interface Card (NIC), also known as a network adapter or LAN adapter, is a hardware component that allows a computer or other device to connect to a network and communicate with other devices.

Credits: https://levens.fr/523572/Card-10-100-1000-Mbps-Low-Profile-LAN-Adapter-For-Desktop-PC

The Physical Arrival (Layer 1 & 2)

As electrical or optical signals arrive at the NIC, the hardware’s controller performs Frame Delimitation. It identifies where a frame starts and ends, verifies the Checksum (FCS) to ensure the data wasn’t corrupted in flight, and checks the Destination MAC address.

The Ring Buffer

The most important concept for a packet-sniffer developer is the RX (Receive) Ring Buffer. Imagine a circular conveyor belt in your RAM.

The NIC places a packet on a spot on the belt.
The NIC sends a “signal” (an interrupt) to the CPU.
The Kernel (the Driver) walks over, picks up the packet from that spot, and moves it into the OS networking stack.
The Belt keeps spinning, ready for the next packet.

How an ethernet packet goes into OS kernel

Keep Reading!

That’s it, we have now understood how the electrical signals entering the host NIC gets converted into a packet and then how that packet flies through NIC to main system RAM making it accessible for Operating System (OS) kernel to process it for user.

In next blog, I’ll talk about this raw ethernet packet, what’s inside it and how can we parse it to read the raw HTTP data.

Inside AWS Nitro: The System Design Behind 100 Gbps Performance for EC2

Shehroz Ali — Thu, 29 Jan 2026 22:34:05 +0000

For years, virtualization came with a hidden cost: the ‘Hypervisor Tax.’ Every time your application sent a network packet, your CPU had to stop what it was doing, context-switch, and play the role of a traffic cop. It was a bottleneck that stood between your code and the raw power of the hardware.

The Problem: The “Virtualization Tax”

Before Nitro, AWS used a traditional hypervisor (specifically a customised version of Xen ). In this setup, every time a virtual machine (VM) wanted to send a packet of data, it had to go through a “middleman.”

Dom0 (The Privileged VM): The physical server ran a specialized VM called “Domain 0” or Dom0. This was a full-blown Linux environment that had direct access to the physical hardware (NICs, Disks).
Context Switching: When a customer VM (DomU) sent a network packet, the CPU had to “stop” what the customer was doing and “switch” to the Dom0 code to process that packet. This constant back-and-forth is known as a context switch.
Resource Stealing: Dom0 required its own CPU cores and memory to function. On a large server, you might lose 10% to 20% of the physical hardware capacity just to run the management software. This is the Virtualization Tax.

Overview of how Hypervisor talks to CPU for processing network packets

Jitter: Because the host CPU is busy “managing” the network for 50 different VMs, latency becomes inconsistent. A packet might be delayed because the CPU was busy processing a storage request for a different customer.

The Solution: AWS Nitro Offloading

The Nitro System effectively “breaks apart” the hypervisor. It takes all the work that Dom0 used to do — networking, storage, and security — and moves it onto a separate piece of hardware: The Nitro Card.

Hardware Separation: The Nitro Card is a physical PCIe card with its own processor (ASIC) and memory. It is physically separate from the main motherboard where the customer’s CPU (Intel/AMD/Graviton) sits.

Credits: https://dev.to/choonho/nitro-card-why-aws-is-best-46ph

Zero-CPU Networking: When a VM sends a packet, it goes directly to the Nitro Card via SR-IOV (Single Root I/O Virtualization). The host CPU never has to “touch” the packet. It simply drops it into a memory queue, and the Nitro Card picks it up.

The Nitro Card for VPC

These are independent System-on-Chips (SoCs) connected via the PCIe bus. They run their own operating systems and are responsible for specific tasks like networking, storage, and management.

SR-IOV Implementation

It uses Single Root I/O Virtualization (SR-IOV) to create “Virtual Functions” (VFs). This allows multiple VMs on the same host to have direct, high-speed paths to the hardware without going through a software switch.

The Magic: Direct Memory Access (DMA)

There is a physical controller (the DMA Controller ) on the Nitro Card. This circuit has the electrical authority to take control of the PCIe bus and move data from the system’s RAM directly into the card’s own memory.

Hardware diagram for DMA controller inside NIC

1. Step One: The OS Kernel Writes to RAM (CPU Action)

When an application inside your VM wants to send a packet (e.g., a “Hello World” message), it doesn’t know about hardware. It just hands the message to the OS Kernel. The CPU takes that message and wraps it in standard network headers (TCP, IP, Ethernet). The CPU writes this completed packet into a specific “buffer” in the System RAM (the slice of memory assigned to your VM).

2. Step Two: The “Doorbell” (The Hand-off Signal)

Once the packet is sitting in the RAM, the CPU needs to tell the Nitro Card, “Hey, I’ve left a package for you in the lobby.”

MMIO Write: The CPU writes a tiny bit of data (a “doorbell” signal) directly to a specific address that is physically mapped to the Nitro Card over the PCIe bus.

3. Step Three: DMA Hardware Takeover (No CPU)

The Fetch: The DMA Controller inside the Nitro Card reads the memory address provided by the CPU and reaches across the PCIe Bus.

Direct Copy: It copies the packet data from the System RAM directly into the Nitro Card’s local memory.

The core innovation of the AWS Nitro System is the physical decoupling of the networking data plane from the host CPU. By shifting virtualization tasks to custom hardware, AWS eliminates the “virtualization tax” and provides performance that is nearly indistinguishable from bare metal.

Why AI is a Physics Problem: Heat, Size, and the Rise of Nvidia CUDA

Shehroz Ali — Fri, 02 Jan 2026 15:41:59 +0000

The CPU optimizes for Latency through complex Branch Prediction and Out-of-Order execution; the GPU optimizes for Throughput by stripping Control Logic to maximize ALU density. This blog explores that silicon-level trade-off and how CUDA bridges the gap.

To understand the difference between a CPU and a GPU, it helps to think of them not just as “chips,” but as two different types of workers in a factory.

The Best Analogy: The Chef vs. The Assembly Line

The CPU (The Master Chef): A CPU is like a world-class chef. This chef is incredibly smart and can follow a complex recipe to make anything from a 5-course French dinner to a chocolate souffle. However, the chef is only one person (or a small team). They do things one at a time but with great precision and logic.
The GPU (The Burger Flippers): A GPU is like a massive line of 1,000 junior cooks who only know how to flip burgers. They aren’t “smart” enough to cook a 5-course meal, but if you need to flip 1,000 burgers at the exact same time, they will finish the job much faster than the master chef ever could.

The Core

A Core is a physical component on a processor which handle and computes instructions. Let’s talk about difference in CPU vs. GPU core and how they handle instructions.

CPU core

A CPU core is like a collection of Control Unit, Arithmetic Logic Unit (ALU) and Cache, this combined makes a single core in a CPU.

1 Core = 1 ALU + 1 Control Unit + 1 Cache

Because each core has its own “Control Unit” (the boss), it can work on a completely different task than the core next to it. So multiple cores in a CPU can work on different set of complex instructions simultaneously at a same time.

GPU core

A GPU core doesn’t have its own Control Unit, but rather it only contains a basic ALU with a cache. Hundreds and thousands of cores in a GPU is controlled by 1 Control Unit.

1 Core = 1 ALU + 1 Cache

1000s Cores -> 1 Control Unit

The difference

CPU: 8 chefs, each with their own recipe book, cooking 8 different meals.
GPU: 1 drill sergeant (Control Unit) screaming “JUMP!” at 1,000 soldiers (ALUs) at the same time. The soldiers don’t have their own brains; they just follow the one sergeant.

If you gave a GPU a task where every core had to do something different (Core 1 adds, Core 2 subtracts, Core 3 divides), the GPU would break down.

Because they share a Control Unit, they all have to perform the same instruction at the exact same time.

CPU core vs. GPU core

GPUs are stupid at making decisions, but good at simple math

If you have one giant complex math problem, the CPU will finish it first. If you have 10,000 tiny additions to do (like brightening every pixel in a photo), the GPU will finish the whole batch before the CPU even gets through the first hundred.

CPU has less workers but each of them are intelligent

This is because in a CPU, every core has its own dedicated Control Unit. It’s like 8 people each having their own brain. They can each decide to do something different.

In a GPU, one Control Unit is shared by a group of, say, 32 or 64 ALUs. This creates a hardware limitation, so they have massive energy to do math but are dumb at making complex decisions due to shortage of Control Unit.

GPU has massive workers but each of them are dumb

So GPU has thousands of these tiny cores, but each of them are controlled by 1 Control Unit, and then all cores perform the same instruction on their own in parallel.

Why can’t we make GPUs intelligent?

It’s a great question — if the master chef (CPU) is so much better at everything, why not just hire 1,000 of them?

The answer comes down to three cold, hard physical limits: Size , Heat , Memory and Cost.

The Size Problem

A “master” CPU core is physically massive compared to a “junior” GPU core.

CPU Core: Packed with complex features like branch prediction (guessing what you’ll do next) and huge “waiting rooms” for data (Cache).

100x in physical size due to complex setup

GPU Core: Stripped down to just the math parts.

In the same physical space where you can fit one high-end CPU core, you can often fit over 500 GPU cores. To fit 1,000 “master” CPU cores, your computer chip would have to be the size of a dinner plate.

The Heat Problem

Complex ALUs are “power hungry.” They run at very high speeds (4–5 GHz) and use a lot of electricity to power all that “smart” logic.

If you tried to pack 1,000 of those into a single chip, the amount of heat generated would be high enough to melt the silicon instantly.
GPUs stay cool(er) because their thousands of cores are much simpler and run at lower speeds (around 1–2 GHz). They are like 1,000 lightbulbs compared to 10 massive industrial spotlights.

Visual comparison of ideal GPU vs. real GPU (size)

The Memory Problem

Imagine having 1,000 master chefs in one kitchen. They would all be screaming for ingredients at the same time.

A CPU core needs a constant stream of complex instructions.
Current memory technology (RAM) isn’t fast enough to feed 1,000 “smart” cores simultaneously. They would spend 99% of their time just sitting there waiting for data to arrive, making them a waste of space.

The Cost Problem

Electricity: A “smart” CPU core uses significantly more power than a “dumb” GPU core because it’s constantly running complex “prediction” circuits.
Heat: 1,000 smart cores would require an industrial-grade cooling system (like liquid nitrogen or massive fans).
Infrastructure: To run a chip that complex, you would need a specialized motherboard and a power supply as big as a microwave.

Heterogeneous Computing: Nvidia CUDA

Heterogeneous Computing is the “teamwork” approach to computer design. Instead of trying to make one chip that is good at everything, it puts different types of specialized processors together on the same system (or even the same chip) to handle specific tasks.

The hardest part of heterogeneous computing isn’t the hardware — it’s the Software.

A CPU and a GPU speak different languages.
To make them work together, programmers have to write special code (using tools like OpenCL or CUDA ) to tell the computer: “Send this math to the GPU, but keep the logic on the CPU.”

High-level architecture of Nvidia CUDA technology

One OS Thread, Millions of Goroutines: The Magic of Go’s Scheduler Explained

Shehroz Ali — Sat, 29 Nov 2025 13:11:45 +0000

Go achieves high concurrency through its scheduler, which manages millions of goroutines on a small number of OS threads. Here’s how it works.

Overview of Go’s scheduler model

Go’s concurrency model lets you run millions of goroutines with just a few OS threads. This post explains how the scheduler multiplexes goroutines onto threads, how it handles blocking and non-blocking I/O, and how the G-M-P model enables efficient concurrency.

Go uses goroutines, not threads. Important differences:

Threads (traditional):

Managed by OS
Heavy (~1–2MB stack each)
Limited (typically 100s-1000s)
Context switching is expensive (OS kernel involved)

Goroutines:

Managed by Go runtime
Lightweight (~2KB stack initially)
Can spawn millions
Context switching is cheap (user-space)

GOMAXPROCS

OS threads are limited by GOMAXPROCS.

// Default: GOMAXPROCS = number of CPU cores
// You can set it:
runtime.GOMAXPROCS(4) // Use 4 OS threads

Example on an 8-core machine:

8 OS threads
Can handle thousands of concurrent goroutines
Go scheduler multiplexes goroutines onto threads

How the OS thread handles multiple goroutines

The OS thread doesn’t run all goroutines (let’s say Gn in the diagram below) at the same time. The Go scheduler switches between them.

Overview of Go concurrency model

Let’s zoom-in here and try to get a full clear picture how the Go runtime scheduler actually juggles between goroutines on a single OS thread.

There can be two types of operation being performed by goroutine (Gn):

Non-blocking (HTTP/network)
Blocking (disk I/O)

Non-blocking Task

Let’s say the goroutine (G1) waiting to be scheduled needs to perform a HTTP call over the internet, here’s how the scheduler will work:

The goroutine (G1) initiates the HTTP call
It yields control to the scheduler
The scheduler switches to another goroutine (G2)
The OS thread continues with other work

Visual Timeline:

Code-level view:

// G1's code
func handler1() {
    // G1 is running on OS Thread 1
    resp, err := http.Get("https://api1.com")
    // ↑ At this point:
    // 1. Creates socket
    // 2. Sends request
    // 3. Registers with poller
    // 4. Yields to scheduler (gopark)
    // 5. G1 marked as "waiting"
    // 6. Scheduler switches to G2

    // Later, when response arrives:
    // 1. Network poller detects it
    // 2. Scheduler marks G1 as "runnable"
    // 3. Scheduler switches back to G1
    // 4. G1 continues here with response

    fmt.Println(resp.StatusCode)
}

// G2's code
func handler2() {
    // G2 runs when G1 yields
    // Could do anything: CPU work, another HTTP call, etc.
}

// G3's code
func handler3() {
    // G3 runs when G2 yields or completes
}

Blocking Task

Let’s say the goroutine (G2) waiting to be scheduled needs to perform a disk I/O like a write operation, here’s how the scheduler will work:

The goroutine (G2) writes some heavy data to disk (say in GBs)
The thread is blocked in the kernel
Scheduler cannot preempt it (it’s in kernel)
Go creates a new OS thread for other goroutines

Visual Timeline:

Code-level view:

// When you call os.Open()
func Open(name string) (*File, error) {
    // 1. Enter syscall
    runtime.entersyscall()
    defer runtime.exitsyscall()

    // 2. Make actual syscall
    fd, err := syscall.Open(name, ...)

    // 3. If blocking detected, scheduler
    // already created new thread
    return newFile(fd, name), err
}

The three components: G, M, P

Go’s scheduler uses three main components:

G = Goroutine (your code)

M = Machine (OS thread)

P = Processor (execution context)

What is a Processor (P)?

A Processor (P) is an execution context that:

Holds a local run queue of goroutines
Binds to an OS thread (M) to execute goroutines
Manages resources for running goroutines
Acts as a bridge between goroutines and threads

Overview of Processor (P) in Go

How Processor (P) manages OS threads?

Consider two Goroutines; G1 and G2, both waiting in queue of P1 (Processor). P1 is binded to OS thread 1 which is running on Core 1. Now:

G1 wants to performs a network call (non-blocking)
G2 needs to open a file from disk (blocking)

Here’s how Go scheduler will manage:

Runs G1 from P1 queue on OS thread 1
Goes to Network poller (epoll/kqueue), G1 waiting, OS thread 1 is free
Scheduler runs G2 from P1 queue on OS thread 1
G2 performs syscall() and is blocked in kernal
Scheduler detects OS thread 1 is blocked
Scheduler creates new OS thread, call runtime.newosproc()
Scheduler detaches P1 from OS thread 1 and binds to OS thread 2
Now Go Scheduler runs G3…Gn from P1 on OS thread 2

Example on a logical 4-core machine:

No of Processor = GOMAXPROCS = No of logical CPU cores.

So, on 4 core machine, it will have P1…P4, where each P is binded to OS thread which is equal to GOMAXPROCS = 4 (CPU cores).

P = Processsor, M = OS thread, G = Goroutine

A Complete Guide to Distributed Tracing in Kotlin and Spring Boot with OpenTelemetry and Grafana…

Shehroz Ali — Thu, 23 Oct 2025 22:49:15 +0000

A Complete Guide to Distributed Tracing in Kotlin and Spring Boot with OpenTelemetry and Grafana Loki

In this blog, I’ll teach you how you can build end-to-end distributed tracing for your backend microservices in Kotlin, OpenTelemetry, Spring Boot and Grafana Loki.

By following the guide, you’ll be able to correlate hundreds and thousands of telemetry data (traces & logs) emitted by your backend services and achieve end-to-end observability to reduce your MTTRs for production incidents and make your developers life easier.

To give you a basic understanding of what we’ll be going to achieve, here’s a system-level diagram to help you understand all the components functioning together:

end-to-end observability

Capturing HTTP traffic within Spring Boot:

First, we need to enable HTTP network logging for our Spring Boot app, for that we’re going to use Zalando’s open-source Logbook which automatically logs HTTP requests and responses on your (preferred output writer). It works by intercepting HTTP traffic in your app and writing TRACE log using Slf4j which is a fascade for Logging in Java. In this example, we’re going to use Logback which is a logging library with auto-configuration support that comes already with spring-starter-web dependency.

First let’s configure Logback to capture INFO, ERROR and TRACE logs and outputs on console:

Setting up Logback appenders:

First we added, Console-Info to capture logs at INFO level which will likely be our classes/service logs emitting some business/user insights/actions (which later devs will use to understand code behaviour during debugging).

We also added another Console-Error to capture ERROR logs emitted by our classes to indicate an exception/error message in logs.

Lastly we added Console-Trace to capture TRACE logs. Since Zalando’s Logbook library write TRACE logs for HTTP traffic, we’ve passed the Console-Trace appender to logger org.zalando to use Console-Trace appender to write TRACE logs on System output. Here’s the final logback-logging.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<included>

    <appender name="Console-Info" class="ch.qos.logback.core.ConsoleAppender">
        <!-- Targeting System.out -->
        <target>System.out</target>
        <!-- JSON Log Formatting -->
        <filter class="ch.qos.logback.classic.filter.LevelFilter">
            <level>INFO</level>
            <onMatch>ACCEPT</onMatch>
            <onMismatch>DENY</onMismatch>
        </filter>
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <fieldNames>
                <message>[ignore]</message>
                <levelValue>[ignore]</levelValue>
                <version>[ignore]</version>
            </fieldNames>
            <provider class="net.logstash.logback.composite.loggingevent.LoggingEventPatternJsonProvider">
                <pattern>
                    {
                    "message": "%replace(%.-150message){'${MASKPATTERNS}', ' *****'}"
                    }
                </pattern>
            </provider>
            <customFields>
                {"service": "${logHost}"}
            </customFields> <!-- Optional: Add custom fields -->
        </encoder>
    </appender>

    <appender name="Console-Error" class="ch.qos.logback.core.ConsoleAppender">
        <!-- Targeting System.out -->
        <target>System.err</target>
        <!-- JSON Log Formatting -->
        <filter class="ch.qos.logback.classic.filter.LevelFilter">
            <level>ERROR</level>
            <onMatch>ACCEPT</onMatch>
            <onMismatch>DENY</onMismatch>
        </filter>
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <fieldNames>
                <message>[ignore]</message>
                <levelValue>[ignore]</levelValue>
                <version>[ignore]</version>
            </fieldNames>
            <provider class="net.logstash.logback.composite.loggingevent.LoggingEventPatternJsonProvider">
                <pattern>
                    {
                    "message": "%replace(%.-150message){'${MASKPATTERNS}', ' *****'}",
                    "stack_trace": "%replace(%ex{short}){'${MASKPATTERNS}', ' *****'}"
                    }
                </pattern>
            </provider>
            <customFields>
                {"service": "${logHost}"}
            </customFields> <!-- Optional: Add custom fields -->
        </encoder>
    </appender>

    <appender name="Console-Trace" class="ch.qos.logback.core.ConsoleAppender">
        <!-- Targeting System.out -->
        <target>System.out</target>
        <!-- JSON Log Formatting -->
        <filter class="ch.qos.logback.classic.filter.LevelFilter">
            <level>TRACE</level>
            <onMatch>ACCEPT</onMatch>
            <onMismatch>DENY</onMismatch>
        </filter>
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <fieldNames>
                <message>[ignore]</message>
                <levelValue>[ignore]</levelValue>
                <version>[ignore]</version>
            </fieldNames>
            <provider class="net.logstash.logback.composite.loggingevent.LoggingEventPatternJsonProvider">
                <pattern>
                    {
                    "message": "#asJson{%message}"
                    }
                </pattern>
            </provider>
            <customFields>
                {"service": "${logHost}", "pod": "${podName}"}
            </customFields> <!-- Optional: Add custom fields -->
        </encoder>
    </appender>

    <!-- LOG everything at INFO level -->
    <root level="info">
        <appender-ref ref="Console-Info"/>
    </root>

    <logger name="org.zalando.logbook" level="INFO" additivity="false">
        <appender-ref ref="Console-Trace"/>
    </logger>

In application.yml file:

logging:
  config: 'classpath:logback-logging.xml'

Structured JSON Logging for Grafana Loki:

Here comes the important detail, since we’re going to structure logs on Grafana Loki based on labels and fields for filtering and querying, we’re going to emit logs as JSON output. For that we need to add this dependency for Logback (maven central).

        <dependency>
            <groupId>net.logstash.logback</groupId>
            <artifactId>logstash-logback-encoder</artifactId>
            <version>6.6</version>
        </dependency>

This will allow us to output logs as JSON on system output and later we can relabel fields on Grafana Agent (or Alloy) for Loki.

Writing HTTP logs as JSON:

Now since our Logback supports writing logs in JSON format, we’ve added this line in our Console-Trace to write Zalando’s HTTP log as JSON on output. The %message captures the entire TRACE level log and output on console with label message like this:

{
 message: <zalando's TRACE log including request, response, headers>
}

Later we can extract out individual fields like status, type, body, path and display on Grafana Loki as fields for filtering and querying logs.

Configuring Logbook in Kotlin:

Now, since our Logback is all ready to emit logs (INFO, ERROR, TRACE) on System output, now it’s finally time to configure Logbook for capturing HTTP traffic inside our app.

Below is the code for configuring Logbook in Kotlin for Spring Boot, add dependency as well:

        <dependency>
            <groupId>org.zalando</groupId>
            <artifactId>logbook-spring-boot-starter</artifactId>
            <version>3.10.0</version>
        </dependency>

As it is a spring-starter dependency it already comes with lots of things pre-configured, let’s tweak it according to our needs:

@Configuration
open class LogbookAutoConfiguration(
    private val jsonBodyFilter: JSONBodyFilter,
    private val sinkConfig: SinkConfiguration,
) {

    @Bean
    open fun logbook(filterConfig: MDCFilter?, objectMapper: ObjectMapper): Logbook {
        return Logbook.builder()
            .correlationId(filterConfig)
            .sink(sinkConfig.defaultSink(objectMapper))
            .bodyFilter(BodyFilter.none())
            .condition(exclude(requestTo("/actuator/**")))
            .build()
    }
}

We’ve excluded /actuator to log because this will be our Kubernetes liveness/readiness probe endpoint for health check, so k8s will hit this endpoint every after X mins/secs to keep pod up and healthy, therefore we excluded to avoid de-bloating our Loki storage.

@Component
open class JSONBodyFilter {

    open fun runFilter(): BodyFilter {
        return JsonBodyFilters.replaceJsonStringProperty(
            setOf("token", "refreshToken", "scopes"),
            " ****"
        )
    }
}

Here, we’ve masked these fields in our response/request bodies as it contains JWT tokens of users as a security practice, you might don’t need it depending upon your infra/team security policies.

@Component
open class SinkConfiguration {

    open fun defaultSink(objectMapper: ObjectMapper): Sink {
        val formatter: HttpLogFormatter = JsonHttpLogFormatter(objectMapper)

        val writer: HttpLogWriter = DefaultHttpLogWriter()

        return DefaultSink(formatter, writer)
    }
}

Here we’re using Zalando’s default HTTP log writer to write TRACE logs on output.

@Component
open class MDCFilter : CorrelationId {
    override fun generate(request: HttpRequest): String {
        val correlationId = UUID.randomUUID().toString()

        val httpMethod = request.method
        val httpPath = request.path
        val userId = request.headers["userId"]?.first()
        val clientKey = request.headers["X-Client-Key"]?.first()
        val ipAddress = request.headers["x-forwarded-for"]?.toString()
        val userAgent = request.headers["user-agent"]?.toString()

        MDC.put("httpMethod", httpMethod)
        MDC.put("httpPath", httpPath)
        MDC.put("userId", userId)
        MDC.put("correlationId", correlationId)
        MDC.put("ClientKey", clientKey)
        MDC.put("ipAddress", ipAddress)
        MDC.put("userAgent", userAgent)

        return correlationId
    }
}

This is an important component, here we’ve extended CorrelationId class by Zalando and override generate() method, so whenever internally this method invokes for correlationId generation which will be attached to each HTTP log, we’re also adding the same correlationId in our MDC along with some other fields as well. This means whenever a HTTP request comes inn, here’s what will happen:

Generate unique UUID as correlationId
Attach correlationId in HTTP request/response TRACE log
Also populate same correlationId in MDC
Whenever we log (info, error) at class-level, same correlationId is attached because MDC is shared, allowing us to correlation HTTP logs with class-level logs

And, that’s it! We’re good to go to start writing HTTP logs on console/output propagating HTTP path, ip address, method and your own unique header fields correlating with your class-level logs.

Using Grafana Agent to ingest Kubernetes Pod logs:

Since, we’re running our Spring Boot app in Kubernetes and writing logs on stdout (system output), k8s maintain a file on the node with all logs data at /var/log/pods/__//0.log we can configure Grafana Agent to ingest this log file and perform some parsing to transform raw JSON logs into structure logs and forward it to Loki tenant.

This is a sample config for Grafana Alloy (similar to Grafana Agent as well):

// Discover pods running on this node
discovery.kubernetes "pods" {
  role = "pod"
}

// Discover the actual log file paths for pods
local.file_match "pod_logs" {
  paths = ["/var/log/pods/*/*/*.log"]
}

// Relabel to add useful metadata
discovery.relabel "pod_labels" {
  targets = discovery.kubernetes.pods.targets

  // Extract namespace, pod, and container from the file path
  rule {
    source_labels = [" __path__"]
    regex = "/var/log/pods/([^/]+)/([^_]+)_([^/]+)/(.+)/(.+)\\.log"
    target_label = "namespace"
    replacement = "$1"
  }

  rule {
    source_labels = [" __path__"]
    regex = "/var/log/pods/([^/]+)/([^_]+)_([^/]+)/(.+)/(.+)\\.log"
    target_label = "pod"
    replacement = "$2"
  }

  rule {
    source_labels = [" __path__"]
    regex = "/var/log/pods/([^/]+)/([^_]+)_([^/]+)/(.+)/(.+)\\.log"
    target_label = "container"
    replacement = "$4"
  }
}

// Read logs and forward to Loki
loki.source.file "pod_logs" {
  targets = local.file_match.pod_logs.targets
  forward_to = [loki.write.default.receiver]
}

// Loki write target
loki.write "default" {
  endpoint {
    url = "https://<your-loki-endpoint>/loki/api/v1/push"
  }
  // Optional: authentication
  // basic_auth {
  // username = "<user>"
  // password = "<password>"
  // }
}

Grafana Alloy (or Agent) runs as daemon sets on your k8s nodes, so each daemon set read pod log file on the node, labels it to pod, namespace and container and push it to Loki tenant over HTTPs (make sure your Loki is accessible within cluster) or if outside cluster use VPC PrivateLink to connect to Loki if running in different AWS network/account.

Grafana Alloy to Loki for logs

Multi Tenant Loki Dashboards:

For better developer experience, we’re going to push class-level logs (info, error) to “service-logs” tenant, and for HTTP logs we’re going to push in another tenant “http-logs” so devs can filter/query based on fields.

Each Loki tenant is a “logical” partition on your storage backend (we’re using S3 bucket for our logs storage), so it writes each tenant logs in its own separate chunk on S3. Each tenant is served separately on UI/frontend so engineers can debug faster and helps reduce cognitive load when scanning through logs.

Multi tenant Loki dashboards

Pushing HTTP logs to separate tenant on Loki:

Now since we understand the underneath architecture of tenants in Loki, we’ll configure our Grafana Alloy (or Agent) config to filter out HTTP logs and push it to new tenant on Loki.

#########################
# TEAM A — Only HTTP logs (org.zalando)
#########################
loki.source.file "team_a_logs" {
  targets = [{
    __path__ = "/var/log/myapp/*.log"
  }]

  pipeline {
    # Step 1: Parse JSON logs
    stage.json {
      expressions = {
        level = "level"
        logger_name = "logger_name"
        msg = "message"
      }
    }

    # Step 2: Keep only logs where logger_name == org.zalando
    stage.keep {
      expressions = {
        logger_name = "org.zalando"
      }
    }

    # Step 3: Optional — add labels for Loki queries
    stage.label {
      values = {
        level = "level"
        logger = "logger_name"
        tenant = "team-A"
      }
    }

    # Step 4: Send to Loki tenant A
    stage.output {
      forward_to = [loki.write.http_logs.receiver]
    }
  }
}

loki.write "http_logs" {
  endpoint {
    url = "https://loki.example.com/loki/api/v1/push"
  }
  tenant_id = "team-A"
}

#########################
# TEAM B — Only class-level logs (INFO, DEBUG, ERROR)
#########################
loki.source.file "team_b_logs" {
  targets = [{
    __path__ = "/var/log/myapp/*.log"
  }]

  pipeline {
    # Step 1: Parse JSON
    stage.json {
      expressions = {
        level = "level"
        logger_name = "logger_name"
        msg = "message"
      }
    }

    # Step 2: Keep only certain log levels
    stage.keep {
      expressions = {
        level = "~^(INFO|DEBUG|ERROR)$"
      }
    }

    # Step 3: Add useful labels
    stage.label {
      values = {
        level = "level"
        logger = "logger_name"
        tenant = "team-B"
      }
    }

    # Step 4: Send to Loki tenant B
    stage.output {
      forward_to = [loki.write.service_logs.receiver]
    }
  }
}

loki.write "service_logs" {
  endpoint {
    url = "https://loki.example.com/loki/api/v1/push"
  }
  tenant_id = "team-B"
}

Tenant service_logs: Kept only logs with log level INFO, DEBUG, ERROR dropped TRACE level logs (i.e. HTTP logs).

Tenant http_logs: Kept only logs with logger_name equals to org.zalando i.e. HTTP logs, dropped other class-level logs (info, debug, error).

Distributed Tracing: OpenTelemetry

Okay, now, we’re almost about to wrap up. The last thing we need to add is OpenTelemetry in our Spring Boot app so we can enable distributed tracing throughout our request lifecycle across microservices.

The good thing is OpenTelemetry for JVM offers automatic instrumentation by running a JAR agent inside your docker container. So before running our app JAR in docker, we can download OTel agent JAR and pass as a JVM arg with our app, this will enable automatic instrumentation for tracing inside our app and will also propagate trace_id in MDC which can be correlated with logs as well.

# === Stage 1: Build the application ===
FROM maven:3.9.8-eclipse-temurin-21 AS build

# Set working directory
WORKDIR /app

# Copy source and build
COPY pom.xml .
COPY src ./src
RUN mvn clean package -DskipTests

# === Stage 2: Runtime image ===
FROM eclipse-temurin:21-jre

# Set working directory
WORKDIR /app

# Copy the Spring Boot fat JAR from the build stage
COPY --from=build /app/target/*.jar app.jar

# Download the latest OpenTelemetry Java agent
# (Alternatively, you can include a specific version in your repo for consistency)
ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar /app/opentelemetry-javaagent.jar

# Expose the application port
EXPOSE 8080

# Environment variables for OpenTelemetry
ENV OTEL_SERVICE_NAME=my-spring-service \
    OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \
    OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod,team=backend \
    OTEL_INSTRUMENTATION_LOGBACK_MDC_ENABLE=true

# Start the app with the OTel Java Agent attached
ENTRYPOINT ["java", "-javaagent:/app/opentelemetry-javaagent.jar", "-jar", "/app/app.jar"]

Transporting Otel Telemetry over OTLP protocol to Grafana Tempo:

The Otel agent JAR transports its telemetry data over OTLP protocol. Here we can either transport telemetry data to an Otel Collector acting as a central fascade for various tracing backends like (Tempo or Jaegar) or we can directly transport telemetry over OTLP protocol to Tempo as it supports OTLP for ingestion.

Emitting Otel telemetry over OTLP to Tempo

Testing:

Done! That’s all, now finally we’re done with setting up our Grafana stack (Alloy, Tempo, Loki), we can start capturing and emitting HTTP layer and business-logic layer logs to Grafana Loki correlated with OpenTelemetry traces on Tempo based on trace_id allowing us to achieve end-to-end distributed tracing across entire system.

Here how it looks like:

HTTP log (request):

Grafana Loki (http_logs tenant)

HTTP log (response):

Grafana Loki (http_logs tenant)

Class-level log (business-logic layer):

Grafana Loki (service_logs tenant)

Full request trace on Grafana Tempo using trace_id

Grafana Tempo

From Monolith to Modularity: Modernizing Identity Platform at Scale

Shehroz Ali — Wed, 11 Jun 2025 17:06:44 +0000

We rebuilt the heart of our identity system — live, at scale, and without a single disruption. Here’s how we transformed a legacy monolith into a modular, high-trust authentication platform for millions.

Introduction

Bazaar’s Identity Platform is the authentication nucleus of our product ecosystem — orchestrating access for every user touchpoint, from our Grocery app and Rider interface to internal admin portals. It’s not just a login service; it governs:

Sign-up, login, OTP verification
Session lifecycle management
Role-based access control
OpenID Connect (OIDC) integration
Multi-factor authentication

At peak, it processes 7000+ requests per minute , handling millions of sessions across guest and registered users. Its stability defines the health of the entire platform. Any change to token behavior would ripple across every consumer product and backend — making correctness, reliability, and backward compatibility non-negotiable.

⚠️ The Problem: Token Renewal at Scale Was Cracking

Historically, our identity system issued JWT access and refresh tokens to both guests and registered users. Clients relied on a shared API endpoint to refresh tokens every n minutes.

This approach began to show its cracks:

🚨 Guest token renewals overwhelmed the identity-service with unauthenticated traffic.
🧩 Token lifecycle logic became entangled with authentication and session logic.
🧱 Domain boundaries blurred , making the system increasingly hard to evolve.
⚙️ Scaling stress — over 1.9M+ active guest sessions stored and rotated in DB.

This wasn’t just a performance bottleneck — it was an architectural red flag. The system needed to evolve or it would constrain our future growth.

Inspired by Global-Scale Identity Platforms

We studied battle-tested identity systems to inform our design:

Amazon : Session-token IDs for opaque anonymous flows
Spotify : Isolation between guest discovery and user-authenticated APIs
Uber : JWT enrichment at the edge via Envoy to decouple downstream auth

These insights helped define our guiding principles.

Design Goals

Reduce guest token renewal traffic on identity-service
Introduce stateless guest sessions to remove DB dependency
Preserve full backward compatibility for legacy clients
Establish clear token domains: Guest, User, and Service
Safely deliver change using TDD, BDD, and progressive rollout

From Monolith to Modular: Restructuring Identity as a Scalable Platform

Before any token migration could succeed, we needed to modernize the very structure of our Identity Platform.

What started as a monolithic mudball — entangling session logic, token management, login flows, and user models — was reimagined into a vertically sliced architecture, with clear boundaries, modularity, and domain ownership.

🎯 Why Vertical Slicing?

Our legacy system suffered from tight coupling between concerns — login flows touched session storage, token validation logic was scattered, and shared models leaked across contexts.
Engineers had a hard time making localised changes without risking unrelated functionality.
Adding support for new flows (like partner logins or stateless sessions) meant touching unrelated parts of the system.

🧩 Our Modular Design Approach

We restructured the platform into independent, domain-aligned vertical slices , such as:

GuestService
UserService
SignupService
OTPFlowHandler
SessionLifecycleManager

Each module:

Owned its models, rules, business logic, and persistence.
Was exposed via explicit interfaces (REST or internal contracts).
Was testable, deployable, and evolvable in isolation.

This modularization accelerated development, simplified testing, and reduced the blast radius of changes — giving us the foundation to implement the new token lifecycle with confidence.

🔄 Future-Ready Foundation

This shift wasn’t just about scaling what we had — it enabled:

Easier onboarding of engineers by navigating clear module boundaries.
A pathway to microservices, as each vertical slice could become its own service.
Faster incident resolution due to localised ownership and observability.

🔄 Our Engineering Approach (Token Migration)

This wasn’t a feature refactor — it was a core identity infrastructure rewrite , under live, high-scale traffic. We chose to evolve the system incrementally and surgically.

🔁 1. Stateless Guest Session Tokens

Instead of persisting or rotating refresh tokens for guests:

We issued opaque session tokens signed with a platform-wide secret
Tokens had a long but bounded TTL
Result: 70%+ drop in guest renewal traffic to identity-service

2. Enriched JWTs via Spring Gateway

Based on the accessLevel claim (GUEST, USER, SERVICE), we:

Injected custom scopes and metadata into headers at the Gateway
Allowed downstream services to stay stateless and avoid JWT parsing

3. Middleware Unification

All backend services received a uniform header structure , regardless of caller type. This centralised auth logic while keeping the gateway lean.

4. Seamless Backward Compatibility

Legacy clients still expecting guest refresh tokens (refreshToken = "GUEST") were gracefully intercepted and served valid tokens from the new guest_session API — no app updates required.

Engineering Practices That Enabled Safe Change

TDD & BDD : Outside-in test design enabled refactoring behind robust test coverage
Trunk-based development : Incremental PRs with clear ownership
Ping-pong pairing : Rapid peer iteration kept velocity high without compromising safety
Stateless-first mindset : Reduced session state complexity at scale
accessLevel claims : Cleanly routed and enforced logic boundaries across the stack

Outcomes & Impact

Despite refactoring one of the most sensitive systems in our infrastructure, we shipped without a glitch. Here’s what we achieved:

Zero downtime during rollout
70% reduction in guest traffic to identity-service
Full backward compatibility — no client-side changes
Simplified operations — no need to manage guest session DB entries
Clear domain boundaries for Guest, User, and Service tokens
Enabled partner token flows with SERVICE scoped JWTs
Robust test suite for long-term maintainability

Final Thoughts

Rebuilding identity wasn’t about rewriting a few endpoints.

It was about evolving a monolith into a modular, testable, and scalable platform.

It was about building trust — at platform level , at engineering level , and at user level.

How we solved SignUp/Login problem for Mint App using Uber’s USL — Unified SignUp Login Model

Shehroz Ali — Sat, 09 Jul 2022 12:17:01 +0000

How Uber’s engineering model inspired me to increase customer acquisition for our app by 6% weekly

So lately, we have ran into a questionnaire based problem, and here is the User story overview:

Ali discovered about Mint from Internet

Ali likes to be the part of rockstar Mint Family as he likes Mint’s vision on bettering cities and urban climate infrastructure

Ali headed towards for Sign Up for Mint App on mymintrewards.com/signup

Ali successfully subscribed to beta version of Mint App as an early adopter

Weeks later, Mint team emailed Ali for onboarding on Mint’s App via Google Play download link

Ali successfully downloaded Mint App

What will Ali see?

Now this is what which has made us to stop and operate for past 5 months and we ended up scratching our heads — i.e. how to keep wait-listing still live (persistently storing data somewhere which we can be use later to allow user to login) till we finish launching our pilot technology. This wasn’t an easy task as we need to figure out a way somehow that can keep both things up-and-gun running in parallel till our other teams finish setting up plans and actions on how to operate and signing up MOUs with initial Brands and Companies.

The Problem

Problem is that, if a user has already signed up for Mint account via website than how we can unlock user’s Mint account for beta app using the same data? i.e. converting website form data into real-time user data. What will user see when he first time open the app?

Nested Problem

Wait — here’s an another problem. We can’t directly ask for some sort of unique auth key (say password, OAuth, etc.) which can also be used later to grant access to user to login into Mint app. The reason was we have some time launching the full-scale app till we fulfill our technological resources so it can be able handle strong continuous traffic and we also want to minimize latency so we can drive excellent in-app user experience. This was the main purpose of attaching subscriber form on the website so we can start onboarding users with moderate traffic giving us space to breath so we can make better informed decisions. Even we built form using no-code SaaS app to save our time plus resources and abstract out all the integration layers so we can really focus on the Business instead.

In a nutshell, our goal is to convert that incoming form responses and turning into real-time relational database for allowing users to access their Mint account via our app.

So how we solved the problem?

Unified SignUp Model (USL)

We taken inspiration from Uber’s USL model derived by their excellent engineering team (a round of applause for them). We thoroughly studied their model and learnt how Uber solved their Login/SignUp problem for their food, delivery and riding apps seamlessly using USL pattern and successfully built an all-in-one smooth in-app experience whether you are a new user or just returning back. This was designed specially for Uber Eats, Uber Ride, Uber Delivery and other Uber products specially those requiring faster database communications for speedy transactions. We observed that somehow Uber had been experiencing the same problem within its multiple apps which we were facing at Mint and i.e. making Business Logic layer decides where to navigate user whether he/she is a completely new user or already signed up via mymintrewards.com/signup or just returning back — this was quite similar to what Uber was also experiencing. We replicated the model and it worked.

Let me give you a high-level overview of how we altered Uber’s USL according to our Business Logic and built a seamless onboarding experience for Mint App.

This is what I’ve come up with

The Final Solution

You open the app — you will be asked to enter your email or phone number. Once you entered, our Business logic layer will decided where to navigate you and you will automatically be taken to where you belong. Boom, done. We made it simple for you. We dropped all those fancy fields, buttons and screens to minimize the end-burden on front-end client and improve the overall user experience enabling users quickly access their Mint account within 2-steps. Usability done right.

Nobody wants to wait — Be quick, Be seamless

Today, end-user looks for value. He didn’t care about all of those underneath things going around. He didn’t care about your technical debt. He has nothing to do with it. The only goal end-user wants to achieve from a successful product is the solution to his problem aka value — real value. This should be his first word of mouth when first time opening an app — Woah!. I believe achieving this couldn’t be hard as technology today allows us if you rightly putted all the puzzle pieces together. The only thing you need to look for is converting Business model into actual solution — seamless, smooth and quick.

Seeyou, again.

Today is not about writing code — Today is all about ‘Velocity’

Shehroz Ali — Thu, 23 Jun 2022 13:36:00 +0000

Today is not about writing code — Today is all about ‘Velocity’

Yes, ‘Velocity’ — the same term that you used to be taught in your late high-school or mid-school physics classes. Maybe, some of you don’t like physics (just like me too) — well, but sometimes it doesn’t mean that it carries the same exact meaning to the actual context where we pointing. Right? So let’s talk about what ‘Velocity’ and ‘Software Engineering’ has to do all about and what their relationship tells us.

Source: Pexels

Referencing

Amazon famously commits code to production every 11.6 seconds and they’re always learning really fast. Pause for a second here and imagine the velocity they moving. Seems more faster than last time I pulled my phone to book an Uber? Exactly. We don’t know how much lines do every merge contains (until we ourselves works with the team at Amazon) but surprisingly what we see is their velocity to ship to production faster, 11.6 seconds. How is that even possible? Let’s try to break and see what’s happening underneath the hood.

Business vs. Tech — Really?

Right tooling. Yes. This is where I want you all to focus. If you really want to build better Continuous Delivery pipelines so you can test, build and release faster than you once again really need to think-off your delivery pipelines. DevOps enable teams to set things to automation and ships the new build to production environment through various CI and CD tools (such as Jenkins, Fastlane, ArgoCD, Ansible, Terraform, etc.). In 2022 today, building shouldn’t be considered as the only primary goal for a startup, it’s secondary, but what’s important today is your ‘Velocity’ from both business and technical perspective. Not every founder is a also a ‘technical’ founder and not every founder cares a lot about those underlaying ‘technical debt’ or your excellent-driven technology stack. Technology is just a tool we use today to build businesses for tomorrow. Technology helps us drive better customer acquisition by delivering values. It’s really the business first. Technology helps us to eliminate painful tasks by automating them and enabling teams to move faster and safer. It really help us achieve those defined business goals to increase profitability and scale faster.

Wonders did by the amazing Software Community

So, listening to all of this, I would say that it’s the right time to re-pay a visit to the wonderful SaaS market and pick the right tools for the right job. Thanks to the wonderful beautiful community of developers and engineers that empowers other engineers and developers to put puzzle pieces together to get things done quickly in seconds or minutes.

For the trade-offs people

Like, as always, I use to add the word ‘trade-offs’ so you need to be-aware of it too. I mean, let’s be honest, its depends on us where gonna compromise and how much ‘technical debt’ we gonna bear. I mean maybe that could charge an extra buck from your bank account but on the other side it is really helping teams to get things done quickly (such as SonarCloud) enabling teams to move forward and ship quality-code faster. If you really getting more value than you might expected and is flourishing your business model leading to bigger customer acquisition and much bigger rounds then I must say that you can mark it as a green signal.

It’s all about shipping value today — not code

Today is all about how much value we getting. As being a technical in nature, every new product or app that I tries left different impressions on me. Some might really excites me. Some might doesn’t and some might doesn’t sounds any meaning to me. Regardless of experiencing and building multiple apps and software, I must add here that your product should at the end delivers some value — value in terms of real value that makes people genuinely happy and stick to your product.

That’s a wrap! Feel free to drop your thoughts below. I would be really happy listening to all your feedbacks.

Until we met again, Seeyou. Have a wonderful day!