DEV Community: Sandesh Ojha

Building a Kubernetes Operator from Scratch with operator-sdk

Sandesh Ojha — Mon, 08 Jun 2026 03:45:21 +0000

GitHub Repo: SandeshOjha06/k8-operator — Clone this to follow along with the code.

The Problem

Standard Kubernetes primitives keep containers running, but they are fundamentally ignorant of your application's business logic. If you deploy a complex database or a custom backend, raw Deployments and StatefulSets are not enough. They cannot execute automated schema migrations, sequence complex initialization steps, or prevent a developer from manually scaling a pod via kubectl scale and accidentally breaking application state.

That is the exact problem Operators solve. An Operator is a custom Go controller running inside the cluster that extends the Kubernetes API by encoding human operational knowledge directly into software. Instead of writing a runbook for your team to follow when things go wrong, you build an Operator that constantly observes the cluster, detects configuration drift, and forces compliance without any human intervention.

What We're Building

We are going to build a fully functional control plane for a custom resource called a WebApp.

Instead of manually wiring up and managing standard Kubernetes objects, users submit a single declarative YAML file defining just two spec fields: their desired replicas and the container image. Our Operator will:

Catch that custom API request and dynamically translate it
Automatically provision both a core Deployment and a NodePort Service
Continuously watch for configuration drift — if an unauthorized user modifies the running Deployment, the Operator instantly detects the deviation and reverts it
Report live application health back via a native Kubernetes Status subresource, exposing the current Phase and AvailableReplicas

Scaffolding with operator-sdk

We don't write an Operator from an empty directory. We use the operator-sdk CLI to generate the project scaffold so we can focus entirely on application logic.

Initialize the project:

operator-sdk init --domain sandesh.dev --repo github.com/sandesh-ojha/webapp-operator

This scaffolds the standard Kubebuilder project layout, including the main.go entrypoint and a comprehensive Makefile that handles compiling, testing, and deploying.

Generate the CRD and controller:

operator-sdk create api --group apps --version v1 --kind WebApp --resource --controller

--resource generates the Go structs that define your custom API schema
--controller scaffolds the Reconciler file where the active management logic will live

This generates two critical files:

api/v1/webapp_types.go — defines the Go structs representing the schema of our custom WebApp resource
internal/controller/webapp_controller.go — houses the Reconciler, the control loop where our logic executes

The SDK also generates // +kubebuilder: comment markers above our controller functions. These are not standard comments — they are compiler directives. When we run make manifests, the SDK reads these markers to automatically generate the strict RBAC YAML that grants our Operator permission to read, create, and update standard Kubernetes Deployments and Services.

Defining the CRD

Open api/v1/webapp_types.go. This is where we define the schema for our custom resource.

Kubernetes strictly separates what the user wants from what actually exists. We mirror this by defining two distinct Go structs: WebAppSpec and WebAppStatus.

type WebAppSpec struct {
    Replicas int32  `json:"replicas"`
    Image    string `json:"image"`
}

type WebAppPhase string

const (
    PhaseScaling WebAppPhase = "Scaling"
    PhaseRunning WebAppPhase = "Running"
)

type WebAppStatus struct {
    Phase             WebAppPhase `json:"phase,omitempty"`
    AvailableReplicas int32       `json:"availableReplicas"`
}

WebAppSpec holds the data the user submits in their YAML. WebAppStatus holds the live state our Operator reports back: the AvailableReplicas currently running and a Phase (Scaling or Running) indicating overall health.

Further up the file, just above the main WebApp struct, we add the kubebuilder markers:

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Phase",type="string",JSONPath=".status.phase"
// +kubebuilder:printcolumn:name="Replicas",type="integer",JSONPath=".spec.replicas"
// +kubebuilder:printcolumn:name="Available",type="integer",JSONPath=".status.availableReplicas"
// +kubebuilder:resource:scope=Namespaced,shortName=wa
type WebApp struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   WebAppSpec   `json:"spec,omitempty"`
    Status WebAppStatus `json:"status,omitempty"`
}

Two of these markers are vital for a production-grade Operator:

+kubebuilder:subresource:status — tells Kubernetes to expose a dedicated /status endpoint for this resource. This is critical: it allows our Operator to update the status safely without accidentally modifying the user's desired spec or triggering an infinite reconcile loop.

+kubebuilder:printcolumn — maps our internal JSON fields to human-readable columns in the terminal. When a user types kubectl get webapps, they immediately see the Phase, desired Replicas, and Available replicas — not just the resource name.

And the WebAppList type required by the SDK:

// +kubebuilder:object:root=true
type WebAppList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []WebApp `json:"items"`
}

func init() {
    SchemeBuilder.Register(&WebApp{}, &WebAppList{})
}

The Reconciler

If the CRD is the API, the Reconciler is the brain. Open internal/controller/webapp_controller.go.

This is where we implement the infinite control loop that constantly drives the cluster toward our desired state. We'll break the Reconcile function down exactly as it executes sequentially: Observe → Create → Correct → Report.

RBAC Markers

Before the function itself, we declare the permissions our controller needs. These markers generate the RBAC manifests automatically when you run make manifests:

// +kubebuilder:rbac:groups=apps.sandesh.dev,resources=webapps,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=apps.sandesh.dev,resources=webapps/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps.sandesh.dev,resources=webapps/finalizers,verbs=update
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete

Step 1 — The Function Signature and the Fetch Block

func (r *WebAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    logger := logf.FromContext(ctx)

    app := &appsv1.WebApp{}
    err := r.Get(ctx, req.NamespacedName, app)
    if err != nil {
        if apierrors.IsNotFound(err) {
            return ctrl.Result{}, nil
        }
        return ctrl.Result{}, err
    }
    // ...

Notice what req ctrl.Request receives — not the actual WebApp object or its state, only a name and a namespace. Kubernetes is highly asynchronous. By the time the Reconciler runs, the object might have changed again. Therefore, the very first step is to Get the absolute latest state of our WebApp resource directly from the API. If it returns a NotFound error, the user deleted the resource and we exit safely.

Step 2 — Deployment Reconciliation: Creation & Owner References

Next, we check if the underlying Nginx pods exist by attempting to fetch a standard Kubernetes Deployment with the same name as our WebApp:

    foundDep := &k8sappsv1.Deployment{}
    err = r.Get(ctx, types.NamespacedName{Name: app.Name, Namespace: app.Namespace}, foundDep)

    if err != nil && apierrors.IsNotFound(err) {
        dep, err := r.deploymentForWebApp(app)
        if err != nil {
            return ctrl.Result{}, err
        }

        if err := r.Create(ctx, dep); err != nil {
            return ctrl.Result{}, err
        }
        return ctrl.Result{Requeue: true}, nil
    }

If the Deployment is missing, we build it. Inside our deploymentForWebApp helper, there is one critical line:

func (r *WebAppReconciler) deploymentForWebApp(app *appsv1.WebApp) (*k8sappsv1.Deployment, error) {
    ls := map[string]string{"app": app.Name}
    replicas := app.Spec.Replicas

    dep := &k8sappsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      app.Name,
            Namespace: app.Namespace,
        },
        Spec: k8sappsv1.DeploymentSpec{
            Replicas: &replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: ls,
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{Labels: ls},
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{{
                        Name:  "webapp",
                        Image: app.Spec.Image,
                        Ports: []corev1.ContainerPort{{
                            ContainerPort: 80,
                            Protocol:      corev1.ProtocolTCP,
                        }},
                    }},
                },
            },
        },
    }

    // This is the critical line — sets the Owner Reference
    if err := ctrl.SetControllerReference(app, dep, r.Scheme); err != nil {
        return nil, err
    }
    return dep, nil
}

ctrl.SetControllerReference(app, dep, r.Scheme) sets an Owner Reference. Kubernetes needs to know that our WebApp is the parent and this Deployment is the child. When a user runs kubectl delete webapp sandy-app, the Kubernetes garbage collector reads this reference and automatically deletes the child Deployments and Services. Without this, you create zombie resources that run forever.

Step 3 — Deployment Reconciliation: Drift Detection

If the Deployment does exist, we don't just move on. We execute drift detection logic. This is what separates a fire-and-forget script from a true Operator.

    desiredReplicas := app.Spec.Replicas
    desiredImage    := app.Spec.Image
    actualImage     := foundDep.Spec.Template.Spec.Containers[0].Image

    if *foundDep.Spec.Replicas != desiredReplicas || actualImage != desiredImage {
        logger.Info("Drift detected! Updating Deployment",
            "DesiredReplicas", desiredReplicas,
            "ActualReplicas", *foundDep.Spec.Replicas,
            "DesiredImage", desiredImage,
            "ActualImage", actualImage,
        )

        foundDep.Spec.Replicas = &desiredReplicas
        foundDep.Spec.Template.Spec.Containers[0].Image = desiredImage

        if err := r.Update(ctx, foundDep); err != nil {
            return ctrl.Result{}, err
        }
        return ctrl.Result{Requeue: true}, nil
    }

We compare the Desired state (from our WebApp) against the Actual state (from the running Deployment). If an unauthorized user uses kubectl scale to crank replicas up to 10, or swaps the image to a vulnerable version, this if block catches it, overwrites the corrupted values in memory, and pushes the correction back to the cluster via r.Update().

Step 4 — Service Reconciliation

Once the compute layer is aligned, we repeat the exact same pattern for networking. We check if the Service exists, and if not, we create it. Because we need this exposed locally on Minikube, we provision a NodePort:

    foundSvc := &corev1.Service{}
    err = r.Get(ctx, types.NamespacedName{Name: app.Name + "-service", Namespace: app.Namespace}, foundSvc)

    if err != nil && apierrors.IsNotFound(err) {
        svc, err := r.serviceForWebApp(app)
        if err != nil {
            return ctrl.Result{}, err
        }
        if err := r.Create(ctx, svc); err != nil {
            return ctrl.Result{}, err
        }
        return ctrl.Result{Requeue: true}, nil
    }

And the helper:

func (r *WebAppReconciler) serviceForWebApp(app *appsv1.WebApp) (*corev1.Service, error) {
    ls := map[string]string{"app": app.Name}

    svc := &corev1.Service{
        ObjectMeta: metav1.ObjectMeta{
            Name:      app.Name + "-service",
            Namespace: app.Namespace,
        },
        Spec: corev1.ServiceSpec{
            Selector: ls,
            Type:     corev1.ServiceTypeNodePort,
            Ports: []corev1.ServicePort{{
                Protocol:   corev1.ProtocolTCP,
                Port:       80,
                TargetPort: intstr.FromInt(80),
            }},
        },
    }

    if err := ctrl.SetControllerReference(app, svc, r.Scheme); err != nil {
        return nil, err
    }
    return svc, nil
}

Step 5 — Status Updates

Finally, when we are certain the resources exist and are correctly configured, we report the application's health back to the user:

    available := foundDep.Status.AvailableReplicas
    app.Status.AvailableReplicas = available

    if available == app.Spec.Replicas {
        app.Status.Phase = appsv1.PhaseRunning
    } else {
        app.Status.Phase = appsv1.PhaseScaling
    }

    if err := r.Status().Update(ctx, app); err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{}, nil
}

We don't count pods ourselves — we read the AvailableReplicas reported by the native Kubernetes Deployment controller, copy that into our WebApp status, determine the Phase, and push the update.

⚠️ Critical: Notice we use r.Status().Update() instead of r.Update(). Kubernetes intentionally separates the Spec endpoint from the Status endpoint. If you update Status using the main r.Update() method, Kubernetes sees a modification to the primary object and instantly triggers another Reconcile loop — an infinite crash loop. The dedicated Status endpoint prevents this. This is why the +kubebuilder:subresource:status marker exists.

Testing on Minikube

With the logic complete, let's deploy the Operator and watch it enforce desired state. Open two terminal windows.

Terminal 1: Install the CRD and start the Operator

make install
make run

You'll see controller-runtime logs start up and wait for events.

Terminal 2: Deploy the Custom Resource

Create my-app.yaml:

apiVersion: apps.sandesh.dev/v1
kind: WebApp
metadata:
  name: sandy-app
  namespace: default
spec:
  replicas: 3
  image: nginx:1.25-alpine

Apply it:

$ kubectl apply -f my-app.yaml
webapp.apps.sandesh.dev/sandy-app created

Check the custom resource. Notice how our printcolumn markers format the output with live state:

$ kubectl get webapps
NAME        PHASE     REPLICAS   AVAILABLE
sandy-app   Running   3          3

Verify the Operator provisioned the underlying resources:

$ kubectl get pods
NAME                         READY   STATUS    RESTARTS   AGE
sandy-app-7c8585559-2mxgk    1/1     Running   0          45s
sandy-app-7c8585559-8pabc    1/1     Running   0          45s
sandy-app-7c8585559-xy9jz    1/1     Running   0          45s

Demo: Drift Detection

Simulate an unauthorized user bypassing the custom resource and manually scaling the Deployment:

$ kubectl scale deployment sandy-app --replicas=10
deployment.apps/sandy-app scaled

Immediately look at Terminal 1. The Operator catches it instantly:

INFO  Drift detected! Updating Deployment  {"DesiredReplicas": 3, "ActualReplicas": 10}

Check the pods — the Operator has already crushed the unauthorized change:

$ kubectl get pods
NAME                         READY   STATUS        RESTARTS   AGE
sandy-app-7c8585559-2mxgk    1/1     Running       0          2m
sandy-app-7c8585559-8pabc    1/1     Running       0          2m
sandy-app-7c8585559-xy9jz    1/1     Running       0          2m
sandy-app-7c8585559-zbt99    0/1     Terminating   0          12s
sandy-app-7c8585559-plk22    0/1     Terminating   0          12s

Kubernetes is terminating the excess pods. Within seconds, only the original 3 remain.

While this recovery is happening, our Status update block reports the degraded state in real time:

$ kubectl get webapps
NAME        PHASE     REPLICAS   AVAILABLE
sandy-app   Scaling   3          10

Once the cluster fully converges, the phase automatically switches back to Running.

What I Learned / What's Next

Building this Operator demystified a lot of the "magic" of Kubernetes. The biggest hurdle wasn't Go syntax — it was wrapping my head around the asynchronous, declarative nature of the Reconcile loop. You don't write scripts that execute step-by-step; you write a state machine that constantly compares what is versus what should be and corrects the difference.

Mastering this control loop pattern was exactly what I needed for my next step. For the upcoming Fall LFX Mentorship term, my target is containerd. Since containerd relies entirely on this same highly concurrent, API-driven Go architecture to manage low-level Linux primitives, building this Operator gave me the exact foundation to start contributing to the core daemon and the runtime shim.

If you have questions or want to dig into the code, the full repo is at SandeshOjha06/k8-operator. Drop a comment below — happy to discuss the controller-runtime internals.

I built a Redis clone in C: From blocking I/O to epoll to crash safe persistence

Sandesh Ojha — Sat, 23 May 2026 03:21:47 +0000

I had been hearing about sockets, TCP connections, and networking for years without truly understanding what any of it meant at the code level. Building C-dis changed that. I learned the full socket lifecycle, what happens on both sides of a connection, how the kernel manages buffers, and how to handle non-blocking I/O states inside a manual event loop.What surprised me most was that a single-threaded architecture using epoll could handle tens of thousands of requests per second: no threads, no locking, no complexity. Architectural efficiency beats brute-force processing more often than people realize.The problem: One client at a timeMy first version was a simple blocking server. It worked, but the limitation became obvious immediately.

while (1) {    
    int client_fd = accept(server_fd, (struct sockaddr *)&client_addr, &client_len);    

    // Server is completely frozen here until this client finishes
    handle_client(client_fd, ht);    
    close(client_fd);
}

c
While handle_client is running, the server cannot accept new connections, cannot read from other clients, and cannot do anything else. One client holds the entire server hostage. A second client connecting during this time sits in the kernel's backlog queue, waiting silently with no feedback. In production, this means a slow client degrades the experience for every other user on the server.The fix isn't to add threads. The fix is to stop blocking.The data engine: Building a hash table from scratchBefore writing the network layer, I needed somewhere to store data. The obvious choice was a simple array with linear search, but that is O(n) lookup. For a database, that is unacceptable.I looked at how Redis 2.0 approached this. Hash tables. O(1) average lookup, regardless of how many keys are stored. I implemented one from scratch using the djb2 hash function:

static unsigned int hash(const char *key) {    
    unsigned long int hashval = 5381;    
    int c;    

    while ((c = *key++))        
        hashval = ((hashval << 5) + hashval) + c;    

    return hashval & (HASH_SIZE - 1);
}

The (hashval << 5) + hashval is equivalent to hashval * 33, a multiplier chosen because it distributes keys evenly across buckets with minimal clustering. The & (HASH_SIZE - 1) replaces modulo with a bitwise AND, which is faster when the table size is a power of two.When two keys hash to the same bucket (a collision) they chain together in a linked list.

Lookup walks the chain comparing keys until it finds a match or hits NULL. With a good hash function and a reasonable load factor, chains stay short and lookup stays effectively O(1).select vs epoll: The architecture that makes this fast to handle multiple clients without threads, I needed I/O multiplexing: a way to ask the kernel "which of these sockets has data ready?" and act only on those.I implemented select first, then replaced it with epoll. The difference taught me more about systems design than any other part of this project.The problem with select:Every time you call select, you rebuild the entire watch list from scratch and hand it to the kernel. The kernel scans every file descriptor in the set (whether active or idle) and returns a modified bitmask showing which ones are ready. With 1,000 connected clients and only 1 active, the kernel scans all 1,000 every time. That is O(n) per event loop iteration.

while (1) {    
    fd_set readfds;    
    FD_ZERO(&readfds);    
    FD_SET(server_fd, &readfds);    

    for (int i = 0; i < MAX_CLIENTS; i++)        
        if (clients[i] != -1)            
            FD_SET(clients[i], &readfds);  // rebuild every iteration    

    select(max_fd + 1, &readfds, NULL, NULL, NULL);    
    // now scan all fds again to find which ones triggered
}

How epoll fixes it:epoll moves the interest list into the kernel permanently. You register a file descriptor once with epoll_ctl. The kernel maintains it. When you call epoll_wait, it returns only the file descriptors that are actually ready, not all of them.
C

// register once
epoll_ctl(epfd, EPOLL_CTL_ADD, client_fd, &ev);

// wait: returns only ready fds
int n = epoll_wait(epfd, events, 64, -1);
for (int i = 0; i < n; i++) {    
    // every event here is guaranteed ready    
    handle(events[i].data.fd);
}

This is a synchronous benchmark. Each command waits for a response before sending the next. It measures worst case single client throughput. Pipelining multiple commands per connection would push this significantly higher.Zero memory leaks across 40,000 operations verified with valgrind --leak-check=full. Every malloc in the hash table has a matching free in ht_destroy.What is nextThe next project is a container runtime: implementing Linux namespaces, cgroups, and pivot_root in C to build a minimal docker run from scratch. Understanding how containers actually work at the kernel level, not just how to use them.The code for C-dis is on GitHub. If you are building something similar or have questions about the epoll implementation, I am happy to talk through it in the comments below!

SHELL PROJECT - LEXER

Sandesh Ojha — Thu, 23 Apr 2026 16:28:55 +0000

I've spent the past few weeks going deep on Unix and Linux systems programming — reading TLPI, working through CS:APP, writing C daily. Today I started the project that ties it all together: building a shell from scratch in C.
The shell has a modular structure. Each component has one job. The first component I wrote today is the lexer.
**
What a lexer does**
The user types a string. Your program needs structured data. The lexer bridges that gap.
It takes a raw input string like:
'''bash
ls -la | grep txt > out.txt
'''
And converts it into an array of typed tokens:

[WORD:"ls"] [WORD:"-la"] [PIPE] [WORD:"grep"] [WORD:"txt"] [REDIR_OUT] [WORD:"out.txt"] [EOF]

Each token has a type and a value string for word tokens. The parser takes this token array and figures out what it means.

The implementation
The lexer is a state machine. It reads characters one at a time and decides what to do:

whitespace — finish any word in progress, skip
| — emit a PIPE token
> — peek at the next character: >> emits REDIR_APPEND, > alone emits REDIR_OUT
< — emit REDIR_IN
anything else — accumulate into the current word

#include "shell.h"

int lexer(const char *input, Token *tokens, int max_tokens) {
    int  i = 0;
    int  t = 0;
    int  w = 0;
    char word[MAX_INPUT];

    while (input[i] != '\0' && t < max_tokens - 1) {

        if (input[i] == ' ' || input[i] == '\t') {
            if (w > 0) {
                word[w] = '\0';
                tokens[t].type = TOKEN_WORD;
                strncpy(tokens[t].value, word, MAX_INPUT - 1);
                t++;
                w = 0;
            }
            i++;
            continue;
        }

        if (input[i] == '|') {
            if (w > 0) {
                word[w] = '\0';
                tokens[t].type = TOKEN_WORD;
                strncpy(tokens[t].value, word, MAX_INPUT - 1);
                t++;
                w = 0;
            }
            tokens[t].type     = TOKEN_PIPE;
            tokens[t].value[0] = '\0';
            t++;
            i++;
            continue;
        }

        if (input[i] == '>') {
            if (w > 0) {
                word[w] = '\0';
                tokens[t].type = TOKEN_WORD;
                strncpy(tokens[t].value, word, MAX_INPUT - 1);
                t++;
                w = 0;
            }
            if (input[i + 1] == '>') {
                tokens[t].type = TOKEN_REDIR_APPEND;
                i += 2;
            } else {
                tokens[t].type = TOKEN_REDIR_OUT;
                i++;
            }
            tokens[t].value[0] = '\0';
            t++;
            continue;
        }

        if (input[i] == '<') {
            if (w > 0) {
                word[w] = '\0';
                tokens[t].type = TOKEN_WORD;
                strncpy(tokens[t].value, word, MAX_INPUT - 1);
                t++;
                w = 0;
            }
            tokens[t].type     = TOKEN_REDIR_IN;
            tokens[t].value[0] = '\0';
            t++;
            i++;
            continue;
        }

        word[w++] = input[i++];
    }

    if (w > 0) {
        word[w] = '\0';
        tokens[t].type = TOKEN_WORD;
        strncpy(tokens[t].value, word, MAX_INPUT - 1);
        t++;
    }

    tokens[t].type     = TOKEN_EOF;
    tokens[t].value[0] = '\0';
    t++;

    return t;
}

Why separate lexer from the parser?
While designing, i learned about splitting responsibilities.

The lexer only cares about about the character or sequence. It knows nothing about a valid structure. That is the parser's problem.

The parser inquires about what the token or sequence mean and not about the raw characters.

If you combine them into one function you get something complex and hard to test. Separate them and each is small enough to hold in your head at once. I can test the lexer by printing its output before the parser ever runs.

I wrote 1 byte to a 10GB file and used almost no disk space

Sandesh Ojha — Sun, 19 Apr 2026 14:03:05 +0000

I used to think of a file as a solid block of data. 10GB file, 10GB on disk. That's just something you'd normally assume, right?
I was wrong.
I'm a CS student currently going deep on systems programming — reading TLPI, writing C daily, learning how the OS actually works under the hood. This week I was working through file I/O and had one of those moments where something you thought you understood turns out to be completely different one level down.
You can seek past the end of a file and write there. The filesystem won't fill the gap with real data. It just remembers that the gap exists.

#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>

int main() {
    int fd = open("huge_file", O_WRONLY | O_CREAT | O_TRUNC, 0644);
    if (fd == -1) { perror("open"); exit(1); }

    // seek to the 10GB mark without writing anything
    if (lseek(fd, 10LL * 1024 * 1024 * 1024, SEEK_SET) == -1) {
        perror("lseek"); exit(1);
    }

    // write exactly one byte
    if (write(fd, "X", 1) == -1) {
        perror("write"); exit(1);
    }

    close(fd);
    return 0;
}

Run this and check the file two different ways:

ls -lh huge_file    # 10G
du -h huge_file     # 4.0K

The file reports 10GB in size. It uses 4KB of actual disk. Both numbers are telling the truth.
ls -lh shows the logical size — where the last byte lives relative to the start of the file. The filesystem knows the file is 10GB in the sense that if you read from byte 0, you'll eventually reach byte 10GB.
du -h shows allocated blocks — the physical disk space actually consumed. There's one 4KB block holding your single X. Everything before it is a hole: a range of offsets the filesystem tracks in metadata without backing it with real storage. Reading from a hole returns zero bytes. No disk access needed — there's nothing to read.

This isn't just a curiosity. It's the mechanism behind tools you use every day.
When you create a virtual machine in VirtualBox or QEMU and tell it "this VM gets a 100GB disk," you're not consuming 100GB immediately. The disk image is a sparse file. It grows as the VM writes data. The "100GB" is just a ceiling.
Docker layers work on the same idea — copy-on-write means a layer only allocates blocks when data actually changes. Database engines use sparse files for pre-allocated tables that fill in over time without ever needing to write zeros to reserve space.

The thing that stuck with me isn't the trick itself. It's what it reveals about how the filesystem actually thinks.
A file isn't a sequence of bytes on disk. It's a mapping from logical offsets to physical blocks — and some offsets don't map to anything. The OS constructs the illusion of a contiguous sequence on demand.
Once you see that, other things start clicking too. Why copying a sparse file with plain cp can suddenly consume far more disk than the original — it materializes the holes. Why rsync has a --sparse flag. Why checking disk usage with ls alone will mislead you.
Systems programming keeps doing this to me. I think I understand something, and then one level lower the actual mechanism is completely different from the abstraction I was using. The abstraction is useful. But knowing what's underneath makes you a better engineer, not just someone who knows how to use the tools.