DEV Community: Aditya Sharma

How Linux Handles a `fork()` When the Parent Has Gigabytes of Memory

Aditya Sharma — Sat, 17 Oct 2026 11:09:49 +0000

A process with 8 GB of resident memory calls fork(). On a modern Linux system, the call returns in a few milliseconds. No 8 GB copy was made. The child is running.

But something happened. The kernel did real work, and more real work is coming. Where did the cost go?

What `fork()` Actually Creates

fork() creates a new process with its own process ID, its own file descriptor table, its own signal state, and critically its own virtual address space. The child's address space is initially identical to the parent's: same virtual address ranges, same mappings, same content.

What it does not have is its own copy of every physical memory page the parent mapped. The two processes share the underlying physical pages. This is the mechanism that makes fork() cheap at creation time, and it is called copy-on-write (COW).

The distinction between a virtual address space and the physical memory it references is the key to understanding everything that follows. Processes operate on virtual addresses. The MMU translates those to physical addresses using page tables. Two processes with different virtual address spaces can point at the same physical pages, and neither process needs to know.

What Happens Inside the Kernel

When a process calls fork(), the kernel enters kernel_clone() (the common implementation behind fork(), vfork(), and clone() in Linux 5.x+), which calls copy_process(). This function allocates a new task_struct for the child and copies or shares the parent's resources according to the clone flags.

For memory, the path goes through copy_mm(), which calls dup_mm(). This function allocates a new mm_struct the kernel's per-process memory descriptor and duplicates the parent's virtual memory areas (VMAs) and populated page-table entries.

VMAs are the kernel's records of what is mapped where. Each VMA describes a contiguous range of virtual addresses: its start and end, its permissions, whether it is file-backed or anonymous, and associated flags. A process with a large heap, several shared libraries, a stack, and a few mmap() regions may have dozens or hundreds of VMAs. Duplicating them means copying the metadata, not the pages they describe.

Page tables are the hardware-level translation structures that map virtual addresses to physical page frames. The kernel duplicates only the page-table entries that are actually populated that is, entries for pages that have been faulted in and are resident in physical memory. Virtual address space that has been reserved with mmap() but never accessed has no page-table entries yet, so it contributes nothing to page-table duplication cost at fork() time. The work is proportional to the number of resident, mapped pages, not to the total virtual address space size.

This matters for interpreting fork() latency. A process that has mmap()d several gigabytes but touched very little of it has a small page-table footprint and forks quickly. A process that has touched the bulk of its allocation has a large page-table footprint and forks more slowly, even if the two processes have the same virtual address space size.

During duplication, the kernel also marks all writable private PTEs as read-only in both parent and child. This write-protection is what arms the COW mechanism. A page that neither process writes after fork() stays shared and stays read-only in both page tables.

Copy-on-Write in Detail

After fork() returns, the parent and child share physical pages, and both have read-only PTEs for all private mappings that were writable. The moment either process tries to write to one of those pages, the hardware detects a write to a read-only PTE and raises a page fault.

The kernel's page-fault handler dispatches to do_wp_page() in mm/memory.c. This function checks whether the faulting page is truly shared between multiple processes or whether it can be made writable without copying.

If the page has only one reference because the other process has already gotten its own copy, or has exited do_wp_page() can simply make the PTE writable and return. No copy is made. The page is already exclusive.

If the page is genuinely shared, the handler allocates a new physical page, copies the content of the old page into it, updates the faulting process's PTE to point to the new page with write permission, and decrements the reference count on the old page. The write that triggered the fault then completes normally. The other process's PTE still points at the original page, still read-only.

This is the full COW sequence:

Both PTEs marked read-only at fork() time.
Write attempt triggers a hardware page fault.
Kernel identifies a COW fault in do_wp_page().
If page is still shared: allocate new page, copy content, update PTE, resume.
If page is now exclusive: make PTE writable, resume without copying. The copy happens at write time, not at fork() time. The cost is paid per-page, the first time each page is modified after forking.

Anonymous pages heap allocations, stack, mmap(MAP_ANONYMOUS) behave this way. File-backed mappings (shared libraries, memory-mapped files) are different: they are backed by the page cache. A write to a private file-backed mapping similarly triggers a COW fault, but the resulting private copy is no longer associated with the underlying file.

Where the Real Cost Actually Appears

fork() is not free, even before any writes happen.

Page-table duplication is the primary upfront cost for processes with large resident footprints. On x86-64, a four-level page table tree must be allocated and populated for the child, covering all pages that are resident in the parent. This is real memory allocation and copying work. A process with a large working set -- many pages actually touched and resident has a proportionally larger page-table footprint. VMA metadata also scales with the number of distinct mapped regions: a process with many small, fragmented mappings is slower to fork than one with the same RSS organized into fewer, larger regions.

The practical consequence is that fork() latency is more tightly coupled to the number of resident, populated pages than to the total virtual address space size or the total allocated memory. A 32 GB process that has only touched 200 MB of its allocation may fork faster than a 4 GB process that has fully populated its working set.

Memory pressure increases after fork() because the child's page tables themselves consume physical RAM immediately. For a process with a large resident footprint, page tables can occupy a meaningful amount of memory that tools like top or htop do not surface directly. The Pss and Size fields in /proc/<pid>/smaps give a clearer picture.

Post-fork writes bring the deferred cost home. If the child modifies a large fraction of the inherited pages, it triggers a COW fault for each one. Each fault allocates a new physical page and copies its content. Modify enough pages and the total work approaches what a full upfront copy would have cost, distributed across the child's execution as latency spikes rather than paid at creation time.

Transparent Huge Pages (THP) can change the per-fault cost, but the behavior is not as simple as "one write copies 2 MB." When a COW fault occurs on a huge-page-backed anonymous mapping, the kernel may try to preserve the huge-page mapping by allocating a new 2 MB huge page and copying the original. If a suitable huge page is not available, or if the fault affects only part of a huge page, the kernel may instead split the huge page into regular 4 KB pages and handle the fault on the relevant small page alone. The outcome depends on memory availability, the kernel version, and the configuration. The net effect is that THP can increase per-fault cost compared to small pages when a full huge-page copy does occur, which is worth checking with grep AnonHugePages /proc/<pid>/smaps if you are diagnosing unexpectedly high post-fork memory growth.

A Reproducible Experiment

This C program allocates and touches a large anonymous region before fork(), measures the fork() call in isolation, and compares a read-heavy child against a write-heavy child.

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <sys/resource.h>
#include <sys/wait.h>
#include <time.h>
#include <unistd.h>

#define REGION_SIZE (1UL << 30)  /* 1 GB */
#define PAGE_SIZE   4096

static long elapsed_us(struct timespec *a, struct timespec *b) {
    return (b->tv_sec - a->tv_sec) * 1000000L +
           (b->tv_nsec - a->tv_nsec) / 1000L;
}

int main(int argc, char *argv[]) {
    int write_heavy = (argc > 1 && argv[1][0] == 'w');

    /* Allocate and touch every page to make them all resident before fork() */
    char *mem = malloc(REGION_SIZE);
    if (!mem) { perror("malloc"); return 1; }
    for (size_t i = 0; i < REGION_SIZE; i += PAGE_SIZE)
        mem[i] = 1;

    printf("Region resident. Forking...\n");
    fflush(stdout);

    struct timespec t0, t1;
    clock_gettime(CLOCK_MONOTONIC, &t0);
    pid_t child = fork();
    clock_gettime(CLOCK_MONOTONIC, &t1);

    if (child < 0) { perror("fork"); return 1; }

    if (child == 0) {
        /* Child: read all pages or write all pages */
        volatile char sink = 0;
        if (write_heavy) {
            for (size_t i = 0; i < REGION_SIZE; i += PAGE_SIZE)
                mem[i] = 2;
        } else {
            for (size_t i = 0; i < REGION_SIZE; i += PAGE_SIZE)
                sink += mem[i];
        }
        (void)sink;

        struct rusage ru;
        getrusage(RUSAGE_SELF, &ru);
        /* Use fflush + _exit to ensure output is written without running
           stdio cleanup (which could allocate and trigger additional faults) */
        printf("Child (%s): majflt=%ld minflt=%ld\n",
               write_heavy ? "write" : "read",
               ru.ru_majflt, ru.ru_minflt);
        fflush(stdout);
        _exit(0);
    }

    /* Parent: report fork latency */
    printf("fork() latency: %ld us\n", elapsed_us(&t0, &t1));

    int status;
    waitpid(child, &status, 0);
    return 0;
}

Compile and run both modes:

gcc -O2 -o fork_test fork_test.c
./fork_test r   # read-heavy child
./fork_test w   # write-heavy child

Interpreting minflt. The write-heavy child will show substantially more minor faults than the read-heavy child. Each COW fault appears as a minor fault (no disk I/O required). However, minflt counts all minor faults, including those triggered by stdio, stack growth, and other kernel activity not exclusively COW copies. It is a useful signal that COW activity is occurring, but not an exact count of pages physically copied. The read-heavy child should show a much smaller minor-fault count because reads against shared COW pages do not trigger faults.

You can also inspect memory after the child is running (before it exits) using:

grep -A 6 "Private_Dirty\|Shared_Clean\|AnonHugePages" /proc/<child_pid>/smaps | head -60

Private_Dirty shows pages the child has copied and modified. Shared_Clean shows pages still shared with the parent. After a write-heavy run, Private_Dirty grows significantly. After a read-heavy run, it stays near zero.

What this experiment establishes. It shows that fork() latency for a fully-resident 1 GB allocation is distinct from post-fork memory access cost, and that write-heavy children incur significantly more minor faults than read-heavy ones. It does not establish universal numbers: results vary with kernel version, architecture, THP configuration, system memory pressure, and CPU cache state. Treat the output as a starting point for understanding your specific system's behavior, not as a benchmark to generalize from.

Common Misconceptions

"fork() instantly copies the entire process." No. Physical page contents are not copied at fork() time. Page-table entries for resident pages are duplicated and write-protected. Physical pages are copied lazily on write.

"Copy-on-write makes fork() free." No. Duplicating page-table entries for a large resident working set takes measurable time and consumes kernel memory. The work is real, just smaller than a full physical copy.

"The child immediately consumes additional RAM equal to the parent's RSS." No. Immediately after fork(), the child shares physical pages with the parent. Private memory grows gradually as COW faults occur. The child's page tables do consume additional physical RAM right away, but that is typically much smaller than the RSS.

"Every write after fork() copies a page." No. If the page has become exclusive because the other process already triggered a COW fault and got its own copy, or has exited the kernel can make the PTE writable without copying.

"Virtual address space size alone predicts fork() latency." No. The cost at fork() time is driven by the number of resident, populated page-table entries, not by the total virtual address space or the total amount allocated. A large sparsely-touched allocation forks cheaply.

Engineering Implications

Prefork servers (Apache's prefork MPM, Gunicorn, many Ruby application servers) fork worker processes from a parent that has already loaded application code and warmed data structures. The intent is for workers to share that memory via COW. It works well as long as workers read more than they write. Workers that heavily modify inherited data structures mutating caches, updating per-request counters in shared structures progressively diverge from shared pages and consume additional physical memory proportional to the pages they dirty.

Diagnosing unexpected memory growth after fork() starts with /proc/<pid>/smaps. Compare Shared_Clean, Private_Clean, and Private_Dirty across the parent and a worker over time. Growing Private_Dirty in workers indicates COW divergence. If AnonHugePages is significant and workers show high Private_Dirty, THP may be increasing per-fault copy cost; madvise(MADV_NOHUGEPAGE) on specific regions can disable THP selectively for those areas if that is the bottleneck.

posix_spawn() avoids duplicating the parent's page tables altogether. It creates a new process running a new executable directly. For patterns where fork() is immediately followed by exec(), posix_spawn() can reduce overhead because there is no COW setup to pay for and immediately discard. Whether this matters depends on the parent's resident footprint and spawn frequency; for many applications the difference is not significant, but it becomes relevant when forking frequently from a large process.

If fork() latency is a bottleneck, measure it in isolation (as the experiment above does), inspect page-table overhead with pmap -x or /proc/<pid>/smaps, and consider whether the process's resident working set can be reduced or whether large contiguous mappings can replace many fragmented ones.

The Core Principle

Linux defers the cost of copying process memory. It does not eliminate it. fork() shifts physical page copying from process creation time to the first write to each page in the child. The page-table duplication cost is paid upfront and scales with the number of resident, populated pages. The physical copy cost is paid on demand and scales with the number of pages the child actually modifies.

These two costs respond to different things and are diagnosed with different tools. Understanding the distinction is what separates a developer who knows that fork() "uses copy-on-write" from one who can explain why their prefork server's workers are consuming hundreds of additional megabytes after a few minutes of traffic.

What Is a Zombie Process in Linux? How Processes Become Stuck in the Process Table

Aditya Sharma — Fri, 16 Oct 2026 18:08:57 +0000

Imagine running a program in Linux, watching it finish, and then discovering that it still exists in the process table.

It consumes no CPU. It is no longer executing instructions. It cannot perform any useful work. Yet, when you inspect the system, its process entry is still there.

This is a zombie process.

Zombie processes reveal something fundamental about how Linux manages process lifecycles. A process finishing its work and the operating system completely removing its process record are two separate events. Linux deliberately keeps a small amount of information about a terminated process until its parent collects that information.

Understanding why this happens requires looking beyond commands like ps and top. It means understanding how processes terminate, how exit status is communicated between parent and child processes, what the kernel preserves after termination, and what happens when a parent fails to collect that information.

Let's examine the mechanism from the inside out.

1. What Exactly Is a Zombie Process?

A zombie process is a process that has terminated, but whose parent has not yet collected its termination status.

When a Linux process finishes, the kernel does not necessarily remove every trace of it immediately. It retains a minimal process-table entry containing information that the parent may still need, especially the process's exit status.

The process is no longer running. Its execution has ended, but its termination has not yet been fully acknowledged by its parent.

Linux represents this state with the process state Z, commonly displayed as Z (zombie).

A zombie is sometimes called a defunct process. Both terms describe a process that has already terminated but still has a record awaiting collection.

There is an important distinction here:

A running process is actively executing or waiting to execute.
A sleeping process is waiting for an event or resource.
A stopped process has been suspended.
A zombie process has already terminated and cannot resume execution.

A zombie is not a program that has frozen. It is a program that has finished, while the operating system is still preserving information about its termination.

2. How a Process Becomes a Zombie

To understand zombie processes, first understand the parent-child relationship between processes.

When a program creates another process using a system call such as fork(), Linux creates a child process. The original process becomes the parent, and the newly created process becomes its child.

For example, imagine a server that launches a separate worker to process a request.

Parent process

Creates and supervises a worker

fork() creates a child

Child process

Executes its task

Child terminates with exit status

Zombie state

Exit status awaits collection by the parent

Parent calls wait() or a related system call

Process record released

The important part is what happens after the child terminates.

Step 1: The child executes

The child performs its work. It may read a file, process a request, calculate a result, or run a short-lived command.

During this time, it has an execution context managed by the kernel.

Step 2: The child terminates

When the program calls exit() or returns from main(), the process begins termination. It can also terminate because of a signal, such as SIGTERM or SIGKILL.

The kernel releases most resources associated with the process, including its address space and other execution resources.

However, the kernel must preserve enough information for the parent to learn how the child ended.

Step 3: The kernel retains the process record

The child transitions into the zombie state if its termination status has not yet been collected.

The kernel preserves information such as the process ID, termination status, and accounting information needed by the parent and system process-management mechanisms.

The zombie does not continue running in the background. Its instructions are no longer executing, and its former memory space is no longer retained as a live process address space.

Step 4: The parent collects the exit status

The parent can call wait(), waitpid(), or another related waiting interface to collect the child's termination status.

Once the kernel has delivered that information and the process can be fully reaped, the zombie entry is removed.

The core idea is simple: process termination and process reaping are separate stages.

3. What Happens Inside the Linux Kernel?

The zombie state makes more sense when viewed as a lifecycle-management problem.

Linux must support processes that finish independently of when their parents are ready to inspect their results. A child might terminate while its parent is busy serving another request, waiting on I/O, or performing unrelated work.

The kernel therefore needs a way to preserve the child's exit information until the parent can retrieve it.

On modern Linux systems, process metadata is represented internally by kernel structures, including task_struct. This structure tracks information about a task, including its state and relationships with other tasks. The exact internal lifecycle and cleanup details depend on kernel implementation.

When a child exits, Linux performs termination and resource-cleanup operations. It does not simply erase every reference to the process immediately. The remaining task information allows the parent to identify the terminated child and collect its status.

The parent-child relationship matters because the kernel needs to know which process is responsible for collecting the child's termination information.

The kernel also has to handle races between a child exiting and a parent waiting for it. For example, the parent might call waitpid() just before the child exits. In that case, the parent can wait until the child terminates, and the kernel can report the termination when it becomes available.

If the child has already terminated, the waiting call can collect the status immediately.

This is why zombies are not necessarily evidence of a bug. A short-lived zombie can appear during normal process management, then disappear as soon as the parent reaps it.

A zombie that remains for a long time is more interesting because it may indicate that the parent is not handling its children correctly.

4. Let's Create a Zombie Process in Linux

The easiest way to understand the behavior is to reproduce it.

The following Python program creates a child process using os.fork(). The child exits immediately, while the parent deliberately waits without collecting the child's status.

Run this on a Linux machine or in a Linux virtual machine. Do not run it in a production environment.

import osimport timepid = os.fork()if pid == 0:    # Child process    print(f"Child exiting: PID={os.getpid()}")    os._exit(0)else:    # Parent process    print(f"Parent PID={os.getpid()}")    print(f"Child PID={pid}")    print("Waiting without reaping the child...")    time.sleep(60)    # Collect the child's exit status    os.waitpid(pid, 0)    print("Child reaped.")

Save the program as zombie_demo.py and run it:

python3 zombie_demo.py

The parent prints the child's PID and sleeps for 60 seconds. The child exits immediately.

During that interval, the child should normally appear as a zombie because the parent has not called waitpid() yet.

Open another terminal and inspect it:

ps -o pid,ppid,stat,cmd -p <CHILD_PID>

Replace <CHILD_PID> with the child PID printed by the program.

You may see output similar to this:

    PID    PPID STAT CMD
  24817   24816 Z    [python3] <defunct>

The exact PID and command formatting will differ across systems.

Here's what matters:

PID identifies the child.
PPID identifies its parent.
STAT shows Z, indicating zombie state.
<defunct> is a descriptive label used by ps for a terminated process that has not yet been reaped.

Now wait for the parent program to finish sleeping. It calls os.waitpid(pid, 0), collecting the child's termination status. The zombie should disappear from the process table.

This demonstration illustrates the distinction between a child exiting and the parent reaping it.

Important: os._exit(0) is used in the child to terminate immediately without running normal Python interpreter cleanup. That is useful after fork() in a demonstration like this because the child should not continue through the parent's Python runtime shutdown behavior.

5. Why Does Linux Need `wait()` and `waitpid()`?

At first, keeping a record of a dead process may seem unnecessary. Why not delete the process immediately and let the parent figure things out later?

Because the parent may need to know whether the child succeeded, failed, or terminated because of a signal.

Consider a parent process that launches a worker to compress a file.

The worker exits with status 0 if it succeeds and a nonzero status if it encounters an error. The parent can use the exit status to decide whether to mark the job as complete or retry it.

Without a mechanism to retrieve the child's termination status, the parent would lose an important part of process coordination.

This is the role of the wait family of system calls.

`wait()`

The wait() system call waits for a child process to change state in a way that can be reported, and collects its termination status when the child has terminated.

At the C level, a simple example looks like this:

int status;
pid_t child = wait(&status);

If a terminated child is available, wait() can reap it and return its PID. If no eligible child has terminated yet, the caller normally blocks until one does, unless configured otherwise or interrupted.

`waitpid()`

waitpid() gives the parent more control over which child it waits for and how it waits.

For example:

waitpid(child_pid, &status, 0);

This waits for the specified child. The options parameter can also be used to request nonblocking behavior or other supported waiting modes.

In Python, the equivalent in our demonstration is:

os.waitpid(pid, 0)

The distinction is not merely about waiting. For a terminated child, collecting its status is the operation that allows the kernel to finish removing the zombie's remaining process record.

A parent can also collect a child's exit status through related interfaces such as waitid().

6. Why Zombie Processes Can Become a Problem

A single zombie usually consumes very little in the way of resources. Most of the child's resources have already been released.

So why do system administrators care about them?

The issue is not typically CPU usage or a large block of leaked memory. It is the accumulation of unreaped process records.

Each zombie retains a process-table entry and a process ID until it is reaped. Linux has finite limits on the number of tasks and process IDs that can exist. If a program repeatedly creates child processes and never collects their exit status, zombies can accumulate.

Eventually, the system may encounter difficulty creating new processes because it has reached a relevant process or task limit.

This can affect services that rely on spawning workers, running subprocesses, or launching helper programs.

Common causes include:

A parent process that never calls wait() or a related interface.
A server that creates many short-lived child processes but does not reap them.
Incorrect signal handling, such as mishandling SIGCHLD.
A subprocess-management bug in a long-running application.
A parent process that is blocked indefinitely and therefore never gets around to collecting child statuses.

One important nuance: receiving SIGCHLD does not automatically reap a child. The signal notifies the parent about a child-state change, but the application still needs appropriate child-status handling. Depending on signal configuration and platform semantics, some child statuses may be handled automatically, but applications should not assume that simply receiving a signal removes every zombie.

7. How to Find Zombie Processes

Linux provides several standard tools for inspecting process state.

Use `ps`

To list processes whose state is zombie:

ps -eo pid,ppid,stat,cmd | awk '$3 ~ /^Z/'

This lists the process ID, parent process ID, state, and command for entries whose state field begins with Z.

If you want to inspect one particular process:

ps -o pid,ppid,stat,cmd -p <PID>

Look for Z in the state column or <defunct> in the command description.

Use `top`

Run:

top

Depending on your system and configuration, top shows a summary of zombie processes in its process-state statistics. You can also inspect individual process states.

A zombie may not behave like a normal running process in the list because it has already terminated.

Inspect the parent process

Finding the zombie is only half the investigation. You also need to identify its parent.

Use the PPID field from ps:

ps -o pid,ppid,stat,cmd -p <ZOMBIE_PID>

Then inspect the parent:

ps -fp <PARENT_PID>

The parent is usually the process that must collect the child's exit status. Investigating the parent's logs, subprocess code, and signal handling is generally more useful than focusing on the zombie itself.

8. Can You Kill a Zombie Process?

This is one of the most common misunderstandings about zombies.

You generally cannot kill a zombie process because it has already terminated.

A signal such as SIGKILL cannot make the child terminate a second time. There is no running process left to execute a signal handler or stop.

For example, sending a signal to the zombie's PID does not solve the underlying issue:

kill -9 <ZOMBIE_PID>

The zombie's parent must normally collect the exit status so that the kernel can remove the remaining record.

The usual troubleshooting process is:

Identify the zombie's parent using PPID.
Determine why the parent is not reaping its children.
Fix or restart the parent if appropriate and safe.
Verify that the zombie disappears.

If the parent exits, the orphaned child is generally adopted by an appropriate system process or configured subreaper. That process can then collect the child's termination status. On many systems, this ultimately involves PID 1, but the exact adoption behavior depends on namespaces and subreaper configuration.

Restarting or terminating a parent can disrupt active work, so it should not be done blindly on a production system. The right fix is usually to correct the parent's child-process management rather than repeatedly trying to kill zombies.

9. Zombies vs. Orphan Processes

Zombie and orphan processes are often confused because both involve parent-child relationships. However, they describe different situations.

Property	Zombie process	Orphan process
Has the child terminated?	Yes	Not necessarily
Is it executing?	No	It may be running or sleeping
What happened to the parent?	The parent has not collected the child's termination status	The parent exited while the child was still alive
Typical handling	Parent reaps the child	Child is adopted by a suitable process or subreaper

An orphan process is not automatically a zombie. If its original parent exits while it is still running, the child can continue executing under a new parent.

If that orphan later terminates, its new parent can collect its exit status.

The distinction is simple: an orphan describes a living child whose parent has gone away; a zombie describes a terminated child whose exit status has not been collected.

10. How Developers Should Prevent Zombies

For developers building servers, worker systems, command-line tools, or applications that spawn subprocesses, zombie prevention is a process-management responsibility.

Always collect child statuses

In C or C++, use the appropriate wait() or waitpid() logic for children you create. In Python, use subprocess APIs that wait for or communicate with the child when needed.

For example, a Python program that starts a subprocess should not simply launch children indefinitely and forget about them. It should manage their lifecycles and ensure their statuses are collected.

Handle multiple children correctly

A parent can have several children terminate close together. A robust SIGCHLD handler or event loop must account for the possibility that multiple child statuses are available when it responds.

A common approach is to repeatedly call waitpid() with WNOHANG until no more terminated children are available. This prevents a parent from reaping only one child when several have already exited.

Use the right subprocess abstraction

High-level process-management libraries often make waiting and cleanup easier, but they do not remove the need to understand lifecycle behavior.

For example, in Python, subprocess.run() waits for the command to complete. When using subprocess.Popen, you need to manage the child process appropriately, including calling methods such as wait() or communicate() when suitable.

Test failure paths

Zombie bugs often appear when workers fail unexpectedly, receive signals, or terminate under load. Test these cases explicitly rather than checking only the successful execution path.

Monitor process state

If a service repeatedly launches subprocesses, monitor the number of zombie processes over time. A brief zombie state is normal; a steadily increasing count can indicate a child-reaping bug.

The Bigger Picture

Zombie processes are not failed programs that continue running in secret. They are a consequence of Linux separating process termination from the collection of a process's exit status.

The child finishes executing. The kernel releases most of its resources but preserves the information required by the parent. The parent calls a waiting interface, collects the status, and the remaining process record can be removed.

That design makes parent-child process coordination reliable, even when children finish before their parents are ready to handle the result.

For developers, the lesson is practical: creating a process is only one part of managing it. A complete process lifecycle includes starting the child, handling its execution, and collecting its termination status.

The next time you see a process marked <defunct>, don't try to kill it first. Find its parent, understand why the child was not reaped, and fix the lifecycle management.

That's how you solve the problem instead of treating its symptom.

What is Overfitting: How Can an AI Model Score 99% on Training Data and Still Fail in the Real World?

Aditya Sharma — Thu, 15 Oct 2026 17:15:51 +0000

Imagine building an AI model to detect fraudulent transactions.

You train it on thousands of historical transactions. The results look incredible. It correctly classifies 99% of the training examples.

You deploy it, expecting it to catch fraud reliably.

But something goes wrong.

Legitimate transactions get flagged. New fraud patterns slip through. The model that looked almost perfect during training performs disappointingly on transactions it has never encountered before.

How can a model that performed so well suddenly become unreliable?

The problem might not be the model's architecture, its computing power, or even the quality of the training algorithm.

It might have learned the wrong things.

This is called overfitting, one of the fundamental problems in machine learning. It occurs when a model learns patterns that work extremely well on its training data but fail to generalize to new, unseen examples.

Understanding overfitting means understanding a critical distinction: learning the training data is not the same as learning the underlying problem.

1. The Difference Between Learning and Memorizing

Consider a student preparing for an examination.

One student memorizes the answers to every question in a practice paper. When the exact questions appear in the examination, the student performs exceptionally well.

Another student understands the underlying concepts. Even when the examination introduces unfamiliar questions, the student can reason through them.

Both students may achieve the same score on the practice paper. But only one has demonstrated that their knowledge transfers to new situations.

Machine learning models can behave in a similar way.

During training, a model receives examples and adjusts its internal parameters to reduce prediction errors. The goal is not simply to reproduce the training examples. It is to learn patterns that remain useful when new data arrives.

An overfitted model can become excessively specialized in its training examples. It may learn meaningful relationships, but it also learns accidental details, noise, and coincidences that do not hold outside the dataset.

For example, imagine training an image classifier to distinguish wolves from dogs.

Suppose most wolf images in the training dataset contain snow, while most dog images show grass or houses.

A model might discover that snowy backgrounds are strongly associated with wolves.

Its training accuracy could be excellent. But it has learned a shortcut rather than a reliable way to distinguish the animals.

When shown a wolf standing on grass, or a dog playing in snow, it may make the wrong prediction.

The model has learned a correlation in its training data, but that correlation is not a dependable solution to the actual task.

This is the central danger of overfitting: a model can learn patterns that explain its training data without learning patterns that generalize.

2. Why Does Overfitting Happen?

Overfitting is not caused by a single problem. Several factors can contribute to it.

An Overly Complex Model

A model with too much capacity relative to the available data may be able to fit not only meaningful patterns but also random fluctuations in the training examples.

Imagine fitting a curve to a small collection of data points.

A simple model might capture the general trend while missing some individual points. A much more flexible model might pass through every point, including those that are there because of measurement errors or random variation.

The second model fits the observed data more closely, but that does not necessarily make it better at predicting new observations.

The same principle applies to neural networks, decision trees, and other machine learning algorithms.

However, complexity alone does not guarantee overfitting. The outcome also depends on the amount and quality of data, the model's inductive biases, the training procedure, and the regularization techniques used.

Too Little or Unrepresentative Training Data

A model can only learn from the examples it receives.

Suppose a medical image classifier is trained using images collected from a single hospital. The images may share particular scanners, acquisition settings, patient demographics, or image-processing conventions.

The model may learn patterns associated with those conditions.

When deployed at another hospital with different equipment or patient populations, its performance may decline.

The problem is not necessarily that the model memorized individual images. It may have learned relationships that were valid in the original dataset but are not sufficiently representative of the broader population.

Small datasets make it easier for accidental patterns to appear convincing. Unrepresentative datasets create a different but related problem: the model may never have learned the variation it will encounter in production.

Training for Too Long

Machine learning models often improve as training progresses, but the improvement does not necessarily continue indefinitely.

Early in training, a model may learn broad, useful patterns. As training continues, it may begin fitting increasingly specific details of the training data.

At some point, its performance on unseen data may stop improving even while its training performance continues to get better.

This is one reason training duration matters.

However, overfitting is not inevitable in every long training run. Its likelihood depends on the model, dataset, optimizer, regularization, and other training conditions.

Noisy Labels and Accidental Patterns

Training data is rarely perfect.

Human annotators make mistakes. Sensors introduce measurement errors. Datasets contain duplicates, inconsistent labels, and unusual examples.

A sufficiently flexible model can learn some of these errors instead of treating them as noise.

Imagine a dataset containing photographs of cats and dogs, but a few images are incorrectly labeled. A model that tries to fit every example may learn to reproduce those incorrect labels.

The same issue appears in financial forecasting, medical diagnosis, fraud detection, and natural language processing.

When the training data contains unreliable signals, a model can spend capacity fitting details that have little value in the real world.

3. Training Accuracy Can Be Misleading

If training accuracy does not tell us whether a model generalizes, what should we measure?

The first step is to evaluate the model on data that was not used to fit its parameters.

Machine learning workflows commonly divide data into three sets.

Training set: The model uses this data to learn its parameters.

Validation set: Developers use this data to compare model configurations, tune hyperparameters, select checkpoints, and make other training decisions.

Test set: This data is reserved for a final evaluation of the selected model.

These sets serve different purposes.

If a model achieves 99% accuracy on its training set but only 78% on a representative validation set, the gap is a warning sign. The model performs much better on the examples it learned from than on unseen examples.

This is often called a generalization gap.

But there is an important qualification: a gap alone does not prove overfitting. Training and validation sets may differ in difficulty, class balance, or data distribution. Evaluation metrics may also hide problems affecting particular groups or classes.

The split must be designed carefully.

For example, randomly splitting medical images can produce misleading results if images from the same patient appear in both training and validation sets. The model might benefit from patient-specific characteristics instead of learning patterns that generalize to new patients.

In that situation, splitting by patient is more appropriate.

Similarly, when predicting future events, randomly splitting observations across time can allow information from the future to influence training. A chronological split is often more suitable.

A validation set is useful only when it represents the kind of unseen data the model will encounter.

4. The Hidden Danger: Data Leakage

Sometimes, a model appears to generalize well because the evaluation process accidentally gives it access to information it should not have.

This is known as data leakage.

Consider a model designed to predict whether a customer will cancel a subscription.

The dataset includes customer activity, account details, and a field indicating whether the customer has already completed the cancellation process.

If that field is accidentally included as an input, the model may achieve excellent evaluation results.

But the feature reveals the answer rather than helping the model predict it.

A less obvious example occurs during preprocessing.

Suppose developers normalize a dataset using statistics calculated from the entire dataset before splitting it into training and test sets. Information from the test set has now influenced the preprocessing applied to the training data.

Depending on the transformation and task, this can introduce leakage and make evaluation less reliable.

The safer approach is to split the data first, fit preprocessing steps using only the training set, and then apply the learned transformations to validation and test data.

Leakage can also happen through duplicated records, overlapping users, future information, or features created after the event the model is supposed to predict.

It is important to distinguish leakage from ordinary overfitting. Overfitting occurs when the model fits training-specific patterns too closely. Leakage occurs when information improperly crosses the boundary between training and evaluation, or when an input contains information that would not legitimately be available at prediction time.

The two can coexist, and both can produce models that fail after deployment.

5. How Do We Reduce Overfitting?

There is no single technique that eliminates overfitting in every situation. Effective solutions depend on why the model is overfitting.

Collect More Representative Data

More data can help the model distinguish recurring patterns from random variation.

But the quality and diversity of the data matter as much as its quantity.

Adding thousands of nearly identical examples may provide less value than collecting examples from new environments, devices, users, or conditions that the model must handle.

For the wolf-versus-dog classifier, the dataset should include wolves and dogs in a variety of backgrounds, lighting conditions, poses, and environments.

The objective is to expose the model to the relevant variation instead of allowing a single accidental feature to dominate.

Use Regularization

Regularization introduces constraints or penalties that discourage certain forms of complexity in a model.

For example, L1 and L2 regularization penalize large or numerous parameter values in different ways. These techniques can discourage a model from relying too heavily on particular parameters.

Decision trees can be regularized by limiting their depth or requiring a minimum number of examples in a leaf.

Neural networks can use techniques such as dropout, which randomly disables selected activations during training to discourage excessive dependence on particular units.

These methods work differently, but they share a broad objective: encourage solutions that are less likely to depend on fragile, training-specific details.

Regularization is not a substitute for representative data or proper evaluation. A heavily regularized model can still perform poorly if the data is biased or the task is poorly defined.

Use Data Augmentation

Data augmentation creates modified versions of training examples while attempting to preserve their relevant meaning.

For image classification, this might involve flipping images, cropping them, or changing brightness within reasonable limits.

For speech recognition, it may involve controlled changes to audio. For text, augmentation requires more care because a seemingly minor change can alter the meaning or correctness of a sentence.

The goal is to make the model less dependent on incidental details.

For example, showing an animal classifier the same species against different backgrounds can help discourage it from using background appearance as the primary signal.

Augmentation must be appropriate for the task. A transformation that changes the label or removes important information can make training worse.

Use Early Stopping

Early stopping monitors performance on validation data while training progresses.

If validation performance stops improving, developers can stop training and restore the checkpoint that performed best on the validation set.

The idea is simple: don't continue optimizing the training objective indefinitely if doing so no longer improves performance on unseen data.

In practice, developers often use a patience threshold because validation metrics can fluctuate from one checkpoint to another.

Early stopping does not guarantee the best possible model, but it can be a useful and computationally inexpensive defense against overfitting.

Simplify the Model When Necessary

A model that is too flexible for the available data may benefit from a simpler architecture or stronger constraints.

For a decision tree, reducing its depth may prevent it from creating branches that fit individual training examples.

For other models, developers may reduce the number of parameters, limit feature complexity, or choose an architecture with more appropriate inductive biases.

However, making a model simpler is not automatically better. An overly simple model may fail to capture meaningful patterns, resulting in underfitting.

The objective is not to build the smallest model possible. It is to find a model that captures useful structure without depending excessively on accidental details.

6. The Opposite Problem: Underfitting

Overfitting is only one side of the problem.

A model can also be too simple to learn the underlying patterns in the data. This is called underfitting.

Imagine trying to predict a complex relationship using a model that can represent only a straight-line trend. If the true relationship is much more complicated, the model may perform poorly on both training and validation data.

A useful diagnostic is to examine both training and validation performance.

Low training error and substantially higher validation error: This can indicate overfitting.
High training error and high validation error: This can indicate underfitting, although poor data quality or optimization problems can also cause it.
Low training and validation error: This suggests the model is fitting the training data and generalizing well on the validation set, though further evaluation may still be necessary.

These patterns are diagnostic clues, not absolute rules.

The challenge is to balance model flexibility with generalization. A model must be powerful enough to capture meaningful patterns, but not so dependent on its training examples that it fails when conditions change.

7. Why Passing a Test Set Still Doesn't Guarantee Success

Suppose a model performs well on its training, validation, and test sets.

Is it ready for production?

Possibly, but there is another question: how closely do those datasets resemble the real environment?

A randomly sampled test set from historical data may not represent future users, new devices, changing customer behavior, or evolving fraud techniques.

This problem is especially important when the data distribution changes over time. A model trained on yesterday's patterns may struggle when tomorrow's environment behaves differently.

This is often discussed under the broader concept of distribution shift.

For example, a fraud detection model may learn patterns associated with previously observed fraud campaigns. Once deployed, attackers can change their behavior, making some learned patterns less useful.

Even a model that generalizes well to a held-out test set can become less effective as the world changes.

That is why production systems need more than a one-time accuracy score.

Depending on the application, developers may monitor prediction quality, false positives, false negatives, calibration, data drift, and performance across relevant groups.

They should also establish procedures for investigating degradation and retraining when appropriate.

In high-stakes applications, evaluation must reflect the actual consequences of errors. A medical screening system, for instance, cannot be judged adequately by overall accuracy alone if false negatives and false positives have very different costs.

A model is not finished when training ends. Its reliability must be assessed in the context where it will be used.

The Bigger Picture

Overfitting exposes a fundamental challenge in machine learning.

A model is trained on a finite collection of examples, but it is expected to make predictions about cases it has never seen.

Training accuracy tells us how well the model fits those examples. It does not, by itself, tell us whether the model has learned patterns that will remain useful beyond them.

That is why representative datasets, careful validation, leakage prevention, regularization, and production monitoring matter so much.

A model that achieves 99% accuracy may be excellent. Or it may have learned shortcuts, memorized accidental details, benefited from leakage, or been evaluated on data that does not reflect real-world conditions.

The score alone cannot tell us which.

The real test of a machine learning model isn't how perfectly it remembers its training data. It's how reliably it handles the data it has never seen.

Escape Analysis: How Can a Compiler Eliminate a Heap Allocation Without Changing Your Code?

Aditya Sharma — Wed, 14 Oct 2026 18:08:23 +0000

Consider this Java method:

double distanceBetween(double x1, double y1, double x2, double y2) {
    Point a = new Point(x1, y1);
    Point b = new Point(x2, y2);
    return Math.sqrt(
        Math.pow(b.x - a.x, 2) + Math.pow(b.y - a.y, 2)
    );
}

Two Point objects are created, used in a calculation, and then discarded. The source code calls new, which in Java means heap allocation, object initialization, and eventually garbage collection. But if you look at this from a different angle: both objects are born at the start of the method, used for arithmetic, and are completely useless after the return statement. Nothing outside the method ever sees them. No other thread can observe them. They exist solely to hold two doubles each.

Does the runtime actually need to allocate heap memory for them? That question is what escape analysis is designed to answer.

What It Means for an Object to Escape

Before a compiler can eliminate an allocation, it needs to prove that the allocation can be eliminated. The key concept is whether an object escapes its local scope.

An object escapes when a reference to it can be observed from outside the context in which it was created. There are several ways this happens:

Return. A method that returns a reference to a locally created object immediately causes it to escape. The caller now holds a live reference, so the object must exist somewhere accessible after the method returns. Stack frames are discarded on return, which means the object cannot safely live in one.

Shared state. Storing a reference in a field of an object that outlives the current scope, or in a static field, allows code anywhere in the program to observe the object. The compiler must assume the object could be read at any time.

Opaque callees. When a method passes an object to another function and the compiler cannot fully analyze what that function does with the reference, it must conservatively assume the object might escape. If the callee stores the reference in a field or returns it to its own callers, the assumption would be violated.

Non-escaping objects are those where the compiler can prove that every reference to the object stays within a bounded scope. It is created, used, and the last reference to it disappears, all within a region the compiler can reason about.

The Point objects in the opening example are non-escaping. The references a and b are local variables. They are never returned, never stored in any field, never passed to an opaque method. The compiler can trace every use of them.

How the Compiler Reasons About References

Escape analysis is not a single yes-or-no check. It is a data flow analysis that the compiler runs over the program's intermediate representation, tracking how references to each object propagate through the code.

The classic formulation, and the one underlying HotSpot's C2 compiler, builds a connection graph: a representation of the relationships between object references, the fields through which they flow, and the boundaries across which they might cross. Each object in the graph carries an escape state. HotSpot's C2 uses three states: NoEscape, for objects whose references are entirely contained within the method; ArgEscape, for objects passed as arguments but not observed to escape further; and GlobalEscape, for objects accessible from outside the method or thread.

The analysis is conservative. If the compiler cannot prove that a reference stays local, it assumes it might escape. Conservative analysis is safe: it never eliminates a necessary allocation. But it also means some allocations that a human could see are unnecessary will be retained because the compiler's model of the code is not complete. Inlining is one of the most important enablers of escape analysis. When a callee is inlined into the caller, the compiler gains full visibility into what the callee does with every argument, which allows escape states to be refined. HotSpot's C2 requires inlining to occur before scalar replacement can work. If a method is not inlined because it is too large or was not compiled at the right optimization tier, the analysis may classify its arguments as escaping, even if they do not.

The Optimization: Scalar Replacement

The natural first guess is that escape analysis moves objects from the heap to the stack. For some compilers and runtimes, that is exactly what happens. The Go compiler does this: if it proves that a value will not outlive the current function, it allocates the value in the goroutine's stack frame rather than on the heap. You can observe this directly with go build -gcflags="-m", which annotates each allocation decision.

HotSpot's C2 JIT takes a different path. According to the OpenJDK wiki: "C2 does NOT replace a heap allocation with a stack allocation for non globally escaping objects." Instead, C2 performs scalar replacement.

Scalar replacement decomposes an object into its constituent fields and eliminates the object itself. Rather than allocating a block of memory and storing field values in it, the compiler treats each field as an independent local variable. The object never materializes as a contiguous region of memory. No allocation happens. No garbage collector ever needs to reclaim it.

The transformation for the Point example looks conceptually like this. Before scalar replacement:

// What you wrote
Point a = new Point(x1, y1);
double ax = a.x;
double ay = a.y;

After scalar replacement, the compiler replaces all field accesses with synthetic local values:

// What the compiler works with internally
double a_x = x1;  // scalar substitute for a.x
double a_y = y1;  // scalar substitute for a.y

The Point object never exists. The field reads (a.x, a.y) become references to local values that can be kept in registers. Aleksey Shipilëv's JVM Anatomy Quarks describes this precisely: HotSpot "does not do stack allocations per se, but it does approximate that with Scalar Replacement."

From there, register allocation can place those scalar values in CPU registers for the duration of the computation. The two doubles that would have been heap-allocated objects end up as register values, with no allocation, no initialization overhead, and no GC involvement.

A Concrete Walk-Through

Take the full distance method from the opening:

double distanceBetween(double x1, double y1, double x2, double y2) {
    Point a = new Point(x1, y1);
    Point b = new Point(x2, y2);
    return Math.sqrt(
        Math.pow(b.x - a.x, 2) + Math.pow(b.y - a.y, 2)
    );
}

For scalar replacement to apply, C2 needs to:

Inline the Point constructor so it can see that no reference to the new object leaks out of the construction.
Determine that a and b have escape state NoEscape: no return of the references, no field stores into external objects, no passing to opaque methods.
Replace new Point(x1, y1) with two scalar variables a_x = x1, a_y = y1.
Rewrite every a.x and a.y access to use those scalars.
Eliminate the allocation instruction entirely. After this transformation, the method compiles to arithmetic on four double values, none of which requires heap memory. The behavior observed by any caller is identical: they receive the same double result. The only change is internal to the representation of the computation.

Lock elimination is a related optimization that escape analysis enables. If an object is non-escaping, no other thread can hold a reference to it, so any synchronized blocks on that object are unnecessary. The compiler can remove them.

Where the Analysis Fails

Escape analysis has hard limits, and understanding them is as important as understanding what it can do.

The most common barrier is an opaque method call. If distanceBetween passed a to any method that C2 has not inlined, the compiler would have to assume the reference might escape. The Point would remain heap-allocated. This is why HotSpot's OpenJDK scalar replacement documentation explicitly lists non-inlined methods as a case where scalar replacement is prevented.

Interface dispatch is a related problem. When a method is called through an interface, the compiler cannot always determine which concrete implementation will run, which makes it impossible to inline the callee. Without inlining, the escape state of the argument is unknown.

Some operations are identity-sensitive. Java's == operator on objects tests reference identity. If the compiler replaces an object with scalars, two allocations that previously produced distinct references might appear identical or might behave incorrectly if the program depends on them being different objects. The compiler must verify that no code in scope depends on the object's identity before it can safely decompose it.

Synchronization presents a harder constraint. If code synchronizes on an object that the compiler is considering for scalar replacement, the semantics of Java's memory model require that the lock be observable. Lock elimination is sound only when the object is genuinely non-escaping, so the compiler must be certain before removing it.

Conservative analysis also means that HotSpot will retain an allocation if any uncertainty exists. The analysis can handle many cases that appear complex in source code, but it cannot analyze beyond what it can observe. Dynamic loading of classes, reflection, and native code are all effectively opaque.

The Performance Picture

Removing allocations reduces pressure on the garbage collector by producing fewer objects that need to be tracked and collected. In throughput-sensitive code that creates many short-lived objects in tight loops, this can meaningfully reduce GC pause frequency and time spent in allocation paths. Keeping data in registers rather than memory also reduces memory traffic.

That said, allocation in modern JVMs is fast. HotSpot's thread-local allocation buffers (TLABs) allow most allocations to complete with a simple pointer bump. The cost of a single allocation is small. The benefit of elimination comes from the aggregate effect over many allocations, particularly in hot loops, and from the downstream reduction in GC work.

Whether scalar replacement improves a specific workload depends on the compiler's ability to apply it, the allocation rate of the relevant code, and the GC configuration. Profiling with allocation-aware tools is the right way to measure this, not inference from source code alone.

The Developer's Relationship With This Optimization

The existence of escape analysis and scalar replacement changes what source-level object creation actually means. Writing new Point(x, y) in Java does not guarantee that a heap allocation will occur at runtime. Whether one does depends on the JIT compiler's escape analysis, whether the relevant methods were inlined, and the escape state the analysis assigned to the object.

This is intentional. The language specification defines what the program must compute and what observable side effects it must produce. It says nothing about whether intermediate objects need physical representation. The JIT compiler is free to choose any implementation that preserves observable behavior.

Several practical points follow from this:

Write code that is clear and correct. Avoiding allocation for its own sake by using awkward data structures or manual field decomposition will not necessarily produce faster code and will make the compiler's job harder if it disrupts inlining.

Measure before concluding. Allocation sampling profilers can show which allocations actually reach the heap at runtime. An allocation that appears in source code may not appear in profiles if scalar replacement has already eliminated it.

Do not assume Java and Go behave identically. Go's escape analysis operates at compile time and makes stack-allocation decisions that are baked into the generated machine code. HotSpot's analysis operates in the JIT compiler and uses scalar replacement rather than stack allocation. The mechanisms, the triggering conditions, and the diagnostics are different.

Check compiler output when it matters. Go's -gcflags="-m" flag reports escape decisions. HotSpot's -XX:+PrintEliminateAllocations flag reports scalar replacement. These are the ground truth; source code is not.

The Deeper Insight

Escape analysis reflects a broader principle in compiler design: a language's specification describes the required observable behavior of a program, not its required internal representation. The compiler is free to transform how data is stored, how operations are sequenced, and how objects are physically represented, as long as anything the program is permitted to observe remains consistent with the specification.

A heap allocation is a mechanism for giving data a lifetime that outlives a function call. When the compiler can prove that a particular datum does not need that extended lifetime, the allocation is unnecessary. The datum can be represented as local values, stored in registers, and discarded the moment the function returns. The program computes the same result. The method's caller observes the same returned value. Nothing that any correct program could observe has changed.

What the compiler has changed is what never needed to exist in the first place.

Sources

OpenJDK HotSpot Escape Analysis Wiki — authoritative description of C2's escape analysis algorithm and its explicit statement that C2 does not perform stack allocation.
Aleksey Shipilëv: JVM Anatomy Quarks #18 — Scalar Replacement — detailed technical explanation of scalar replacement in HotSpot C2.
HotSpot Escape Analysis and Scalar Replacement Status (OpenJDK) — analysis of escape analysis coverage and limitations in C2.
Oracle Java Magazine: Escape Analysis in the HotSpot JIT Compiler — accessible explanation with GC log evidence.
Segment Engineering: Allocation Efficiency in High-Performance Go Services — practical explanation of Go's stack-allocation-first approach and escape analysis flags.
Go compiler escape analysis flags — go build -gcflags="-m" documentation.

NTLM Relay Attacks: How Can Attackers Authenticate as You Without Knowing Your Password?

Aditya Sharma — Tue, 13 Oct 2026 18:01:27 +0000

A password can remain secret, the authentication exchange can be cryptographically valid, and an attacker can still authenticate as you.

No password cracking. No stolen plaintext credentials. Sometimes, not even a vulnerability in the target application.

The attacker simply forwards an authentication exchange between two systems and lets the legitimate user do the cryptographic work for them.

This is called an NTLM relay attack, and it exposes a subtle weakness in how authentication can be used across network connections. The problem isn't necessarily that NTLM fails to verify the credentials. It's that a valid authentication exchange can sometimes be redirected to a service the user never intended to authenticate to.

To understand how that happens, we need to look at what NTLM actually proves, what it leaves unverified, and how an attacker can exploit that gap.

NTLM Doesn't Send Your Password

NTLM is a Windows authentication protocol that uses a challenge-response mechanism. Instead of sending a user's plaintext password across the network, it allows a server to verify a response calculated using password-derived secret material.

A simplified authentication exchange works like this:

Negotiation: The client and server establish the parameters for authentication.
Challenge: The server sends the client a challenge value.
Response: The client calculates a response using the challenge and its NTLM credentials.
Verification: The server, or an appropriate authentication authority, validates the response.
Authentication: If verification succeeds, the client can access resources according to the account's permissions.

The important detail is that the response depends on the server's challenge. An attacker who captures a response from an earlier exchange generally cannot simply replay it against a new challenge and expect it to work.

That sounds reasonably secure, and it is an important property of challenge-response authentication.

But there's another question the protocol must answer: is the client authenticating to the service it actually intended to reach?

A valid response can prove possession of the necessary credential material without necessarily proving that the authentication is bound to the intended server or communication channel.

That distinction is where NTLM relay attacks begin.

The Attack: Let Someone Else Authenticate for You

Imagine three systems on a corporate network:

The victim: A Windows computer logged in as a domain user.
The attacker: A system capable of receiving or inducing the victim's NTLM authentication.
The target: A server that accepts NTLM authentication and grants the victim certain permissions.

The attacker wants to access the target as the victim. They don't know the password, but they may not need it.

Here's the basic sequence:

Victim                  Attacker                  Target
  |                        |                        |
  |--- Authentication ---->|                        |
  |                        |--- Authentication ---->|
  |                        |<------ Challenge -------|
  |<------- Challenge -----|                        |
  |                        |                        |
  |-------- Response ----->|                        |
  |                        |---- Response --------->|
  |                        |                        |
  |                        |<-- Authentication OK ---|
  |                        |                        |
  |                        |--- Access as victim --->|

This diagram is simplified, but it captures the essential idea: the attacker forwards messages between the victim and the target while the victim generates the cryptographic response.

Let's walk through what happens.

First, the attacker gets an authentication attempt.

The victim must be induced to authenticate to a service controlled by the attacker, or the attacker must otherwise be able to obtain the victim's authentication exchange. The method depends on the environment and may involve coercing a machine to connect to another system.

Next, the attacker forwards the exchange to the target.

Rather than independently generating a valid NTLM response, the attacker relays the victim's authentication messages to the target. The target provides a challenge, which is forwarded to the victim.

The victim generates the response.

Because the victim has the necessary credentials, their system calculates the appropriate response to the challenge. The attacker forwards that response to the target.

The target validates the response.

If the target accepts the exchange and the relevant relay protections are absent, authentication can succeed as the victim.

The attacker has not necessarily learned the password or cracked a hash. They have forwarded a live authentication exchange to a service that accepts it.

The resulting access depends on the target, the account's permissions, and the protections enforced by the protocol and service.

Why Does the Relay Work?

The interesting part is that the attacker doesn't have to break the cryptography.

The weakness lies in the context in which authentication is accepted.

NTLM's challenge-response mechanism helps prevent straightforward replay of a captured response against a different challenge. However, NTLM authentication by itself does not universally bind the exchange to the specific server or protected communication channel the client intended to use.

If the attacker can relay the exchange while the target accepts it without the necessary protections, the target may have no sufficient way to distinguish the relayed authentication from an acceptable authentication attempt.

Think of it as a distinction between proving your identity and proving who you're communicating with.

A system may verify that the authentication response is valid but fail to verify that the exchange belongs to the intended connection.

This is why NTLM relay is not the same as password cracking or pass-the-hash. Password cracking attempts to recover a password. Pass-the-hash abuses password-hash material to authenticate. Relay attacks forward authentication messages between systems, potentially without recovering the password or obtaining the hash.

The technique is also conditional. Not every NTLM exchange can be relayed successfully. The target protocol, account privileges, service configuration, and protections such as signing and Extended Protection all affect the outcome.

When a Relay Becomes a Bigger Problem

NTLM relay becomes particularly serious when the target can grant access to sensitive resources or perform privileged operations on behalf of the authenticated user.

One example is Active Directory Certificate Services (AD CS), Microsoft's infrastructure for issuing and managing certificates in Windows environments.

Certificates can be used for authentication, which makes certificate enrollment services a potentially valuable target. Under certain configurations, an attacker who relays a privileged account's NTLM authentication to an inadequately protected enrollment endpoint may be able to obtain a certificate that enables further impersonation.

Microsoft addressed this risk in its guidance on mitigating NTLM relay attacks against AD CS.

The key word is configuration. AD CS is not automatically vulnerable simply because it is deployed. The risk depends on how the enrollment services are exposed and whether the appropriate authentication protections are enabled.

This example illustrates how relay attacks can become part of a larger intrusion. A successful relay may provide the foothold needed to access resources, request credentials or certificates, or move further through a network, depending on the target's permissions and configuration.

The attack doesn't magically grant domain administrator privileges in every environment. It turns an accepted authentication exchange into access that the attacker may otherwise have been unable to obtain.

Why Changing the Password Isn't Enough

If an attacker steals a password, changing it is an obvious response. If a password hash is compromised, credential rotation is also important.

But NTLM relay exploits a different problem.

The attacker can forward a live authentication exchange without learning the password. Changing the password alone does not fix a target service that accepts relayed authentication without adequate protections.

The solution has to address the authentication flow itself.

SMB Signing

Server Message Block (SMB) signing adds integrity protection to SMB messages. When signing is required and correctly enforced, it can prevent common SMB relay attacks because the attacker cannot freely create or modify signed messages without the necessary session key.

This is particularly relevant to SMB relay. It is not a universal defense against relay attacks targeting other protocols.

Extended Protection for Authentication

Extended Protection for Authentication (EPA) helps bind authentication to the intended service or communication channel, making it harder to reuse authentication from one connection in another context.

Two related concepts are important:

Channel binding: Associates authentication with the underlying TLS channel.
Service binding: Helps ensure that authentication is associated with the intended service identity.

The exact protections available depend on the protocol, application, and configuration. Simply enabling TLS does not guarantee protection against NTLM relay if authentication is not properly bound to the connection.

Reduce NTLM Usage

Where operationally feasible, organizations should reduce NTLM authentication and move to stronger alternatives such as Kerberos in supported environments.

Legacy applications, workgroup configurations, and other dependencies may make immediate removal difficult. Administrators should audit NTLM usage before restricting it and address systems that still depend on the protocol.

Harden Services and Limit Privileges

High-value services, including file servers and certificate enrollment endpoints, should enforce the protections supported by their protocols. Unnecessary exposure should be removed, and authentication configurations should be reviewed rather than assumed secure by default.

Least privilege matters, too. A relayed authentication exchange can only grant the attacker the access the target permits for that identity. Limiting administrative privileges and restricting where privileged accounts authenticate can reduce the impact of a successful relay.

No single mitigation prevents every possible relay scenario. Effective defense combines protocol-level protections, hardened services, reduced NTLM usage, and carefully controlled permissions.

The Real Lesson Behind NTLM Relay

NTLM relay attacks expose a subtle distinction in security engineering: verifying an authentication response is not always the same as verifying the context in which that response is being used.

The protocol can validate the response without the attacker ever learning the password. If a target accepts that authentication without adequate channel or service protections, a legitimate exchange can become a path to unauthorized access.

That's why protecting credentials is only part of authentication security. Systems must also ensure that authentication is being used with the intended service and under the right conditions.

Sometimes, an attacker doesn't need to steal your password. They just need to redirect the proof that you know it.

Sources

What Is BYOVD? How Can a Signed Driver Give Malware Kernel-Level Access?

Aditya Sharma — Mon, 12 Oct 2026 04:19:18 +0000

In July 2022, a ransomware actor walked into a corporate network, found a system with endpoint protection properly configured, and disabled it. Not by exploiting a zero-day in the antivirus software. Not by finding a misconfiguration in the firewall. They used a driver signed by miHoYo, the developer of Genshin Impact, one of the world's most popular video games.

The driver was legitimate. Its certificate was valid. A valid signature is the entry requirement for the kernel, though other security policies can still block a driver from loading. This one passed. And it could terminate any process on the machine, including every security tool running on it, because it executed in the kernel.

This is the central paradox of Bring Your Own Vulnerable Driver, or BYOVD: a driver can be legitimately signed, accepted by Windows, and still become a weapon. Understanding why requires understanding what a driver signature actually proves, and what it deliberately does not.

The Kernel Is Not Just Another Layer

When Windows runs, it splits execution into two distinct privilege levels: user mode and kernel mode. User-mode code runs in a sandboxed environment. A browser, a text editor, an antivirus agent: they all operate in user mode, separated from each other and from the OS itself. If a user-mode process crashes, Windows can contain the damage. It cannot reach across and terminate another protected process, modify arbitrary memory, or directly touch hardware.

Kernel mode is different. Code running in kernel mode executes with the full privileges of the operating system. It can read and write any memory address, terminate any process, and access hardware directly. There is no sandbox. A crash in kernel mode is a system crash. A successful attacker in kernel mode has, in practice, won.

Drivers live in kernel mode. A graphics driver, a storage controller driver, a network interface driver: they all run with those elevated privileges because that is what their job requires. Controlling hardware at a low level demands direct access to memory, I/O ports, and system structures that user mode simply cannot touch.

This creates an obvious security concern. If arbitrary code can run in kernel mode, any attacker who gets their code loaded as a driver owns the machine. Microsoft's answer to this was Kernel Mode Code Signing, which became mandatory on 64-bit Windows starting with Vista: to load into the kernel, a driver binary must carry a valid digital signature from a trusted certificate authority.

The assumption embedded in that policy is reasonable. Getting a certificate requires a real identity. Signing a driver is a deliberate act that creates accountability. Unsigned code stays out.

The assumption is also incomplete.

What a Signature Proves, and What It Does Not

A digital signature on a kernel driver answers a specific question: has this binary been modified since it was signed, and does the certificate chain trace back to a trusted authority? It is an integrity and identity check. It says nothing about whether the code's implementation is secure, whether its privileged operations are safe to expose, or whether an attacker could send it the right inputs to make it do something it was never intended to do.

Before a driver can run on modern Windows, it generally must pass through the Windows Hardware Quality Labs process, where it is signed by Microsoft after testing. But WHQL testing focuses on stability and compatibility. It does not perform the kind of adversarial security review needed to guarantee that every function the driver exposes is safe against malicious inputs. A driver can pass certification, ship with legitimate hardware or software, and still contain vulnerabilities that allow attackers to abuse its kernel-level access.

There are also two distinct ways a driver can become a BYOVD target. The first is a genuine vulnerability: a memory corruption bug, a logic error, a path that allows arbitrary code execution or memory manipulation. The second, and arguably more common in recent incidents, is dangerously over-exposed functionality: the driver intentionally provides operations like process termination or arbitrary memory reads, but with insufficient access controls, so any process that can reach its device interface can issue those commands. Both categories end up in the same place: a signed, trusted driver doing things on behalf of an attacker.

The Attack, Step by Step

BYOVD requires an attacker who has already obtained administrative privileges on the target system. That is the prerequisite that matters most. This is not a technique for gaining initial access; it is a technique for expanding what you can do once you are in, particularly for defeating security tools that are deliberately hardened against user-mode tampering.

Once the attacker has admin access, the attack follows a consistent pattern.

The vulnerable driver, a .sys file, is dropped onto disk. It does not need to be installed; it just needs to be present. The attacker then registers it as a kernel service, typically through the Windows Service Control Manager, and starts it. Because the driver carries a valid signature and the signing certificate is trusted, Windows loads it into the kernel without raising an alarm based on signature alone.

At this point, the driver is resident in kernel mode, and the attacker's user-mode code can communicate with it through the driver's device interface using I/O control codes, or IOCTLs. These are essentially function calls into the driver from user mode. A legitimate driver exposes IOCTLs for its intended purpose: a hardware driver might expose IOCTLs for device configuration or status queries. A vulnerable or over-exposed driver may also happen to expose IOCTLs that can terminate processes, read or write kernel memory, or perform other privileged operations.

The attacker sends the right IOCTL to the loaded driver. The driver, executing in kernel mode, carries out the request. If the target is process termination, the driver calls kernel APIs like ZwTerminateProcess in its own privileged context, bypassing the process protection mechanisms that would prevent a user-mode process from doing the same thing directly. Protected processes, including security agents specifically designed to resist user-mode tampering, can be killed this way.

What the attacker gains from this depends entirely on what the driver can do. BYOVD does not automatically grant unlimited kernel-level control. The attack is bounded by the driver's own capabilities and what it exposes. But terminating endpoint detection and response processes is typically sufficient for the next phase of an intrusion, whether that is deploying ransomware, exfiltrating data, or establishing persistence.

Case Study: The Genshin Impact Driver

In August 2022, Trend Micro published a detailed incident report documenting a ransomware campaign that used exactly this technique. The driver was mhyprot2.sys, the anti-cheat component shipped with Genshin Impact and signed by miHoYo.

Anti-cheat systems are a particularly fertile source of BYOVD candidates. To do their job, they need kernel-level access to inspect game memory, detect unauthorized modifications, and terminate processes. The functionality they need is precisely the functionality attackers want.

The mhyprot2.sys driver exposed IOCTLs that allowed a user-mode process to enumerate modules loaded into any given process and to terminate any process by PID using ZwTerminateProcess, the kernel's own process termination interface. The driver performed these operations in kernel mode, on behalf of whoever sent the right command.

Genshin Impact does not need to be installed on a victim's machine for this to work. The driver runs independently of the game. In the Trend Micro incident, the attacker dropped two files onto a compromised user's desktop: mhyprot2.sys and a malicious executable called kill_svc.exe. The executable installed the driver, then issued IOCTLs through its device interface to work through a list of processes to terminate. That list included the antivirus services running on the machine. With endpoint protection down, the attacker deployed ransomware through a shared network folder.

The driver in this attack was the version built in August 2020. At the time of Trend Micro's report, the code signing certificate was still valid. The developer had been informed of the vulnerability before the attack, but the certificate had not been revoked.

That last point is important. The response to a vulnerable driver is not just to patch it. A newer, patched version of the driver does not invalidate the old signed binary. As long as the original certificate remains valid and the old binary is not explicitly blocked, attackers can continue using the older version indefinitely.

Why Signing and Revocation Cannot Solve This Cleanly

The logical response to a vulnerable signed driver is to revoke its certificate. This would prevent Windows from loading the signed binary. In practice, it is often not done promptly, and sometimes not done at all.

Certificate revocation is blunt. Revoking a certificate affects every binary signed with it, including legitimate software currently in use. A hardware vendor with thousands of deployed devices, all using drivers signed with the same certificate, faces a significant compatibility problem if revocation breaks those drivers for their customers. The commercial and support costs are real, which creates reluctance to act quickly.

Microsoft's answer to the revocation problem is the vulnerable driver blocklist: a list of specific driver binaries, identified by hash, file name, version number, and signing certificate, that Windows should refuse to load. The blocklist allows a much more surgical response than certificate revocation. Since Windows 11 22H2, the blocklist is enabled by default on all devices and can be toggled via Windows Security. It is also enforced independently when HVCI (Hypervisor-Protected Code Integrity), Smart App Control, or S mode is active on a device. Prior to 22H2, the blocklist was only active on systems running HVCI or S mode. On Windows 10 systems and older Windows 11 configurations without these features enabled, the blocklist is not guaranteed to be in effect.

The blocklist has a structural limitation that BYOVD exploits by design: it is reactive. A driver cannot be added to the blocklist until a vulnerability is known, reported, confirmed, and processed. During that window, the vulnerable driver remains usable. In late 2022, security researcher Will Dormann documented that the blocklist had not received significant updates in nearly three years on some configurations, leaving many known-vulnerable drivers usable well after their vulnerabilities had become public. Microsoft has since committed to updating the blocklist with each major Windows release and through servicing updates, but the fundamental reactive nature of the mechanism remains.

This is also why attackers choose BYOVD over writing a kernel exploit from scratch. A novel kernel vulnerability is expensive to develop, risky to use (it may destabilize the system), and will be patched when it becomes known. A collection of signed, legitimate drivers with documented kernel capabilities and no current blocklist entry is a catalog of ready-made tools that an attacker can bring to any adequately privileged intrusion.

Defenses That Actually Help

Keeping HVCI enabled is the most structurally important control. HVCI uses the CPU's virtualization capabilities to isolate code-integrity enforcement into a protected environment, separate from the kernel itself. Even if the kernel is compromised, the hypervisor layer that HVCI relies on can continue to enforce that only verified, known-good code runs in kernel mode. A kernel driver cannot tamper with that enforcement because it operates at a privilege level below where HVCI's protections live.

HVCI and the vulnerable driver blocklist are distinct controls that work well together. The blocklist specifies which drivers should be refused; HVCI makes it harder for a compromised kernel component to circumvent whatever policy is in place. Without HVCI, a sophisticated attacker who has already loaded a driver may be able to interfere with kernel-level enforcement mechanisms. With HVCI active, the integrity verification itself is harder to undermine.

For enterprises, Windows Defender Application Control allows organizations to deploy custom driver allowlists and blocklists. The ASR rule "Block abuse of exploited vulnerable signed drivers" can add a layer of detection, though Microsoft's documentation notes it does not prevent a driver already on the system from loading.

Detection is harder than prevention but still meaningful. Legitimate drivers are generally not dropped into temporary user directories and registered as services from user desktop folders. A driver installed via the Service Control Manager from an unusual path warrants investigation. Monitoring for unusual IOCTL patterns, unexpected process terminations of security agents, and driver loads from non-standard locations gives defenders something to work with even when prevention has failed.

Endpoint protection products with kernel-level self-protection (tamper protection), when properly configured, can raise the bar for what a BYOVD attack needs to accomplish. Some products monitor for attempts to kill their own processes through kernel-level calls and can alert or resist those attempts. This does not make them immune, but it creates observable signal.

The least complicated defense is also the most consistently undervalued: limiting administrative privileges. A BYOVD attack begins after the attacker has administrative access. Restricting who has local administrator rights, enforcing just-in-time access for privileged operations, and monitoring for unexpected privilege escalation reduces the pool of machines on which BYOVD becomes an option.

Trusting Origin vs. Trusting Behavior

The core problem with driver signing as a security boundary is that it conflates two different properties: where the software came from and what the software does. A signature is evidence of origin and integrity. It cannot be evidence of safe behavior, because safe behavior depends on implementation details that signing does not and cannot inspect.

BYOVD is the gap between those two properties made into an attack technique. A driver signed by a legitimate manufacturer, shipped with a popular video game, passing every certificate check Windows performs, can still call kernel APIs that terminate protected processes and disable the security tools meant to detect an intrusion.

The defensive implication is that a policy of "only allow signed kernel code" is necessary but not sufficient. The question has to extend further: which signed code, and does it remain safe given what we know about its implementation today? That is a harder question to answer, and the blocklist, revocation infrastructure, and HVCI together represent the current best approximation of an answer.

The driver was trusted. The certificate was valid. The security tools were running. And the ransomware still deployed.

Primary Sources

Trend Micro Research: Ransomware Actor Abuses Genshin Impact Anti-Cheat Driver to Kill Antivirus (August 2022)
Microsoft Learn: Microsoft recommended driver block rules
Microsoft Learn: Windows driver signing tutorial
Microsoft Support: Vulnerable driver blocklist KB5020779
CrowdStrike: Falcon Prevents Multiple Vulnerable Driver Attacks in Real-World Intrusion (December 2024)
MITRE ATT&CK: T1068 - Exploitation for Privilege Escalation / Bring Your Own Vulnerable Driver
Expel: The Feature That Stops BYOVD

What Is a Bootkit? How Can Malware Run Before Your Operating System Even Starts?

Aditya Sharma — Sun, 11 Oct 2026 10:13:13 +0000

Reinstalling Windows does not always remove a sufficiently sophisticated attacker from a compromised machine.

That statement deserves a second read. We tend to think of reinstalling an operating system as wiping the slate clean. But the operating system is not the first thing that runs when a computer powers on. Before Windows loads, before any security software initializes, before even the user's login screen appears, a chain of software components executes. If an attacker can implant code into that earlier chain, reinstalling the OS above it may change very little. A typical Windows reinstall replaces the OS files but may leave the EFI System Partition and its boot components untouched. Restoring a truly clean boot chain requires ensuring those earlier components are also known-good, not just the OS above them.

Malware that achieves this is called a bootkit. Its defining characteristic is not sophistication or stealth alone, but position: it executes before the software that is supposed to protect the system has any chance to run.

Before the Operating System: What Actually Happens

To understand why this matters, the boot process needs to be concrete rather than abstract.

When a modern computer powers on, the first software to execute lives in the firmware, a low-level program stored in a chip on the motherboard. On systems built in the past decade or so, this firmware implements the Unified Extensible Firmware Interface, or UEFI. UEFI initializes hardware, performs basic hardware checks, then looks for a bootable storage device.

What it finds there is not Windows directly. UEFI looks for a special disk partition called the EFI System Partition, a small FAT32-formatted area that contains executable files for boot management. On a Windows system, this partition holds the Windows Boot Manager, a signed EFI binary typically at \EFI\Microsoft\Boot\bootmgfw.efi. UEFI hands control to the Boot Manager.

The Boot Manager then finds and launches the Windows OS loader (winload.efi), which loads the Windows kernel, its early drivers, and the initial components of the operating system environment. Only after all of that is in place does Windows begin enforcing its own security controls, loading antivirus software, initializing user sessions, and doing the work we associate with a running operating system.

The sequence matters:

UEFI firmware
    ↓
Windows Boot Manager (EFI System Partition)
    ↓
Windows OS Loader
    ↓
Windows Kernel
    ↓
Operating system security controls

Each component executes in this order, and each stage depends on the integrity of what came before.

Execution Order as a Security Boundary

This sequence establishes a security hierarchy. Any component that runs earlier in the chain has the opportunity to shape the environment for everything that follows. That is not a flaw in the design; it is inherent to how a staged boot process works.

The critical implication: if an attacker can insert code into an early stage of this chain, that code executes before the operating system's security controls exist. It runs with privileges that no user-level malware possesses. It can modify memory, intercept system calls, alter the kernel's view of the filesystem, or disable security components before those components have initialized. By the time Windows is fully running and security software starts scanning for threats, the bootkit has already established itself.

This is qualitatively different from a rootkit that hides in the operating system itself. A bootkit does not need to defeat Windows security; it operates before Windows security is even a concept.

It is also distinct from threats that implant directly into the motherboard's UEFI firmware chip. A firmware implant modifies the UEFI program itself, surviving even a complete disk replacement. A bootkit, in the sense discussed here, operates at the level of the EFI System Partition and boot components on disk, which is still deeply persistent but is not the same category of threat. The distinction matters for understanding what remediation looks like.

How a Bootkit Gets Installed

A bootkit does not arrive on a system through magic. Installing one requires an existing foothold and typically elevated privileges. The attacker needs administrative access to write to the EFI System Partition, modify boot configuration, or install a machine owner key into firmware variables.

The attack lifecycle generally looks like this: the attacker gains initial access through some conventional means, a phishing email, an exploited vulnerability, or compromised credentials. From that position, they escalate privileges if necessary, then execute a bootkit installer. The installer places malicious components on the EFI System Partition and modifies boot configuration so that the malicious component executes during startup.

On subsequent reboots, the bootkit executes before the operating system. It may load a kernel driver that tampers with the operating system as it initializes, intercept system functions, hide files or registry keys associated with the attacker's infrastructure, or disable security components entirely. The operating system that eventually loads appears to function normally, but it is operating under conditions the attacker has already arranged.

Persistence is the key advantage. Conventional malware can often be removed by cleaning infected files, scanning and disinfecting the system, or reinstalling the OS. A bootkit that has embedded itself in the boot chain survives all of those steps, provided the EFI System Partition and its components are not also cleaned and restored. Even a fresh OS installation is loaded by the same compromised boot components if the partition they live on is left intact.

Secure Boot: What It Is and What It Guarantees

Secure Boot was designed specifically to address this category of threat. Understanding it requires distinguishing it from related but distinct technologies.

Secure Boot is a mechanism in UEFI firmware that verifies the cryptographic signature of each component before executing it. When a system with Secure Boot enabled starts, the firmware checks whether the Windows Boot Manager's signature validates against a database of trusted keys stored in the firmware. If the signature is invalid or the component is not trusted, the firmware refuses to execute it.

This extends through the chain: the Boot Manager verifies the OS loader, and the OS loader verifies kernel components. The idea is that at each step, trust is explicitly established through cryptographic signatures before execution proceeds.

The Trusted Platform Module, or TPM, is a separate but related component. A TPM is a dedicated security chip that can store cryptographic keys and support platform integrity measurements. Measured Boot is a feature that uses the TPM to record cryptographic hashes of each boot component as it executes. Windows itself, working with the TPM, logs these measurements to a record called the Boot Configuration Log. Those measurements can be checked later to detect whether any boot component differed from what was expected. Measured Boot does not necessarily prevent a compromised component from running, but it creates a record that can be audited.

Neither Secure Boot nor TPM-based Measured Boot is the same as Windows Defender or conventional endpoint protection software. Antivirus tools operate within the running operating system. Before the kernel initializes, they have no visibility into what has occurred in earlier boot stages, which means a bootkit that acts before that point can potentially disable or evade them before they load. That said, endpoint protection can still observe artifacts after the OS is running: changes to EFI System Partition files, unexpected kernel drivers, registry modifications, and event log entries that reflect what the bootkit did during setup or installation.

Secure Boot's protection rests on a fundamental premise: only code signed by trusted keys should run during boot. The question of who holds those trusted keys, and how the trust decisions are made, turns out to be where the complexity lives.

BlackLotus: When Secure Boot Was Not Enough

In March 2023, researchers at ESET published an analysis of a bootkit they named BlackLotus. It was, as the researchers documented, the first publicly confirmed in-the-wild bootkit capable of bypassing UEFI Secure Boot on fully patched Windows 11 systems.

The critical word is "bypassing," but the mechanism requires precise explanation.

BlackLotus did not break the cryptography of Secure Boot. It did not directly modify UEFI firmware. Instead, it exploited a vulnerability in legitimate, Microsoft-signed Windows boot components to subvert the Secure Boot policy from within the trusted environment.

The specific vulnerability is CVE-2022-21894, known as "Baton Drop." Microsoft issued a patch for this vulnerability in January 2022. The patch addressed the flaw in future versions of the Windows Boot Manager. But Secure Boot's trust model contains a structural challenge: the list of revoked, no-longer-trusted components, called the Secure Boot DBX database, must also be updated to exclude old vulnerable versions. In the period when BlackLotus was operating in the wild, those older signed Windows Boot Manager binaries had not yet been added to the revocation list.

This created an opening. BlackLotus's installer brought its own copies of the older, vulnerable-but-validly-signed boot manager binaries onto the target system. Because Secure Boot still trusted those older signatures, they were permitted to execute. The vulnerable code could then be exploited to truncate the Secure Boot policy, effectively stripping its enforcement. With Secure Boot policy neutralized, BlackLotus enrolled its own Machine Owner Key into the firmware's key database, allowing its self-signed malicious components to be trusted on subsequent reboots.

In May 2023, Microsoft released CVE-2023-24932, a separate Secure Boot bypass vulnerability that addresses the attack path BlackLotus used to circumvent the earlier CVE-2022-21894 patch. The fix updates the Windows Boot Manager and adds the vulnerable binaries to the DBX revocation list. Importantly, the initial update shipped with protections disabled by default, because enabling it requires irreversible changes to the boot environment that can render older boot media, recovery drives, and installation media unbootable. Microsoft deployed the full mitigations in a staged rollout across several phases. Simply installing Windows updates was not sufficient to close the attack path; administrators had to follow explicit configuration guidance to enable and complete the protections.

The attack chain, documented by ESET and independently confirmed by Eclypsium, required:

Running an installer with administrative privileges on the target system.
The installer placing malicious files on the EFI System Partition and placing old vulnerable Boot Manager binaries alongside them.
On first reboot: exploiting CVE-2022-21894, enrolling the attacker's key, then rebooting again.
On all subsequent reboots: the self-signed bootkit executing early in the boot process, loading a kernel driver and a user-mode HTTP downloader for command-and-control communication. The bootkit was capable of disabling BitLocker, Hypervisor-Protected Code Integrity (HVCI), and Windows Defender. The NSA and CISA released a mitigation guide specifically addressing the threat.

This is not a claim that every Windows computer was or is vulnerable to BlackLotus. The attack required administrative access to install. The specific bypass depended on the revocation state of firmware databases, which has since been addressed through security updates and the staged CVE-2023-24932 remediation process. But the case is instructive because it demonstrated a specific and documented mechanism by which Secure Boot could be undermined using trusted, signed components rather than by breaking cryptography.

Remediation was also more complex than removing malware files. Simply deleting the bootkit's components from the EFI System Partition would not restore a clean boot chain. Defenders needed to verify the integrity of boot components, apply the CVE-2023-24932 mitigations including DBX revocation updates to block the vulnerable signed binaries, and in some cases rebuild the system from trusted media to establish a known-good starting state.

Why Signed Does Not Mean Safe

BlackLotus illustrates a principle that goes beyond that specific bootkit: the integrity of Secure Boot depends not just on whether components are signed, but on whether the list of trusted and revoked components is accurate and up to date.

Secure Boot's DBX revocation database is the mechanism for excluding previously trusted but now-dangerous components. If revocation is not applied promptly, validly signed but vulnerable binaries remain usable as attack tools even after patches are issued for the underlying vulnerability. This creates a gap between "software vulnerability patched" and "attack vector actually closed" that can persist for as long as revocation updates are delayed or not fully enabled.

It is also worth distinguishing between a bootkit that operates on the EFI System Partition and a threat that modifies the UEFI firmware chip itself. LoJax, documented by ESET in 2018 and attributed to an APT group, was the first publicly documented in-the-wild UEFI rootkit to achieve persistence by modifying firmware modules on the chip. Such a threat survives disk replacement and potentially requires specialized tools to detect and remediate. BlackLotus did not operate at that layer, which, while still serious, placed it in a different and somewhat more remediable category.

Detection and Recovery

Detecting a bootkit is inherently difficult because the threat operates in an environment that most security tools cannot inspect during startup. Endpoint protection software that runs within Windows has no visibility into what happened before the kernel initialized. However, it can observe artifacts after the fact: unexpected files or timestamps in the EFI System Partition, registry changes that reflect bootkit installation, event log entries showing that Defender or HVCI was disabled, and the presence of unauthorized kernel drivers. Microsoft Defender for Endpoint can generate alerts for known BlackLotus indicators, though a bootkit that successfully disables Defender before those detections fire will reduce or eliminate that visibility. TPM-backed Measured Boot matters here precisely because its logs record what happened before the OS loaded, and those records exist outside the environment the bootkit can easily tamper with.

Indicators that defenders may be able to observe include:

Unexpected files in the EFI System Partition, particularly in directories that legitimate Windows boot processes do not use. BlackLotus, for example, stored components in a directory that is not part of the normal Windows boot file layout.

Changes to UEFI firmware variables, including unexpected entries in the Machine Owner Key database or Boot Services key list.

Disabled security features that were previously enabled, including HVCI or Windows Defender, without a corresponding user action.

Network activity from early boot components or kernel drivers that would not normally communicate externally.

For suspected boot-chain compromise, the appropriate response is not simply to run an antivirus scan. Restoring a clean boot environment involves verifying the integrity of boot components, applying firmware variable updates including DBX revocation updates, and potentially rebuilding the system from trusted installation media to establish a known-clean starting state. Guidance from CISA, the NSA, and Microsoft should inform incident response in cases where bootkit infection is suspected.

Preventive measures are clearer:

Keep UEFI firmware and operating system boot components updated, and follow the specific configuration steps Microsoft published for CVE-2023-24932. Applying the Windows update alone does not automatically enable all Secure Boot protections; the staged rollout requires deliberate action.

Maintain Secure Boot in its enabled configuration. Disabling Secure Boot to accommodate other use cases removes this protection. Review whether those tradeoffs are necessary.

Apply Windows security updates promptly. CVE-2022-21894 was patched in January 2022; systems that had applied that update and subsequently completed the CVE-2023-24932 mitigations were in a substantially better position when BlackLotus appeared in the wild.

Consider TPM-backed Measured Boot where appropriate, and review whether platform integrity measurements are being logged and monitored.

Limit administrative privileges. BlackLotus required administrative access to install. Reducing unnecessary administrative access limits the opportunities for initial installation.

The Chain of Trust Is Only as Strong as Its Weakest Link

The Windows boot process is built as a chain of trust. Each component verifies the next before passing control. The chain is supposed to ensure that by the time Windows is running, every step that preceded it has been verified as legitimate.

Bootkits attack this model by compromising an early link. Once a malicious component has executed and established its position in the startup sequence, it can shape every subsequent stage. Security controls that the operating system relies on can be disabled, hidden, or subverted before they initialize.

Secure Boot addresses this threat by requiring cryptographic verification at each step. But, as BlackLotus demonstrated, the value of that verification depends on whether the trusted component list is accurate, current, and correctly maintained. A vulnerability in a signed boot component that remains trusted because revocation has not been applied is an opening that can be exploited without breaking any cryptographic primitive.

The deeper lesson is about where security boundaries actually exist. We tend to locate security in the operating system, in the software layer that users interact with. But an operating system that loads on top of compromised boot components is building on an untrustworthy foundation. Security must begin before the operating system. When the mechanism that starts the OS cannot be trusted, everything built on top of it is suspect too.

Primary Sources

What Is Kerberoasting? How Can Attackers Crack Service Account Passwords Without Exploiting a Vulnerability?

Aditya Sharma — Sat, 10 Oct 2026 18:07:32 +0000

An attacker with a regular domain account requests a service ticket. The domain controller issues it. No vulnerability was exploited. No alarm fires. And if the service account's password is weak enough, the attacker may eventually recover it offline without ever making another authentication request.

This is Kerberoasting: an attack that exploits the gap between how Kerberos was designed to work and how service accounts are often deployed and managed.

The Kerberos Pieces Readers Actually Need

Kerberos is a network authentication protocol designed to let parties prove their identities without sending passwords over the wire. Understanding Kerberoasting requires a working mental model of a few components.

The KDC (Key Distribution Center) is a trusted server, in Windows environments it is a domain controller, that issues cryptographic tickets. The KDC contains two logical parts: an Authentication Server (AS) that handles initial login and a Ticket-Granting Server (TGS) that handles subsequent service access.

Tickets are the central concept. A ticket is a blob of data that proves to a recipient that the holder authenticated successfully. Tickets are issued by the KDC and cannot be forged by ordinary clients.

The TGT (Ticket-Granting Ticket) is issued when a user authenticates. Think of it as a session token that proves identity to the KDC. The client uses it to request access to individual services without re-entering credentials.

Service tickets are issued in response to requests for specific services. They are what the client actually presents to an application server to gain access.

Service Principal Names (SPNs) are identifiers that associate a specific service instance with a specific account. An SPN might look like HTTP/reporting.corp.example.com or MSSQLSvc/dbserver01.corp.example.com:1433. When a client wants to reach a service, it specifies the SPN and the KDC figures out which account is responsible for that service.

Service accounts are domain accounts that run services rather than belonging to human users. A SQL Server instance might run under svc-sql, a reporting tool under svc-reporting. These accounts have SPNs registered against them.

A Normal Kerberos Service-Ticket Request

When a user wants to access an application, here is what happens at the protocol level:

The client authenticates to the domain and receives a TGT. This involves proving knowledge of the user's password using cryptography.
The client sends a ticket-granting service request to the KDC, presenting the TGT and specifying the SPN of the service it wants to reach.
The KDC finds the account associated with that SPN and issues a service ticket. Critically, part of this ticket is encrypted using a key derived from the service account's password. The client receives the ticket but cannot use that encrypted portion as the intended recipient. Only the service itself, which knows its own account password, can decrypt that portion.
The client presents the service ticket to the application. The application decrypts the relevant portion, verifies it, and grants access. The important detail: the KDC is doing exactly what it is supposed to do. It issued a legitimate ticket. The client made a legitimate request. Nothing abnormal has occurred from the protocol's perspective.

The question is what happens to that ticket afterward.

How Kerberoasting Uses This Against You

An attacker who has already obtained access to a valid domain account, perhaps through phishing, password reuse, or another initial compromise, can enumerate service accounts by querying for accounts with SPNs registered. This is ordinary LDAP directory enumeration, the kind any legitimate administrator might perform.

The attacker then requests service tickets for those SPNs. Again, this is a normal Kerberos operation. Any authenticated domain user can request a service ticket for a service they are nominally allowed to reach. The KDC issues the ticket.

Now the attacker has ticket material that includes a portion encrypted with a key derived from the service account's password. They take this material offline.

Offline means: no more authentication requests. No exposure to lockout policies. No network traffic to the domain controller. The attacker uses a cracking tool to generate password guesses, derives candidate keys from each guess, and tests whether each candidate key correctly produces the expected cryptographic output from the ticket.

If the service account's password is Password123 or ServiceAcct2019, a dictionary-based approach may find it in minutes or hours. If the password is a 32-character random string, testing billions of guesses still will not find it in any practical timeframe.

Account lockout does not help here. Lockout policies protect against online attacks, where each guess requires sending a request to an authentication system that can count failures and lock accounts. Offline cracking makes no such requests.

What Is Actually Being Cracked?

This is worth slowing down on, because "cracking a ticket" sounds more dramatic than what is happening.

The attacker is not breaking the encryption, reading arbitrary data out of the ticket, or extracting a password through some direct operation. They are performing a password guessing attack against cryptographic material.

The relevant portion of the service ticket was encrypted using a key derived from the service account's password through a defined key-derivation function. The attacker's process is:

Guess a password
↓
Derive a candidate key using the same process the KDC uses
↓
Test whether the candidate key produces output consistent with the captured ticket
↓
If yes: password found. If no: try the next guess.

The encryption type matters here. Older Kerberos configurations may still issue tickets using RC4 encryption, where the key is derived directly from the account's NT hash. More modern configurations favor AES-based encryption types that involve a slower key-derivation process. Slower key derivation increases the computational cost of each guess, which matters at scale.

Neither approach makes offline cracking impossible against weak passwords. Both make weak passwords fundamentally risky, and both make strong passwords effectively safe.

The practical consequence: Kerberoasting is an attack against weak passwords protected by ticket-derived keys. It is not an attack against Kerberos itself.

A Concrete Example

Imagine a company called Meridian Analytics. They have an internal reporting application that runs under a service account named svc-reporting. The account has an SPN registered: HTTP/reports.meridian.internal. The password on this account was set three years ago by a sysadmin who followed the then-current policy: eight characters, mixed case, a number. Something like Reports1.

Cassandra, an attacker who has compromised a regular employee's domain credentials through a phishing attack, authenticates to the domain and receives a TGT. She then queries Active Directory for accounts with SPNs registered. svc-reporting appears on the list.

Cassandra sends a TGS request for HTTP/reports.meridian.internal. The domain controller issues a service ticket. From the KDC's perspective, this is unremarkable: authenticated user requested a ticket for a service she may well need to use.

Cassandra extracts the relevant encrypted material from the ticket and runs it through a password-cracking tool with a wordlist and rules. Within a few hours, the tool finds Reports1.

Now Cassandra has the credentials for svc-reporting. The question of what she can do with them depends entirely on what that account can access.

Why the Impact Can Be Serious

In a well-administered environment, svc-reporting would have only the permissions it needs to run the reporting service and access the data it requires. If Cassandra recovers that password, she can impersonate the reporting service.

But real environments are often not well-administered. Service accounts accumulate permissions over time. They get added to groups for "temporary" access that never gets revoked. In some organizations, the same service account is used across multiple services, meaning one recovered password opens multiple doors. Some service accounts have been granted Domain Admin or local administrator rights on many machines.

The attack does not guarantee privilege escalation. The outcome depends entirely on what the compromised account can reach. A service account with minimal, scoped permissions is a much less interesting target than one with broad access.

This also explains why Kerberoasting works as a lateral movement or privilege escalation technique rather than purely as an initial compromise tool. An attacker who already has a foothold, perhaps a regular user account, can use Kerberoasting to look for service accounts with weak passwords and elevated privileges, then move from a limited starting position toward more powerful access.

Detection and Investigation

Defenders have some visibility into Kerberoasting, though not complete visibility into what is happening offline.

Windows Security Event 4769 (A Kerberos service ticket was requested) is generated when the KDC issues a service ticket. A burst of 4769 events for multiple different SPNs from a single account in a short time window is worth investigating. So is an unusual account requesting tickets for services it has never accessed before, or accessing services that are not part of its normal workflow.

The encryption type in these events can also be informative. Kerberoasting tools sometimes request RC4-encrypted tickets even in environments that normally use AES, because RC4 has been more tractable for offline cracking. Observing service-ticket requests with unexpectedly downgraded encryption types can flag activity worth reviewing.

The challenge with detection is false positives. Service-ticket requests are normal. Applications request many tickets. Administrators run scripts that enumerate services. A single unusual request means very little. Detection requires a behavioral baseline: what does normal service-ticket activity look like for this account, from this machine, at this time of day?

Investigation benefits from correlating authentication telemetry with other signals. An account that suddenly begins requesting tickets for many service accounts, from a workstation that has never made such requests, shortly after a known phishing delivery, is a much clearer signal than isolated authentication events.

Defenses That Address the Root Cause

Kerberoasting is a symptom of a broader problem: service accounts with human-guessable passwords and excessive privileges. The defenses follow from that understanding.

Use long, randomly generated service-account passwords. A 64-character random password makes offline cracking economically infeasible. The password does not need to be remembered by a human; it just needs to be stored securely and used programmatically. This is the highest-impact single change.

Prefer group Managed Service Accounts (gMSAs) where supported. gMSAs are a Windows feature where the domain controller manages the service account's password automatically. The password is long, random, and rotated on a schedule without manual intervention. Services using gMSAs are largely immune to Kerberoasting because recovering a 128-character random password offline is not practically feasible.

Apply least privilege to service accounts. A service account should have exactly the permissions it needs and nothing more. This limits the damage if a password is eventually recovered. Review group memberships and permissions regularly.

Audit SPNs and service-account ownership. Know which accounts have SPNs registered, which services they represent, and whether those SPNs are still needed. Orphaned service accounts, ones where the service was decommissioned but the account and its SPN remain, are easy targets.

Review encryption type configurations. Modern environments should prefer AES-based Kerberos encryption. Legacy RC4 support may still be needed for older systems, but its presence where it is not needed unnecessarily increases the tractability of offline cracking.

Monitor and baseline service-ticket activity. Establish what normal looks like, then alert on meaningful deviations. Combine authentication logs with endpoint telemetry to build fuller context around anomalies.

Disabling Kerberos is not a viable mitigation. The protocol is foundational to Windows domain authentication. Blindly blocking all service-ticket requests would break most of the services that depend on authentication. The response to Kerberoasting is not to break the protocol but to remove the weakness it exploits.

The Protocol Did Its Job; The Weakness Is Elsewhere

Kerberos issued a legitimate ticket in response to a legitimate request. From the protocol's perspective, nothing went wrong.

The problem is that this ticket contains material encrypted with a key derived from a password that a human chose. The protocol assumed that the password protecting the key would be strong enough to withstand offline guessing. In many environments, service accounts have weak passwords because no one is guessing them interactively, so the usual heuristics about password strength were not applied.

Kerberoasting works because it turns ticket-derived cryptographic material into an offline oracle: a way to test password guesses without ever returning to the domain. The protocol enables this because service authentication requires the client to receive material that the service will eventually decrypt. The attack grabs that material before the service sees it.

The design assumption that service-account passwords are strong enough to resist this treatment was reasonable in principle. In practice, service-account passwords often were not set with this threat in mind.

That is the lesson Kerberoasting teaches: authentication protocols are designed with assumptions about secrets. When those assumptions about secret quality break down, protocol operations that were intended to be safe can become attack primitives. The weakness is not the cryptography. It is the password standing behind it.

What Is DNS Tunneling? How Can Hackers Hide Data Inside DNS Queries?

Aditya Sharma — Fri, 09 Oct 2026 05:20:30 +0000

Every time a browser visits a website, a DNS query leaves the machine. The query contains a domain name. That name resolves to an address. The connection proceeds. Nothing interesting happened.

But the name in that query is arbitrary text. It is structured, subject to protocol constraints, and travels through the internet's resolution infrastructure. And it can carry information that has nothing to do with looking up an address.

That is the premise of DNS tunneling: using DNS queries and responses as a relay channel for information that has nothing to do with name resolution.

What DNS Normally Does

Before exploring how the protocol can be abused, it helps to have a clear picture of how it actually works.

When a client wants to resolve a hostname, it sends a query to a recursive resolver, typically provided by an ISP, a corporate network, or a service like 1.1.1.1 or 8.8.8.8. The recursive resolver does the work of finding the answer. If the resolver does not already have the answer cached, it works through the DNS hierarchy.

The resolution process starts at the root. Root name servers know which authoritative servers are responsible for each top-level domain. Those TLD servers know which authoritative servers are responsible for specific domains. The authoritative server for a domain knows the actual records: the IP addresses, mail exchangers, text records, and other data associated with names in that zone.

In practice, caching shortcuts this process constantly. If the recursive resolver already has a cached answer, the query never reaches root or TLD servers. The same applies at every level. What matters for understanding tunneling is the endpoint: a query under a specific domain ultimately reaches the authoritative server for that domain, and the operator of that authoritative server can see every query for every subdomain.

A DNS query contains a name and a record type. The response contains one or more resource records with the requested data. The response travels back through the same path it came from, returning to the client.

The Insight: A DNS Name Can Carry Data

A DNS domain name is divided into labels separated by dots. The name mail.example.com has three labels: mail, example, and com. Each label can be up to 63 octets long. The full wire-format domain name, including the label-length prefix bytes and the terminating zero-length root label, cannot exceed 255 octets.

These constraints are meaningful but not severe. They leave room for labels that contain encoded information rather than readable hostname identifiers.

Consider a query like:

Y2F0YWxvZzEy.session-id-8a2f.attacker-controlled.example

The first label looks like Base64-encoded text. It might be a fragment of a file, a short command, or a sequence number identifying part of a larger dataset. The second label might identify a session or client. The final two labels reach the authoritative server for attacker-controlled.example.

This is not a special DNS record type. It is not a protocol extension. It is a normal-looking DNS query carrying information encoded into the structure of the name being queried. Any resolver that processes the query and forwards it toward the authoritative server is, without knowing it, relaying that information toward an attacker-controlled endpoint.

The Communication Channel

The channel works in both directions.

Sending data out: A compromised endpoint encodes information into subdomain labels and sends DNS queries. The queries travel to the configured recursive resolver, which performs normal resolution. Eventually, the query reaches the authoritative server for the domain the attacker controls. The server sees the query and decodes whatever information the labels contain.

Compromised host
    → recursive resolver (performs normal resolution)
    → attacker's authoritative DNS server (receives and interprets the query)

Receiving data back: The authoritative server returns a response. The response can carry data in resource records. TXT records accept arbitrary text content and are a common choice because they do not imply a specific structured meaning. The response travels back through the recursive resolver to the client, which decodes the record content.

The recursive resolver is not complicit in this. It resolves names and returns responses, which is its job. It does not interpret attacker-defined application payloads; it forwards DNS traffic. The authoritative endpoint, controlled by the operator of the tunnel, is where the protocol data is actually interpreted and acted upon.

Different implementations make different design choices. Some use TXT records for responses, some use CNAME or NULL records, some use other types. The query name carries outbound data; the response carries inbound data. The channel is bidirectional but asymmetric. Queries have more consistent universal support than some response record types, and caching behavior can affect how frequently unique queries are needed.

Why It Can Survive Network Controls

Organizations that block direct outbound TCP and UDP connections often still permit DNS. Applications need to resolve names. Servers need to respond to queries. DNS traffic flows constantly and its presence is expected.

This is the environment DNS tunneling exploits. Traffic that uses DNS looks like DNS traffic. At a superficial level, a query to encoded-data.subdomain.example.com resembles a query to mail.example.com. Both are standard DNS queries to a valid-seeming domain.

But there are important distinctions to make here. The fact that DNS traffic is permitted does not mean it is trusted in any deep sense, and it does not mean it is uninspected. These are different properties:

Permitted means the firewall allows the protocol to pass.
Trusted means the contents are accepted as safe or unexamined.
Inspectable means the traffic can be decoded and analyzed by monitoring systems. Many organizations monitor DNS queries and can detect anomalies. What varies is whether that monitoring actually happens, how deep it goes, and whether anything is tuned to catch tunnel-like patterns.

The visibility situation changes with encrypted DNS. DNS over HTTPS (DoH) and DNS over TLS (DoT) encrypt the query content, which prevents passive interception by on-path network monitoring. An organization that routes all DNS through a monitored recursive resolver retains visibility. One that allows endpoints to send DoH directly to external resolvers may lose it. But encrypted DNS is not itself tunneling, and not all use of DoH or DoT is malicious. The point is that encryption affects where and whether monitoring is possible.

What the Tunnel Actually Carries

The most common malicious applications fall into two categories.

Command-and-control: A compromised machine needs to receive instructions from an operator. It periodically queries a domain the operator controls, and the responses encode instructions. This is attractive because it uses infrastructure that is often permitted and may not immediately raise alarms.

Data exfiltration: A machine that has accessed sensitive information encodes and sends it out in fragments over many queries. Large amounts of data require many queries, and the encoding overhead reduces efficiency significantly. DNS is not a high-throughput protocol. A tunnel carrying DNS traffic as its substrate faces inherent constraints: query latency, encoding overhead, label-length limits, resolver behavior, and the risk of detection from high query volume.

This is not a channel designed for bulk transfers. It is better suited to low-bandwidth communication: small commands, status signals, short credentials, session tokens, or slow incremental exfiltration of modest amounts of data. Implementations that need to send larger payloads fragment the data across many queries and reassemble at the receiving end, accepting the latency and overhead that implies.

Protocol Constraints and What They Mean

The constraints on DNS names shape what tunneling can practically carry.

A single label is limited to 63 octets. In a name encoded using a URL-safe Base64 variant, that is roughly 47 bytes of original data per label before encoding overhead. A domain name can have multiple labels before reaching the fixed portion, and the full wire-format name cannot exceed 255 octets.

These limits mean every query carries a small and bounded amount of data. Encoding schemes like Base32, Base64 variants, or hexadecimal are common choices, not because the protocol requires them, but because they produce characters that are compatible with DNS label character restrictions. Different tunneling implementations make different choices. No encoding method is universal.

Caching complicates repeated queries. If a recursive resolver caches the response to a query, a subsequent identical query may be served from cache rather than forwarded to the authoritative server. This means the authoritative endpoint may not see the repeated query. Tunneling implementations that need to send unique data per query often use unique labels to prevent cache hits, accepting that each query generates a new lookup and network traffic.

Query and response sizes matter. Large DNS responses may trigger UDP truncation and fallback to TCP, which affects timing and may be more visible to monitoring systems. Implementations designed to stay under truncation thresholds are more likely to use common query types and keep responses small.

A Concrete Conceptual Example

Consider what a normal DNS lookup looks like:

Query:  www.example.com  (type A)
Answer: 93.184.216.34

Now contrast that with a tunneling query:

Query:  bWVzc2FnZQ.client42.attacker-controlled.example  (type TXT)
Answer: "cmVzcG9uc2U=" (TXT record content)

The first label, bWVzc2FnZQ, is an illustrative encoded-looking value. It might represent a fragment of a command, a file chunk identifier, or a short message. The label client42 might identify the client or session. The authoritative server for attacker-controlled.example receives this query, extracts the relevant labels, decodes them, and returns an encoded response in the TXT record.

The client receives the TXT record, decodes the content, and processes the instruction or data.

These are hypothetical labels. They demonstrate the mechanism. A real implementation would add session management, fragmentation logic, error handling, and likely encryption of the payload itself. But the DNS infrastructure underneath is doing exactly what it is designed to do: resolving names and returning records.

How Defenders Detect DNS Tunneling

Detection focuses on behavioral signals that distinguish tunnel traffic from normal resolution activity. No single signal is reliable in isolation.

Label entropy and length: Legitimate subdomain labels are usually human-readable or follow consistent patterns. Labels that look like Base64, hexadecimal, or random character sequences stand out, especially when they appear in high volume or under a single domain. High label entropy is a meaningful signal but not proof of malicious intent. Generated hostnames in CDN and cloud services can produce similar patterns.

Query volume and domain concentration: A compromised host sending many queries to a single domain stands out if that domain is not one the host normally contacts. Similarly, a domain that receives a disproportionate share of queries from a single endpoint warrants investigation.

Unique subdomain ratio: Legitimate domains typically resolve a manageable set of hostnames. A domain that receives thousands of queries each with a different first label is behaving unlike a normal web service.

TXT record frequency: TXT records are less commonly queried in most environments than A or AAAA records. An endpoint that makes heavy use of TXT queries to unusual destinations is worth examining, though TXT records have many legitimate uses and this signal requires context.

Periodic beacon-like patterns: Queries at regular intervals, consistent with a client checking in for instructions, can be detected by frequency analysis. But periodic queries are also normal for many legitimate services.

Response size anomalies: TXT records returning unusual amounts of data, or responses that are consistently near truncation limits, may indicate non-standard usage.

Behavioral correlation: DNS anomalies are most meaningful when correlated with other endpoint behavior. A process generating unusual DNS queries alongside other suspicious activity is far more significant than DNS patterns in isolation.

Effective detection means combining these signals, establishing baselines for each environment, and having the process and network context to investigate alerts. A high-entropy subdomain label sent by a software update service checking for a new version is not a threat. The same pattern from an unknown process on an endpoint that should not be making external DNS queries is.

How Organizations Defend Against It

Detection without response is incomplete. Practical defenses involve controlling the DNS infrastructure itself as well as monitoring its traffic.

Route all endpoint DNS through monitored recursive resolvers. This is the single most effective control. If DNS queries pass through a resolver that logs them, behavioral analysis is possible. If endpoints can send DNS directly to arbitrary external resolvers, visibility is lost.

Restrict direct outbound DNS access. Many organizations allow port 53 outbound from all endpoints. Restricting this to specific resolvers, and blocking direct access to external DNS infrastructure, removes a significant degree of freedom from an attacker.

Monitor and alert on anomalous query patterns. This requires establishing what normal looks like. Anomaly detection without a baseline produces too many false positives to act on. With a baseline, deviations stand out.

Use response-policy zones or DNS filtering where appropriate. Blocking queries to known-malicious domains prevents contact with existing attacker-controlled infrastructure. This does not help with newly registered or unknown domains, but it addresses a meaningful subset of threats.

Investigate suspicious endpoints. DNS patterns are one piece of evidence. Correlating with process telemetry, network connections, and filesystem activity gives a full picture of what is happening on an endpoint.

Consider the implications of encrypted DNS. If endpoints use DoH to external resolvers directly, the organization loses DNS visibility unless it can inspect that traffic at the application layer or enforce routing through an internal DoH resolver that it controls.

Blocking specific record types or refusing TXT queries is a tempting but fragile defense. Tunneling implementations can adapt. The underlying capability exists as long as the endpoint can reach an authoritative server it or its operator controls.

The Lesson DNS Tunneling Teaches

DNS tunneling demonstrates something general about protocol trust.

A firewall rule that permits DNS traffic makes a decision about protocols and ports. It does not make a decision about what the traffic is carrying. A policy that allows DNS does not imply that DNS traffic will only ever be used for name resolution. The protocol's design allows it to carry arbitrary text in names and in response records. Attackers who understand this can use it.

The same principle applies elsewhere. Protocols that are permitted because they are necessary can carry data they were not designed to transport. HTTP, HTTPS, ICMP, and DNS all have examples. The common thread is that allowing a protocol does not constrain what applications can build on top of it.

Effective network security eventually requires not just asking "what protocol is this?" but also "what is this traffic actually doing?" DNS is essential infrastructure. But treating it as uniformly trustworthy because it uses port 53 is treating the container as a guarantee about the contents.

The channel into a domain's authoritative server was always there. DNS tunneling just found a use for it.

What Is Memory-Mapped I/O? How Does the CPU Talk to Hardware Through Memory?

Aditya Sharma — Thu, 08 Oct 2026 05:06:14 +0000

A single store instruction can turn on a hardware timer, start a DMA transfer, or tell a network controller that there is work waiting.

From the software side, it looks like writing to a memory address. From the hardware side, it is a command to a device. Nothing in the instruction itself makes this distinction. The same mechanism the CPU uses to write to RAM is also used to talk to hardware. Understanding why requires taking a closer look at what an "address" actually means to a processor.

The Assumption Worth Questioning

Ask most programmers what a memory address refers to and they will say: a location in RAM. A byte or word stored somewhere in physical memory. That is a reasonable first approximation for the code they write most of the time.

But a CPU executing a load or store instruction does not inherently know it is accessing RAM. It issues a transaction for a particular address. The rest of the system decides what that address refers to.

On modern systems, the physical address space is divided among multiple resources. Some address ranges map to RAM. Others map to firmware storage. Others map to device registers. When the CPU performs a transaction targeting an address in a device region, something other than RAM responds.

This is Memory-Mapped I/O: the technique of assigning device registers to address ranges in the processor's physical address space so that ordinary load and store operations can access hardware.

Devices Need a Communication Channel

Every device in a computer, a network interface, a timer, a display controller, a USB host controller, needs some way to receive instructions and report state back to software. The CPU needs to be able to say "start transmitting this packet" or "what is the current time?" and receive an answer.

Several mechanisms exist for this. On x86 systems, there is a separate I/O port address space with dedicated IN and OUT instructions. But port-mapped I/O is not universal, and even on x86 most modern devices prefer the alternative: mapping their registers into the memory address space where they can be reached with ordinary load and store instructions.

This has advantages. No special instructions are needed. Memory-access code paths in compilers and processors are mature and efficient. Drivers can be written using the same constructs as ordinary memory access, with appropriate care for the semantics involved.

The tradeoff is that device memory does not behave like RAM, and software needs to understand the difference.

Registers, Not Storage

When a device is mapped into the address space, it exposes a set of registers. These are not memory cells in the ordinary sense. They are hardware interfaces. A register at a given offset might represent the device's current status, a control field, a data buffer, a command trigger, or a configuration setting. The meaning is defined by the device specification.

Take a simplified hypothetical controller that exposes three registers:

offset 0x00: control register
offset 0x04: status register
offset 0x08: data register Writing a value to the control register might enable the device or start an operation. Reading the status register might tell software whether the device is idle or busy. Writing to the data register might enqueue bytes for transmission. These operations are defined by the hardware, not by anything the CPU knows about memory.

The address range assigned to this device could be something like 0xFE000000 through 0xFE000FFF, reserved in the system's address map for that device. The exact range depends on the platform and how the system firmware or hardware configures it.

The Path From Instruction to Device

What actually happens when software reads from or writes to one of these addresses?

Software typically works with virtual addresses. Before any memory transaction happens, a virtual address may need to be translated to a physical address. The Memory Management Unit handles this translation using page tables that the operating system maintains. Device regions have to be mapped into the virtual address space with appropriate attributes before software can access them, and the kernel sets this up before handing a mapping to a driver.

Once the physical address is determined, the CPU issues a transaction on its interconnect. Modern systems route transactions through buses, bridges, and interconnects depending on the destination. The system's address map determines which component should respond to each physical address range. A transaction targeting an address assigned to RAM goes to the memory controller. A transaction targeting a device region is routed toward that device.

The device receives the transaction, decodes the address to determine which of its registers is being accessed, and responds. For a read, it returns the current value of that register. For a write, it accepts the value and changes its internal state or triggers an action.

This routing is a hardware-level function. The CPU does not know it is talking to a device rather than RAM. The address falls in a range the device is configured to respond to, and so the device responds.

Side Effects Are the Point

This is where device registers diverge sharply from ordinary memory.

Reading a normal RAM location returns the stored value without any other consequence. Reading a device register can have side effects. On some devices, reading a status register clears a pending condition. Reading a data register can consume a byte from a hardware FIFO. Some registers are write-only; attempting to read them returns undefined or dummy values.

Writes to RAM store a value. Writes to device registers can do almost anything: start a hardware operation, configure a mode, acknowledge an interrupt, trigger a reset, initiate a DMA transfer. Writing the same value twice may have different effects depending on the device's current state. Some registers latch state on write. Others are level-sensitive.

Writing to a control register twice with the same value is not equivalent to writing once. The first write might arm a trigger; the second might fire it. Or the first might enable a device and the second might not change anything. The semantics are defined entirely by the device.

This is why MMIO regions cannot be treated as ordinary memory.

Compilers and `volatile`

Most programming languages allow the compiler to optimize memory accesses when it can determine that an access is unnecessary. If you write the same value to a location twice and nothing reads it in between, the compiler might eliminate one of the writes as redundant. If you read a value in a loop and the compiler can determine it is not being modified by the executing code, it might cache the value in a register instead of re-reading memory each iteration.

These optimizations are correct for ordinary memory. They are wrong for device registers.

A device register can change independently of the executing code. The status register might flip from busy to idle because the hardware finished an operation. If the compiler caches the initial status in a register and never re-reads the hardware location, the software will miss that change.

The volatile qualifier in C and C++ is intended to tell the compiler that a variable or location cannot be assumed stable between accesses from the compiler's perspective, and that every access in the source code must translate to an actual access in the generated machine code.

But there is an important boundary here. volatile constrains the compiler. It says: do not elide or reorder this access in the generated code. It does not say anything about how the CPU itself executes instructions, or how memory transactions propagate through a system. Modern CPUs can reorder memory operations relative to the order in the instruction stream, subject to their architectural memory model. A system with multiple components may have ordering constraints that extend beyond the CPU.

volatile alone does not provide memory ordering guarantees between the CPU and a device. That requires explicit memory barriers or ordering primitives appropriate to the architecture.

Memory Barriers and Ordering

Consider a driver preparing to hand work to a device. It writes data to a shared memory buffer that the device will read via DMA. Then it writes to a device control register to tell the device the data is ready.

This sequence seems obvious: data first, then notification. But modern CPUs and memory systems can reorder transactions in ways that are not visible to the programmer. The CPU might issue the register write before the data writes have propagated to wherever the device will see them. The device might observe the notification before the data is coherent.

Memory barriers instruct the CPU to impose an ordering requirement. A write barrier establishes an ordering constraint between memory operations before and after the barrier, with the exact guarantees defined by the architecture's memory model.

This is a separate concern from compiler optimization. Even with volatile ensuring the compiler generates the accesses in the correct order, a memory barrier may be needed to ensure the CPU and memory system honor that ordering relative to external observers like a device.

Device driver code in production operating systems carefully places memory barriers at the points where ordering between software-visible memory and device-visible operations is required. Getting this wrong produces intermittent failures that can be difficult to reproduce, because the race window may only materialize under specific timing conditions.

Caching and Memory Attributes

Normal RAM accesses benefit from CPU caches. The cache exploits temporal locality: recently accessed data is likely to be accessed again, so keeping it close to the CPU reduces latency.

Ordinary device registers generally cannot be treated like cacheable RAM. A cache that returns a stale value for a status register might miss a state change the hardware made moments ago. And writes to control registers that are buffered in a cache and not immediately sent to the device can delay or corrupt device operations.

Systems handle this through memory type and attribute mechanisms. When a device region is mapped into the virtual address space, the mapping is configured with attributes appropriate for device memory: typically uncacheable, possibly with specific ordering properties. These attributes are part of the page table entry or memory-type configuration. The hardware respects them to ensure that accesses to device regions are not cached or reordered in ways that would violate device semantics.

This does not mean that all device memory is always uncached. Some devices provide memory regions that behave more like ordinary RAM and can be cached with appropriate coherence management. Framebuffers and device memory used for bulk data transfer can sometimes benefit from write-combining, a mode that buffers writes and flushes them to the device in larger transactions for efficiency. The appropriate attributes depend on the device and the access pattern.

The important point is that the kernel configures mappings to match what the device requires, rather than treating device memory as interchangeable with RAM.

The Operating System's Role

Software cannot just pick an address and write to it expecting a device to respond. The physical address ranges assigned to devices are fixed by the hardware platform, and on modern systems they are discovered through mechanisms like PCI configuration space or device trees.

The operating system learns about devices and their address ranges during initialization, typically by reading platform firmware or enumerating the bus. It maps device regions into the kernel's virtual address space with the appropriate memory attributes. It implements device drivers that know the semantics of each device's registers: which offsets correspond to which functions, what values mean, what sequences of operations are required to accomplish a task.

User programs do not normally access device registers directly. Direct hardware access from user space would be a security problem: a program could misconfigure devices, corrupt system state, or interfere with other processes' hardware. Instead, user programs invoke system calls or interact with kernel-provided abstractions, and drivers translate those requests into the appropriate register-level operations.

When a driver needs to perform MMIO, it uses the mapping the kernel established and accesses it through whatever mechanism the operating system provides, typically a pointer to the mapped region with appropriate use of volatile and barriers.

A Concrete Path Through a Network Transmission

To make this concrete, consider the sequence involved in sending a packet through a simplified network controller.

The driver prepares the packet data in memory and sets up a descriptor that tells the device where the data lives and how large it is. This preparation requires that the data and descriptor be visible in memory to the device, which may involve a DMA-coherent allocation and appropriate memory barriers.

The driver then writes to a transmit-request register exposed by the device via MMIO. This write is an ordinary store instruction targeting an address in the device's mapped region. The transaction travels through the system's interconnect, the device decodes it, and the corresponding register receives the value.

The device, responding to the register write, reads the descriptor via DMA, fetches the packet data from memory, and begins the transmission. Software does not actively participate in the transmission itself.

When the transmission completes, the device can either update a status register that the driver polls, or assert a hardware interrupt that causes the CPU to enter an interrupt handler. Either way, the driver reads the status via another MMIO access and records the outcome.

The packet transmission was initiated by a single store instruction to a device register. That instruction did not move any packet data. It told the device to look in a known memory location and proceed. The store instruction was the command; the device register was the command interface; MMIO was the mechanism that made a load/store instruction into a hardware communication.

Addresses All the Way Down

The CPU does not know whether an address points to RAM, a device register, or something else. It issues transactions. The system's hardware and configuration determine what responds.

MMIO works because of this. By assigning device registers to address ranges, systems designers give software a uniform mechanism for interacting with hardware. The same load and store instructions that read and write memory also read and write device registers, firmware regions, and other system resources. The address encodes the intent; the hardware resolves it.

What the address actually means is a question the system's address map answers. And the answer is not always RAM.

What Is Copy-on-Write? How Can a Process "Copy" Memory Without Actually Copying It?

Aditya Sharma — Wed, 07 Oct 2026 06:27:32 +0000

When a process calls fork(), the operating system creates a child process with what appears to be an identical copy of the parent's memory. If the parent had two gigabytes of working memory, did the kernel just allocate another two gigabytes?

The answer is no. And the reason why reveals something genuinely elegant about how operating systems manage memory.

What "Memory" Actually Means to a Process

Before getting to the trick, it helps to have a precise mental model of what a process's memory actually is.

A process does not own physical RAM the way a warehouse owns inventory. Instead, it operates through an abstraction: the virtual address space. Every memory address the process works with is a virtual address. When the CPU needs to read or write memory, it translates that virtual address to a physical address using a structure called a page table.

The page table is a mapping:

Virtual address
      ↓
Page table
      ↓
Physical address

Memory is divided into fixed-size chunks called pages. A typical page is 4 KB. The page table maps virtual pages to physical pages (sometimes called frames). The CPU's memory management unit performs this translation automatically on every memory access.

The critical insight: two different processes can have virtual addresses that map to the same physical page. Their virtual address spaces are separate, but the underlying physical memory can be shared. This is not a bug or a security hole when the kernel controls it deliberately. It is the foundation that makes Copy-on-Write possible.

What `fork()` Actually Does to Memory

When a process calls fork(), the kernel creates a child with a logically equivalent virtual address space. The child sees the same variables, the same stack, the same heap, the same loaded libraries.

The naive approach: allocate new physical pages for everything, copy all the contents. For a process with gigabytes of memory, that means gigabytes of allocation and copying, taking real time and consuming real resources.

The Copy-on-Write approach: do not allocate new physical pages immediately. Instead, arrange for parent and child to reference the same physical pages through their respective page tables. Both virtual address spaces look identical, but they are temporarily pointing at the same underlying physical memory.

This works because the kernel controls the page-table mappings. The pages that both processes now share are marked with a protection that prevents writes through those mappings. If either process tries to modify a shared page, the CPU detects the permission violation and the kernel gets involved.

The processes have separate virtual address spaces. They just happen to have page-table entries pointing to the same physical pages, with those entries temporarily protected against modification.

The Moment It Matters: One Process Writes

The interesting part happens when a process actually tries to modify a shared page.

Suppose after fork(), the state looks like this:

BEFORE WRITE

Parent VA ──┐
            ├──→ Physical Page X
Child VA ───┘

Both virtual-page mappings point to the same physical page. Both are marked read-only in the relevant page-table entries.

Now the child executes a write to that virtual address. The CPU checks the page-table permissions. The mapping says: not writable. The CPU raises a page fault.

Here is an important distinction: a page fault is not inherently an error. It is a hardware mechanism that tells the kernel the current memory access needs handling. The kernel examines the fault, determines it is a legitimate Copy-on-Write situation, and takes action.

The kernel's response, step by step:

Allocate a new physical page.
Copy the contents of the original shared page into the new page.
Update the child's page-table mapping to point to the new physical page.
Make the child's mapping writable.
Leave the parent's mapping pointing to the original physical page, unchanged.
Resume the faulting instruction. The child's write now completes against its own private copy. The parent is unaffected. The result:

AFTER CHILD WRITES

Parent VA ─────→ Physical Page X

Child VA ──────→ Physical Page Y
                  (copy of X)

The CPU detected the protection violation and signaled the kernel. The kernel performed the allocation, copy, and table update. The whole sequence is transparent to both processes. From their perspective, they each have independent writable memory. From the kernel's perspective, most of the memory is still shared.

Why This Is Faster Than Copying Everything

Return to the original question. A process with two gigabytes of memory calls fork().

Without Copy-on-Write, the kernel would need to allocate two gigabytes of new physical pages and copy every byte before the child can run. The child then calls exec() to load a different program entirely, replacing its address space. The two-gigabyte copy was unnecessary.

With Copy-on-Write, fork() sets up page-table structures and marks shared pages appropriately. The physical memory is not duplicated at all yet. When the child then calls exec(), it replaces its address space with the new program. Most of the parent's physical pages were never copied because the child never wrote to them.

The combination of fork() and exec() is an extremely common Unix pattern. A shell spawns a child and immediately executes another program. Copy-on-Write makes this practical: the expensive duplication that seems implied by fork() mostly never happens.

When a process does actually modify memory after fork(), only the specific pages that were written get their own copies. If a two-gigabyte process modifies twenty megabytes worth of data, the system copies roughly twenty megabytes, not two gigabytes. The work is deferred until it is actually necessary, and then limited to exactly what became necessary.

Page Faults Are Not Just Errors

This is worth a moment on its own because it contradicts what many developers learn early.

A page fault means the CPU encountered a memory access that could not be completed by a straightforward lookup in the page table. That can happen for several reasons:

The page has not been loaded into physical memory yet (demand paging).
The page needs to be brought in from disk.
The access violates a permission, which might indicate Copy-on-Write.
The access truly is invalid, in which case the kernel delivers a segmentation fault to the process. The kernel inspects each page fault and decides what to do. A Copy-on-Write fault is handled transparently. A demand-paging fault loads the page and resumes execution. Only if the access is genuinely illegal does the process receive a signal.

Page faults are the operating system's hook for managing virtual memory lazily. They allow the kernel to defer work until the hardware tells it work is necessary. Copy-on-Write is one application of this mechanism. Demand paging is another. The page fault itself is just the hardware saying: "I need your help with this access."

What Happens If Both Processes Keep Writing

Copy-on-Write does not eliminate copying. It defers and minimizes it.

If a parent and child both write extensively to shared pages, both will eventually end up with their own independent physical pages. The initial sharing was useful because much of the memory might never have been touched. But write-heavy workloads can result in the full cost eventually being paid.

Copying also happens at page granularity. If a process modifies one byte within a page, the entire page typically needs to be copied before that write can proceed. This means Copy-on-Write can occasionally produce more copying than a naive byte-level approach if many single-byte writes land on many different shared pages. The performance benefit depends on how much sharing remains useful over time.

The kernel needs to know when a physical page is no longer shared. It keeps track of how many virtual-page mappings reference each physical page. When a page fault triggers a Copy-on-Write, the kernel checks this reference information. If only one process still references the page, that process can take ownership directly without allocating a new page, because there is nothing else to protect.

Copy-on-Write as a General Pattern

The same conceptual approach appears throughout systems design wherever deferred copying is useful.

Filesystem snapshots can record the current state of a volume without immediately duplicating all its data. Blocks are shared between the original filesystem and the snapshot until one of them modifies a block. Then that block gets its own copy. The snapshot reflects the old state. The live filesystem reflects the new state. Storage is consumed only as changes accumulate.

Virtual machine snapshots work similarly. The snapshot and the running VM can share memory pages and disk blocks until modification requires divergence.

Some container storage systems and copy-on-write filesystems use similar strategies for efficient image layering.

The common thread: start with shared immutable state, detect modification, diverge on demand. The implementations differ, and the mechanism for detecting modification differs, but the logical structure is the same as process Copy-on-Write.

Security and Isolation

Copy-on-Write relies on the kernel correctly enforcing the boundary between shared physical pages and independent writable mappings. The protection comes from:

page-table permission bits that prevent writes through shared mappings
CPU hardware that enforces those permissions on every access
kernel code that correctly handles Copy-on-Write faults and updates page tables accurately If the kernel handles a Copy-on-Write fault incorrectly, a process might write to physical memory that another process is still reading as shared. That would violate process isolation. The correctness of Copy-on-Write is therefore a correctness requirement for the kernel's virtual-memory system, not just a performance optimization.

This is why bugs in Copy-on-Write handling can be serious. A race condition or incorrect reference count in the Copy-on-Write path could potentially allow one process to corrupt another's memory. The physical page sharing that makes Copy-on-Write efficient also makes correctness essential.

Performance in Practice

The performance characteristics of Copy-on-Write are worth being honest about.

Benefits: fork() completes quickly even for large processes. Initial memory consumption after fork() is much lower than if everything were copied immediately. Workloads that fork() and exec() immediately pay almost no copying cost at all.

Costs: Each page fault during a Copy-on-Write operation takes real time: the CPU traps into the kernel, the kernel allocates a new page, copies the old page's contents, updates the page table, and resumes execution. For workloads that write to many shared pages, these page faults accumulate. In latency-sensitive contexts, the occasional pause for page copying can be noticeable.

Copy-on-Write is an optimization for common cases, not a guarantee that memory never needs to be copied. Its benefit is proportional to how much sharing remains useful after fork(). For processes that immediately diverge by modifying many pages, it buys less than for processes that mostly read shared state or quickly call exec().

The Mechanism, Assembled

The apparent paradox at the start of the article has an answer now.

fork() creates a child process whose virtual address space looks identical to the parent's. The kernel does not immediately allocate new physical pages for everything. It arranges for the parent's and child's page-table entries to reference the same physical pages, marking those entries as not writable through those mappings. The processes have separate virtual address spaces, but temporarily share physical memory.

When either process writes to a shared page, the CPU detects the permission violation and raises a page fault. The kernel examines the fault, determines it is a Copy-on-Write situation, allocates a new physical page, copies the original contents, updates the faulting process's page-table entry to point to the new page, marks it writable, and resumes execution. The two processes now have independent copies of that page.

The abstraction works because the virtual-memory system gives the kernel complete control over what each process sees when it accesses a given virtual address. Physical pages can be shared or separated, read-only or writable, present or absent, all without the process necessarily knowing. The hardware enforces the permissions and notifies the kernel when something needs to change.

Copy-on-Write works because the operating system does not ask "what if this memory changes?" It asks the hardware to tell it when the change actually happens, and then does exactly as much work as that particular change requires.

What Is Jump-Oriented Programming? How Can Attackers Hijack Control Flow Without RET?

Aditya Sharma — Tue, 06 Oct 2026 06:05:41 +0000

Protecting return addresses is not the same as protecting control flow.

That statement might seem strange. Return addresses are the classic target of stack-based exploitation. Stack canaries detect their corruption. Shadow stacks maintain protected copies and validate every return. These are meaningful, well-motivated defenses.

But they protect a specific subset of control-flow transfers. A program has many ways to change where it executes next, and a return is only one of them.

Jump-Oriented Programming emerged as a response to exactly this gap. If return addresses become harder to abuse, what happens when an attacker builds malicious computation around a different primitive entirely?

Control Flow and Its Moving Parts

Execution in a program is not purely sequential. The instruction pointer advances through instructions, but branches, calls, and returns interrupt that sequence constantly.

Some of these transfers are direct: the destination is encoded in the instruction itself. A direct call to a known function, a direct jump to a labeled location, a direct branch over a conditional block. For a direct transfer, the destination encoded by the instruction is fixed rather than being read from attacker-controlled runtime state. Whether execution reaches that instruction can still depend on program state.

Indirect transfers are different. An indirect call reads a destination from a register or memory location and transfers execution there. An indirect jump does the same. The destination is not fixed in the instruction; it depends on whatever value is in that register or memory location at runtime.

This is what makes indirect transfers interesting from an exploitation perspective. If an attacker can influence the state that an indirect transfer reads, they can influence where execution goes. The instruction itself is correct and legitimate. The destination becomes attacker-controlled.

Returns are one example of an indirect backward transfer. They read a return address from the stack. But indirect jumps and indirect calls are forward-edge indirect transfers, and they are present throughout compiled code: virtual function dispatch, function pointers, computed gotos, jump tables for switch statements, callbacks stored in data structures.

Why ROP Mattered

Return-Oriented Programming established that an attacker does not need to inject new executable code to accomplish something useful. If the attacker can corrupt return addresses, they can chain together short sequences of existing instructions, each ending in a RET, by controlling what the stack contains. Each RET reads the next attacker-chosen address and transfers there. The chain assembles computation from fragments that already exist in legitimate executable memory.

This was significant because it bypassed NX and DEP entirely. Those protections mark data regions non-executable, blocking shellcode placed on the stack or heap. But ROP uses only existing executable code. There is nothing to mark non-executable.

Shadow stacks address ROP directly. By maintaining a protected copy of legitimate return addresses and comparing against them at each RET, a shadow stack can detect when the return target has been corrupted. If the value on the normal stack differs from the shadow-stack record, the return is blocked.

That protection is real. It specifically targets the primitive ROP depends on: the ability to substitute an attacker-chosen address wherever a RET expects to find a legitimate one.

Which raises an obvious question. Can code-reuse techniques be built without RET?

Jump-Oriented Programming

Jump-Oriented Programming is a code-reuse technique that chains existing instruction sequences through indirect jumps rather than returns. The attacker does not need to corrupt return addresses for each transition. Instead, each gadget ends in an indirect jump, and the destination of that jump can be influenced by attacker-controlled register or memory state.

Like ROP, JOP requires no new executable code. It reuses instructions already present in legitimate executable memory. The program's existing code, and the libraries it links against, contain indirect-jump instructions throughout their compiled output. Some of those can be useful gadgets.

The important conceptual difference:

ROP chains gadgets by controlling what appears on the stack. Each RET pops the next address.

JOP chains gadgets by controlling what register or memory value an indirect jump uses as its destination. Each JMP [register] goes wherever that register points.

A shadow stack that validates return addresses does not inherently validate the destinations of indirect jumps. Those are a different category of control-flow transfer, operating through a different mechanism. A defense scoped to backward-edge returns does not automatically cover forward-edge indirect jumps.

What Makes a Useful JOP Gadget

A useful gadget for JOP performs some computation and then reaches an indirect jump whose target the attacker can influence.

That second requirement is the constraint. Not every indirect jump is usable. If the jump reads from a register that the attacker cannot control at the point the gadget executes, it is not useful. The attacker needs some primitive that lets them set up machine state, specifically the register or memory location that the gadget's indirect jump will use, in advance of reaching that gadget.

This is substantially more complex than finding ROP gadgets. In ROP, the chaining mechanism is the stack. The stack is easy to reason about: each RET reads from the current stack pointer, and the attacker controls the stack layout. Every gadget inherits the same chaining mechanism.

In JOP, gadgets may read from different registers. Maintaining useful state across the chain requires careful attention to which gadgets modify which registers and whether the next gadget's indirect jump target can still be set correctly by the time execution reaches it.

The Dispatcher

This complexity is why JOP introduces a design concept that ROP does not need: a dispatcher gadget.

In a ROP chain, the chaining mechanism is automatic. RET reads from the stack. As long as the stack is controlled, every gadget chains to the next without additional structure.

In a JOP chain, the attacker needs a way to repeatedly determine which gadget executes next. A dispatcher is a gadget that uses attacker-controlled state to select and transfer execution to the next gadget in the chain. The dispatcher does not perform useful computation itself. Its job is to advance the chain.

Conceptually, a dispatcher might use a register pointing into a table of gadget addresses and advance through that table on each dispatch. The computation gadgets perform useful operations, then return control to the dispatcher through an indirect jump. The dispatcher then selects the next computation gadget.

Attacker-controlled dispatch state
           ↓
     Dispatcher gadget
           ↓
    Computation gadget A
    (performs operation, ends in indirect jump)
           ↓
     Dispatcher gadget
           ↓
    Computation gadget B
           ↓
     Dispatcher gadget
           ↓
    Computation gadget C

This structure requires finding both useful computation gadgets and a suitable dispatcher. The dispatcher has to be usable: it has to read attacker-controlled state to select targets, and those targets have to be reachable through an indirect jump. Finding a dispatcher in a real binary is often harder than finding computation gadgets.

JOP vs ROP

The comparison is worth making precisely.

ROP depends on corrupting return addresses. The stack is the chaining medium. Each RET pops the next gadget address. Shadow stacks protect this: by maintaining a protected independent record of legitimate return addresses, they can catch the substitution.

JOP uses indirect jumps. Return addresses are not the chaining mechanism, so a defense specifically protecting return addresses does not inherently address the indirect jumps JOP relies on. This does not mean JOP bypasses all defenses, but it does mean it operates through a different class of control-flow transfer.

The defenses relevant to JOP are those that protect indirect branches: CFI policies constraining valid indirect-jump targets, and hardware mechanisms that enforce constraints on where indirect jumps can go.

Shadow Stacks and CFI Are Different Things

This distinction is worth stating clearly because the shadow-stack article and this one are closely connected.

Shadow stacks protect backward-edge control flow. A return should go to where the corresponding call came from. The shadow stack records that expected destination and validates it.

CFI is broader. It attempts to constrain control-flow transfers to destinations that are valid according to a policy or control-flow graph. Depending on implementation, this can cover indirect calls, indirect jumps, and returns. The question CFI asks is "is this destination allowed?" not "does this return match its corresponding call?"

A system with shadow stacks but no forward-edge CFI has strong protection for returns and limited protection for indirect jumps. A system with CFI but no shadow stacks might constrain indirect-jump targets while leaving return addresses less specifically protected.

JOP is primarily a problem for forward-edge protection. Shadow stacks do not directly constrain where indirect jumps can go. CFI does, to varying degrees depending on how precise the policy is.

How CFI Changes JOP

CFI constrains which destinations are valid for indirect control transfers. When applied to indirect jumps, it reduces the set of usable JOP gadgets.

A coarse-grained CFI policy might allow indirect jumps to land on any function entry point. That still leaves many potential gadget targets across a binary and its libraries. The set of usable gadgets shrinks compared to an unconstrained binary, but it may not shrink to zero.

A fine-grained CFI policy tries to restrict each indirect-jump site to a much smaller set of legitimate targets based on the program's intended structure. This can reduce the availability of JOP gadgets significantly, since many instruction sequences the attacker might want to reach are not in the permitted set for any indirect-jump site.

The protection depends on implementation. Fine-grained CFI requires accurate static analysis of the program's intended control-flow structure, has compatibility constraints with dynamic behavior, and carries some overhead. Real deployments involve tradeoffs.

The concept of control-flow bending describes a scenario where an attacker constructs an attack using only destinations that satisfy the CFI policy. With a coarse policy, enough usable-but-unintended targets may still exist. With a fine policy, the available set shrinks enough to make construction extremely difficult.

Intel CET and IBT

Intel CET provides two distinct control-flow protection mechanisms, and keeping them separate matters here.

The Shadow Stack component protects backward-edge control flow by tracking legitimate return addresses.

Indirect Branch Tracking (IBT) addresses a different problem. IBT requires that valid indirect branch targets be preceded by an ENDBR instruction. An indirect jump or call that lands on a location without ENDBR raises a control-protection fault.

IBT is relevant to JOP because JOP relies on indirect jumps. If every valid indirect-branch destination is required to carry an ENDBR marker, the attacker needs gadgets that begin with or include ENDBR, and any gadget not at an intended indirect-branch target cannot be reached without triggering a fault.

This reduces the pool of usable JOP gadgets. Instruction sequences in the middle of functions, or at locations that compilers never intended to be indirect-branch targets, become unreachable through protected indirect jumps.

IBT and the Shadow Stack are complementary. Shadow Stack covers backward edges. IBT constrains forward-edge indirect branches. A system deploying both addresses more of the control-flow surface than either does alone.

JOP in the Code-Reuse Landscape

Code-reuse techniques have evolved as defenses developed. Return-to-libc predates ROP: instead of chaining gadgets, redirect execution to an existing function like system. ROP generalized this, enabling arbitrary computation from gadget fragments. SROP exploited the signal-return mechanism for bulk state restoration. JOP extends the code-reuse idea to indirect jumps when return-address defenses make ROP harder.

None of these techniques is universally applicable. Each requires specific primitives: the ability to corrupt the relevant control-flow data, some way to influence the destination of the relevant indirect transfer, and enough usable fragments in the target binary to construct something useful.

JOP is generally more difficult to construct than ROP. Finding a usable dispatcher, maintaining consistent register state through a chain that does not have a uniform chaining mechanism, and doing all of this while satisfying any CFI or IBT constraints imposed by the target, makes JOP construction a harder engineering problem than building a ROP chain in an unprotected binary.

Why Control Flow Has to Be Treated as a System

The defense against JOP is not one thing. ASLR makes gadget addresses harder to predict. NX blocks executing injected code. Stack canaries detect some stack corruption. Shadow stacks protect return addresses specifically. CFI constrains indirect-transfer targets. IBT marks valid indirect-branch landing sites. Memory-safe languages eliminate the memory-corruption primitives that make control-flow hijacking possible in the first place.

Each mechanism addresses a different part of the problem. ASLR addresses knowledge of addresses. NX addresses code injection. Shadow stacks address return-address substitution. CFI and IBT address the destinations of indirect transfers. None of them, individually, closes the entire attack surface.

JOP illustrates this clearly. It does not defeat shadow stacks: it simply operates through a class of control-flow transfer that shadow stacks are not designed to protect. That is not a flaw in shadow stacks. It is a reminder that a defense designed around one assumption does not automatically handle every other assumption.

The Point

Protecting RET does not mean you have protected control flow.

This is the deeper architectural insight. A program's execution can be influenced through returns, but also through indirect jumps and indirect calls. Those transfers all read destinations from program state that can potentially be corrupted. Protecting one class of them, through whatever mechanism, leaves the others as potential attack surface.

Security mechanisms are scoped to specific assumptions about how execution can be manipulated. Shadow stacks assume the threat is return-address corruption. CFI assumes the threat is indirect-transfer destinations outside the intended control-flow graph. IBT assumes the threat is indirect branches to unintended locations.

When attackers encounter a defense targeting one primitive, they look for another. JOP is what that looks like when the primitive being avoided is RET. The response is not to find a single new defense that covers everything, but to build the layered combination of mechanisms that closes more of the control-flow surface together than any of them does alone.

DEV Community: Aditya Sharma

How Linux Handles a `fork()` When the Parent Has Gigabytes of Memory

What fork() Actually Creates

What Happens Inside the Kernel

Copy-on-Write in Detail

Where the Real Cost Actually Appears

A Reproducible Experiment

Common Misconceptions

Engineering Implications

The Core Principle

What Is a Zombie Process in Linux? How Processes Become Stuck in the Process Table

1. What Exactly Is a Zombie Process?

2. How a Process Becomes a Zombie

Step 1: The child executes

Step 2: The child terminates

Step 3: The kernel retains the process record

Step 4: The parent collects the exit status

3. What Happens Inside the Linux Kernel?

4. Let's Create a Zombie Process in Linux

5. Why Does Linux Need wait() and waitpid()?

wait()

waitpid()

6. Why Zombie Processes Can Become a Problem

7. How to Find Zombie Processes

Use ps

Use top

Inspect the parent process

8. Can You Kill a Zombie Process?

9. Zombies vs. Orphan Processes

10. How Developers Should Prevent Zombies

Always collect child statuses

Handle multiple children correctly

Use the right subprocess abstraction

Test failure paths

Monitor process state

The Bigger Picture

What is Overfitting: How Can an AI Model Score 99% on Training Data and Still Fail in the Real World?

1. The Difference Between Learning and Memorizing

2. Why Does Overfitting Happen?

An Overly Complex Model

Too Little or Unrepresentative Training Data

Training for Too Long

Noisy Labels and Accidental Patterns

3. Training Accuracy Can Be Misleading

4. The Hidden Danger: Data Leakage

5. How Do We Reduce Overfitting?

Collect More Representative Data

Use Regularization

Use Data Augmentation

Use Early Stopping

Simplify the Model When Necessary

6. The Opposite Problem: Underfitting

7. Why Passing a Test Set Still Doesn't Guarantee Success

The Bigger Picture

Escape Analysis: How Can a Compiler Eliminate a Heap Allocation Without Changing Your Code?

What It Means for an Object to Escape

How the Compiler Reasons About References

The Optimization: Scalar Replacement

A Concrete Walk-Through

Where the Analysis Fails

The Performance Picture

The Developer's Relationship With This Optimization

The Deeper Insight

Sources

NTLM Relay Attacks: How Can Attackers Authenticate as You Without Knowing Your Password?

NTLM Doesn't Send Your Password

The Attack: Let Someone Else Authenticate for You

Why Does the Relay Work?

When a Relay Becomes a Bigger Problem

Why Changing the Password Isn't Enough

SMB Signing

Extended Protection for Authentication

Reduce NTLM Usage

Harden Services and Limit Privileges

The Real Lesson Behind NTLM Relay

Sources

What Is BYOVD? How Can a Signed Driver Give Malware Kernel-Level Access?

The Kernel Is Not Just Another Layer

What a Signature Proves, and What It Does Not

The Attack, Step by Step

What `fork()` Actually Creates

5. Why Does Linux Need `wait()` and `waitpid()`?

`wait()`

`waitpid()`

Use `ps`

Use `top`

Compilers and `volatile`

What `fork()` Actually Does to Memory