DEV Community: Walter Hrad

How Operating Systems Schedule Processes

Walter Hrad — Sun, 26 Jul 2026 12:30:18 +0000

Your computer is running dozens of programs right now. A browser, a terminal, a music player, background daemons, system services. If you open a task manager and count, the number of processes is usually somewhere between a hundred and several hundred on a typical desktop system. On a server it can be in the thousands.

Your CPU has a fixed number of cores. A modern consumer machine might have eight or sixteen. A server might have sixty-four or one hundred twenty-eight. The number of processes almost always exceeds the number of cores, often by a factor of ten or more.

Something has to decide which processes run on which cores and for how long. That something is the scheduler, and it is one of the most consequential pieces of software running on your machine. Every response time you experience, every latency spike, every moment where your music skips while something else is happening, every time a background task quietly finishes without you noticing: all of it reflects decisions the scheduler made, thousands of times per second, invisibly.

This is an attempt to explain how those decisions actually get made.

What scheduling is solving

The fundamental problem is simple to state. You have N processes and M cores, where N is much larger than M. At any given moment, you have to pick which M processes are running and which are waiting. You also have to decide when to switch, because switching too rarely makes interactive programs feel sluggish, and switching too often wastes CPU time on the overhead of switching itself.

The tricky part is that different processes have wildly different needs.

A video player needs to decode and render frames on a predictable schedule. Missing a deadline means a dropped frame. The work it needs to do is not huge, but it needs to happen at regular intervals without fail.

A database running a complex query needs sustained CPU time. Interrupting it frequently to let other things run just means the query takes longer. It does not care about latency in the way an interactive program does.

A text editor sitting idle needs almost no CPU at all. When you press a key it needs to respond immediately, but between keystrokes it should consume nothing.

A background compression job needs CPU when nothing else does. It should get out of the way when something more urgent needs the core.

No single scheduling policy handles all of these well simultaneously. Real schedulers are multi-part systems that make different decisions for different classes of workloads and try to keep everyone reasonably satisfied.

The basics: time slices and context switches

The core mechanism of a preemptive scheduler is the time slice, sometimes called a quantum. The scheduler assigns each process a time slice, a maximum amount of CPU time it gets before being preempted and replaced with another process.

When a time slice expires, the hardware timer fires an interrupt. The CPU stops executing the current process, saves its state (registers, program counter, stack pointer, everything needed to resume it later), and transfers control to the kernel's scheduler code. The scheduler picks the next process to run, restores that process's state, and resumes execution. From the process's perspective, nothing happened. It just ran for a while, then ran some more later. The gap in the middle is invisible to it.

This save-and-restore operation is a context switch. Context switches are not free. Saving and restoring registers takes a small number of cycles. More significantly, switching processes means the new process's data is likely not in cache. The CPU caches that were warm for the previous process are now filled with data that is useless for the new one. The new process will incur cache misses until its working set loads into cache. On a modern processor where cache misses can cost hundreds of cycles, this cache warming cost can be substantial.

This is why time slices are not arbitrarily short. If you set a 1 microsecond time slice on a 3 GHz processor, you get 3000 cycles of useful work followed by a context switch that might cost 10,000 cycles in cache misses alone. The overhead dwarfs the useful work. Real schedulers use time slices in the range of a few milliseconds to tens of milliseconds, balancing responsiveness against overhead.

The time slice length is also not universal. Different processes and different situations call for different slice lengths. Interactive processes often get shorter slices so they can respond quickly to input and then yield. CPU-bound background processes often get longer slices to minimize context switch overhead and let them make sustained progress.

Preemptive versus cooperative scheduling

There are two broad approaches to scheduling: cooperative and preemptive.

In a cooperative scheduler, processes voluntarily yield the CPU when they are done with their current work or when they choose to wait for something. The scheduler only gets control when a process explicitly gives it up. Early versions of Windows and the classic Mac OS used cooperative scheduling. The obvious problem is that a misbehaving or buggy process that never yields can freeze the entire system, because the scheduler never gets a chance to run anything else.

In a preemptive scheduler, the operating system can forcibly take the CPU away from a running process at any time, using a hardware timer interrupt. The process does not get a vote. This is what all modern general-purpose operating systems use. It means that no single process can monopolize a CPU core indefinitely, and the operating system remains in control of resource allocation regardless of what individual processes do.

Preemption requires hardware support: a timer that generates interrupts at configurable intervals, and a mechanism for those interrupts to transfer control to privileged kernel code even when user code is running. Both of these have been standard in hardware since the 1970s. The concept of preemptive multitasking is not new, but the sophistication of the policies built on top of it has grown enormously.

Priority: not all processes are equal

The simplest form of scheduling policy is round-robin: give each process a time slice, then move to the next, cycling through all processes continuously. Round-robin is fair in the sense that every process gets equal time, but fairness is not always what you want. A background indexing job should not consume the same share of CPU as the video call you are actively participating in.

Priority changes this. Each process has a priority value, and higher-priority processes get scheduled preferentially over lower-priority ones. In the simplest version, the scheduler always runs the highest-priority runnable process. A lower-priority process only runs if no higher-priority process wants the CPU.

Pure strict priority scheduling has a problem called starvation. If there are always high-priority processes that want to run, low-priority processes never get CPU time. They starve. A background task assigned a low priority on a heavily loaded system might not run for hours or at all.

The fix is aging: a process that has been waiting for a long time gets its effective priority gradually increased, until it eventually gets scheduled. Once it runs, its priority returns to its base level. This prevents starvation without abandoning the concept of priority entirely.

On Linux, user processes have priorities called nice values, ranging from negative 20 (highest priority) to positive 19 (lowest priority). The name comes from the idea that a process with a high nice value is being nice to other processes by voluntarily accepting less CPU time. Normal processes run at nice value 0. You can start a process with a specific nice value using the nice command, or change a running process's nice value with renice. The kernel translates nice values into internal scheduling weights.

The difference between I/O-bound and CPU-bound processes

Understanding how schedulers actually behave requires understanding the distinction between I/O-bound and CPU-bound processes.

A CPU-bound process is one that mostly wants to compute. It rarely waits for anything external. Given CPU time, it uses all of it. Compiling code, running numerical simulations, transcoding video: these are CPU-bound workloads.

An I/O-bound process is one that spends most of its time waiting for input or output. A web server waiting for network requests, a text editor waiting for keystrokes, a database process waiting for disk reads. These processes are runnable only intermittently. Most of the time they are blocked, waiting for something to happen.

This distinction matters enormously for scheduling. A CPU-bound process, if given a long time slice, will use all of it productively. An I/O-bound process, if given a long time slice, will use a tiny fraction of it and then block while waiting for I/O. When the I/O completes and the process becomes runnable again, you want it to run quickly, because the user or the network or whatever was waiting for its response is waiting.

Good schedulers detect this distinction and use it. A process that consistently uses its entire time slice before blocking is probably CPU-bound. A process that consistently blocks well before its time slice expires is probably I/O-bound. I/O-bound processes are rewarded with higher scheduling priority or faster scheduling after their I/O completes, because getting them back on the CPU quickly is what makes interactive programs feel responsive.

Linux's scheduler tracks how long each process has been waiting versus how long it has been running, and uses this to adjust scheduling decisions. A process that has been sleeping (waiting for I/O) for a long time gets scheduled quickly after it wakes up, because the assumption is that it needs to respond to something and every millisecond of additional waiting is latency that the user or the system will notice.

The runqueue and how the scheduler picks what runs next

Internally, the scheduler maintains data structures that track the state of every process. The key categories are:

Running: the process is currently executing on a CPU core.

Runnable: the process has work to do and is ready to run, but is waiting for a CPU core to become available.

Sleeping: the process is waiting for something, typically I/O, a timer, or a synchronization event. It is not consuming CPU and should not be scheduled until whatever it is waiting for happens.

Zombie: the process has exited but its exit status has not yet been collected by its parent. It exists only as a record.

The scheduler only concerns itself with runnable processes. Its job is to pick from the runnable pool and decide who runs on each core.

The data structure holding runnable processes is the runqueue. Naive implementations use a single sorted list of processes ordered by priority. The scheduler picks the head of the list each time. The problem is that inserting into and removing from a sorted list takes time proportional to the list length, and when the runqueue has thousands of entries, this overhead adds up across thousands of scheduling decisions per second.

Real schedulers use more efficient data structures. Linux's Completely Fair Scheduler uses a red-black tree, a self-balancing binary search tree that supports insertion, deletion, and minimum lookup all in O(log n) time. With thousands of processes, this is the difference between acceptable overhead and a scheduler that itself becomes a bottleneck.

The Completely Fair Scheduler

Linux's main scheduler for normal processes is called the Completely Fair Scheduler, or CFS. It was introduced in 2007 and replaced a previous O(1) scheduler that, despite its efficiency, had problems with fairness and interactivity.

The central concept in CFS is virtual runtime. Each process has a virtual runtime counter that tracks how much CPU time it has consumed, adjusted for its weight (which reflects its nice value). Higher-weight processes accumulate virtual runtime more slowly for the same amount of real CPU time, which means they stay near the front of the scheduling queue and get scheduled more often. Lower-weight processes accumulate virtual runtime faster and drop to the back of the queue more quickly.

The scheduler always picks the process with the lowest virtual runtime to run next. In theory, if all processes had equal weight and ran continuously, their virtual runtimes would converge: every process would get exactly equal CPU time, and the system would be perfectly fair. In practice, processes sleep and wake, have different weights, and vary in how much CPU they actually use, so the virtual runtime values diverge and reconverge continuously as the scheduler responds to what each process actually does.

When a process wakes up from sleep, CFS does not bring it back with the same virtual runtime it had when it slept. If it did, a process that slept for a long time would have a virtual runtime far below everything else and would immediately monopolize the CPU, trying to catch up on all the time it missed. Instead, CFS sets the waking process's virtual runtime to something close to the minimum virtual runtime currently in the runqueue. This gives it a scheduling boost (it will be picked to run soon) without letting it steal disproportionate time.

The time slice in CFS is not a fixed value. It is derived from the number of runnable processes and the target latency: the maximum time within which every runnable process should get at least one turn. If the target latency is 6 milliseconds and there are 3 runnable processes, each gets a 2 millisecond slice. If there are 60 runnable processes, each gets a 100 microsecond slice, which is very short and would cause excessive context switch overhead. CFS introduces a minimum granularity (typically 0.75 milliseconds) to prevent slices from becoming too small, accepting that as the number of processes grows, the latency guarantee degrades gracefully rather than catastrophically.

CFS handles the I/O-bound versus CPU-bound distinction through the wakeup mechanism. When an I/O-bound process wakes up, it has a low virtual runtime relative to the current minimum (because it has not been consuming CPU). It gets scheduled promptly. When it runs, it does a small amount of work and blocks again before its time slice expires. Its virtual runtime barely increases. The next time it wakes up, it is still near the front of the queue. The net result is that I/O-bound processes naturally get low latency without the scheduler needing to explicitly classify them.

Real-time scheduling

CFS handles normal processes. But some workloads need stronger guarantees than "you will run within the next few milliseconds on a loaded system." Audio processing, kernel drivers, industrial control systems, anything with hard timing requirements needs a different approach.

Linux provides real-time scheduling policies for this: SCHED_FIFO and SCHED_RR.

SCHED_FIFO is a first-in, first-out policy for real-time processes. A SCHED_FIFO process runs until it voluntarily yields, blocks, or is preempted by a higher-priority real-time process. There is no time slice. If it does not yield, it keeps running. This means a SCHED_FIFO process set to the highest real-time priority can, if it runs continuously and never blocks, prevent all other processes on that CPU from running at all. This is by design: real-time processes are supposed to be trusted to behave well and not abuse this power.

SCHED_RR adds a time slice to SCHED_FIFO. Real-time processes of the same priority take turns, each getting a fixed slice before yielding to the next. Processes at higher priority still preempt processes at lower priority immediately.

Real-time processes in Linux have priorities in the range 1 to 99, entirely separate from and above the nice value system used for normal processes. A process at any real-time priority level will always preempt a normal CFS process, regardless of that normal process's nice value. You need root (or the CAP_SYS_NICE capability) to set a process to a real-time scheduling policy, because a misconfigured real-time process can lock up a system.

The kernel also provides a real-time throttling mechanism: a configurable fraction of CPU time (95 percent by default) is reserved for real-time processes, with the remaining 5 percent reserved for normal processes. This prevents a runaway real-time process from making the system completely unresponsive, though it reduces the latency guarantees you can make for real-time workloads.

Multicore scheduling and the cache affinity problem

Everything so far has discussed scheduling as if there is one CPU. Real systems have multiple cores, and this introduces a new set of problems.

The obvious approach is to maintain one global runqueue and have each core pull from it whenever it needs work. This works but creates contention: every scheduling decision on every core requires acquiring a lock on the shared runqueue. On a system with many cores all scheduling frequently, this lock becomes a bottleneck.

The more common approach is per-core runqueues. Each core has its own runqueue and schedules from it independently. This eliminates the global lock. The problem is load imbalance: one core might have ten runnable processes while another sits idle.

Load balancing fixes this. At regular intervals, and when a core becomes idle, the scheduler looks across all runqueues and migrates processes from overloaded cores to underloaded ones. This balances work across the system but introduces its own cost: migrating a process from one core to another means that process's cache data, which was warm on the original core, is now cold on the new core. The process will incur cache misses until it warms up the new core's caches.

This is the cache affinity problem. All else being equal, it is better to keep a process on the same core it has been running on, because its data is more likely to still be in that core's caches. The scheduler tries to respect this. When it balances load, it prefers to move processes that have been sleeping (their caches are cold anyway) rather than processes that have been actively running. It migrates work only when the imbalance is significant enough to justify the cache disruption cost.

Modern processors also have non-uniform memory access (NUMA) architectures, where cores are grouped into nodes, and each node has its own local memory that is faster to access than memory attached to other nodes. A process running on a core in node 0, accessing memory that was allocated on node 1, pays a latency penalty on every memory access compared to accessing node 0's memory. Linux's scheduler is NUMA-aware and tries to keep processes on the same node as their memory, balancing load within a node before resorting to cross-node migration.

The scheduler and system calls

Scheduling is not just about the periodic timer interrupt. Processes frequently put themselves to sleep voluntarily, and the scheduler has to handle this correctly.

When a process calls read on a file and the data is not in cache, the process has to wait for the disk. It calls into the kernel, which starts the I/O operation and then calls the scheduler to pick something else to run. The calling process is moved from runnable to sleeping. When the I/O completes, a hardware interrupt fires, the interrupt handler marks the sleeping process as runnable again (placing it back on the runqueue), and the scheduler will pick it up the next time it makes a scheduling decision.

The same sequence happens for network I/O, for waiting on locks held by other processes, for sleeping via the sleep system call, for waiting on a child process to exit, and for dozens of other blocking operations. In every case, the process voluntarily gives up the CPU by making a system call that blocks, and the scheduler fills that core with something else while it waits.

This voluntary yielding is what makes I/O-bound processes efficient users of the system. A process that is waiting for the network is not consuming CPU. The core is free for something else. A CPU-bound process that never blocks never voluntarily yields, so it only gives up the CPU when the timer fires and the scheduler preempts it.

Scheduler latency and what it means for applications

Scheduler latency is the time between a process becoming runnable and the time it actually starts running. For interactive applications, this directly translates to responsiveness. If you press a key and the text editor is runnable but the scheduler does not get to it for 50 milliseconds, you will notice.

CFS's target latency tries to bound this. With a 6 millisecond target and a moderate number of processes, every runnable process should get a turn within 6 milliseconds. In practice, system load, real-time processes, and other factors can push this higher.

For applications that care about latency, there are tunable kernel parameters. The sched_latency_ns and sched_min_granularity_ns parameters control CFS's target latency and minimum granularity. Reducing them makes CFS switch more frequently, reducing latency at the cost of more context switch overhead. Increasing them reduces overhead at the cost of higher latency.

The kernel also provides scheduling hints through control groups (cgroups), which allow you to assign processes to groups and control the share of CPU time each group gets. A group of high-priority processes can be given 80 percent of CPU time, with everything else sharing the remaining 20 percent. This is how container runtimes like Docker implement CPU limits: they put each container's processes in a cgroup and set a CPU quota.

Priority inversion and how it is handled

Priority inversion is a scheduling problem that emerges when priorities and locks interact in unexpected ways.

Imagine a low-priority process L holds a mutex. A high-priority process H tries to acquire the same mutex and blocks, because L has it. Now a medium-priority process M becomes runnable. The scheduler runs M instead of L, because M has higher priority than L, and H is blocked so it cannot run anyway. L never gets scheduled to finish its work and release the mutex, because M keeps the CPU. H, the highest-priority process, is stuck waiting for L, which is stuck waiting for M to stop running.

The high-priority process has effectively been degraded below the medium-priority process, which is backwards. This is priority inversion.

The canonical solution is priority inheritance. When H blocks waiting for a mutex held by L, L temporarily inherits H's priority. L is now treated as a high-priority process and gets scheduled ahead of M. L finishes, releases the mutex, its priority returns to its original value, and H can run. The inversion is avoided.

Priority inheritance adds complexity to the locking subsystem and has its own edge cases (priority inheritance with multiple locks can create chains that are expensive to maintain), but it is widely implemented because priority inversion can cause severe problems. The Mars Pathfinder mission in 1997 experienced a real-time system reset caused by priority inversion in its onboard software, which contributed to interest in understanding and solving the problem in embedded and real-time systems.

The idle task and what happens when nothing needs to run

Every core always has something to run, even if that something is doing nothing. The scheduler has a special idle task for each core that runs whenever no other process is runnable on that core.

The idle task is not a normal process. It is a kernel thread that runs at the lowest possible priority and executes a loop that repeatedly asks the hardware to enter a low-power state. On modern processors, this is done with instructions like HLT on x86, which halt the core until the next interrupt arrives. The interrupt wakes the core, the interrupt handler runs, and if something became runnable (I/O completed, a timer fired, a process was migrated from another core), the scheduler picks it up. If nothing became runnable, the core returns to the idle loop.

This is how modern CPUs achieve power efficiency under low load. Cores that have nothing to do are in a low-power sleep state most of the time, waking only when work arrives or when maintenance tasks run. Power management goes further than this (cores can reduce their clock frequency, reduce voltage, and enter progressively deeper sleep states), and the scheduler interacts with the power management system to make these decisions well: a core that the scheduler knows will be needed again soon should not enter a deep sleep state that takes a long time to exit.

What all of this means from the outside

You do not normally need to think about the scheduler. It makes its decisions quietly and the results are usually good enough. But knowing how it works changes how you interpret certain things.

When your system feels sluggish despite low CPU usage, the problem is often scheduling latency. Some process that matters to you is runnable but not getting scheduled quickly. This can happen when a high-priority real-time process is holding the CPU, when the target latency is too high for the number of runnable processes, or when a NUMA migration has degraded a process's memory access patterns.

When a program that should be fast is not, and profiling shows that it is spending time sleeping rather than computing, the issue might be that it is making many small I/O calls, each of which puts it to sleep and pays the wakeup latency. Batching those calls into fewer, larger ones reduces the number of times the process goes through the sleep and wake cycle.

When CPU usage on a multicore system is not evenly distributed, load balancing is not working as well as it could. This can happen when processes have strong cache affinity and the scheduler is reluctant to migrate them, or when the workload is not easily parallelizable and one core ends up with all the work.

When a low-priority background task seems to be taking CPU away from a more important foreground process, the nice value system lets you explicitly tell the scheduler to deprioritize it. Running a compilation job at nice value 19 means it will only get CPU when nothing else wants it, without requiring you to carefully manage when it runs.

The scheduler is not magic. It is a set of algorithms making decisions based on limited information about what each process needs and what the system should optimize for. Those decisions are usually correct. When they are not, knowing what the scheduler is actually doing is what lets you diagnose and fix the problem rather than blaming the computer for being slow.

The gap between simple and real

Textbook descriptions of scheduling make it sound straightforward. Runqueue, time slice, context switch, repeat. The reality is considerably more involved. Cache affinity, NUMA topology, real-time guarantees, priority inversion, power management, cgroup quotas, load balancing across heterogeneous core types (which matters on processors with efficiency cores and performance cores, like ARM big.LITTLE and Intel's hybrid architecture): all of this has to be handled correctly and efficiently, in software that runs thousands of times per second on every machine in the world.

Linux's scheduler is several thousand lines of code and has been actively developed for decades. It has gone through multiple complete rewrites as hardware has changed and as understanding of the tradeoffs has improved. The version running today reflects lessons learned from systems that failed in production, benchmarks that exposed unexpected regressions, and careful study of where the previous approaches were wrong.

The scheduler that decides when your program runs is one of the most carefully engineered pieces of software in existence, and it does its job so well that most programmers never have to think about it. That invisibility is the point. The goal is not to be noticed. The goal is to keep everything running smoothly, allocating a finite resource fairly and efficiently, without any of the thousands of processes running on your machine ever knowing how much work went into the decision.

How Virtual Memory Works and What Your OS Is Hiding From Your Program

Walter Hrad — Fri, 17 Jul 2026 06:31:29 +0000

Every program you have ever written has been lied to. Not maliciously, and not in a way that causes you harm. But from the moment your program starts running to the moment it exits, the operating system is maintaining an elaborate fiction about the nature of memory, and your program has no idea.

The fiction is this: your program believes it has access to a large, flat, contiguous address space that belongs entirely to it. On a 64-bit system, that address space is theoretically 16 exabytes. Your program can read from address 0x1000, write to address 0x7fffffffe000, and jump to code at address 0x400000, and it does so with the complete confidence that those addresses are real, that the memory at those addresses is yours, and that nothing else is touching it.

None of that is true in the way your program thinks it is. The addresses are not real physical memory addresses. The memory is not necessarily yours in the sense of being physically reserved. Other programs are running in what appears to be the same address space but is not. The OS is intercepting every single memory access and translating it, in hardware, before your program sees the result.

Understanding how this works is not just an academic exercise. It explains behavior that is otherwise confusing: why processes are isolated from each other, how memory mapped files work, what a page fault actually is, why malloc does not immediately consume RAM, how copy-on-write makes fork fast, what a segmentation fault really means at the hardware level, and why certain access patterns are dramatically slower than they should be.

The problem virtual memory solves

To understand why virtual memory exists, you need to understand the world before it.

In the early days of computing, programs ran directly on physical memory. A program was loaded at some physical address, and it used physical addresses to access its data. This created several serious problems.

The first was relocation. If your program was written to run at physical address 0x1000, and something else was already loaded there, your program could not run. Every program had to be written to run at a specific location, or it had to be compiled with relocation in mind and then adjusted at load time, which was complex.

The second was isolation. If two programs were running simultaneously and sharing physical memory, there was nothing preventing one program from reading or writing the memory of the other. A bug in one program could corrupt another. A malicious program could read passwords out of another program's memory. The only security was trusting that programs were well-behaved.

The third was fragmentation. As programs were loaded and unloaded, physical memory developed holes. You might have 100 megabytes of free physical memory but no contiguous 50-megabyte region to load a program that needed 50 megabytes. The memory existed but was not usable.

The fourth was the mismatch between program needs and available RAM. A program might need more memory than the machine physically had. Without some mechanism to handle this, programs were simply limited to what physically existed.

Virtual memory was designed to solve all of these problems simultaneously. It does so by inserting an indirection layer between the addresses a program uses and the physical memory those addresses correspond to.

Pages and page tables

The fundamental unit of virtual memory is the page. A page is a fixed-size block of memory, typically 4096 bytes (4 kilobytes) on most systems, though some architectures support larger page sizes. All memory management in a virtual memory system happens at page granularity. You never map a single byte. You map a page, which means 4096 bytes all mapped together.

The data structure that records the mapping from virtual pages to physical pages is the page table. Every process has its own page table. The page table is a structure in physical memory, maintained by the operating system, that the hardware uses to translate virtual addresses to physical addresses.

When your program accesses a virtual address, the hardware does not use that address directly. Instead, it looks up the virtual page number (derived from the address) in the page table, finds the corresponding physical page number, combines that with the offset within the page, and produces the physical address that actually goes onto the memory bus.

A virtual address on a 64-bit system with 4KB pages breaks down like this: the bottom 12 bits are the page offset (since 2^12 = 4096, these bits index into the page). The remaining bits above that identify the virtual page number. The translation hardware looks up the virtual page number in the page table, retrieves the physical page number, and appends the original offset bits to form the physical address.

This translation happens for every single memory access. Every instruction fetch, every data load, every data store. The hardware does this transparently, in a few nanoseconds, without any involvement from your program. Your program issues a virtual address, and the physical address is what actually happens.

The translation lookaside buffer

If every memory access required looking up the page table, and the page table itself is in memory, every memory access would require at least one additional memory access just for the translation. That would roughly halve memory throughput. For most programs it would be far worse than halving, because page tables are hierarchical structures that require multiple lookups.

The solution is the Translation Lookaside Buffer, or TLB. The TLB is a small, fast cache built into the CPU that stores recent virtual-to-physical page mappings. When the hardware needs to translate a virtual address, it checks the TLB first. If the mapping is there (a TLB hit), the translation is done in a single cycle or two. If the mapping is not there (a TLB miss), the hardware has to walk the page table in memory, which takes many cycles, and then cache the result in the TLB for future use.

TLBs are small. A typical L1 TLB might hold 64 entries. An L2 TLB might hold a few hundred to a few thousand. Each entry covers one page, which is 4KB. A 64-entry TLB covers 256KB of address space with zero-cost translations. Code and data that fits within 256KB will have excellent TLB behavior. Code and data that is scattered across many pages will cause TLB misses and pay the page table walk cost repeatedly.

This is why TLB pressure is a real concern for performance-sensitive code. A program with a large working set, data scattered across thousands of pages accessed in no particular order, will spend significant time on TLB misses. Large pages (2MB or 1GB on x86-64, configured through huge pages on Linux) are one mitigation: each TLB entry covers far more memory, so the same TLB can cover a much larger working set without misses.

Hierarchical page tables

A flat page table would be straightforward but impractical. On a 64-bit system with 4KB pages, the virtual address space has 2^52 pages (ignoring the bits that are not used in practice). A flat array with one entry per page would itself be enormous, larger than the physical memory of most machines. And each process would need its own copy.

The solution is a hierarchical page table, also called a multi-level page table. Instead of a single flat array, the page table is a tree structure. The virtual page number is split into multiple parts, and each part indexes into a different level of the tree. Only the parts of the tree that are actually needed are allocated.

On x86-64, the current standard uses a four-level page table structure, with a fifth level available on newer processors for even larger address spaces. A virtual address is split into four 9-bit indices and a 12-bit page offset. The four indices index into four levels of page table directories. Each level of directory contains 512 entries (2^9). An entry at each level either points to the next level's directory or marks itself as absent.

The tree is sparse. For a typical process, most of the address space is unmapped. Only the regions that have been mapped (code, data, stack, heap, memory-mapped files) have entries in the tree. The directories covering unmapped regions do not need to exist. The kernel only allocates page table memory for the address space regions that actually exist.

When a process terminates, the kernel walks its page table and frees all the allocated directories. When a process forks, the kernel can share page table entries between parent and child (with modifications to support copy-on-write, which we will get to). The hierarchical structure makes both of these operations practical.

What happens when you access unmapped memory

Your program's address space contains many virtual addresses that are not mapped to any physical page. The stack has a bottom. The heap has a current limit. There are gaps between the code segment, the data segment, and the stack. Most of the theoretical 128 terabytes of usable virtual address space (on current x86-64 implementations) is unmapped.

What happens when your program attempts to access an unmapped address?

The hardware attempts the page table walk and finds either a missing entry in one of the page table levels, or a page table entry marked as not present. The hardware raises an exception: a page fault. Control transfers from your program to the operating system's page fault handler.

The page fault handler examines the faulting address and the context of the access. It has several possible responses.

If the address is simply invalid, not in any region that the process has legitimately mapped, the kernel sends a signal to the process. On Unix-like systems, this is SIGSEGV, the segmentation fault signal. By default this terminates the process and optionally produces a core dump. The error message "Segmentation fault (core dumped)" that every C programmer has seen is the result of this sequence: invalid memory access, page fault, kernel sends SIGSEGV, process terminates.

If the address is in a legitimate region but the page is not currently in physical memory (because it was never loaded, or because it was swapped out), the kernel fetches the page from wherever it lives (disk, a file, a swap partition), allocates a physical page to hold it, updates the page table entry to point to that physical page and mark it present, and resumes the process from the faulting instruction. The process never knew anything happened. From its perspective, it issued a memory access and got back a result. The fact that the OS intervened and fetched data from disk is invisible.

If the address is in a legitimate region and the page is present but the access violated the permissions on the page (for example, writing to a read-only page), the kernel again sends a signal. This is the basis for detecting certain bugs and for implementing copy-on-write, which we will discuss shortly.

Page faults are the mechanism through which the operating system maintains control over memory. Every unmapped or missing or protected page gives the kernel an opportunity to intervene, manage resources, enforce isolation, and implement features transparently.

Demand paging: memory that does not exist until you use it

When you call malloc and allocate a hundred megabytes, the C library might ask the kernel for that memory via the brk or mmap system call. The kernel grants the request and marks a hundred megabytes of virtual address space as belonging to your process. But it does not immediately allocate a hundred megabytes of physical RAM.

The pages in that region are marked as reserved but not present. Physical memory is not assigned until your program actually touches a page.

When your program writes to the first byte of that region, a page fault occurs. The kernel sees that the fault is in a valid region, allocates one physical page (4KB), zeros it out, maps the virtual page to the physical page, and resumes your program. That one page is now in RAM. The other 99,999 pages in the hundred-megabyte allocation are still not in RAM, because you have not touched them yet.

This is demand paging. Memory is paged in on demand, as it is accessed. It is the reason that allocating a large buffer does not immediately consume RAM. It is the reason that a program can start up fast even if it has a large initialized data segment: the initialization happens lazily as pages are first touched.

Demand paging also means that the amount of physical RAM a process actually uses can be much less than the virtual address space it has mapped. A program that allocates a hundred-megabyte buffer but only uses ten megabytes of it consumes roughly ten megabytes of RAM, not a hundred. The kernel only has physical pages for what was actually touched.

The trade-off is that the first access to each new page incurs a page fault, which involves a kernel mode switch and some amount of work. For programs that allocate and then sequentially use large buffers, this is usually fine: the faults are amortized over a lot of useful work, and the kernel may even prefault pages when it can predict access patterns. For programs that have performance requirements and cannot tolerate unpredictable latency spikes, demand paging can be a problem, which is why there are system calls like mlock that force pages to be faulted in and pinned in physical memory.

Copy-on-write: how fork is not as expensive as it sounds

On Unix-like systems, the primary way to create a new process is fork. Fork creates a copy of the current process: same code, same data, same heap, same stack, same file descriptors. On a process with a gigabyte of resident memory, you might expect fork to take a while and consume another gigabyte of RAM.

In practice, fork is fast and does not immediately consume the memory you expect.

The mechanism is copy-on-write. When fork is called, the kernel creates a new process with its own page table, but instead of copying all the physical pages from the parent, it makes the child's page table point to the same physical pages as the parent. Both parent and child share the same physical memory.

To prevent one process from modifying the other's data, all those shared pages are marked read-only in both page tables. If either process reads from a shared page, no fault occurs and no copy is made. If either process writes to a shared page, a page fault occurs. The fault handler sees that this is a copy-on-write fault on a shared page, allocates a new physical page, copies the content of the original page into the new page, updates the faulting process's page table to point to the new page with write permission, and resumes the write. The other process continues to use the original page.

The result is that fork is cheap when the parent and child don't write to much memory after forking. On a typical Unix server that uses fork to handle requests and then immediately calls exec to replace the child with a new program, almost no copying happens: the exec replaces the entire address space, so the copy-on-write pages are never actually copied.

Copy-on-write is also why modifying a file that has been memory-mapped as a private mapping does not affect the underlying file. The pages start shared with the file's backing store. When you write to them, the copy-on-write mechanism creates private copies. The file on disk is untouched.

Memory-mapped files: the unified interface

Memory-mapped files are one of the more useful features built on top of the virtual memory mechanism. The mmap system call allows you to map a file (or a portion of a file) directly into your process's address space. Once mapped, reading from the mapped region reads from the file. Writing to the mapped region (if it's a writable mapping) writes to the file.

From your program's perspective, you just read and write memory. There are no read or write system calls. No buffers to manage. No seeking. You access the file by its virtual address.

Under the hood, the kernel sets up virtual-to-physical mappings that point to pages backed by the file. Pages are not loaded until they are accessed: demand paging applies here too. When you access a page of the mapped file that is not in memory, a page fault occurs, the kernel fetches the page from the file on disk, maps it into physical memory, updates your page table, and resumes your program. When memory is tight, the kernel can evict file-backed pages from RAM without needing to write them to swap, because the file itself is the backing store and can be read again later.

This unified model (files and anonymous memory managed the same way through the same page fault mechanism) is one of the cleaner aspects of the Unix memory model. Everything that uses memory goes through the same machinery.

Shared memory between processes works through the same mechanism. Two processes can both mmap the same file (or the same anonymous region created with special flags). Their page tables point to the same physical pages. Writes by one process are visible to the other without any copying. This is the most efficient form of interprocess communication: no data moves, no kernel involvement after the initial setup, just two processes looking at the same physical memory through different virtual addresses.

Swap: extending RAM onto disk

When physical RAM is full and a process needs more, the kernel has to make room. It does this by evicting pages: taking physical pages that are currently in RAM and moving them elsewhere.

For pages backed by files (code segments, memory-mapped files), eviction is free. The data is already on disk. The kernel can simply remove the page table mapping, mark the page as not present, and the physical page is now available for something else. If the evicted page is accessed again, a page fault occurs and the kernel reloads it from the file.

For anonymous pages (heap data, stack data, anything not backed by a file), eviction requires writing the data somewhere before reclaiming the physical page. That somewhere is swap space: a dedicated area on disk (or a swap file) where the kernel stores evicted anonymous pages.

When an anonymous page is evicted to swap, the kernel records the swap location in the page table entry, marks the page as not present, and reclaims the physical page. If the process accesses that virtual address later, a page fault occurs, the kernel reads the page back from swap into a physical page, updates the page table, and resumes the process. This is completely transparent to the program.

Swap allows the total memory demand of all running processes to exceed the amount of physical RAM. A system with 16GB of RAM and 32GB of swap can run programs that collectively need up to 48GB of memory, as long as not all of it is actively needed at the same time.

The cost is performance. RAM access takes nanoseconds. Disk access takes microseconds to milliseconds. When a process accesses pages that have been swapped out, it waits for disk I/O, which can be orders of magnitude slower than RAM. A system that is swapping heavily (sometimes called thrashing) can become almost unusably slow: programs that should run in milliseconds take seconds because every memory access misses and requires a disk read.

Modern SSDs have made swap significantly more usable than it was with spinning disks. An NVMe SSD with microsecond access times makes swap a reasonable overflow buffer. On spinning disks, heavy swap usage was often catastrophic.

What the virtual address space actually looks like

For a running process on a Linux 64-bit system, the virtual address space is organized into several distinct regions, each with a specific purpose.

At the low end of the address space, the first few pages are typically unmapped. This is intentional: it ensures that null pointer dereferences (accessing address zero or nearby addresses) cause a page fault and a segfault rather than silently succeeding. A null pointer is not a valid memory address in the virtual address space of any process.

Above the null guard region sits the text segment: the compiled code of your program, loaded from the executable file. This region is mapped from the executable file, readable and executable but typically not writable (to prevent the program from accidentally modifying its own code at runtime).

After the text segment comes the data segment: global and static variables that have explicit initial values. These are loaded from the executable file and are readable and writable.

The BSS segment follows: global and static variables that are zero-initialized. The kernel does not actually store all these zeros in the executable file; instead, the BSS region is marked as zero-initialized anonymous memory, and the demand paging mechanism zeroes physical pages as they are first accessed.

The heap grows upward from above the BSS. The C library's malloc implementation manages this region, requesting more space from the kernel (via brk or mmap) as needed and subdividing it for individual allocations. The current top of the heap is the program break.

In the middle of the address space are memory-mapped regions: shared libraries, memory-mapped files, and large anonymous allocations. When you dynamically link against libc or any other shared library, the library's code and data are mapped into this region. All processes that use the same shared library map the same physical pages for the library code (read-only), sharing it without duplication, while having their own private copies of the library's writable data.

The stack lives at the high end of the address space and grows downward toward lower addresses. It starts at a high address and expands toward the heap as function calls are made and local variables are allocated. The operating system maps the stack region with a guard page at the bottom: an unmapped page that catches stack overflows by causing a page fault when the stack tries to grow past it.

Above the stack, the kernel reserves the very top of the address space for itself. On Linux, the upper half of the 64-bit address space is always mapped to kernel memory. When a system call occurs, the kernel code runs in this region, which is present in every process's page table but only accessible in kernel mode. This is why system calls are fast compared to switching to a separate kernel process: the kernel is already mapped into your process's address space, just inaccessible from user mode.

Why two processes can have the same address

One of the things that confuses people new to operating systems is running the same program twice and noticing that both instances report the same addresses for their variables. Run a simple C program that prints the address of a local variable in two terminals and you will see the same hexadecimal value in both.

This makes no sense if you think of virtual addresses as physical locations. Two different things cannot be at the same location.

It makes complete sense once you understand that each process has its own page table. The virtual address 0x7ffe1234abcd in process A maps to some physical page via A's page table. The same virtual address in process B maps to a completely different physical page via B's page table. Same virtual address, same program layout, different physical memory. The isolation is enforced in hardware by the MMU using each process's page table when translating addresses for that process.

This is also why process isolation is robust. Process A cannot access process B's memory by guessing virtual addresses, because A's page table simply does not have mappings for B's physical pages. A's page table is authoritative for A's address space. B's memory does not exist in A's page table. Any attempt by A to access an address that is not mapped in its own page table results in a page fault and, if the address is not in any valid region, a segfault.

The kernel's role in all of this

The kernel is the entity that maintains page tables and handles page faults. Every time the hardware raises a page fault, execution transfers to the kernel's fault handler. The kernel decides what to do: allocate a page, load from disk, send a signal to the process, or kill the process.

The kernel also decides which physical pages to evict when memory is tight. This decision is made by the page replacement algorithm. Linux uses a variant of the clock algorithm and LRU approximation, tracking which pages have been recently accessed and preferring to evict pages that have not been used recently. Getting page replacement right matters a lot for system performance under memory pressure.

Every process context switch requires loading the new process's page table base address into the hardware. On x86-64, this is done by writing the address of the top-level page table directory into the CR3 register. This causes the TLB to be flushed (since the translations it cached belong to the previous process), which is one reason context switches have a real performance cost: the new process starts with a cold TLB and pays for TLB misses until its working set warms up. Tagged TLBs on modern hardware can avoid full flushes by tagging entries with a process identifier, but this optimization has limits.

System calls that involve memory (mmap, munmap, brk, mprotect, mlock) modify the kernel's data structures describing the process's virtual memory regions, and may modify the page table itself. These operations have to be performed carefully because they affect hardware state that is live while the process is running.

Memory protection and what mprotect is actually doing

The permissions on a page (readable, writable, executable) are stored in the page table entry. Every entry has bits indicating which operations are permitted. When the hardware translates a virtual address, it also checks whether the access being performed matches the permissions in the entry. An attempt to write to a read-only page, or to execute data from a non-executable page, causes a page fault just like an access to an unmapped page.

The mprotect system call changes the permission bits in page table entries for a range of virtual memory. This is how the kernel implements write protection for code segments (preventing accidental modification of running code), and it is how security features like W^X (write XOR execute, the policy that a page can be writable or executable but not both) are enforced in hardware.

This mechanism is also what makes stack canaries and address space layout randomization work as security mitigations. ASLR randomizes the base addresses of the stack, heap, and shared libraries each time a program runs, so an attacker cannot hard-code addresses in an exploit. The virtual memory system makes this straightforward: the page table mappings simply start at different base addresses each run.

The abstraction and when it leaks

Virtual memory is one of the most successful abstractions in computing. The majority of programmers use it every day without thinking about it, and it works correctly essentially all the time. The abstraction is clean and well-maintained.

But it leaks in specific circumstances, and knowing when it leaks is useful.

Performance anomalies from TLB pressure show up in programs that scatter accesses across huge address spaces or that access many sparsely populated pages in a short time. The fix is usually to compact the working set: access memory in a pattern that keeps the active pages within what the TLB can cover.

Page fault latency matters for programs with real-time requirements. The cost of a page fault is unpredictable from the program's perspective: it might be a quick kernel allocation, or it might require reading from a slow disk. Programs that cannot tolerate latency spikes (audio processing, trading systems, kernel drivers) use mlock to pin pages in RAM and mlockall to prevent any page faults during critical sections.

The mismatch between virtual allocation and physical consumption can mislead profiling. A process with 10GB of virtual address space might only have 500MB of physical pages. Tools like /proc/PID/status on Linux report both: VmSize for virtual, VmRSS for resident (physical). Understanding which number you're looking at matters when debugging memory issues.

Swap can make performance problems invisible until they become catastrophic. A program running fine with 16GB of RAM can suddenly become unusable when a new workload pushes total memory demand past the physical limit and heavy swapping begins. Monitoring RSS (resident set size) rather than virtual size gives a better picture of actual physical memory consumption.

Copy-on-write makes fork behave in ways that surprise people. A parent process with 1GB of resident memory forks, and immediately the parent modifies its entire heap. Every modified page triggers a copy-on-write fault and a page copy. Total physical memory for the parent and child is now nearly 2GB. If you're using fork in a memory-intensive process, patterns of memory modification after fork have real physical memory implications.

What this means for writing software

You do not need to think about virtual memory for most of what you write. The abstraction holds. malloc works. Arrays work. Pointers work. The OS handles the machinery.

But certain classes of problems become much clearer once you have this model.

Understanding why sequential memory access outperforms random access is trivial once you know about cache lines, TLB coverage, and prefetching. The hardware is optimized for the case where physical pages are accessed in order. Random access across a large sparse virtual address space hits TLB misses and cache misses at every step.

Understanding why two processes can share a shared library without duplicating its memory is immediate once you know page tables and physical page sharing. The library code is one set of physical pages, mapped read-only into every process that uses it. Understanding this makes you appreciate why linking dynamically against common libraries costs less physical memory than statically linking everything.

Understanding why a container or virtual machine does not immediately consume all the RAM it is allocated is clear once you know demand paging. Allocating 4GB to a container allocates virtual address space. Physical RAM is only consumed as it is actually written.

Understanding why a segfault on a null pointer dereference happens at address 0x8 rather than 0x0 makes sense once you know that struct field offsets are added to the base pointer: a null pointer to a struct with a field at offset 8 produces an access to virtual address 8, which is in the unmapped null guard region and causes a fault.

The operating system is doing an enormous amount of work on behalf of every program running on a machine, work that is invisible precisely because it is designed to be invisible. Virtual memory is one of the core mechanisms through which that invisibility is maintained. Once you see it, a lot of things that seemed like magic become straightforward engineering, and a lot of things that seemed like inexplicable behavior become obvious.

What Actually Happens Inside a CPU When It Runs Your Code

Walter Hrad — Tue, 14 Jul 2026 07:45:59 +0000

Most programmers spend their careers at a comfortable altitude above the hardware. You write code, the compiler handles the translation, and somewhere beneath all that abstraction, a processor runs instructions. The details of how that happens feel irrelevant to the daily work of writing software, and for most tasks, they are.

But there are moments when those details stop being irrelevant. When your code is slower than it should be in ways that make no sense from the language level. When a loop that does half the work somehow takes the same amount of time. When adding an unrelated variable somewhere makes another part of your code faster. When multithreaded code produces results that shouldn't be possible. These things are explicable, but only if you know what the hardware is actually doing.

This is an attempt to explain what the hardware is actually doing.

The instruction set is not the machine

The first thing to get straight is the relationship between the instructions you write (or that your compiler writes on your behalf) and what the CPU physically does.

Most programmers are vaguely familiar with the idea of assembly language. x86, ARM, RISC-V. Instructions like MOV, ADD, JMP. The mental model is that these instructions are the thing the CPU executes, one at a time, in order.

That mental model was accurate for processors built before the 1990s. It has been increasingly fictional ever since.

Modern high-performance CPUs use the instruction set as a front-end interface, a compatibility layer that looks like it works the way programmers expect, while the actual execution engine underneath does something far more complicated and far more interesting. The x86 instructions your compiler produces are not executed directly. They are decoded into a different, lower-level set of operations called micro-ops, and it is those micro-ops that the execution engine actually runs. The instruction set architecture is a contract with software. The implementation behind that contract is almost entirely hidden.

Understanding what the implementation actually does requires starting with a basic model and then breaking it repeatedly until the real picture emerges.

The basic model: fetch, decode, execute

A simple processor works in a loop. It fetches the next instruction from memory, decodes it to figure out what operation it represents, and executes it. Then it moves to the next instruction and repeats. This is the classic fetch-decode-execute cycle that every computer science course covers.

The problem with this model, from a performance standpoint, is that each step takes time. Fetching from memory is slow. Decoding takes hardware. Executing takes more hardware. If you do each of these steps in strict sequence for each instruction, you spend most of your time waiting for one step to finish before the next can start.

The first major solution to this is pipelining.

Pipelining: running multiple instructions at once

A pipeline divides the fetch-decode-execute cycle into multiple stages and runs them in parallel on different instructions. While one instruction is being executed, the next is being decoded, and the one after that is being fetched. It is the assembly line applied to computation.

A simple five-stage pipeline has stages for instruction fetch, instruction decode, operand fetch, execution, and write-back of results. At any given clock cycle, five different instructions are in flight simultaneously, each at a different stage of processing. The throughput of the pipeline approaches one instruction completed per clock cycle, even though each individual instruction takes five cycles to move through all stages.

This works well when instructions flow through the pipeline without interruption. The complications arise when they cannot.

Pipeline hazards

A hazard is any situation that prevents the pipeline from advancing smoothly. There are three kinds.

A structural hazard happens when two instructions need the same hardware resource at the same time. Two instructions both needing the multiplier unit during the same clock cycle, for example. The processor has to stall one of them.

A data hazard happens when one instruction depends on the result of a previous instruction that hasn't finished executing yet. If you add two numbers and then immediately use the result in another addition, the second instruction needs a value that the first hasn't produced yet. A naive pipeline would read a stale value from the register file and produce the wrong answer.

Processors handle data hazards in a few ways. One is stalling: just hold the dependent instruction in place and wait until the value is ready. This wastes cycles. A more sophisticated approach is forwarding, also called bypassing: rather than waiting for the result to be written back to the register file and then read out again, the processor creates a direct path from the output of one execution unit to the input of another, delivering the value as soon as it is computed without the detour through the register file. Forwarding eliminates many stall cycles at the cost of more wiring in the processor.

A control hazard is the most expensive kind. It happens at branches: jumps, if statements, loops. When the processor encounters a branch instruction, it doesn't know yet which instruction comes next. It might need to jump to a completely different location in the code, or it might continue to the next sequential instruction. But the pipeline doesn't stop fetching instructions while the branch is being resolved. It has to fetch something. If it guessed wrong, it has to throw away the work it did on the incorrectly fetched instructions and start over from the right location.

Branch prediction: guessing what comes next

Branch misprediction is expensive. In a deep pipeline, the processor might have ten or twenty instructions in flight that are on the wrong path. Discovering the misprediction, flushing those instructions, and fetching from the correct location can waste fifteen or more cycles. At three gigahertz, fifteen cycles is five nanoseconds. Do that a million times a second and you've lost a significant fraction of your compute budget on wrong guesses.

Branch predictors are dedicated hardware units whose job is to guess correctly as often as possible. Modern predictors are sophisticated enough to achieve accuracy above 95 percent on typical code, sometimes above 99 percent.

The simplest predictors use static rules: always predict that backward branches are taken (because they're usually loop back-edges) and forward branches are not taken. These rules are right often enough to be useful but miss a lot of real patterns.

Better predictors maintain history tables that track the recent behavior of each branch. A two-bit saturating counter per branch remembers whether the branch was taken or not taken in recent executions and uses that to predict the next time. This handles branches that are consistently taken or not taken, and recovers from occasional exceptions without immediately flipping the prediction.

Modern processors use much more sophisticated schemes: perceptron predictors that weight multiple history bits, TAGE predictors that use tagged geometric history tables, and hybrid approaches that combine multiple prediction mechanisms and select among them based on which has been most accurate for each branch. The branch predictor in a current high-end processor is an elaborate piece of engineering that represents decades of research.

This is why branch-heavy code can be slower than it appears it should be, and why making branches more predictable is a legitimate optimization. If you have code that branches randomly, the predictor cannot learn the pattern, and you pay the misprediction penalty frequently. If you sort your data before processing it, branches that depend on data values become more predictable, and you pay that penalty rarely.

Out-of-order execution: doing work before it's asked for

Pipelining helps with the sequential flow of instructions. Out-of-order execution handles something different: the fact that even within a sequence of instructions, some can proceed while others are waiting.

Consider a sequence of instructions where the first one is a memory load, and the next few don't depend on the loaded value. A strictly in-order processor would stall on the load, waiting for memory to respond, before doing anything else. An out-of-order processor looks ahead in the instruction stream, finds instructions that don't depend on the load, and executes them while the load is still in flight.

The mechanism that makes this work is the reorder buffer. Instructions are fetched and decoded in order, then placed into a buffer that tracks them. The execution engine picks instructions from this buffer in whatever order their dependencies allow, potentially executing later instructions before earlier ones. When instructions complete, their results go back into the reorder buffer. Instructions are then retired in order, committing their results to the architectural state of the machine in the original program order.

This last part matters. From the perspective of the program, instructions happen in order. Results are visible in program order. Exceptions are handled in program order. The out-of-order execution is completely hidden behind this ordered retirement. The hardware does speculative work out of order and then presents the results as if everything happened in the sequence the programmer expected.

The depth of the reorder buffer determines how far ahead the processor can look for independent work. Processors with larger reorder buffers can keep more instructions in flight simultaneously and tolerate longer latencies from slow operations. A current high-end x86 processor might have a reorder buffer depth of 500 or more instructions.

Superscalar execution: doing multiple things simultaneously

Out-of-order execution allows the processor to find work to do while waiting. Superscalar execution allows the processor to do more than one piece of work at the same time.

A superscalar processor has multiple execution units. Multiple arithmetic logic units for integer operations, multiple floating-point units, multiple load units, multiple store units, a branch unit. In any given clock cycle, if the processor has multiple independent instructions ready to execute, it can dispatch several of them simultaneously to different execution units.

This is instruction-level parallelism, and extracting it is one of the primary jobs of a modern processor. The hardware looks at the stream of instructions, figures out which ones are independent of each other, and runs them in parallel without any intervention from the programmer or compiler.

The limit on how much parallelism can be extracted is set by the actual dependencies in the code. If every instruction depends on the result of the previous one, there is no parallelism to find. If instructions are largely independent, the processor can keep many execution units busy simultaneously. Real code sits somewhere between these extremes, and the processor's ability to find and exploit whatever parallelism exists determines a large part of its performance on that code.

The memory hierarchy: the part that changes everything

So far we've been talking as if memory access is a minor inconvenience that forwarding and out-of-order execution can mostly hide. In reality, memory is the dominant performance constraint for most programs, and understanding it is at least as important as understanding the execution pipeline.

A modern processor runs at several gigahertz. A DRAM memory access takes roughly 60 to 100 nanoseconds. At three gigahertz, one nanosecond is three clock cycles. A memory access that takes 80 nanoseconds takes 240 clock cycles. That is 240 cycles in which the processor is trying to do useful work while waiting for data that hasn't arrived yet.

Out-of-order execution can hide some of this. If there's other work to do while the memory access is in flight, the processor does it. But there's a limit to how much other work is available, constrained by the reorder buffer depth and the actual dependencies in the code. For programs that are dominated by memory accesses with not much independent work in between, the processor spends a lot of those 240 cycles doing nothing useful.

The solution is the cache hierarchy.

Caches

A cache is a small, fast memory that sits between the processor and main memory. When the processor requests data, it first checks the cache. If the data is there (a cache hit), it comes back quickly. If it's not there (a cache miss), the processor has to go to the next level of the hierarchy.

Modern processors have multiple levels of cache. L1 cache sits closest to the execution units, is very small (typically 32 to 64 kilobytes per core), and responds in three to five clock cycles. L2 cache is larger (typically 256 kilobytes to a few megabytes per core) and slower, maybe ten to fifteen cycles. L3 cache is shared among all cores on a chip, might be tens of megabytes, and takes thirty to forty cycles to access. Beyond that is main memory at hundreds of cycles.

The effectiveness of caches depends on locality. Temporal locality: if you access a memory location, you're likely to access it again soon. Spatial locality: if you access a memory location, you're likely to access nearby locations soon. Caches exploit both. Recently accessed data stays in cache. Data is fetched in cache lines (typically 64 bytes), so when you access one byte, the surrounding 63 bytes come along for free.

This is why the memory access pattern of your code matters so much. Iterating through an array sequentially is fast because you access each cache line once and then move on, and the hardware prefetcher can predict the pattern and fetch lines before they're requested. Iterating through a large array with a random access pattern is slow because every access is likely to miss the cache and require a trip to main memory.

Prefetching

Hardware prefetchers watch the pattern of memory accesses and try to predict what data will be needed next, fetching it into cache before the processor asks for it. A simple stride prefetcher detects that you're accessing memory in regular steps and prefetches ahead along that stride. More sophisticated prefetchers detect more complex patterns.

Prefetchers work well for predictable access patterns and fail for unpredictable ones. A linked list traversal, where each node contains a pointer to the next node at an address that depends on the contents of the current node, defeats most hardware prefetchers because each address is only known after the previous access completes. The processor has to wait for each pointer load to resolve before it can request the next one, with no ability to prefetch ahead.

This is one concrete reason why arrays are often faster than linked lists even for operations that should theoretically favor linked lists. Arrays have predictable access patterns that prefetchers can handle. Linked lists have pointer-chasing patterns that they cannot.

Speculative execution and what it means

Out-of-order execution and branch prediction together produce something called speculative execution: the processor executes instructions before it knows whether those instructions should execute at all.

When the branch predictor guesses that a branch is taken, the processor starts executing down the predicted path before the branch condition has been evaluated. If the prediction was correct, that work is committed and everything is fine. If it was wrong, all that work is discarded.

The processor also speculates across loads. If a load instruction is waiting for its address to be computed, and there's a later instruction that doesn't appear to depend on that load's result, the processor might execute the later instruction first. If that turns out to be valid, great. If the ordering mattered in some way the processor didn't detect, it has to replay the affected instructions.

Speculative execution is essential to extracting performance from modern hardware. It is also the source of an entire class of security vulnerabilities that were discovered starting in 2017 with Spectre and Meltdown.

The core issue is that speculative execution leaves microarchitectural traces even when its results are discarded. If speculative code brings data into cache, that data stays in cache even if the speculation is rolled back. By carefully measuring cache access times, an attacker can infer what data was loaded speculatively, including data the attacker should not have access to. The timing of a cache hit versus a cache miss is measurable, and that timing difference becomes a covert channel through which secrets can leak.

Fixing these vulnerabilities properly requires either preventing the speculation that causes them (with significant performance cost) or preventing the microarchitectural traces from being observable (which is extremely difficult to do completely). The hardware and software mitigations that have been deployed since 2017 represent an enormous engineering effort and have permanently changed how people think about the security implications of performance optimizations.

Register renaming: eliminating false dependencies

One more piece of the puzzle worth understanding is register renaming.

Programs use a limited number of architectural registers, the ones that have names in the instruction set. In x86-64, there are sixteen general-purpose integer registers. Code that does a lot of work has to reuse these registers. Register reuse creates what look like data dependencies: if instruction A writes to register R1 and instruction B later writes to the same register R1, a naive processor would think B has to wait for A because they share a register. But if B doesn't actually use A's result, this dependency is false. They could run in parallel without any correctness issue.

Register renaming eliminates this problem by mapping architectural registers to a much larger pool of physical registers. When instruction A writes to R1, it gets mapped to physical register P37. When instruction B writes to R1, it gets mapped to physical register P52. The processor tracks these mappings and uses the physical registers internally. A and B now have no register conflict and can run in parallel.

Modern processors have physical register files of 200 or more entries. This is far more than the sixteen architectural registers that code refers to. The extra registers exist to absorb the false dependencies that would otherwise prevent parallel execution. Register renaming, combined with the reorder buffer and out-of-order execution, is what allows the processor to find and exploit instruction-level parallelism even in code written without any awareness of the underlying hardware.

Putting it together: why some code is fast and some is not

With all of this in place, a few patterns become explainable.

Sequential memory access is fast because caches and prefetchers are designed for it. Random memory access is slow because every miss goes to main memory. A program whose hot path consists of sequential array iterations will run close to memory bandwidth limits. A program whose hot path consists of pointer chasing through a linked list or a tree will be limited by memory latency, with the processor spending most of its time waiting for each pointer load to resolve before it can proceed.

Branches that follow predictable patterns are cheap. Branches that are genuinely unpredictable are expensive. A branch that depends on whether a data value is positive or negative will be predictable if the data is sorted and unpredictable if it is random. Sorting data before processing it is sometimes fast not because the sorting saves computation but because it makes subsequent branches predictable.

Tight loops with lots of independent iterations are fast because the processor can keep many instances of the loop body in flight simultaneously, hiding latencies and keeping the execution units busy. Loops with tight serial dependencies between iterations are slow because each iteration cannot start until the previous one completes.

Code that thrashes the instruction cache, jumping around to many different locations, can run slower than dense straightforward code even if the scattered code does technically less work, because instruction cache misses stall the fetch stage of the pipeline.

Functions that are called very frequently benefit from staying resident in the L1 instruction cache. Code that is cold, called rarely, will have instruction cache misses when it runs. This is usually fine for cold code but means you shouldn't measure the performance of cold code and assume it represents what hot code will do.

The gap between what you write and what runs

The thing I find most interesting about all of this is how large the gap has grown between the programming model and the physical reality.

When you write a for loop, you're describing a sequential operation. The processor executes that loop with multiple iterations in flight simultaneously, with instructions from different iterations running in parallel on different execution units, with the branch predictor guessing when the loop ends before the loop condition has been evaluated, with data fetched from cache lines that arrived before they were requested. The sequential programming model is a fiction that the hardware maintains on your behalf, because the fiction is useful and the reality would be impossibly complex to program against directly.

This is mostly a good thing. The abstraction is remarkably effective and lets programmers reason about correctness without tracking the physical state of hundreds of in-flight instructions. But the abstraction leaks in performance-sensitive situations, and when it does, knowing what's underneath is the difference between debugging by intuition and debugging by understanding.

The processor is not a passive executor of your instructions. It is an aggressive, speculative, parallel machine that is constantly trying to do more work than you asked for, in an order you didn't specify, and then presenting the results as if it did exactly what you wrote. Most of the time that works perfectly. Understanding when and why it doesn't is what separates people who write fast code from people who write code and hope it's fast enough.

Every Time You Run npm install, You Are Trusting Hundreds of Strangers

Walter Hrad — Mon, 06 Jul 2026 07:44:43 +0000

Most developers have a ritual with new projects. You clone the repo, change into the directory, type npm install, and watch the terminal fill up with download progress. It is automatic. Comfortable. So routine that you stop thinking about it entirely.

That comfort is exactly what attackers have been counting on.

This year has been brutal for the JavaScript ecosystem in ways that have not gotten enough attention outside of security circles. The Axios library, which sits in roughly 174,000 downstream packages and pulls in somewhere around 100 million weekly downloads, was compromised in March 2026 through stolen maintainer credentials. A malicious dependency called plain-crypto-js was injected into two versions, and it downloaded multi-stage payloads including a remote access trojan onto machines belonging to developers who had no idea anything had changed. CISA issued an advisory. Most people just ran npm update and moved on.

That was not an isolated incident. It was part of a pattern that has been accelerating for over a year.

The Shai-Hulud Problem

In September 2025, a campaign compromised 18 widely used packages including debug and chalk. If you have ever worked on a Node.js project you have used those packages. They are not optional utilities most teams choose to include. They are woven into the dependency trees of virtually everything. Combined weekly downloads at the time exceeded 2.6 billion. The attackers did not need to trick anyone into installing something new. They just had to poison something developers were already trusting implicitly.

The campaign that became known as Shai-Hulud took things further. What made it especially dangerous was its self-propagation mechanism. Once a developer installed an infected package, the malicious preinstall script would harvest that developer's npm tokens and GitHub credentials, then use those credentials to publish poisoned versions of every package that developer maintained. The worm spread not through a traditional exploit chain but through the trust relationships baked into the npm ecosystem itself. By the time multiple waves of the campaign had played out, it had touched 796 packages with a combined 132 million monthly downloads.

The June 2026 Miasma attack introduced a refinement called the Phantom Gyp technique. An attacker places a 157-byte binding.gyp file inside a compromised package. That file triggers npm's implicit node-gyp rebuild call, an automatic C/C++ compilation step that npm runs for any package containing that file. The critical detail is that this happens with no lifecycle script required and with no protection from --ignore-scripts, which was the flag developers had been told would protect them. The Miasma worm used this technique to compromise 57 packages in a second wave after the initial infection, all within a two-hour window, targeting Red Hat's official @redhat-cloud-services namespace.

These were not obscure packages from unknown maintainers. They were packages from organisations developers had every reason to trust.

Why the Architecture Makes This So Difficult

The underlying problem is structural. When you run npm install, the package manager hands execution rights to every package in your dependency tree. Not just the packages you chose to install directly. Every transitive dependency. Packages you have never read, maintained by people you have never heard of, several levels deep in a tree you probably have not audited. The average npm project pulls in 79 transitive dependencies.

Each of those packages can define preinstall, install, and postinstall lifecycle scripts. Those scripts run automatically and silently with the full permissions of the user running the install. On a developer workstation that might mean access to AWS credentials sitting in environment variables, SSH keys, GitHub tokens stored by the git credential manager, CI/CD secrets, and cloud provider configurations. On a build server the situation is often worse, because build servers tend to have elevated access to production infrastructure by design.

GitHub themselves described this as the single largest code-execution surface in the npm ecosystem. That framing is worth sitting with. Not the largest vulnerability. The largest code-execution surface. Meaning the attack does not require finding a bug. It requires getting code into a package, which is significantly easier.

Attackers have three main ways to do that. First is account compromise, where a maintainer's credentials get stolen and used to publish a malicious version of a legitimate package. Second is dependency confusion, where an attacker publishes a public package with the same name as an internal private package, exploiting the way some package managers resolve scoped dependencies. Third is typosquatting, where a malicious package is published under a name close enough to a popular one that a developer with a typo in their install command will pull it in instead.

In late May 2026, Microsoft Threat Intelligence documented a dependency confusion campaign where a threat actor published malicious packages under organisational scopes that mirrored real internal corporate namespaces. The packages spoofed internal enterprise infrastructure URLs in their package.json to appear legitimate. One maintainer account involved had been sitting dormant since April 2024, when it was registered under bug bounty research branding. Two years later the same account was pushing obfuscated credential-stealing payloads. The patience involved is not reassuring.

What Actually Happens When You Get Hit

The Shai-Hulud payload execution chain is worth understanding in detail because it illustrates how far beyond simple credential theft these attacks have evolved.

The infection starts during npm install, where the malicious preinstall hook executes a loader script without any user interaction. The payload unpacks through multiple decoding layers, using ROT-based obfuscation variants followed by AES-128-GCM decryption. It then downloads the Bun JavaScript runtime and detonates the final payload. Before continuing, the malware validates the execution environment and can restrict itself to CI/CD environments only, which means some developer machines may never see any obvious signs of infection while build pipelines are being quietly drained.

The self-propagation step is where it turns into a worm. The malware republishes packages owned by the compromised maintainer using forged provenance metadata, which means it spreads wearing the identity of a trusted publisher. In one campaign there was also a destructive tripwire: if the malware detected interaction with a planted decoy token, it triggered a command that wiped the victim's home directory.

The stolen credentials cascade in a specific way. Stolen npm tokens enable further package poisoning. Stolen GitHub tokens enable repository manipulation. Stolen AWS credentials enable cloud access. Each layer of compromise opens the next one, and by the time a security team detects unusual activity in one place, multiple systems are often already affected.

In 2025 alone, attackers published nearly 455,000 malicious npm packages. Over 99 percent of all open source malware now targets npm according to Sonatype's 2026 report. This is not a niche concern for enterprise security teams. It is the environment every JavaScript developer is working in every day.

The npm v12 Response

GitHub announced earlier this month that npm v12, shipping in July 2026, will flip three defaults that have been permissive since npm was created.

The first and most significant change is that install scripts will be blocked by default. Running npm install will no longer automatically execute preinstall, install, or postinstall scripts from dependencies unless they are explicitly allowed in your project. This includes native node-gyp builds triggered by binding.gyp files, which closes the Phantom Gyp vector directly. The second change blocks Git dependencies by default. The third blocks remote URL dependencies like HTTPS tarballs.

The mechanism for managing this is a new command called npm approve-scripts. You run it, it shows you which packages in your dependency tree have scripts that are not yet covered by your policy, and you approve the ones you have actually reviewed. Those approvals get committed to your package.json and checked into source control, creating a documented record of which packages your team has explicitly trusted to execute code at install time.

The security argument for this is clear. Only about 2 percent of npm packages actually need install scripts to function. The vast majority use them as a convenience shortcut for build steps that could be performed explicitly. For those 2 percent, adding them to an explicit allowlist is a one-time cost. For the other 98 percent, blocking execution by default removes the attack surface entirely.

The friction argument against it is also real. Paul McCarty, a vulnerability researcher who has been tracking npm supply chain attacks closely, put it plainly: when the choice is between a build that works and a build that is less prone to malware, the former will always win. Developers under deadline pressure who encounter blocked scripts are likely to approve everything with a blanket flag just to make the warnings go away, which defeats the purpose entirely. The tools exist to do this correctly, but tooling availability and tooling adoption are different things.

What You Should Actually Do Before July

If you are running the npm CLI today, you are relying entirely on publisher-side defences and assuming nothing slips through the registry. That assumption has been wrong repeatedly this year.

The most immediate thing you can do is upgrade to npm 11.16.0, which enables advisory mode for the v12 changes. In advisory mode, warnings surface but nothing breaks yet. That gives you visibility into which packages in your dependency tree have scripts running during install, without forcing you to fix everything before you understand the scope of the problem.

Run npm approve-scripts --allow-scripts-pending in read-only mode first. It lists every package whose scripts are not yet covered by your policy without changing anything. Go through that list and think carefully about which packages actually need to execute code at install time. Most of them do not. Approve the ones that do, pinned to the specific version you have reviewed, and commit the result.

Switch --ignore-scripts on for your CI pipelines now rather than waiting for v12. Anything that breaks when you do that is a package that was silently running code in your build environment, and you should know which packages those are.

Consider moving to pnpm or Yarn Berry if you have not already. Both have been shipping consumer-side security defaults for some time. pnpm v11, released in April 2026, unified its security configuration into a cleaner API and blocks lifecycle scripts by default unless you explicitly allowlist them. The npm CLI is the only major package manager that had not shipped these consumer-side protections until the v12 announcement. The alternatives were ahead of this problem.

For packages your team publishes, set up trusted publishing if you have not already. Trusted publishing ties a package release to a verified CI/CD workflow rather than to credentials that can be stolen from a developer's machine. It does not prevent every attack vector but it raises the bar significantly for account compromise scenarios.

Pin your dependency versions in CI. A package that was clean when you last audited it is not guaranteed to be clean after an update. Automated dependency updates through tools like Dependabot are useful for staying current with security patches, but they should go through a review step rather than merging automatically. The Axios attack left detectable signals in the registry metadata before anyone ran npm install, including a missing trusted publisher block and a missing gitHead field. Tools like Socket and StepSecurity surface these signals automatically. Without those tools, checking a package's metadata manually takes about thirty seconds per package and would have caught that attack before it ran.

The Deeper Shift

What is happening in the npm ecosystem right now is a forced reckoning with a trust model that was designed for a different era. When npm was built, the open source ecosystem operated on a foundation of good faith. Packages were published by developers who wanted to share useful tools. The idea that someone would weaponise that distribution mechanism at industrial scale was not a primary design consideration.

That era is over. The attacks in 2025 and 2026 are industrialised, coordinated, and in some cases state-linked. The Shai-Hulud campaigns showed self-replicating worm behaviour. The dependency confusion attacks showed patience and careful reconnaissance. The Bitwarden impersonation package that was part of one Shai-Hulud wave shows attackers studying which tools developers trust specifically to impersonate them.

npm v12 is a structural response to a structural problem. Blocking install scripts by default, requiring explicit allowlists, and closing the Phantom Gyp loophole are not complete solutions but they remove the path of least resistance that attackers have been using repeatedly. The era of implicit trust in package installation is ending whether developers are ready for it or not.

July is not far away. The question is whether you find out what is running in your dependency tree on your own terms, or whether npm v12 breaks your CI pipeline on the first day it ships and you figure it out under pressure.

The inventory is not that hard. The install is the thing you have been doing automatically for years without thinking about it. It is worth thinking about it now.

How Git Actually Works Under the Hood

Walter Hrad — Sun, 05 Jul 2026 12:26:54 +0000

Most developers use Git every day and understand almost none of it. That's not an insult, it's just the reality of how most people learn tools. You pick up the commands that get you through the day, you memorize the ones that fix the situations you keep breaking, and you build a working mental model that is almost entirely wrong at the mechanical level.

The mental model most people carry looks something like this: Git tracks changes to files. When you commit, it saves a snapshot of what changed. Branches are pointers to different lines of work. That's roughly correct at a surface level, but it skips over the actual machinery in a way that leaves you confused every time something unexpected happens. Why does rebasing rewrite history? Why are commits immutable? Why does detached HEAD state exist? Why can you lose work in ways that feel impossible if Git is just tracking changes?

The answers are all in the object model, and the object model is surprisingly simple once you sit with it.

Git is a content-addressable filesystem

Before any of the version control concepts, Git is a key-value store. You put content in, you get a hash back. You use that hash later to retrieve the content. That's the entire foundation, and everything else is built on top of it.

The hash Git uses is SHA-1, producing a 40-character hexadecimal string. When you run git hash-object on a file, Git takes the content, prepends a small header describing the object type and size, and runs SHA-1 over the whole thing. The resulting hash is both the key and the identity of that content. Two files with identical content will always produce the same hash. A file whose content changes even slightly will produce a completely different hash.

This is the first thing that breaks people's mental models. In most storage systems, identity is location: a file is "that file" because it lives at that path. In Git's object store, identity is content. The path a file lives at is separate metadata, not the file's identity.

All of Git's objects live in .git/objects. Go look at it sometime on a real repository. You'll find subdirectories named with two-character hex prefixes, and inside each one, files named with the remaining 38 characters of various hashes. Each of those files is a compressed Git object. The entire history of your project, every version of every file that ever existed, every commit, every tree, is sitting right there in that directory as a pile of content-addressed blobs.

The four object types

Git has exactly four types of objects: blobs, trees, commits, and tags. That's it. The entire version control system is built from those four things.

Blobs

A blob is file content, nothing else. Not a filename, not a path, not permissions. Just the raw bytes of a file at a particular moment in time.

If you have two files in your repository with the same content, they share a single blob. Git doesn't store duplicates. The filename and location of those files are stored elsewhere. The blob itself is just content.

This is why Git is efficient in ways that surprise people. If you have a thousand files and you change one of them, Git only needs to store one new blob. The other 999 files haven't changed, so their blobs already exist in the object store, and the new tree structure just references them by hash. Nothing is copied.

You can look at a blob directly using git cat-file -p <hash>. Run git ls-files --stage in any repository and you'll see the blob hashes for every file in your current index alongside their filenames and modes. Pick any hash from that list and cat-file it, and you'll see the raw file content.

Trees

A tree is Git's representation of a directory. It contains a list of entries, where each entry is a mode, a type, a hash, and a name. The type is either blob or tree, because directories can contain files and other directories.

A tree for a simple directory with two files and one subdirectory might look like this:

100644 blob a8c6a8d9...    README.md
100644 blob 3f1b2c4e...    main.go
040000 tree 9d2e1f7a...    internal

That tree has a hash. The subdirectory internal is itself another tree object with its own hash. That tree has a hash. Every file in it is a blob with a hash. The whole directory structure of your project at any given moment is represented as a tree of hashes pointing to other hashes.

This structure is a Merkle tree, the same data structure that shows up in Bitcoin and a lot of other systems where you need to verify large amounts of data efficiently. If any blob anywhere in your directory tree changes, its hash changes, which changes the hash of the tree containing it, which changes the hash of any parent tree, which changes the hash of the root tree. The root tree hash is a fingerprint of your entire directory structure at that moment. If two root tree hashes are equal, every file in every directory is byte-for-byte identical.

Commits

A commit object contains four things: a pointer to a root tree, zero or more pointers to parent commits, author and committer metadata, and a commit message.

The pointer to the root tree is what gives a commit its snapshot of the entire project. When you check out a commit, Git reads that commit's tree hash, then recursively resolves all the trees and blobs in it, and reconstructs your working directory from those objects. There's no concept of a "diff" in the object store. Every commit has a full snapshot, but because blobs and trees are deduplicated by content, the actual storage cost of a new commit is only the objects that didn't exist before.

The pointer to parent commits is what makes the history graph. A regular commit has one parent, the previous commit. A merge commit has two or more parents, one for each branch that was merged. The first commit in a repository has no parent.

This is the second thing that breaks mental models. The history graph is not a sequence of diffs. It is a directed acyclic graph of snapshots, where each node contains a full picture of the entire project, and edges point backwards in time to parent commits. When you ask Git to show you what changed between two commits, it reconstructs both snapshots from their respective trees and computes the diff on the fly. The diff is not stored anywhere. It's derived.

Commits are also immutable. Once a commit object exists with a given hash, it cannot be changed, because any change to its content would change its hash, making it a different object. When you amend a commit, you're not modifying the existing commit. You're creating a new commit object and moving the branch pointer to it. The old commit still exists in the object store until it gets garbage collected.

References: the human layer on top of hashes

Hashes are how Git thinks about objects internally. Humans are bad at hashes. References are the layer that makes Git usable for humans.

A reference is a file containing a hash. That's all it is. The file .git/refs/heads/main contains the SHA-1 hash of the commit that the main branch currently points to. The file .git/refs/tags/v1.0.0 contains a hash. The file .git/HEAD contains either a hash (when you're in detached HEAD state) or a symbolic reference to another ref file (when you're on a branch, it contains something like ref: refs/heads/main).

When you create a new commit on the main branch, Git creates the commit object, writes its hash into .git/refs/heads/main, and that's the entire operation of "advancing the branch." The branch didn't grow. The branch pointer moved.

Branches in Git are not containers. They're not timelines. They're not parallel universes of code. They're a single file containing a single hash. When people say a branch is "just a pointer," they're being completely literal. Go look at .git/refs/heads in any repository. Every branch is a file. Open any of those files. It contains exactly one hash.

This is why creating and deleting branches in Git is so cheap compared to other version control systems. There's no data to copy, no history to replicate. Creating a branch is creating a file. Deleting a branch is deleting a file. The commits the branch pointed to are unaffected, they still exist in the object store.

HEAD and what detached HEAD actually means

HEAD is a special reference that tells Git where you currently are. In the normal case, HEAD contains a symbolic reference to a branch, something like ref: refs/heads/main. Git calls this being "attached" to a branch. When you make a commit, Git creates the commit object, updates the branch ref to point to it, and HEAD follows because HEAD points to the branch, not directly to a commit.

Detached HEAD happens when HEAD contains a commit hash directly instead of a branch reference. This happens when you check out a specific commit, a tag, or a remote tracking branch. Git tells you about it because it has consequences: if you make commits in this state, those commits don't belong to any branch. They exist in the object store, and HEAD advances as you commit, but no named reference tracks them. When you switch back to a branch, HEAD moves to that branch, and your detached commits are now orphaned. They'll get garbage collected eventually unless you create a branch pointing to them first.

Once you understand what HEAD actually is, detached HEAD goes from a scary warning to a completely sensible description of what's happening. You're not attached to anything. You're floating at a specific commit with no branch to record where you go from here.

How staging actually works

The staging area, also called the index, is one of the more misunderstood parts of Git. Most people think of it as a temporary holding area for changes on the way to a commit. That's not wrong, but it undersells what it actually is.

The index is a binary file at .git/index that represents a complete snapshot of your project. It contains an entry for every tracked file: the file's path, its mode, its blob hash, and some stat information from the filesystem that Git uses to detect changes without hashing every file on every status check.

When you run git add, Git takes the current content of that file, creates a blob object for it in the object store, and updates the index entry for that file to point to the new blob hash. Nothing else happens. No commit is created. Just a blob object and an updated index entry.

When you run git commit, Git takes the current state of the index, constructs a tree object hierarchy from it, creates a commit object pointing to the root tree and the current HEAD commit as parent, writes the new commit hash to the current branch ref, and updates HEAD.

The index is essentially a proposed tree. It's the snapshot you're about to commit. That's why partial staging makes sense mechanically: you're selectively updating the index to contain some changes but not others, constructing the exact snapshot you want the commit to represent.

How merging works

A merge takes two commits and produces a new commit with both of them as parents. But to know what to put in that new commit's tree, Git has to figure out what changed in each branch relative to their common ancestor.

Finding that common ancestor is the job of the merge base algorithm. Git walks the commit graph backwards from both commits simultaneously, looking for the first commit that appears in both paths. Once it has the merge base, it diffs each branch tip against the merge base to find what changed in each, then applies both sets of changes to produce the merged result.

When the two branches changed different parts of different files, the merge is automatic. When they changed overlapping parts of the same file, you get a conflict, because Git can't automatically decide whose change should win.

A fast-forward merge is a special case where one commit is a direct ancestor of the other. If you're on main and you merge a feature branch, and main hasn't moved since you branched off, there are no divergent changes to combine. Git can just move the main pointer forward to the feature branch's tip. No merge commit is created because no actual merging needed to happen.

How rebasing works and why it rewrites history

Rebasing is where a lot of people get confused, and the confusion usually comes from thinking about branches as containers rather than pointers.

When you rebase a branch onto another, Git takes the commits in your branch that aren't in the target, and replays them one by one on top of the target. For each commit, it computes the diff between that commit and its parent, then applies that diff on top of the current tip of the target, creating a new commit object.

The key word is "new." The replayed commits are new objects. They have different parent hashes (because their parents changed), which means they have different hashes themselves. The original commits still exist in the object store, they're just no longer reachable from any branch. You haven't moved your commits. You've created copies of them in a new position and moved your branch pointer to the last copy.

This is what "rewriting history" means. The commits that exist after a rebase are different objects than the commits that existed before, even if their content is identical. If someone else had pulled your branch before you rebased, they have references to the old commits. After your rebase, those old commits no longer appear in your branch history. Their local branch and your remote branch have diverged. This is why rebasing shared branches causes problems: you've replaced the objects other people are pointing at with new objects, and their local Git has no way of knowing the new ones are meant to replace the old ones.

How git gc and the reflog save you

Git's object store is append-only during normal operation. Every blob, tree, commit, and tag you ever create lives in .git/objects until Git decides to clean up. Objects that aren't reachable from any reference are called loose objects or dangling objects, and they accumulate over time from operations like amending commits, rebasing, and resetting branches.

The reflog is what gives you a window to recover from those operations before cleanup happens. Every time HEAD or a branch ref moves, Git appends an entry to the reflog recording where the ref was before and where it moved to. The reflog for HEAD lives at .git/logs/HEAD. Run git reflog and you'll see a timestamped history of every position HEAD has ever been at in that repository.

When you do a hard reset and realize you needed those commits, or you rebase and want to get back to the pre-rebase state, the reflog is how you find the hash of the commit you want to return to. That commit still exists in the object store. You just need the hash to reach it.

git gc is the garbage collector. It finds objects with no references pointing to them (directly or through the reflog, which has its own expiry) and deletes them. By default, objects that are more than 30 days old and unreachable from any ref or reflog entry are eligible for collection. This is the window you have to recover from mistakes. It's also why truly losing work in Git is harder than it feels in the moment: Git is quite conservative about actually deleting anything.

Packfiles and how Git stores history efficiently

If every object is stored as a separate compressed file, large repositories with long histories would be enormous. Git handles this with packfiles.

A packfile is a single file that contains many objects stored together with delta compression. Instead of storing each version of a file as a complete compressed blob, Git can store one version in full and then store other versions as deltas relative to it. For files that change incrementally over time, this is dramatically more space efficient than storing each version separately.

Packfiles get created by git gc, by git repack, and automatically by Git when the number of loose objects crosses a threshold. When Git reads from a packfile, it reconstructs the requested object on the fly from whatever deltas are needed. From your perspective as a user, this is invisible. You use the same commands, request objects by the same hashes, and Git handles the physical storage transparently.

The index file that accompanies each packfile allows Git to binary search for any object hash in the pack without reading the entire file. This is how Git can efficiently access any object in a repository with millions of commits and hundreds of thousands of files.

What this changes about how you use Git

Understanding the object model doesn't make you memorize fewer commands. It does make the commands make sense in a way that changes how you work.

You stop being afraid of rebasing once you understand it's just replaying commits in a new position, not shuffling some fragile linear history. You stop being afraid of resetting once you know the reflog has your back and objects don't disappear immediately. You stop being confused by detached HEAD because you know exactly what HEAD is and what "detached" describes. You understand why git push --force is dangerous on shared branches, because you understand that it replaces remote refs with hashes pointing to new objects, orphaning whatever the old hashes pointed to.

You also start reading error messages differently. When Git says "your branch and origin/main have diverged," you know that means the commit graphs have branched: the local ref and the remote ref point to different commits that don't have a simple ancestor relationship. When Git says a ref is not valid, you can go look in .git/refs and see for yourself what's there and what isn't. When you need to find a lost commit, you know to check the reflog rather than feeling like something is gone forever.

Git's design is not arbitrary. The object model, the content addressing, the immutability of objects, the lightweight references, the reflog, the packfiles: all of it fits together into a system that is remarkably consistent once you have the right mental model. The learning curve exists because most people learn Git from the outside, picking up commands without ever looking at what the commands are actually doing to the files in .git. Going the other direction, starting with the objects and working outward to the commands, is a slower way to start but a much more durable way to understand.

The next time something goes wrong in Git and your instinct is to blow away the repository and clone fresh, stop and think about what objects exist and what references point to what. The answer to what happened and how to fix it is almost always there, in plain text, sitting in .git.

Nobody Wants Your 70B Parameter Model Anymore

Walter Hrad — Sat, 20 Jun 2026 20:18:16 +0000

For a while the entire AI conversation was about scale. Bigger model, bigger context window, bigger benchmark score, bigger headline. If you weren't training something with a parameter count that needed scientific notation, you weren't really in the game.

That story is quietly falling apart, and I don't think enough people are talking about it.

The thing nobody admits about huge models

Massive general purpose models are genuinely impressive, but most of what they're impressive at, you don't actually need. If you're building a voice assistant for a car dashboard, you don't need a model that can write sonnets about quantum mechanics. You need something that reliably understands "turn the AC down" and runs without melting the car's battery.

That mismatch between what big models offer and what most real products need has been sitting there for years, mostly ignored because everyone was chasing the same leaderboard. What's changed recently is that the tooling finally caught up to the obvious idea: train something small, train it well, and point it at one job.

Quality of data beats size of model

The thing that broke this open wasn't a clever new architecture, it was a boring realization. A handful of labs proved that a model trained on a few billion carefully chosen, high quality tokens can go toe to toe with models many times its size trained on whatever could be scraped off the internet.

That's a strange thing to sit with if you came up believing more data and more parameters was the whole game. It turns out a smaller model trained like a sharp student studying good textbooks beats a bigger model trained like someone speed reading the entire internet once.

This is the same lesson every engineer eventually learns the hard way. Throwing more resources at a problem is rarely as effective as understanding the problem better.

Where this actually matters

A few places where small, specialized models are quietly taking over instead of staying a research curiosity:

On-device assistants. Phones now ship with dedicated AI hardware that can run a few billion parameter model directly, no network call required. That means your voice assistant keeps working in a tunnel, on a flight, or somewhere with terrible signal, which honestly describes a lot of where I live and work. There's something almost funny about it too. For years the assumption was that intelligence lived in the cloud and your phone was just a window into it. Now the phone itself is quietly doing real reasoning, locally, while you're not even thinking about it.

Anything regulated. Hospitals, law firms, anyone handling sensitive data has a real problem sending that data to a third party API. Running a smaller model locally, on hardware you control, sidesteps that entire conversation. No data leaves the building, no compliance headache. A clinic running a specialized model on its own server doesn't need to explain to a regulator where patient data went, because it never went anywhere. That single fact unlocks entire industries that were effectively locked out of the AI conversation until recently, not because the technology wasn't good enough, but because the deployment model was wrong for how those industries are required to operate.

Latency sensitive systems. A self driving car cannot afford a network round trip to decide whether the shape ahead is a pedestrian. Object detection models running locally on quantized weights are the only version of this that makes sense. The same logic shows up in smaller, less dramatic ways too. Industrial sensors deciding in real time whether a machine is about to fail. Cameras on a factory line flagging defects before the part moves to the next station. None of these can wait two hundred milliseconds for a server somewhere to respond. The model has to live where the decision happens.

Cost at scale. If you're running millions of inference calls a day, the difference between a frontier model and a well tuned small model is the difference between a sustainable business and one that burns cash on every request. This is the one that doesn't get talked about enough outside finance meetings, but it's probably the most decisive factor for most companies. A frontier model might be ten times more capable on a benchmark, but if it costs forty times more per call and your use case only needed a fraction of that capability anyway, you were never going to win by using it. Margins don't care how impressive your model is on a leaderboard.

Offline and low connectivity environments. This one is personal for me, working out of Kisumu. A lot of AI products are quietly designed with the assumption that internet access is fast, cheap, and constant. That assumption doesn't hold everywhere, and it especially doesn't hold in a lot of the world outside a handful of countries. A small model that runs entirely on-device doesn't care if the connection drops. It doesn't care if data is expensive that month. For products meant to work in places with patchy infrastructure, this isn't a nice to have, it's the only way the product functions at all.

How you actually shrink a model without ruining it

It's worth pulling back the curtain a bit on how this is done, because "just make it smaller" undersells the engineering involved.

Quantization is the most common technique, and the idea is simple even if the implementation isn't. A model's weights are normally stored as high precision numbers. Quantization reduces that precision, packing the same information into fewer bits. A model that took 400 megabytes can shrink to 100 megabytes this way, and in a lot of cases the accuracy loss is barely noticeable for the task at hand. Some of the more aggressive recent approaches go even further, using ternary weights that only take the values negative one, zero, or positive one, which sounds almost too simple to work, but it does, within the right constraints.

Pruning is the other major lever. Neural networks tend to be wildly overparameterized, meaning a lot of the connections inside them contribute very little to the final output. Pruning identifies and removes those low impact connections, shrinking the model without meaningfully touching its behavior. It's the machine learning equivalent of realizing half your codebase is dead code nobody ever calls.

Then there's knowledge distillation, which I find the most conceptually interesting of the three. You take a large, capable model and use it as a teacher. The small model is trained not just on raw data, but on the larger model's outputs and internal behavior, essentially learning to mimic the bigger model's reasoning on the specific tasks that matter. It's apprenticeship, formalized into a training process. The student doesn't need to know everything the teacher knows, it just needs to get good at the narrow thing it was hired to do.

Stack these techniques together and you get something that sounds almost contradictory on paper: a model a fraction of the size, running a fraction of the cost, that still holds its own against something many times larger, as long as you stay within the lane it was built for.

The part that's relevant to actual developers

The interesting bit for people building things, rather than people writing think pieces about AI, is that this shifts the skill that matters. Prompting a giant general model well is one skill. Picking, fine tuning, and deploying a small specialized model for your exact use case is a different one entirely, and it's closer to traditional engineering than people expect.

It involves real tradeoffs you have to reason about. How much accuracy are you willing to give up for speed. Can you get away with quantizing down to a smaller footprint without your outputs degrading in ways your users will notice. Do you need one model doing everything, or are you better off chaining a few small specialized ones together, each handling a narrow piece of the task.

That last pattern is becoming more common than I expected. Instead of one model trying to reason, call tools, and generate a final response, you split it. One small model handles reasoning, another handles tool calls, another handles the final response generation. Each one is small enough to run cheaply and fast enough to feel instant, and together they cover ground that used to require a single enormous model.

It's a similar idea to microservices, if you squint. Stop building one giant thing that does everything adequately, and instead build several small things that each do one thing well.

Why this matters beyond hype cycles

I think the honest reason this shift is sticking, rather than being another trend that fades, is that it lines up with constraints that don't go away. Power costs money. Bandwidth isn't guaranteed everywhere. Sensitive data shouldn't leave the building it was created in. None of those problems get solved by a bigger model. If anything, they get worse.

A model that runs locally, fast, cheap, and reliably on one specific task isn't a downgrade from a giant general purpose model. It's a different tool built for a different job, the way a screwdriver isn't a worse hammer.

The giant frontier models aren't going anywhere, and they're still pushing the frontier of what's possible, which is exactly what they should be doing. But the actual day to day work of building things that run reliably, cheaply, and close to where people are, that's increasingly happening on models small enough that a few years ago nobody would have bothered writing a paper about them.

That's the part I find genuinely interesting. Not that small models exist, but that betting on them is now the obviously correct engineering decision in a lot more situations than people expected.

What I Learned About the Lightning Network in My First Week of Bootcamp

Walter Hrad — Mon, 15 Jun 2026 08:29:13 +0000

A few weeks ago I sat down for the Bitcoin Lightning Network bootcamp at Zone01 Kisumu with a vague idea that Lightning was "the fast version of Bitcoin." That's about as accurate as saying a smartphone is "the fast version of a brick." It's not wrong, but it skips over the part that actually matters: how it works underneath.

This article is a walkthrough of what clicked for me during the first stretch of the bootcamp, mostly around channels and payments. I'm writing it the way I wish someone had explained it to me on day one, with the confusion left in rather than smoothed over.

The problem Lightning is solving

Before touching any Lightning concepts, the bootcamp made us sit with the actual problem. Every transaction on the Bitcoin blockchain has to be broadcast, verified by miners, and included in a block. That's the whole point of Bitcoin's security model, but it means transactions are slow and, depending on network congestion, expensive. Paying for a coffee with an on-chain transaction is technically possible but practically absurd.

Lightning's pitch is: what if two people who transact with each other often didn't have to touch the blockchain every single time? What if they could settle up on-chain only occasionally, while doing the actual back-and-forth payments off-chain, instantly, for nearly nothing?

That's the entire idea. Everything else is just the engineering required to make that idea trustworthy.

Channels: the unit everything is built on

The first real concept is the payment channel. Two parties open a channel by committing funds into a special on-chain transaction, a 2-of-2 multisig output. Both parties have to sign off on anything that happens with that money from this point forward.

Once the channel is open, the two parties can send funds back and forth between each other by exchanging signed updates that represent the current balance split. None of these updates touch the blockchain. They're just signed messages held by both parties, each one superseding the last.

When either party wants to stop, they broadcast the latest balance state to the chain, and the channel closes with funds distributed according to that final state.

What took me a while to get is that the channel doesn't track "transactions" the way I think of them. It tracks a running balance. If I open a channel with 100,000 sats and send my counterparty 5,000 sats, the new state isn't "a transfer happened," it's "the balance is now 95,000 / 5,000." Every payment is really just both parties agreeing to a new split and signing it.

What happens if someone cheats

This is where the bootcamp slowed down and made sure we actually understood it, because it's the part that makes Lightning trustworthy without a central authority.

Each balance update invalidates the previous one. But what stops someone from broadcasting an old state where they had more funds, the one before they sent money out?

The answer is a punishment mechanism. Every time the channel state updates, both parties generate a new commitment transaction, and as part of agreeing to the new state, each party gives the other a revocation secret for the old state. If someone tries to broadcast an outdated commitment transaction, the other party can use that revocation secret to claim the entire channel balance as a penalty.

So broadcasting an old, more favorable state isn't a sneaky shortcut. It's a way to lose everything. The incentive structure does the enforcement, not a referee.

I found this genuinely elegant once it landed. The system doesn't prevent cheating by making it impossible. It prevents cheating by making it economically suicidal.

Routing payments through people you've never met

Channels alone would only let you pay people you've directly opened a channel with, which isn't useful at scale. The next piece is routing payments across a network of channels you're not directly part of.

Say I have a channel with Alvin, and Alvin has a channel with Ronnie. I can pay Ronnie without ever opening a channel with him directly, by routing the payment through Alvin.

The mechanism that makes this safe is Hashed Timelock Contracts, or HTLCs. Ronnie generates a secret, hashes it, and gives the hash to me (via an invoice). I forward a conditional payment to Alvin: "you get this amount if you produce the preimage of this hash, and you have until this block height to do it." Alvin forwards the same conditional payment to Ronnie. Ronnie, who knows the secret, reveals it to Alvin to claim the payment. Alvin now has the secret and reveals it to me to claim his payment.

The result: the payment moves across the route atomically. Either the whole chain of HTLCs resolves and everyone gets paid, or it times out and everyone reverts. Nobody in the middle can walk away with funds without forwarding them, because the only way to claim the incoming HTLC is to produce a preimage they can only get by paying out the outgoing HTLC.

The timelocks decrease as you go further from the payer, which I initially found unintuitive until I realized why: it gives nodes earlier in the route enough time to claim or fail their HTLC after observing what happens downstream.

Setting up nodes and actually sending a payment

The bootcamp had us spin up LND nodes connected to a Bitcoin Core regtest network, open channels between our nodes, and send payments across them. Doing this hands-on is where a lot of the theory actually solidified.

A few things stood out in practice:

Opening a channel is itself an on-chain transaction, so it needs to be confirmed before the channel is usable. On regtest this is trivial since you can mine blocks on demand, but it's a good reminder that Lightning doesn't eliminate on-chain activity, it just minimizes it.

Creating an invoice on the receiving node generates that hash I mentioned earlier, encoded along with the amount and other metadata into a string you'd recognize as a Lightning invoice (the ones starting with lnbc). Paying that invoice from another node is what kicks off the HTLC chain.

Watching lncli output during a multi-hop payment, and seeing the HTLCs appear and resolve across the route in real time, made the routing concept click in a way that diagrams hadn't. There's something about watching the actual state changes happen on nodes you set up yourself that diagrams can't replicate.

What I'm still working through

Channel balance and liquidity management is the next thing on my list. A channel can only route payments in a direction it has capacity for, which means the network's ability to route payments depends heavily on how liquidity is distributed across channels. Rebalancing, fee policies for routing nodes, and how nodes decide on routes (especially with private channels and onion routing for privacy) are all things I've only scratched the surface of.

I'm also curious about how this connects to the broader infrastructure work I've been doing, particularly around building tools that interact with these nodes programmatically rather than through lncli. That feels like a natural next step.

Closing thought

What surprised me most about this first stretch wasn't any single technical detail, it was how much the whole system relies on aligning incentives rather than adding more rules. Channels, revocation secrets, HTLCs, all of it boils down to making honest behavior the only behavior that makes economic sense.

That's a different kind of engineering than I'm used to from typical software work, where you're usually trying to prevent bad states through validation and access control. Here, the bad states are allowed to exist, they're just made unprofitable. I'm looking forward to digging into the next layer of this.

The Man Who Armed the Internet and Then Spent His Life Trying to Fix It

Walter Hrad — Fri, 05 Jun 2026 08:44:28 +0000

There is a particular kind of person in security who does not fit neatly into the categories the industry likes to use. Not purely a researcher, not purely a builder, not purely an attacker or a defender. HD Moore is that person. He built the most widely used hacking tool in the world, handed it to the public for free, watched it get used in ways that made a lot of people nervous, and then spent the next two decades using that same instinct for exploration to help organizations understand exactly how exposed they really are. His career reads less like a resume and more like a long argument with the security industry about what openness actually means.

To understand him, you have to go back to Austin, Texas, and a teenager with too much time and too many phone lines.

Dialing in the Dark

HD Moore grew up around Austin in the 1990s, which meant he grew up in a particular era of the internet that people who were not there tend to romanticize without fully understanding. Before the web took over everything, before social media, before the endless scroll, there was a different kind of exploration available to anyone with a modem and a phone line. You could just dial numbers. Random numbers. Area code, exchange, four digits, and see what picked up.

Most of the time nothing happened. But occasionally a computer answered. An old UNIX machine somewhere, sitting in a university basement or a corporate server room, configured to accept inbound connections from whoever happened to be curious enough to dial in. Nobody had thought particularly hard about what would happen if a teenager in Austin started working through every number in the 512 area code.

Moore's mother was a medical transcriptionist, and that job came with an unusual household setup. Multiple phone lines. An ISDN line. Two computers. And a mother who went to bed early. He would run a program called ToneLoc across the entire area code, night after night, logging whatever picked up. HVAC systems at department stores. Radio transmission towers. Machines that had no business being reachable by a kid sitting at home, but were, because nobody had locked the door.

He was not doing this to cause damage. He was doing it because it was fascinating. There was a whole world connected by phone lines, and almost none of it knew he was there. That feeling, of discovering something that was just sitting there waiting to be found, never really left him.

This is important context for everything that came later. Moore is not a person who got into security because he wanted to break things. He got into it because he wanted to know what was there.

The Phrack Channel and the Air Force

Moore spent a lot of his early online life in the Phrack IRC channel. Phrack was the hacker magazine, the one that had been running since 1985 and had, by the mid-90s, accumulated a serious body of knowledge about how systems worked and how they could be broken. The IRC channel was where the people who read Phrack talked to each other, and it was, apparently, also where defense contractors went to find entry-level talent.

Someone in that channel sent Moore a message asking if he was looking for work. He was still in high school. The job was with Computer Sciences Corporation, doing work for what was then the Air Force Intelligence Agency, building offensive tools for red teams inside the Air Force. They needed someone who could write exploits, who understood how networks worked at a low level, who was comfortable with the kind of technical problem that does not have clean documentation.

Moore was, by his own admission, not a particularly good programmer at the time. But he understood the material, and he was willing to learn. His first professional experience was getting vague briefs about tools that needed to scan networks for open registry keys or intercept specific kinds of traffic, and then going off and building them. This was before most of the industry had formalized the idea of red teaming. The people doing it were figuring it out as they went.

That job led directly to his next move. After doing a penetration test on a local business and basically walking through every layer of their security without being stopped, he and the team went back to CSC and proposed expanding into commercial pen testing. CSC said no. They were a federal contractor, and that was what they did. So the team left and started their own company: Digital Defense.

The Problem That Made Metasploit Necessary

Digital Defense was doing penetration testing for clients, and the work exposed a problem that the industry did not have a good answer to.

Here is how a pen test worked in the late 1990s: you ran a vulnerability scanner to find out what was running on the network, you matched up the versions against lists of known vulnerabilities, and then you needed to prove to the client that the vulnerability was actually exploitable. It was not enough to tell someone their server was unpatched. Anyone could say that. The value was in demonstrating what an attacker could actually do with it.

That meant you needed working exploits. And getting working exploits was, at the time, genuinely difficult. There were some hacker sites where you could download exploit code, but that code was often old, often undocumented, and potentially carrying malware. Nobody had taken the time to build a clean, trusted, well-organized collection. The commercial options, like Core Impact, existed but were expensive and limited. For a small firm doing pen tests, the practical answer was to write your own, which required constant reinvestment of time and effort.

Moore was accumulating exploits. He had bits and pieces scattered across his machines, some written by him, some shared by people he trusted, none of it organized in a way that made it easy to hand to a teammate or use reliably on a client engagement. The information security community had started to dry up around sharing exploits. The people who had been sharing freely in the 90s were either getting real jobs that made sharing complicated, or running into legal concerns, or simply drifting away. What had been available was disappearing.

Moore's answer was to build a framework. A single application that held exploits in a consistent format, with documentation, with known payloads, without hidden garbage. Something he could trust on a client network. Something he could add to over time. That was the original Metasploit: a practical tool, built to solve a real problem he was having at work, that he happened to release publicly.

What Metasploit Actually Was

The first version was menu-based, terminal-based, and not particularly elegant. You picked an exploit, picked an encoder, picked a payload, and sent it. Functional but rigid.

By Metasploit 2, the architecture had been rethought. The core idea was modularity. An exploit and a payload were separate things, and you could combine them like components. Before this, most exploits came with one or two hardcoded payloads and that was it. What Moore and his collaborators built was a system where any compatible payload could be attached to any compatible exploit, which multiplied the number of possible attack configurations enormously.

The payload metaphor is worth understanding. If the exploit is the needle that gets through a system's defenses, the payload is what the needle is carrying. It could open a command shell. It could establish a persistent connection back to the attacker. It could do almost anything, depending on what you loaded into it. Metasploit made it easy to swap payloads in and out, which meant that security researchers could test not just whether a system was vulnerable, but what an attacker could realistically do with that vulnerability.

This was genuinely useful for pen testers. It was also genuinely useful for people who wanted to attack systems without permission. Moore knew this and released it anyway, which is where the controversy started and never really stopped.

His position was consistent. The information was not secret. The vulnerabilities were already known. Hiding the tools did not make the vulnerabilities go away; it just made it harder for defenders to understand what they were defending against. A security professional who could not replicate what an attacker could do was working with incomplete information. The argument had real merit, and it still does.

The Moral Weight of Building Metasploit

The Darknet Diaries episode about Moore frames this tension honestly. Metasploit was crammed with exploits and payloads that could be used to compromise computers. Anyone with the tool and a target could cause serious damage. And Moore put it on the internet for free.

What makes his position defensible is not that the tool was never misused. It was misused. What makes it defensible is that the alternative, keeping effective security tools restricted to commercial vendors and government agencies, does not actually make systems more secure. It just makes it easier for institutions to charge money for security services while leaving the broader community without the knowledge needed to evaluate the claims those institutions make.

There is also the question of what Metasploit did for education. For the generation of security professionals who came up in the 2000s and 2010s, Metasploit was often the first real penetration testing tool they got their hands on. It was free, it was documented, and it lowered the barrier to actually understanding how exploits worked at a technical level. People who went on to do serious security work, both offensive and defensive, often trace part of their foundation to time spent with Metasploit. You cannot fully evaluate that against the cases where someone misused it and pretend the accounting is straightforward.

Moore built something that changed the industry. The people who benefited from it vastly outnumber the people who used it to cause harm, and the harm that was caused would largely have been possible anyway for anyone with the motivation to find the same exploits elsewhere. What would not have existed otherwise is the community, the shared language, the common framework that allowed the field to develop a shared understanding of offensive techniques.

Rapid7 and What Came After

In 2009, Rapid7 acquired Metasploit. Moore joined the company as chief security officer and stayed on as chief architect of the framework. He had been building Metasploit largely in his spare time, nights and weekends and lunch breaks, and the acquisition meant there would be actual resources dedicated to it full-time.

At Rapid7, Moore continued the kind of large-scale internet scanning work that had been a thread through his whole career. The most significant of these projects was what became known as Project Sonar, which involved scanning the entire public internet to understand what was actually exposed. In 2013, this work produced a notable finding: roughly 50 million networked devices were exposed to the internet through flaws in Universal Plug and Play, a protocol that had been designed for home networks and had no business being accessible from the public internet. This was not a targeted discovery. It was the result of simply scanning everything and looking at what came back.

That kind of work, running probes across the entire address space and analyzing what responds, gave Moore a perspective on internet exposure that almost nobody else had. He had seen the public internet from the outside, systematically, at scale. He knew what was out there and how bad the configuration problems were across every industry.

He left Rapid7 in 2016.

The Problem He Could Not Stop Thinking About

After Rapid7, Moore spent time at Atredis Partners doing research and development. But the problem that kept pulling at him was one he had run into repeatedly over the course of his career: organizations did not know what was on their own networks.

This sounds basic, but it is one of the most persistent and serious problems in enterprise security. You cannot protect what you cannot see. And the larger an organization gets, the harder it is to maintain an accurate picture of every device, every service, every connection. Things get added without being tracked. Old systems never get decommissioned. Consumer devices appear on corporate networks because employees connect personal devices to Wi-Fi. IoT devices get deployed in facilities and forgotten about. The picture that the security team has of their own network is almost always incomplete, and often dramatically so.

Moore had seen this from both sides. As a penetration tester, he had scanned client networks and found things the client did not know were there. As someone who had scanned the entire public internet, he had a clear view of how exposed the outside-facing portion of organizational infrastructure was. The inside of the network was usually worse.

In 2018, he founded what was initially called Rumble Network Discovery to address this directly. The tool was designed to scan a network and identify everything on it, fingerprinting devices precisely enough to give organizations an accurate picture of their attack surface. Not just identifying hosts, but identifying what they were, what was running on them, and what the security implications were.

The company later renamed itself runZero, a name that referenced the idea of starting from zero assumptions about what is on the network and building an accurate picture from scratch. Moore found, consistently, that when his tool ran on a corporate network, it found things the security team had not known were there. PlayStation 4s. Amazon Echo devices. Smart TVs. Weather stations that had updated their firmware and opened unauthenticated telnet services on the local network. The gap between what organizations thought they had and what was actually there was often startling.

Why His Career Makes Sense as a Whole

HD Moore has spent his career doing a version of the same thing. Finding what is there when nobody has looked carefully. As a teenager, it was running ToneLoc across area codes to see what picked up. As a young pen tester, it was building a framework for discovering what vulnerabilities were actually present and exploitable on client networks. At Rapid7, it was scanning the public internet to see what was exposed. At runZero, it is scanning internal networks to show organizations the gap between their assumed state and their actual state.

The thread is curiosity about what exists, combined with a commitment to making that knowledge available rather than hoarding it. Metasploit was not hidden. Project Sonar data was published. The research Moore did on UPnP exposure was shared publicly so that device manufacturers and ISPs could address it. The fingerprint database that runZero uses, Recog, is open source and contributed back to the broader community.

This is a coherent position. Information about how systems work, including information about how they can be broken, is more useful in the open than locked away. The security industry has spent decades trying to argue both sides of this question depending on what was commercially convenient, and Moore has been one of the clearer voices insisting that openness, on balance, makes things better.

He is also a person who clearly finds the work interesting for its own sake. He described the internet to someone once as just a series of numbers. Make up any random 32-bit address and there is probably something there. That observation has a quality of genuine wonder in it that does not come from someone who got into the field to make money or build influence. It comes from someone who dialed random phone numbers as a teenager because they wanted to know what would pick up, and never really stopped asking that question.

What He Leaves Behind

Metasploit is now owned by Rapid7 and maintained by a large team. It remains the dominant open source penetration testing framework. Certifications like OSCP require students to use it. Security courses around the world are built around it. The number of security professionals who developed their foundational skills with Metasploit is enormous.

RunZero has become a real company with funding and customers. The asset discovery problem Moore identified has not gone away. If anything, the explosion of cloud infrastructure, remote work, and IoT devices has made it worse, which means the work he started in 2018 is more relevant now than when he began it.

The Darknet Diaries episode that brought Moore to a wider audience in 2022 has over 400,000 plays. The story it tells is not a simple one. It is not a story about a genius who built something and got rich. It is a story about a person who had a particular way of looking at technical problems, who made a decision to share what he built instead of keeping it to himself, and who spent decades living with the consequences of that decision while continuing to push in the same direction.

That kind of consistency is rare. The security industry is full of people who pivot toward whatever is commercially valuable at a given moment. Moore has been doing essentially the same thing since he was a teenager in Austin dialing random phone numbers at midnight. The tools have changed. The scale has changed. The question he is asking is the same.

What is actually out there? And who else knows about it?

The Lie We Are All Telling Ourselves About AI Code

Walter Hrad — Wed, 27 May 2026 09:56:28 +0000

There is a number that has been circling developer circles for the past few months and nobody seems to know what to do with it. A broad analysis of 470 GitHub pull requests found that AI-generated code was 1.7 times more likely to have major issues like logic errors, and 2.74 times more prone to security vulnerabilities compared to human-written code. (daily.dev) Hardcoded API keys. Plain-text passwords. APIs that do not even exist, hallucinated from outdated training data.
And yet, developer adoption is near-universal. Big Tech is generating anywhere from 25 to 90 percent of new code with AI, depending on the organisation. (Hostinger)
So we have a situation where the tools are objectively producing riskier code, and we are using them more than ever. That is not a contradiction. That is a rationalization.
The most counterintuitive finding of 2026 is this: researchers ran a randomized controlled trial with experienced open-source developers on real codebases doing real tasks, and they were measurably slower when using AI tools. They did not know it. Even after the experiment, they believed AI had helped them. (Hashnode) Broader survey data shows 95 percent of developers report feeling productive while measurably producing lower-quality code. (Hashnode)
That gap between feeling productive and being productive is where a lot of careers are quietly getting damaged right now.
I am not writing this to tell you to stop using these tools. I use them too. The point is that we have collectively agreed to stop asking hard questions about what we are actually shipping. A prototype that works in a demo is not the same thing as a product someone depends on. The "three-month black box" problem is real: projects become unmanageable because nobody on the team actually understands the AI-generated codebase they are now responsible for maintaining. (daily.dev)
Andrej Karpathy coined the term "vibe coding" in early 2025. The idea was straightforward: describe what you want in plain English, let AI write the code, and do not obsess over every line. (Alex Cloudstar) That framing made sense for personal projects and quick experiments. What happened next is that companies started treating it as a production methodology.
By 2026, 41 percent of all code globally is AI-generated. (daily.dev) Some of that is fine. A lot of it is sitting in production systems that someone will have to debug at 2am six months from now, written by an AI that had no stake in whether the thing holds up.
The security angle is the part that should concern anyone working in this industry. Security firm Tenzai tested five popular AI coding tools and found that 45 percent of AI-generated code samples contain OWASP Top-10 vulnerabilities. (Hashnode) These are not obscure edge cases. These are the most well-documented, widely-known vulnerability categories in web development. The kind of mistakes a junior developer would get flagged for in a code review.
What is the fix? Honestly, it is boring. The difference between a vibe-coded prototype and a vibe-coded product is the automated verification layer that runs on every change. Set up tests, type checking, and linting from day one. These quality gates catch AI-generated mistakes before they compound into larger problems. (DEV Community) That is not revolutionary advice. It is just the fundamentals that people started skipping because the AI made everything feel so fast.
One engineer described it well: the best approach is a pendulum method, going back and forth between AI-assisted and hand-written code. Not because AI is bad, but because staying sharp matters. You do not want to go too long without writing something by hand and lose your mental model of what good code actually looks like. (Explainx)
That feels right to me. The developers who will be fine are the ones treating AI as a collaborator they have to supervise, not a replacement for thinking. The ones who will struggle are the ones who have outsourced their judgment along with their keystrokes.
We built careers on understanding systems. That understanding is still the job. The keyboard just got faster

You Understand It. You Just Cannot Build It Yet.

Walter Hrad — Sat, 23 May 2026 11:42:06 +0000

At some point you will understand something completely and still not be able to use it.
This happens more than people admit. You read the documentation. You watch the tutorial. Someone explains it to you and you nod along and you genuinely follow every step. And then you open a blank file and your hands do not know what to do.
This is not a knowledge gap. The knowledge is there. This is something else and I think it matters to name it correctly because if you think it is a knowledge gap you will go read more documentation and that will not help.
The thing that is actually missing is a different kind of familiarity. Not understanding how something works but having used it enough times that your hands remember what to reach for. There is a difference between knowing that useState takes an initial value and returns a pair and actually reaching for useState without thinking when you need reactive state in a component. The first one lives in your head. The second one lives somewhere else, in your fingers maybe, or in whatever part of your brain handles automatic things.
Musicians talk about this. You can understand music theory completely and still not be able to improvise. The theory is in your head but improvisation requires something that bypasses your head. Athletes talk about it too. The gap between understanding a technique and executing it under pressure is not closed by more understanding. It is closed by repetition. A footballer who has studied penalty kicks all week still has to take thousands of them before his body knows what to do when the stadium goes quiet and it actually matters.
Programming culture is weird about this because we treat it as purely intellectual work. If you understand the concept you should be able to apply it. But that is not true and I think a lot of people quietly suffer from believing it. They understand closures but cannot write one without looking it up. They understand recursion but freeze when they need to use it in a real problem. They feel like frauds because the gap between their understanding and their output is visible to them and they think it should not exist.
It should exist. It always exists. For everyone. The only way it closes is by building things, specifically by building things where you do not fully know what you are doing. Not following a tutorial step by step. Not copy-pasting from a project that already works. Actually sitting in front of a blank file and figuring it out slowly, making mistakes, getting stuck, coming back the next day.
The stuck part is important and I want to stay on it for a moment because people treat being stuck as evidence that they are doing the wrong thing. They take it as a signal to go watch another video or find a better explanation. But getting stuck is where the second kind of familiarity forms. When you cannot remember the syntax and you have to look it up for the fourth time, your brain is doing something with that friction. The fifth time will be easier. Not because you read something new but because you went through the process of not knowing and then finding out again. That cycle is doing something that passive reading cannot do.
There is also something that happens when you make a mistake in code you actually wrote versus code you copied. When you copy code and it breaks you troubleshoot the library, the environment, the tutorial. When you write code and it breaks you are forced to understand why, because you cannot blame anyone else. That accountability is uncomfortable but it is also the thing that makes the knowledge stick. You remember the bug you actually caused in a way you will never remember a bug from a tutorial.
I think the reason people avoid this is that getting stuck feels like not knowing. And not knowing feels like failure. So they stay in the tutorial loop where everything works and they always know what to do next and they understand everything perfectly and they cannot build anything on their own. The tutorial loop is comfortable because someone else has already figured out all the places where things could go wrong and smoothed them over before you arrived. Real projects are not like that. Real projects have rough edges everywhere and no one has prepared the path for you.
This is also why side projects that you actually care about are more valuable than exercises you do just to learn. When you care about the thing you are building, you tolerate more confusion. You come back after the third failed attempt because you actually want the thing to exist. That motivation is what gets you through the part of the process where you do not know what you are doing, which is most of the process for most projects.
There is no comfortable path from understanding to building. The discomfort is the path. That probably sounds like something someone would put on a poster but I mean it practically. If you are comfortable, you are probably in a mode that is not building the second kind of familiarity. If you are uncomfortable, you are probably exactly where you need to be. The goal is not to eliminate the discomfort but to develop a tolerance for it, to learn that confusion is a stage and not a verdict.
The move is to build something you do not know how to build yet. Not something slightly outside your comfort zone. Something where you genuinely do not know how to start. Then start anyway and see what happens. Pick something small enough that finishing it is realistic but unfamiliar enough that you will have to figure things out. The specific thing matters less than the act of choosing and starting.
What usually happens is that you figure it out slower than you wanted to, with more mistakes than you expected, and you come out the other side able to do something your hands did not know before. And the next time you sit in front of a blank file it will feel slightly less empty. Not because you learned more. Because you built something once and your body remembers that it survived.
That is the whole game. Understanding gets you started. Building is what finishes it.

Nobody Wants to Read Your Code (And You Don't Want to Read Theirs)

Walter Hrad — Sat, 23 May 2026 11:36:02 +0000

There is a thing that happens when you join a new codebase. You open a file, read maybe fifteen lines, and then you close it. Not because you understood it. Because you stopped wanting to understand it.

I think about this a lot. We talk about clean code, readable code, well-documented code, as if the problem is always on the writing side. But the reading side has its own psychology and almost nobody talks about it honestly.

Reading someone else's code is uncomfortable in a way that is hard to name. It is not just that the logic is unfamiliar. It is that you are stepping inside someone else's head. Their assumptions are baked into every variable name. Their shortcuts make sense only if you know what they were rushing toward. Their weird choice of a for loop where a while loop would feel more natural is probably the result of a bug they fixed at 11pm six months ago and never bothered to refactor. You are not reading code. You are reading a person.

And most of us are not trained for that. We are trained to write, to produce, to ship. Reading feels passive. Reading someone else's bad code feels like losing. Reading someone else's good code feels even worse because then you have to sit with the fact that you would not have done it that way and you are not sure if your way is actually better.

There is also the time problem. When you write code, you have context. You know what the function is for, what came before it, what will come after. When you read, you have to build that context from scratch and the codebase usually does not help you. Comments are either missing or lying. Variable names like data and result and temp are everywhere. The folder structure made sense to the person who created it and to nobody else.

So you do what most people do. You read just enough to do your task. You make your change in the one file you actually understand. You submit your PR and you move on.

The problem is that this pattern compounds. The person after you does the same thing. And the person after them. Over time the codebase becomes a city where everybody knows their own neighborhood and nobody knows the roads in between. When something breaks in the space between neighborhoods, it takes three people and a long afternoon to figure out why.

I do not think the solution is to force people to read more code through code reviews or documentation requirements. Those things help but they do not fix the underlying discomfort. The discomfort is real and it deserves to be taken seriously.

What has actually helped me is reading code the way I read a book I am not enjoying. Not to finish it. Not to extract every piece of information. Just to follow one thread. Pick one function. Understand what it calls. Understand what calls it. Stop there. Do that a few times over a few days and the codebase starts to feel less like a foreign country.

The other thing that helps is accepting that confusion is not a sign of failure. The confusion you feel when you open an unfamiliar file is the same confusion the person who wrote it felt when they started. They just had more time. Time is the only real difference between a file that makes sense and one that does not.

We should probably talk more openly about how hard reading code actually is. Not as a complaint but as an acknowledgment. It is a skill that takes practice and patience in a field that rewards speed. That tension is not going anywhere. Might as well be honest about it.

When "It Works on My Machine" Stops Being Good Enough

Walter Hrad — Tue, 12 May 2026 12:16:43 +0000

There is a version of learning to code where the goal is just to make the thing run. You write it, you test it locally, it does what it is supposed to do, and that feels like the finish line.
For a while, that was me.
ascii-art-web changed that. The project was straightforward on paper — take the ASCII art CLI I had already built and put it behind a Go HTTP server with a browser interface. Same logic, new layer. I figured it would be a lighter week.
It was not.
The standard library in Go handles HTTP cleanly but it does not hide anything from you. Every route has to be defined. Every status code has to be intentional. The audit requirements were specific — 400 for bad input, 404 for missing routes, 500 when the server breaks. Not approximately right. Exactly right. I had written code before that worked in the sense that it produced correct output. This was the first time I had to write code that behaved correctly under conditions I had not planned for.
That is a different skill and I did not fully have it yet.
Getting the frontend and the server to actually talk to each other took more back and forth than I expected. Form data that seemed fine in the browser arriving malformed on the Go side. Routes that I was sure existed returning 404s. Small things that each took longer than they should have to debug because I did not yet have a clean mental model of the full request cycle.
By the end of the week I had something that worked and handled edge cases properly. That felt different from previous weeks. Not just "it runs" but "it holds up."

Then came Docker.
I had heard of Docker the way you hear about things in tech before you understand them — vaguely, as something that matters, without knowing exactly why. The ascii-art-web-dockerize project made the why concrete pretty fast.
The pitch is simple: your app runs the same everywhere because you are packaging not just the code but the environment it needs. A Dockerfile describes that environment. You build an image from it. You run a container from that image. Anywhere that container runs, your app behaves the same way.
In practice the first time you write a Dockerfile for a Go web server there are a few things that will catch you. Getting the build right inside the container. Making sure the port your server listens on is the port you actually expose and map when you run it. These are not hard problems once you understand the model but the model takes a minute to click.
What stuck with me from that week was less the technical detail and more the shift in thinking. Shipping is not the same as building. A lot of people can build something that works. Fewer can package it in a way that works for someone else, on a different machine, without needing to explain anything. That gap is real and Docker is one of the tools that closes it.

If you are learning and you have not touched deployment or containerisation yet — get there sooner than I did. Writing code is one skill. Shipping it is another. Both matter and the second one does not get talked about enough in beginner spaces.