Compiler vs Interpreter vs JIT: What Actually Happens When Code Executes

Most introductory computer science classes explain execution engines with a neat, three-part story:

1. Compilers convert source code into machine code before execution.
2. Interpreters read and run source code line-by-line at runtime.
3. JIT Compilers combine the two by compiling code while the program runs.

While this summary is simple, it glosses over how modern language runtimes actually function. In real-world software engineering, almost no production interpreter runs source code "line-by-line". Compilers perform dozens of intermediate transformations before touching machine code. Modern JIT compilers actively rewrite running code, modify stack frames while functions are executing, and dynamically undo optimizations when their assumptions fail.

Let's look beneath the high-level definitions to see what actually happens in physical memory, CPU registers, and instruction pipelines across all three execution models.

---

1. Ahead-of-Time (AOT) Compilers: Direct Silicon Execution

Ahead-of-Time (AOT) compilers (used by languages like C, C++, Rust, and Go) translate human-readable source files into native machine binaries before the program ever runs.

Source Code (.rs / .c)
       │
       ▼
 [ Lexer / Scanner ]  ──► Token Stream (IDENT, OP, CONST)
       │
       ▼
 [ Parser ]           ──► Abstract Syntax Tree (AST)
       │
       ▼
 [ Type Checker / Semantic Analysis ]
       │
       ▼
 [ Intermediate Representation ] (LLVM IR / SSA Form)
       │
       ▼
 [ Target-Independent Optimization Passes ]
 (Constant Folding, Dead Code Elimination, Inlining, Loop Vectorization)
       │
       ▼
 [ Backend Code Generator ]
 (Instruction Selection, Register Allocation, Instruction Scheduling)
       │
       ▼
 [ Linker ]           ──► ELF / Mach-O / PE Binary

The Optimization Pipeline

Modern compilers do not convert an Abstract Syntax Tree (AST) directly into x86 or ARM assembly. Instead, they translate the AST into an Intermediate Representation (IR), usually structured in Static Single Assignment (SSA) form. In SSA form, every variable is assigned exactly once, which makes data dependencies explicit.

In this intermediate state, LLVM or GCC runs optimization passes:

• Constant Folding and Propagation: Pre-calculating arithmetic operations known at compile time.
• Dead Code Elimination (DCE): Removing code paths that can never be reached or whose outputs are never read.
• Function Inlining: Replacing a function call instruction with the actual body of the called function, eliminating stack frame setup costs.
• Loop Vectorization (SIMD): Packing multiple loop iterations into 128-bit or 256-bit vector registers (AVX2 / NEON) so a single CPU instruction processes multiple data points simultaneously.

The Backend and Register Allocation

Once optimized, the compiler's backend translates IR instructions into target-specific machine instructions.

This requires Register Allocation (using algorithms like Chaitin-Briggs graph coloring or linear scan). The compiler maps an arbitrary number of IR variables to a small set of physical CPU registers (RAX, RCX, RDX, R8-R15). Variables that do not fit into registers are spilled onto the call stack in RAM.

What Happens at Execution?

When you run an AOT binary on Linux (./my_program):

1. The kernel handles the sys_execve system call.
2. The ELF loader parses the binary headers and maps the segments into virtual memory:
   • .text segment: Executable machine code (read-only, executable).
   • .rodata segment: String literals and constants (read-only).
   • .data and .bss segments: Global and static variables (read-write).
3. The kernel sets up the stack pointer (RSP) and jumps the CPU instruction pointer (RIP) directly to _start.

; Native assembly emitted by an AOT compiler for adding two numbers
mov eax, edi        ; load first parameter from EDI register
add eax, esi        ; add second parameter from ESI register
ret                 ; return result in EAX

There is no virtual machine, no interpreter loop, and no runtime translation layer. The CPU executes raw opcodes directly on silicon at maximum clock speed.

---

2. Interpreters: Bytecode and the Dispatch Tax

The traditional mental model of an interpreter "reading source code line by line" only describes very early or specialized script engines (such as early shell scripts or basic calculators).

Why Tree-Walking Interpreters Are Slow

An engine that recursively walks an Abstract Syntax Tree (eval(node)) faces severe physical hardware bottlenecks:

• Cache Misses: AST nodes are scattered across the heap as linked tree objects. Traversing them forces the CPU to constantly fetch memory from slow L3 cache or RAM.
• Call Stack Overhead: Evaluating nested expressions creates deep native C recursion, filling the native stack.

Because of this, modern interpreters (like CPython, Ruby YARV, and Lua) parse source code once during startup and compile it into a linear sequence of bytecode.

Source Code (.py)
       │
       ▼
 [ Lexer + Parser ]  ──► Abstract Syntax Tree (AST)
       │
       ▼
 [ Bytecode Compiler ]──► Linear Bytecode Instructions (e.g. LOAD_FAST, BINARY_OP)
       │
       ▼
 [ Virtual Machine Dispatch Loop ]

The Bytecode Virtual Machine Loop

A bytecode virtual machine is essentially a software-emulated CPU. It maintains its own virtual program counter and evaluation stack.

In CPython, the execution loop is implemented in Python/ceval.c:

// Simplified conceptual bytecode dispatch loop
while (1) {
    opcode = *ip++;
    switch (opcode) {
        case OP_LOAD_FAST:
            PUSH(stack, locals[oparg]);
            break;
        case OP_BINARY_ADD: {
            PyObject *b = POP(stack);
            PyObject *a = POP(stack);
            PyObject *res = PyNumber_Add(a, b);
            PUSH(stack, res);
            break;
        }
        case OP_RETURN_VALUE:
            return POP(stack);
    }
}

The Three Costs of the "Interpreter Tax"

Why is interpreted bytecode typically 5x to 50x slower than compiled native code?

1. The Dispatch Loop Overhead

Every bytecode instruction requires fetching the opcode, decoding it, and jumping to the handler. In a switch-based loop, this central jump point causes constant CPU branch prediction failures.

Advanced interpreters use Direct Threaded Code (using GCC computed gotos: goto *dispatch_table[opcode]) to spread branch prediction across individual handlers, but the dispatch overhead remains substantial.

2. Dynamic Type Inspection

In statically compiled languages, types are fixed at compile time. In dynamic languages, the VM does not know what types are being processed until the moment of execution.

When Python executes a + b, the VM must check:

• Is a a PyLongObject?
• Is b a PyLongObject?
• If not, does a implement tp_as_number->nb_add?
• Does b implement nb_radd?

What takes an AOT compiler a single add eax, ebx CPU instruction requires dozens of conditional branches, pointer dereferences, and type checks in an interpreter.

3. Boxing and Object Allocation

In native C or Rust, an integer is a raw 32-bit or 64-bit value sitting in a CPU register. In Python, an integer is a boxed heap structure (PyLongObject) containing:

• Reference count (8 bytes)
• Type object pointer (8 bytes)
• Size metadata (8 bytes)
• Digit array (4+ bytes)

Adding two integers often requires reading multiple pointers in heap memory, allocating a new PyLongObject on the heap, setting its reference count, and returning its address.

---

3. Just-In-Time (JIT) Compilers: Dynamic Speculation

JIT compilation bridges the gap between the instant startup of an interpreter and the execution speed of an AOT compiler.

A modern JIT (such as Google V8 in Node.js/Chrome, or the Java HotSpot VM) does not just compile everything to machine code upfront. Instead, it uses Tiered Compilation.

              JavaScript / Java Source
                         │
                         ▼
                   [ Bytecode ]
                         │
         ┌───────────────┴───────────────┐
         ▼                               ▼
    [ Tier 0: Interpreter ]     [ Execution Starts Instantly ]
    (Collects Type Feedback)
         │
         │ (Function called frequently: "Warm")
         ▼
    [ Tier 1: Baseline JIT ]    (e.g., V8 Sparkplug / Java C1)
    (Fast 1-pass compilation, removes dispatch loop)
         │
         │ (Loop / Function is "Hot")
         ▼
    [ Tier 2: Optimizing JIT ]  (e.g., V8 TurboFan / Java C2)
    (Speculative Optimization, Inlining, Vectorization)
         │
         ├─── [ Speculation Holds ] ──► Maximum Native Speed
         │
         └─── [ Assumption Violates ] ──► [ Deoptimization / Bailout ]
                                              (Drops back to Interpreter)

The Modern Multi-Tier Architecture (V8 Example)

1. Ignition (Interpreter): Starts executing bytecode instantly. As it executes, it records metadata in Feedback Vectors (such as the observed types of variables at every operation).
2. Sparkplug (Baseline Compiler): If a function runs a few times, Sparkplug compiles the bytecode directly to native machine instructions in a single pass without building an intermediate optimization graph. This eliminates the bytecode dispatch loop immediately.
3. Maglev (Mid-Tier Compiler): In newer V8 releases, Maglev uses a fast Static Single Assignment (SSA) representation to perform basic optimizations and inlining.
4. TurboFan (Optimizing Compiler): If code becomes "hot", TurboFan uses an advanced Sea-of-Nodes intermediate representation to emit highly tuned machine code.

The Power of Speculative Optimization & Hidden Classes

In dynamic languages like JavaScript, objects are dynamic key-value maps. However, engines assign internal descriptors called Hidden Classes (Maps/Shapes) to objects based on their layout.

Consider this JavaScript function:

function calculateTotal(item) {
    return item.price * item.quantity;
}

An interpreter must look up "price" and "quantity" in dictionary hash tables, check types, and perform arithmetic.

When TurboFan compiles this function, it checks the Feedback Vector. If calculateTotal was always passed objects with the same shape (e.g., { price: number, quantity: number }), TurboFan makes a speculative assumption: This function will always receive this exact shape with 31-bit integers (Smis).

TurboFan generates native machine code with a lightweight guard:

; TurboFan optimized machine code (conceptual)
mov rax, [rbp - 8]              ; Load object pointer
cmp [rax - 1], MAP_ITEM_SHAPE   ; GUARD: Is the hidden class still Map_Item?
jne deoptimize_bailout          ; If shape changed, BAIL OUT!
mov edx, [rax + 12]             ; Direct field offset load: item.price
imul edx, [rax + 16]            ; Direct multiplication: item.quantity
ret

By verifying the shape in a single instruction, the engine bypasses property lookups, hash computations, and type checks entirely.

Deoptimization: What Happens During a Bailout?

What happens if you pass an object with a new property or a string value to calculateTotal({ price: "10.50", quantity: 2 })?

The guard check fails (cmp does not match). The CPU jumps to a deoptimization handler:

1. The engine reads the deoptimization table for the compiled code.
2. It reconstructs the unoptimized Ignition interpreter stack frame, reading values out of CPU registers and placing them into virtual interpreter registers.
3. It marks the optimized machine code as invalid.
4. It seamlessly transfers control back to the Ignition bytecode interpreter at the exact instruction that failed.

The program continues without crashing, though performance drops back to interpreted speed for that execution.

On-Stack Replacement (OSR)

What happens if a function is called only once, but contains a loop that runs 10 million times?

function processData() {
    let sum = 0;
    for (let i = 0; i < 10_000_000; i++) {
        sum += i;
    }
    return sum;
}

If the JIT only checked function invocation counts, this loop would spend its entire execution running in the slow interpreter.

Engines solve this with On-Stack Replacement (OSR):

1. Every time a loop jumps back to its start (a loop backedge), the interpreter increments a loop counter.
2. When the counter exceeds the hot threshold, the JIT compiles the loop body while the loop is actively running.
3. At the next loop iteration safepoint, the runtime swaps the active interpreter stack frame with a newly allocated native machine code stack frame.
4. The CPU instruction pointer (RIP) is pointed directly into the compiled native loop, accelerating execution mid-flight.

---

Architectural Comparison

Dimension	Ahead-of-Time (AOT)	Bytecode Interpreter	Just-In-Time (JIT)
Representative Runtimes	Rust (rustc), C (clang/gcc), Go	CPython, Ruby YARV, PHP Zend	V8 (Node/Chrome), JVM HotSpot, PyPy
Startup Latency	Instant (Direct OS binary load)	Instant (Starts bytecode dispatch immediately)	Instant startup in Tier 0, warms up over time
Peak Throughput	Maximum (Full static optimization)	Low (Subject to the interpreter dispatch tax)	Near-AOT (Often matches AOT via speculative inlining)
Memory Footprint	Small (Only executable code mapped)	Moderate (VM engine overhead)	High (VM + Compiler in RAM + JIT Code Cache + IR graphs)
Runtime Adaptability	None (Static binaries cannot adapt to live data)	High (Every operation dynamically inspected)	Highest (Profiles live data patterns and re-optimizes)
Memory Security	Strict W^X (Code pages are read/exec only)	Strict W^X	Requires dynamic executable memory allocation (PROT_EXEC)

---

The Convergence of Execution Models

The lines between compilers, interpreters, and JIT engines continue to blur:

• Statically compiled languages using JIT concepts: Modern C++ and Rust toolchains use Profile-Guided Optimization (PGO). You compile a binary with instrumentation, run realistic workloads, and feed runtime branch data back into the compiler to re-compile an optimized binary.
• JIT runtimes supporting AOT compilation: Java offers GraalVM Native Image to compile JVM bytecode directly into native ELF binaries, trading runtime profiling for instant startup and low memory footprints.
• Interpreted languages adding JIT tiers: Python 3.13 introduced an experimental copy-and-patch JIT compiler, while PHP introduced a JIT compiler in PHP 8.

Summary Mental Models

1. AOT Compilers do all the heavy lifting upfront. They optimize through SSA IR transformations and emit static machine code that executes directly on hardware.
2. Bytecode Interpreters trade raw execution speed for portability and rapid startup. Their main overhead comes from the instruction dispatch loop, dynamic type lookups, and memory boxing.
3. JIT Engines are dynamic multi-tiered systems. They start in an interpreter, measure execution profiles with feedback vectors, speculatively compile hot paths into native machine code, and deoptimize back to bytecode when assumptions change.