[My Debugger Journey] Managing Execution Modes with NASM

Starting from Reset Vector

I've already documented Reset Vector in my blog. You can read this header from my blog in detail.

Every x86-64 processor starts its life in a mode that has more in common with an 8086 from 1978 than with the machine you're actually running. At power-on, the hardware enforces a rigid initialization contract: CS is loaded with a base of 0xFFFF0000, the instruction pointer sits at 0xFFF0, and the resulting physical address — 0xFFFFFFF0 — is the reset vector, the first fetch the processor will ever make. From that point, control belongs to firmware. (Vol. 3A, §9.1.4 — Processor State After Reset).

What makes this interesting from a systems perspective is that the processor isn't truly in Real Mode at this stage — it's in a transitional state where the CS descriptor cache holds a base of 0xFFFF0000, but the visible CS register still reads 0xF000. The first far jump drops A20–A31 and collapses the processor into the lower megabyte, at which point you're in conventional 16-bit Real Mode with the full segment:offset constraint: physical address = segment × 16 + offset, maximum addressable space 1MB. (Vol. 3A, §9.1 — Processor State After Reset; i386 Reference §10.2.3).

Here's a figure from the intel's volume:

We can only know the state of the processor via CR0 register.

Real-Address Mode and the Segmented Memory Model

Real-Address Mode predates Protected Mode entirely. Before the 80286 introduced hardware-level memory protection and privilege rings, it was the only execution mode available on x86 hardware. The addressing model has not changed since the 8086: a 16-bit segment register is shifted left by 4 bits and added to a 16-bit offset to produce a physical address.

Physical Address = (Segment × 16) + Offset

Since both values are 16-bit, the maximum expressible address is 0xFFFF × 16 + 0xFFFF = 0x10FFEF — but the address bus on the original 8086 was 20 bits wide, so anything above 0x000FFFFF wraps. The addressable range is effectively capped at 1MB.

As a concrete example: if CS holds 0x2000 and IP holds 0x1234, the physical address resolves as:

0x2000 × 16 = 0x20000
0x20000 + 0x1234 = 0x21234

This calculation applies to every segment register — CS, DS, ES, SS — and to every memory access the processor makes in Real Mode. There is no page table, no descriptor table, no privilege check. The segment value is a raw base, not an index into any structure.

Calculating 0xFFFF0000 Address

First step is to calculate the address.

In all x86 processors, Real-Address Mode exists for one reason: backward compatibility with the 8086. It predates Protected Mode entirely, and every x86 processor since has initialized into it — not because it is useful at boot, but because the architecture requires it. The addressing model is unchanged from 1978: a 16-bit segment register is shifted left by 4 bits and added to a 16-bit offset, producing a 20-bit physical address. The addressable range is capped at 0x000FFFFF — the first 1MB.

This creates an immediate contradiction. The reset vector sits at 0xFFFFFFF0, the top of the 32-bit address space — well outside what Real Mode addressing can express. A processor that is both in Real Mode and fetching from 0xFFFFFFF0 should not be possible under the standard model.

The resolution is in the segment register's hidden descriptor cache:

Here's the trick: : Upon reset, the CS register’s Visible Selector is set to 0xF000, but the processor forcibly sets the CS Hidden Base Address to FFFF_0000 (Don’t forget Real Mode’s Segment:Offset addresses).

At reset, the visible CS register holds 0xF000, but the processor loads the cache with a base of 0xFFFF0000 rather than the value the normal shift-and-add calculation would produce. This base is hardwired into the reset state and bypasses the Real Mode formula entirely.

Therefore, the processor calculates the Reset Vector by combining the Hidden Base of CS with the value of EIP:

CS Base (0xFFFF0000) + EIP (0xFFF0) = 0xFFFFFFF0

We can also see this in QEMU:

After the reset, Paging (Bit 31) is disabled and Protection (Bit 0) is not enabled. If this Protection bit were 1, then it would indicate that the processer is in Protection Mode. Also we can see the result of CR0 in GDB:

Now the important section will come.

Fetching Instruction from the Reset Vector

Once the processor resolves 0xFFFFFFF0 and places it on the address bus, it has no knowledge of what backs that address. The processor issues the fetch; where it lands is entirely the chipset's decision.

The chipset — historically the Southbridge, on modern platforms the PCH (Platform Controller Hub) — sits between the processor and every peripheral on the board and monitors every address the processor puts on the bus. When it sees a request targeting the high-memory range around 0xFFFFFFF0, it does not forward that request to DRAM. DRAM has not been initialized at this point and would return garbage. Instead, the chipset routes the request to the SPI flash ROM carrying the BIOS or UEFI firmware.

The mechanism that governs this routing is a chipset configuration register: FWH_DEC_EN1 (Firmware Hub Decode Enable 1), located in the LPC (Low Pin Count) Interface Bridge register space. Its function is straightforward — it defines which physical memory ranges the chipset should redirect to the Firmware Hub rather than system RAM. At power-on, this register is preset by hardware to cover the reset vector region, so the very first fetch the processor issues is silently redirected to ROM before any software has had a chance to configure anything. The Intel 5 Series / 3400 Series Chipset datasheet documents this at page 487:

The mechanism works as follows:

• Decoding Ranges: The FWH_DEC_EN1 register contains specific bits (like Bit 7, Bit 6, etc.) that correspond to different memory segments at the top of the 4GB address space (e.g., the range FFFF_0000h to FFFF_FFFFh).

• Redirection: When the system resets, the hardware defaults for these bits are typically set to “Enabled”. This ensures that when the CPU requests 0xFFFFFFF0, the chipset instantly recognizes this address falls within the “BIOS Range.”

• Fetching: Instead of sending the request to the DRAM controller, the chipset asserts the LPC (or SPI) bus signals, activating the Flash ROM chip.

Here's a diagram:

In short, the area the processor wants to read (0xFFFFFF0) is actually located in the BIOS chip, not in RAM, and the instruction is taken from this address in the BIOS chip within the Reset Vector and given to the processor.

Switching from Real Mode to Protection Mode

0x0 - A20 Gate

The 8086 had a 20-bit address bus — lines A0 through A19. This imposed a hard ceiling of 2²⁰ bytes, or exactly 1MB, on the physical address space. There was no 21st line. Any address calculation that produced a value above 0xFFFFF would silently lose bit 20, causing the address to wrap back into the bottom of the memory map. This was not a bug in the 8086 — it was a physical constraint of the hardware.

The problem surfaced with the 80286. Its address bus was 24 bits wide, meaning bit 20 now had a real wire behind it and addresses above 0xFFFFF resolved correctly. Wraparound no longer happened. This broke a class of DOS-era software that had come to depend on it — specifically, programs that constructed segment:offset pairs deliberately producing addresses just above 0x100000, relying on the wraparound to land back in low memory. On the 80286, those same addresses resolved to their actual targets above 1MB instead, and the programs failed.

The hardware fix was the A20 gate: a logic circuit, originally wired through the 8042 keyboard controller, that could force address line 20 to zero regardless of what the processor put on the bus. With the gate closed, bit 20 was always 0, wraparound was preserved, and legacy software ran correctly. With the gate open, bit 20 propagated normally and the full address space above 1MB became accessible.

We can simply open this gate with NASM:

; 0x92 - A20
in al,0x92  ; Get the current value 
or al,0x02  ; Open A20
and al,0xFE
out 0x92,al ; Write the bit again

This is required for Protection Mode.

0x1 - GDT and Switching to Protected Mode

With A20 open, the next requirement before touching CR0.PE is a valid Global Descriptor Table. In Real Mode, segment registers hold raw base values. In Protected Mode, they become selectors — 16-bit indices into the GDT — and the processor uses the descriptor at that index to determine the segment's base address, limit, and access rights. Attempting to set CR0.PE without a loaded GDT means the first segment register access in Protected Mode will reference garbage, producing a #GP or a silent wrong-base calculation.

The GDT used here defines five descriptors:

gdt_start:
    dq 0                                    ; 0x00 null (required)
    dw 0xFFFF, 0x0000
    db 0x00, 10011010b, 11001111b, 0x00     ; 0x08 code32
    dw 0xFFFF, 0x0000
    db 0x00, 10010010b, 11001111b, 0x00     ; 0x10 data32
    dw 0xFFFF, 0x0000
    db 0x00, 10011010b, 10101111b, 0x00     ; 0x18 code64  (L=1)
    dw 0xFFFF, 0x0000
    db 0x00, 10010010b, 11001111b, 0x00     ; 0x20 data64
gdt_end:

The null descriptor at offset 0 is mandatory — the processor reserves it and any selector that resolves to it raises #GP. The 32-bit code and data descriptors (0x08, 0x10) have D/B=1, placing the processor in 32-bit default operand and address size when CS holds 0x08. The 64-bit code descriptor at 0x18 has L=1 and D/B=0 — this is the bit that tells the processor to decode instructions as 64-bit after the final far jump.

The descriptor table is loaded into GDTR with lgdt, then CR0.PE is set and control transfers immediately to the 32-bit code selector:

lgdt [gdt_desc]

mov eax, cr0
or  eax, 1
mov cr0, eax
jmp 0x08:pMode32

The far jump is not optional. Setting CR0.PE does not flush the prefetch queue. Instructions fetched under Real Mode decode rules may still be in the pipeline, and executing them in Protected Mode produces undefined behavior. The far jump serializes the processor and reloads CS from the GDT, putting the descriptor cache into a consistent state for Protected Mode operation.

0x2 - Paging and the Transition to Long Mode

Protected Mode with paging disabled gives a flat 32-bit physical address space — useful for setup, but insufficient for Long Mode. The architecture requires paging to be active before Long Mode can be enabled, and it requires PAE (Physical Address Extension) specifically, since Long Mode uses 4-level page tables with 64-bit entries.

The page table structure is built at fixed physical addresses before any control register is touched:

mov edi, 0x13000
mov ecx, 0x3000 / 4
xor eax, eax
rep stosd            ; zero PML4 + PDPT + PD (3 × 4KB)

mov dword [0x13000], 0x14003  ; PML4[0] → PDPT (present + writable)
mov dword [0x14000], 0x15003  ; PDPT[0] → PD   (present + writable)

The Page Directory is filled with 512 entries, each mapping a 2MB physical region:

    ; [...]
    mov edi, 0x15000
    mov eax, 0x83           ; P=1, W=1, PS=1 (2MB page)
    mov ecx, 512

.pd:
    mov dword [edi],   eax
    mov dword [edi+4], 0
    add eax, 0x200000
    add edi, 8
    loop .pd

The PS bit (bit 7) in a Page Directory entry signals a 2MB huge page, bypassing the Page Table level entirely. The result is a flat identity map covering the first 1GB of physical memory — virtual address equals physical address throughout. This keeps all subsequent absolute references valid without any address fixup.

With the page tables in place, the Long Mode activation sequence:

mov eax, 0x13000
mov cr3, eax            ; point CR3 at PML4

mov eax, cr4
or  eax, (1 << 5)
mov cr4, eax            ; CR4.PAE = 1

mov ecx, 0xC0000080     ; IA32_EFER MSR
rdmsr
or  eax, (1 << 8)
wrmsr                   ; EFER.LME = 1

mov eax, cr0
or  eax, (1 << 31)
mov cr0, eax            ; CR0.PG = 1 → hardware sets EFER.LMA

jmp 0x18:lmode64        ; CS = 0x18 (L=1) → 64-bit decode active

The order is strictly enforced by the hardware. Setting EFER.LME before CR4.PAE is set raises #GP. Setting CR0.PG last is the trigger that causes the processor to atomically assert EFER.LMA — Long Mode Active. Until the far jump loads the 0x18 descriptor with L=1, the processor is technically in compatibility mode: Long Mode paging is active, but instruction decode is still 32-bit. The far jump is what makes it 64-bit.

0x3 - IDT and Exception Handling

The IDT in Long Mode serves the same purpose as the IVT in Real Mode, but the mechanism is entirely different. Each entry is 16 bytes, encoding a 64-bit handler address split across three fields, a code segment selector, an IST index, and a type/attribute byte. The processor locates the IDT through IDTR, loaded with lidt.

The IDT is placed at physical 0x10000. All 256 entries are first populated with a CatchAll handler that does nothing but iretq, then vector 1 — the #DB (Debug Exception) vector — is overwritten with the actual handler address:

IDT_BASE equ 0x10000

setup_idt:
    mov rdi, IDT_BASE
    mov rcx, (256 * 16) / 8
    xor rax, rax
    rep stosq               ; zero the entire IDT

    mov rcx, 256
    mov rdi, IDT_BASE
.fill:
    call .write_gate        ; write CatchAll to every entry
    add  rdi, 16
    loop .fill

    mov rdi, IDT_BASE + (1 * 16)
    mov rax, DbHandler
    call .write_gate_rax    ; overwrite vector 1 with DbHandler

    mov word  [idt_limit],    (256 * 16) - 1
    mov qword [idt_base_addr], IDT_BASE
    lidt [idt_limit]
    ret

Each gate is written with selector 0x18 (the 64-bit code descriptor), IST index 0, and type byte 0x8E: Present, DPL=0, 64-bit interrupt gate. The interrupt gate type means the processor clears IF on entry, preventing nested interrupts from arriving while the handler is running.