[My Debugger Journey] Building a Disassembler with NASM

How a Disassembler Reads Machine Code

Have you ever wondered how a CPU sees and executes your code? Not the high-level abstractions, not the compiler output — the raw bytes sitting in memory. Let's go deep.

Instruction Encodings

In x64 and IA-32 architectures, every instruction is a sequence of bytes following a specific format. An instruction consists of Instruction Prefixes, Opcode, ModR/M, SIB, Displacement and Immediate. We will cover each one.

Here's the figure directly from Intel's Volume 2:

Not every instruction uses all of these fields. Some instructions are a single byte. Others stretch to 15. The decoder has to figure out where one instruction ends and the next begins — purely from the byte stream, with no delimiters, no markers.

Instruction Prefixes

A prefix modifies the behavior of the following instruction without being an instruction itself. Intel groups them into four groups.

• Group 1 — Lock and Repeat: The LOCK prefix (F0) forces exclusive access to shared memory. In a multiprocessor environment, two cores can read and write the same address simultaneously, corrupting data. LOCK holds the memory bus until the operation completes. F2 and F3 are the repeat prefixes — REPNE/REPNZ and REP/REPE/REPZ respectively, typically used with string operations.

• Group 2 — Segment Override: Redirects memory access to a specified segment register. 2E overrides to CS, 36 to SS, 3E to DS, 26 to ES, 64 to FS, and 65 to GS. Largely irrelevant in 64-bit mode where segmentation is mostly flat, but the decoder still has to consume and skip them.

• Group 3 — Operand Size Override: 66 switches the operand size between 16-bit and 32-bit. In a 32-bit context, prefixing an instruction with 66 narrows the operand to 16-bit — this is how mov ax, 1 and mov eax, 1 share the same opcode but differ by a single prefix byte.

• Group 4 — Address Size Override: 67 switches the address size between 32-bit and 64-bit. Rarely used in practice but the decoder must handle it — a disassembler that ignores 67 will miscalculate the size of any displacement or memory operand that follows.

In 64-bit mode, the REX prefix (0x40–0x4F) follows these four groups. REX.W=1 promotes the operand to 64-bit. That 0x48 byte you see before most 64-bit instructions is exactly this — it tells the CPU to treat the following instruction as a 64-bit operation.

Opcode

A primary opcode can be 1, 2, or 3 bytes in length. Most common instructions fit in a single byte — 0x90 is NOP, 0xC3 is RET, 0x89 is MOV. The problem is that x86 only has 256 possible values in a single byte, and that space filled up quickly.

When Intel ran out of room, they introduced escape sequences. 0x0F is the most common — it tells the decoder "don't treat this as an opcode, read the next byte too." So 0x0F 0xA3 together means BT (Bit Test), a two-byte opcode. The 0x0F byte itself carries no instruction — it's purely a table switch.

0x89        →  MOV   (1-byte opcode)
0x0F 0xA3   →  BT    (2-byte opcode, 0x0F is the escape)

But, we cannot escape with these information :>

Mandatory Prefix and Escape Prefix

An opcode also can consist of a mandatory prefix and Escape Prefix:

• Mandatory Prefix: A prefix that is part of the opcode itself, not a modifier. Unlike Group 1-4 prefixes which change behavior, a mandatory prefix changes the identity of the instruction entirely.

• Escape Prefix: A byte that switches the decoder into an extended opcode table. 0x0F is the most common — alone it unlocks a second table of 256 opcodes. When that table also fills up, a second escape byte follows: 0x0F 0x38 and 0x0F 0x3A open two additional tables.

For example, an opcode can start with these values (CVTDQ2PD):

CVTDQ2PD: F3 0F E6 

CVTDQ2PD instruction is one of the instructions that requires a mandatory prefix and begins with 0xF3. This is the mandatory prefix, while the value 0x0F is the escape prefix. Finally, 0xE6 is the command itself.

By the way, the mandatory prefix can also be 0x66 and 0xF2.

But there are instructions that don't requires Mandatory Prefix. Here are a few examples:

0F 28   →  MOVAPS   (move aligned packed single-precision floats)
0F 58   →  ADDPS    (add packed single-precision floats)
0F 54   →  ANDPS    (bitwise AND of packed floats)
0F 59   →  MULPS    (multiply packed single-precision floats)

As you can see, they start with 0x0F (escape prefix).

I've prepared a nasm project that shows the differences between SIMD, AVX2 and X64 instructions:

BITS 64

section .data
    align 16
    xmm_a   dd 1.0, 2.0, 3.0, 4.0
    xmm_b   dd 10.0, 20.0, 30.0, 40.0

    align 32
    avx_a   dd 1, 2, 3, 4, 5, 6, 7, 8
    avx_b   dd 10, 20, 30, 40, 50, 60, 70, 80

section .text
global _start

_start:
    ; X64 Register -> Prefix is 0x48
    mov rax,0x5000000000

    ; Mandatory Prefix Instruction: 0xF3 
    cvtdq2pd xmm1, xmm1

    ; Non-Mandatory Prefix: 0x0F
    movaps xmm0, [xmm_a]   ; xmm0 = [1.0, 2.0, 3.0, 4.0]

    ; AVX2 Register (256-bytes) Prefix: 0xC5
    vmovdqu ymm0, [avx_a]

There are X64, SIMD and AVX2 registers, so we can see the different. Let's see the instructions with objdump:

Focus on the first bytes. They have different prefix.

ModR/M

After the opcode, many instructions include a ModR/M byte. Not all — NOP, RET, HLT need no operands and carry no ModR/M. But any instruction that needs to specify "which register" or "which memory location" requires one.

The ModR/M byte encodes three fields packed into 8 bits:

 7   6   5   4   3   2   1   0
┌───────┬───────────┬───────────┐
│  Mod  │    Reg    │    R/M    │
│  2bit │   3bit    │   3bit    │
└───────┴───────────┴───────────┘

Mod (bits 7:6) — tells the decoder how to interpret R/M:

Mod	Meaning
00	memory, no displacement
01	memory + 1-byte displacement
10	memory + 4-byte displacement
11	register direct, no memory

Reg (bits 5:3) — encodes the register operand index (0–7, extended to 8–15 via REX.R).

R/M (bits 2:0) — encodes the second operand. When Mod=11, it's a register index. When Mod≠11, it encodes the base register for a memory access — or triggers SIB when R/M=4.

A concrete example

Take the instruction mov rbp, rsp. In memory it looks like this:

48  89  E5

• 0x48 — REX prefix, W=1, meaning 64-bit operand
• 0x89 — opcode: MOV r/m, reg
• 0xE5 — ModR/M byte

Breaking down 0xE5 value:

E5 = 1110 0101
     ^^  ^^^  ^^^
     11  100  101
     Mod  Reg  R/M
     =3   =4   =5

Mod=3 → register direct, no memory access, no displacement.

Reg=4 → RSP (source).

R/M=5 → RBP (destination).

Combined with 0x89 (MOV r/m ← reg) and REX.W=1 (64-bit), the decoder produces: mov rbp, rsp.

Building a Disassembler with NASM

As you can see before the chapter, creating a disassembler is really tough task. This means that we can't determine the opcodes with just increasing the bytes. There can be different byte for the instruction. And we must extract them.

In this chapter, i've shared a piece code of my nasm project. This is simple disassembler function, which determines opcodes and instruction for mov rbp,rsp. In this way, we can understand the basic of a disassembler:

; ...

; Messages
str_mov        db "found mov in opcode!", 13, 10, 0
str_mod_reg    db "found reg in ModRM!", 13, 10, 0 
str_mod_disp8  db " [mem+disp8]", 0
str_mod_disp32 db " [mem+disp32]", 0
str_mod_mem    db " [mem]", 0
str_newline    db 13, 10, 0

opcodes db 0x48, 0x89, 0xE5   ; mov rbp, rsp
        db 0x90               ; nop
        db 0x48, 0x31, 0xC0   ; xor rax, rax
        db 0xC3               ; ret

BITS 64
_start64:
    mov  ax, 0x20
    mov  ds, ax
    mov  es, ax
    mov  fs, ax
    mov  gs, ax
    mov  ss, ax
    mov  rsp, 0x90000

    ; ... 

    xor rcx, rcx
    mov rbx, opcodes

.loop:
    mov al, byte [rbx + rcx]

    cmp al, 0x48
    jz .prefix_found

    cmp al, 0xC3
    jz .exit

    cmp al, 0x89
    jz .mov_found

    ; unknown opcode — skip and continue
    inc rcx
    jmp .loop

.mov_found:
    mov rsi, str_mov
    call serial_print

    inc rcx
    mov al, [rbx + rcx]

    mov r8b, al
    shr al, 6
    cmp al, 3
    je .mod_reg

    cmp al, 1
    je .mod_disp8

    cmp al, 2
    je .mod_disp32

    mov rsi, str_mod_mem
    call serial_print
    inc rcx
    jmp .done_print

.mod_reg:
    mov rsi, str_mod_reg
    call serial_print
    inc rcx
    jmp .done_print

.mod_disp8:
    mov rsi, str_mod_disp8
    call serial_print
    add rcx, 2
    jmp .done_print

.mod_disp32:
    mov rsi, str_mod_disp32
    call serial_print
    add rcx, 5
    jmp .done_print

.done_print:
    mov rsi, str_newline
    call serial_print
    jmp .loop

.prefix_found:
    inc rcx
    jmp .loop

.exit:
    movzx rax, byte [rbx + rcx]
    call serial_hex64

.halt:
    hlt
    jmp .halt

And the output:

As you can see, we can determine mov instruction and its registers.

This is really simple disassembler. Let's enhance it with this plan: Remember that we saw Prefixes and escape bytes. In the disassembler we will extract them. i've updated the opcodes variable:

opcodes db 0x48, 0x89, 0xE5 ; mov rbp, rsp
        db 0x90 ; nop
        db 0x48, 0x31, 0xC0 ; xor rax, rax
        db 0xf3, 0x0f, 0xe6, 0xc9 ; cvtdq2pd xmm1,xmm1
        db 0x0f, 0x28, 0x04, 0x25, 0x00, 0x20, 0x40, 0x00 ; movaps xmm0,XMMWORD PTR ds:0x402000
        db 0xC3 ; ret

Now we have different prefixes to extract the opcode. We can handle it with this way:

.loop:
    mov al, byte [rbx + rcx]

    ; Check the mandatory prefix
    cmp al,0xF3 
    jz .found_mandatory

    ; Check the Escape Byte
    cmp al,0x0F
    jz .found_escape

.continue:
    cmp al,0x48
    jz .prefix_found

    cmp al,0xC3
    jz .exit

    cmp al,0x89
    jz .mov_found

    ; unknown opcode — skip and continue
    inc rcx
    jmp .loop

.found_mandatory:
    mov rsi,str_mandatory
    call serial_print

    ; We need to check the escape byte
    inc rcx
    mov al,byte [rbx + rcx]
    cmp al,0x0F
    jz .found_escape

    ; if it is not escape byte, then find the opcode
    jmp .continue

.found_escape:
    mov rsi,str_escape
    call serial_print

    ; Now we can find the instruction
    inc rcx
    mov al,byte [rbx + rcx]

    jmp .continue

Here's the output:

As you can see, there are two messages found escape byte (0x0F). Because two instruction contains 0x0F value:

; cvtdq2pd xmm1,xmm1
db 0xf3, 0x0f, 0xe6, 0xc9 

; movaps xmm0,XMMWORD PTR ds:0x402000
db 0x0f, 0x28, 0x04, 0x25, 0x00, 0x20, 0x40, 0x00 

In this way, we can handle them.

Conclusion

There are many possibilities for a disassembler.

What I’ve shown you here are all very simple examples. As you can see from the first heading, there are many prefix groupings and so on, and as a debugger developer, we need to extract through them. Doing this with languages like NASM is tedious, but it makes the job much easier.