x64 Windows Assembly Fundamentals Part 1: What You Actually Need to Know

Hello everyone. Mirrai here as always. Today I decided to expand a bit more on low-level fundamentals. If you've spent any time in a debugger or reading shellcode you've probably stared at assembly and had no idea what was going on. That's why I made this guide to demystify assembly. We'll cover binary, registers, the Windows x64 calling convention, and how RSP and RIP actually work. By the end you'll be able to read what a debugger is showing you and understand why things are where they are. With that out of the way lets begin with binary. Basic C knowledge will definitely help to understand assembly. I recommend you at least know how functions work in C or programming in general. With that, let's start.

Binary and data storage

Everything your computer does comes down to binary. Text, pictures, videos, everything, all ones and zeros. A bit can store two values: 0 and 1. Two bits can store 4 values 0, 1, 2, 3 etc. Each bit doubles capacity. Eight bits make up a byte.

0000 0000 = 0
0000 0001 = 1
0000 0010 = 2
0000 0011 = 3
1111 1111 = 255

Now when we talk about a 64 bit register it means that register can store a value up the amount that 64 bits or 8 bytes can hold. For 64 the limit is (18,446,744,073,709,551,615). For 32 bit it's (4,294,967,295). For 16 bits it's (65,535). For 8 bits it's (255). To clarify. While those are their max values they can represent numbers of those values +1. For example, 8 bits can hold 0-255 which is 256 max value but it can't store the number 256.

You'll rarely work with raw binary in practice. Hexadecimal is more common because it's a cleaner representation of binary data. One hex digit represents exactly four bits.

0000 = 0x0
0001 = 0x1
1010 = 0xA
1111 = 0xF
1111 1111 = 0xFF = 255

In x64dbg and most debuggers memory is displayed in hex. When you see 41 in memory that's 0x41 which is 65 in decimal which is the letter "A" in ASCII.

Registers

Registers are storage locations inside the CPU itself. They're faster than RAM because they're on the chip. On x64 the general purpose registers are:

RAX  RBX  RCX  RDX
RSI  RDI  RSP  RBP
R8   R9   R10  R11
R12  R13  R14  R15

Each holds 64 bits (8 bytes). You can also access the lower bits of the first eight registers.

RAX(64 bits)  →  EAX (lower 32 bits)  →  AX (lower 16)  →  AL (lower 8)
RBX  →  EBX  →  BX  →  BL
RCX  →  ECX  →  CX  →  CL
RDX  →  EDX  →  DX  →  DL`

You see ECX and lower registers even in x64 (also x86-64). This is because x64 is an extension of x86.

Now I want you to focus on these two registers.

RSP: the stack pointer. Always points to the top of the current stack. It grows down and fills upward. When we do sub rsp, 40 a bit later we are adding more space to the stack or in the specific example we are reserving shadow space and alignment (More on that later). Likewise add rsp, 40 will remove that space. RSP also changes when data is pushed (RSP is subtracted by 8, then the data is stored at the new top). and popped (loads the data from the top into a register, then adds 8 to RSP, effectively removing it)

RIP: the instruction pointer. Holds the address of the next instruction the CPU will execute. You can't write to RIP directly in normal code but controlling it is the goal of most exploitation techniques. Wherever RIP points execution follows.

The Windows x64 Calling Convention
When a function is called in Windows x64, there are strict rules about how arguments are passed and how the stack is set up. This is called the calling convention and if you're reading compiled code or writing shellcode you need to know it.
Integer and pointer arguments go in registers in this order:

1st argument → RCX
2nd argument → RDX
3rd argument → R8
4th argument → R9
5th and beyond → pushed onto the stack

Return values are then stored in RAX after the function ends. So if you see something like:

mov rcx, rax          ; first argument
mov rdx, 0            ; second argument
call SomeFunction

This code passes two arguments and calls the function. The result will be in RAX afterward.

Shadow space is the part that trips people up (Me included). Before any call the caller must allocate at least 32 bytes (0x20) on the stack even if the function takes fewer than four arguments. This is called the shadow space and it exists so the called function has somewhere to spill its register arguments if it needs to. We also allocate 8 bytes to align the stack because the winapi functions need the stack to be 16 byte aligned or the program crashes. BTW if you make your own functions you don't have to follow these conventions but it's wise to do so if you plan to use the winapi in anyway.

Let me give you a more practical example. A program that prints "Hello World" on a messagebox.

BITS 64
default rel
global main
extern ExitProcess
extern MessageBoxA


%define uType 1         ; MB_OKCANCEL

section .data
text_1  db "Hello World", 0
text_2  db "Hello from mirral", 0


section .text
main:
    sub rsp, 40                      ; shadow space + 8 byte alignment

    xor rcx, rcx                     ; hWnd
    lea rdx, text_1                  ; lpText
    lea r8, text_2                   ; lpCaption
    mov r9, uType                    ; uType  
    call MessageBoxA

    xor rcx, rcx
    call ExitProcess

I'll Explain basic assembly syntax like mov, lea, xor in my next post but for now I want you to look at the bigger picture.

If you wanna compile it:

(Replace "hello_world.nasm" which what you saved the code with)

nasm -f win64 "hello_world.nasm" -o "hello_world.obj"
gcc "hello_world.obj" -o "hello_world.exe" -lkernel32 -luser32 -mwindows -s -nostartfiles -nostdlib

This is the messagebox function definition from MSDN:

int MessageBoxA(
  [in, optional] HWND   hWnd,
  [in, optional] LPCSTR lpText,
  [in, optional] LPCSTR lpCaption,
  [in]           UINT   uType
);

You see how it maps to the asm code? The asm is zeroing out RCX, loading the pointer of the text_1 and text_2 to rdx and r8 respectively and moving the Utype value to r9 then calls MessageboxA Then RCX is zeroed and ExitProcess is called which just exits the process (duh). This is what C does under the hood.

Putting it Together

Open x64dbg or your debugger of choice and load any executable, step through with it and watch the registers panel. Every time you step over a call notice which registers change. Watch RSP decrease when something is pushed and increase when something is popped. Watch RIP change with every instruction.

Reading assembly gets easier fast once you can see it happening in real time. The concepts aren't difficult but you might need a while to internalise them. Don't give up though. Despite how it looks persistence is key to understanding assembly.

If you haven't read the buffer overflow series this is a good foundation for it. Part 1 covers the stack in more depth and Part 2 puts all of this to practical use. For part 2 I'll explain assembly syntax like mov, lea etc. Till then see ya. Also leave a comment if you have questions.