Writing Windows Applications in x64 Assembly in 2026

Stefan Hofmann — Fri, 06 Nov 2026 08:45:18 +0000

The compiler's tax, the practical Win32-to-Win64 migration traps — with a 55 KB tray application as the running example.

ClipFlow, the running example for this article, is a Windows tray application: press a hotkey and it loads a text template onto the clipboard; templates can carry $$Name$$ placeholders, a small dialog asks for the values, and the personalized text is ready to paste anywhere. The binary is 55 KB, icon included — one executable, no installer, no runtime, no framework, imports from four system DLLs, and its source contains no networking code.

But the app is not the subject. The subject is the layer underneath, the one every C compiler handles for you. If you wrote 32-bit Windows assembly back in the MASM32 days, this article is your migration path: the x64 delta is smaller than rumored and sharper than expected. The calling convention is the famous half of it. The quiet half — instructions that vanished, addressing that changed meaning, a segment register that moved, exceptions that became metadata, and a type system where half the types widened and the other half did not — is where 32-bit habits break silently.

1. The toolchain: three commands, no C runtime

The assembler is JWasm, the headers are the WinInc set, and the linker comes from the Windows SDK. A complete build:

jwasm -nologo -I%USERPROFILE%\WinInc\Include -win64 -Zp8 clipflow.asm
rc /nologo clipflow.rc
link /nologo /subsystem:windows /entry:start clipflow.obj clipflow.res

-Zp8 matters more than it looks: the SDK structures on x64 assume natural alignment capped at 8 bytes, and a mismatch silently changes every offset in every struct you touch.

There is no C runtime in the binary — the entry point is genuinely yours. For clarity, here is an explicitly stack-managed entry-point sketch (the real build should follow one consistent assembler stack-management strategy):

start:
    and rsp, -16                  ; align for calls; entry is not a normal proc
    sub rsp, 20h                  ; reserve 32 bytes of outgoing shadow space
    call RunApplication
    mov ecx, eax                  ; ExitProcess takes a UINT in ECX
    call ExitProcess              ; does not return

Here the linker explicitly selects start with /entry:start. The manual sub rsp, 20h is essential: alignment alone does not provide the callee's shadow space. In normal framed procedures, let your chosen JWasm invoke strategy manage outgoing arguments rather than mixing it with ad hoc adjustments.

No CRT startup objects, no hidden constructors, no atexit machinery. The first instruction that runs is one you wrote.

The following options illustrate the relevant assembler settings (choose one win64 mode, not both):

option casemap:none — case-sensitive identifiers, as the Windows headers require.
option scoped — procedure-local labels stay local.
option frame:auto — automatically generates prologues/epilogues and unwind metadata for proc frame procedures; the frame designation still matters.
option win64:1 — enables automatic register-parameter homing, but uses the default per-invoke stack adjustment.

2. The Win64 ABI survival kit

In C, the compiler pays a tax on your behalf — on every call, in every function. In assembly you are the compiler, and the tax collector still comes. These are the rules that actually bite.

2.1 One default calling convention instead of a zoo

The first four integer or pointer arguments travel in RCX, RDX, R8, R9 (floats use XMM0–XMM3). Everything from the fifth parameter onward goes on the stack, one argument per 8-byte slot — a DWORD argument still occupies a full slot. And the callee never cleans up: every procedure ends with a plain ret, never ret 8. In 32-bit Windows you had stdcall, cdecl, fastcall and thiscall coexisting; ordinary Windows x64 API calls use a unified default convention (specialized conventions such as __vectorcall also exist). The ret n habit from stdcall is the one that bites first — nothing crashes immediately; the caller's stack is just quietly wrong.

2.2 Shadow space: 32 bytes you owe every callee

Before every call, the caller reserves 32 bytes of stack directly above the return address. The callee is entitled to spill RCX, RDX, R8 and R9 there — its "home slots". From the callee's perspective at entry:

[rsp+0x28]   5th parameter (first stack argument)
[rsp+0x20]   home slot for R9
[rsp+0x18]   home slot for R8
[rsp+0x10]   home slot for RDX
[rsp+0x08]   home slot for RCX
[rsp+0x00]   return address

Skip the shadow space and the crash does not happen in your code — it happens inside the callee, when it writes its parameters home. A crash far from the bug is the signature of this mistake.

2.3 16-byte alignment, or the movaps crash

RSP must be 16-byte aligned immediately before the call; the callee therefore enters with RSP 8 mod 16, and the prologue's pushes restore alignment as they go. The reason is mundane and brutal: SSE. movaps and friends fault on unaligned operands, and plenty of system code uses them. Get this wrong and your process dies inside a DLL you did not write, with a stack trace that is a lie. (It is also why the entry point above aligns RSP before reserving outgoing shadow space.)

2.4 Homing: parameters that survive a call

RCX, RDX, R8 and R9 are volatile. A procedure that needs its parameters after making calls of its own must put them in memory. With the appropriate JWasm Win64 homing option and a framed procedure, the generated prologue can home them into the incoming shadow space (schematically):

    push rbp
    mov rbp, rsp
    push rsi
    push rdi
    sub rsp, N                    ; locals + outgoing shadow space, 16-aligned
    mov [rbp+0x10], rcx           ; parameter 1 → its home slot
    mov [rbp+0x18], rdx
    mov [rbp+0x20], r8
    mov [rbp+0x28], r9

The parameters are stable memory references from then on — and the fifth parameter, if there is one, sits at [rbp+0x30]. This is why a window procedure can still read its LPARAM deep inside a message handler.

2.5 Unwind information is not optional

x64 structured exception handling is table-based. Every non-leaf function should carry unwind data (.pdata/.xdata) so that debuggers walk honest stacks and exceptions can pass through your frames. option frame:auto together with proc frame can generate it; a true leaf function that needs no unwind metadata is exempt. Without it, everything runs fine — until the day it doesn't, and the crash dump blames the wrong function.

2.6 The fine print

Volatile: RAX, RCX, RDX, R8–R11, XMM0–XMM5. Nonvolatile general-purpose registers (RBX, RBP, RSI, RDI, R12–R15) and XMM6–XMM15 must be preserved if modified. In the right framed-procedure mode, JWasm USES can help save and restore these; it is not a blanket guarantee for every register or context.

Variadic calls

Functions such as printf(format, ...) accept a variable number of arguments. Windows x64 uses the normal calling convention: the first four arguments occupy RCX, RDX, R8 and R9 for integer or pointer values, or XMM0–XMM3 for floating-point values. Arguments beyond the fourth go on the stack in 8-byte slots.

The important difference is how floating-point arguments are passed. For variadic calls, a floating-point value in one of the first four positions must appear in both its XMM register and the corresponding general-purpose register, using the same bit representation.

For example, printf("%f %d", 1.5, 7) passes the format pointer in RCX, the double 1.5 in both XMM1 and RDX, and the integer 7 in R8. The register numbers follow argument positions, not separate integer and floating-point sequences.

The caller reserves the usual 32-byte shadow space. The callee can save register arguments into their home slots, allowing a va_list to traverse them together with stack arguments. Standard C default argument promotions still apply: float becomes double, and narrow integer types are promoted to int or unsigned int.

Width discipline

32-bit register writes zero the upper half of the 64-bit register; 8- and 16-bit writes do not:

    mov eax, byteCount            ; upper half of RAX is now guaranteed zero
    sete al                       ; AL = 0/1 — the rest of EAX is stale
    movzx eax, al                 ; so widen explicitly

And the upper halves of return values are only defined where the type is 64 bits wide:

    invoke GetMessageW, addr msg, 0, 0, 0
    .if eax == -1                 ; BOOL: 32-bit — test EAX, never RAX
        ...

    invoke DialogBoxParamW, ...
    .if rax == -1                 ; INT_PTR: 64-bit
        ...

Testing the wrong half of RAX is a bug that assembles, runs, and simply never takes its error branch.

3. From 32-bit to 64-bit: the migration delta

The calling convention above is the famous half of the migration. The quiet half is below — the things that don't announce themselves, because half of them still assemble and then mean something different.

3.1 Instructions that vanished — and what replaces them

Long mode removed more than most people notice:

    pushad                        ; gone entirely — save registers individually
    push ecx                      ; gone — push always moves 8 bytes now
    push 80000000h                ; still assembles — but sign-extends!

That last one is the quiet trap: push with a 32-bit immediate pushes the value sign-extended to 64 bits. push 80000000h puts 0xFFFFFFFF80000000 on the stack, not 0x0000000080000000. If you meant the unsigned value, load through a 32-bit register first — mov eax, 80000000h zero-extends RAX — then push rax. For signed widening from a memory DWORD there is movsxd; for unsigned, mov into the 32-bit register and rely on the zero-extension.

Two more removals worth knowing: the single-byte INC/DEC encodings (0x40–0x4F) were reassigned as REX prefixes, so INC/DEC still exist but with different encodings — invisible unless you hand-assembled opcode tables. And there is no push imm64 at all: the only instruction that holds a full 64-bit constant is mov reg, imm64.

mov has a related encoding trap. mov r64, imm64 can load a full 64-bit constant into a register, but there is no mov m64, imm64: a direct immediate store to a 64-bit memory operand uses a sign-extended 32-bit immediate. For example, mov qword ptr [rbx], -1 writes 0xFFFFFFFFFFFFFFFF, which is intentional. For an unsigned value such as 0x0000000080000000, make the width explicit:

    mov eax, 80000000h            ; zero-extends into RAX
    mov qword ptr [rbx], rax

This avoids relying on an assembler's handling of an out-of-range or ambiguous immediate.

3.2 Addressing went RIP-relative

In 32-bit code, mov eax, offset MyString puts an absolute address in a register. In 64-bit code, memory references with a plain displacement assemble to RIP+disp32 instead — relative to the instruction pointer. lea rax, [MyString] and mov ecx, [counter] are position-independent by construction.

The consequences are pleasant: almost no .reloc fixups for your code, ASLR for free, and your source syntax barely changes. The rule to internalize: ordinary static data references are generally RIP-relative. Restricted absolute forms still exist (including certain mov accumulator forms); a general 64-bit address can be loaded with mov rax, imm64 and then dereferenced.

3.3 The TEB moved from FS to GS

If your 32-bit code read the Thread Environment Block — the self-pointer, stack limits, TLS array, last error — it did so through FS:

    mov eax, fs:[18h]             ; 32-bit: linear address of the TEB
    mov eax, fs:[30h]             ; 32-bit: the PEB

On x64, all of that lives under GS instead:

    mov rax, gs:[30h]             ; 64-bit: TEB self-pointer
    mov rax, gs:[60h]             ; 64-bit: the PEB

The trap: FS-relative addressing still assembles in 64-bit mode. It just no longer points at anything useful in a 64-bit process. Code that reads fs:[18h] compiles, runs, and returns garbage — another member of the "looks fine, means something else" family. In practice you will rarely need the TEB directly; the API covers almost all of it. But if you port code that used it, this is the line to fix.

3.4 Exceptions: runtime chains became static tables

The classic 32-bit SEH prolog — installing a handler at run time by pushing it and re-pointing the FS chain — simply does not exist on x64:

    ; the MASM32-era pattern — impossible on x64
    push handler
    push dword ptr fs:[0]
    mov dword ptr fs:[0], esp

The 64-bit dispatcher does not use the old FS-linked handler chain. It looks up function-table entries in .pdata for the function containing the faulting RIP and reads unwind codes from .xdata that describe the prolog: which registers were pushed, where the frame is, how far to step. You don't install a handler — you declare your prolog to the assembler, before execution ever starts:

MyProc proc frame
    push rbp
    .pushreg rbp
    mov rbp, rsp
    .setframe rbp, 0
    sub rsp, 28h
    .allocstack 28h
    .endprolog                    ; after this, the prolog must not change

JWasm's option frame:auto emits all of this for you — the directives above are what it produces. The mental shift: in 32-bit, SEH was code; in 64-bit, SEH is metadata. A prolog that lies about itself breaks unwinding even when the code "works".

3.5 Types: what widened, and what did not

The dangerous intuition is "everything is 64-bit now". It is not. The type table has two halves, and confusing them is the classic porting bug:

Type	32-bit	64-bit
`LONG`	32 bit	32 bit — unchanged
`DWORD`, `BOOL`, `HRESULT`	32 bit	32 bit — unchanged
`size_t`, `ULONG_PTR`, `LONG_PTR`	32 bit	64 bit
`WPARAM`, `LPARAM`	32 bit	64 bit
pointers	32 bit	64 bit
`HWND`, `HANDLE`	pointer-sized (32-bit)	pointer-sized (64-bit); Windows handles are typically 32-bit-significant

The three practical consequences:

Message parameters. WPARAM and LPARAM are 64-bit now. Reading only the low 32 bits is legitimate when you know the payload is small — ClipFlow's tray handler reads dword ptr parameter2 because a tray message's mouse code lives in the low word — but a pointer passed through LPARAM (a CREATESTRUCT*, a dialog's user data) must be read at full width. Know which messages carry pointers, and never let a truncation habit survive those cases.

SetWindowLong became SetWindowLongPtr. The canonical pattern — storing a pointer in GWLP_USERDATA — needs the Ptr variant on x64. ClipFlow's dialog procedure uses it to carry its TemplateField* from WM_INITDIALOG to the IDOK handler. On 32-bit Windows the plain function took a LONG and macros papered over the difference; on x64, pushing a pointer through the wrong API is the porting bug.

Structures changed shape. Fields holding pointers or LONG_PTR doubled in size, padding moved, and every sizeof you memorized in 32-bit code is wrong. GetTickCount needed a 64-bit sibling (GetTickCount64); window handles remain handle types rather than raw application pointers. Windows currently supports 32-bit-significant handles for 32/64-bit interoperability, but store them in HANDLE/HWND-sized fields rather than relying on truncation. When in doubt, let the assembler compute offsets: [rbp].SomeStruct.SomeField and the header files exist precisely so you don't memorize layout.

3.6 The cheat sheet

Area	32-bit (Win32)	64-bit (Win64)
Calling conventions	stdcall, cdecl, fastcall, thiscall	unified default convention; specialized alternatives exist
Argument registers	ECX, EDX (fastcall only)	RCX, RDX, R8, R9
5th+ arguments	stack, 4-byte slots	stack, 8-byte slots
Stack cleanup	convention-dependent (`stdcall`: callee; `cdecl`: caller)	caller (plain `ret`)
Shadow space	—	32 bytes, mandatory
Stack alignment	less stringent ABI requirements	16-byte before the call
Parameter saving	convention-dependent	homing into shadow space
SEH	`fs:[0]` runtime chain	static `.pdata`/`.xdata` tables
TEB / PEB	`fs:[18h]` / `fs:[30h]`	`gs:[30h]` / `gs:[60h]`
`pushad` / `push r32`	available	removed
`push imm32`	as-is	sign-extended to 64 bits
`mov reg, imm`	up to 32-bit immediate	`mov r64, imm64` supports full 64-bit immediates
`mov mem, imm`	up to 32-bit immediate	64-bit memory destination: sign-extended `imm32` only
Data addressing	often absolute	generally RIP-relative for static memory
EBP	frame pointer by convention	optional, general register
`LONG`	32 bit	32 bit — unchanged
`WPARAM`, `LPARAM`	32 bit	64 bit
Handles	32-bit value	32-bit significant
User address space	typically 2 GB per process (configuration-dependent)	much larger; OS- and configuration-dependent

4. What you actually get

The honest answer to "why not C?" is: you could, and it would be fine.

Assembly does not automatically make a tray application faster. Most of its lifetime is spent waiting for user input. The benefits lie elsewhere:

Minimal dependencies. No C runtime, no application framework, and a small set of Windows API imports.
Minimal startup overhead. No framework initialization; the hotkey becomes available as soon as the application is ready.
A small audit surface. One main assembly source file and a resource script make it practical to examine what the application actually does.

ClipFlow is one example: a 55 KB Windows tray application, written entirely in x64 assembly.

For small, focused Windows utilities, x64 assembly remains a viable choice in 2026. The transition from 32-bit assembly is not a complete reinvention — but the details matter.

Source code and builds: github.com/hofmann-development/clipflow

DEV Community: Stefan Hofmann