DEV Community

Nnamdi Okpala
Nnamdi Okpala

Posted on

How to Write a Windows App in Assembly Language

How to Write a Windows App in Assembly Language

If you are learning low-level programming, operating systems, compilers, kernels, or calling conventions, this is a very useful exercise because it forces you to understand what normally happens underneath C or C++.

In this article, we will build a tiny 64-bit Windows GUI application in assembly using MASM and the Win32 API.

Our first application will simply display a Windows message box.

What We Are Building

The program will call the Windows function:

MessageBoxA(...)
Enter fullscreen mode Exit fullscreen mode

from assembly.

Conceptually, the program looks like this:

Assembly Program
      |
      v
Windows x64 Calling Convention
      |
      v
Win32 API
      |
      +---- user32.dll
      |
      +---- kernel32.dll
      |
      v
Windows
Enter fullscreen mode Exit fullscreen mode

The final executable will be a normal Windows .exe.

Tools We Need

For this tutorial, I recommend Microsoft's assembler:

MASM
Enter fullscreen mode Exit fullscreen mode

More specifically, for 64-bit Windows:

ml64.exe
Enter fullscreen mode Exit fullscreen mode

Install Visual Studio Build Tools and make sure you include:

  • Desktop development with C++
  • MSVC x64 build tools
  • Windows SDK

After installation, open:

x64 Native Tools Command Prompt for VS
Enter fullscreen mode Exit fullscreen mode

This gives you access to tools such as:

ml64.exe
link.exe
Enter fullscreen mode Exit fullscreen mode

Step 1: Create the Assembly File

Create a file called:

hello.asm
Enter fullscreen mode Exit fullscreen mode

Add the following code:

option casemap:none

EXTERN MessageBoxA:PROC
EXTERN ExitProcess:PROC

.data

message db "Hello from Windows x64 Assembly!", 0
title   db "My Assembly App", 0

.code

main PROC

    ; Reserve shadow space and preserve stack alignment.
    sub rsp, 28h

    ; MessageBoxA(
    ;     HWND    hWnd,
    ;     LPCSTR  lpText,
    ;     LPCSTR  lpCaption,
    ;     UINT    uType
    ; )

    xor rcx, rcx
    lea rdx, message
    lea r8, title
    xor r9d, r9d

    call MessageBoxA

    ; ExitProcess(0)

    xor ecx, ecx
    call ExitProcess

main ENDP

END
Enter fullscreen mode Exit fullscreen mode

This is already enough to create a Windows GUI program.

Understanding the Data Section

This part:

.data

message db "Hello from Windows x64 Assembly!", 0
title   db "My Assembly App", 0
Enter fullscreen mode Exit fullscreen mode

creates two null-terminated strings.

The first string contains the message:

Hello from Windows x64 Assembly!
Enter fullscreen mode Exit fullscreen mode

The second contains the title of our message box:

My Assembly App
Enter fullscreen mode Exit fullscreen mode

The trailing 0 terminates each string because MessageBoxA expects C-style strings.

Calling Windows from Assembly

This is where things get interesting.

On 64-bit Windows, the first four integer or pointer arguments are passed through registers.

The Windows x64 calling convention uses:

Argument 1 -> RCX
Argument 2 -> RDX
Argument 3 -> R8
Argument 4 -> R9
Enter fullscreen mode Exit fullscreen mode

So this C function call:

MessageBoxA(
    NULL,
    "Hello from Windows x64 Assembly!",
    "My Assembly App",
    0
);
Enter fullscreen mode Exit fullscreen mode

becomes something like:

xor rcx, rcx
lea rdx, message
lea r8, title
xor r9d, r9d

call MessageBoxA
Enter fullscreen mode Exit fullscreen mode

This is one of the most important things to understand when learning assembly.

A normal high-level function call eventually becomes operations involving registers, memory, the stack, and a machine-level call instruction.

Why LEA Is Used

You may notice this:

lea rdx, message
Enter fullscreen mode Exit fullscreen mode

instead of:

mov rdx, message
Enter fullscreen mode Exit fullscreen mode

MessageBoxA needs a pointer to the string.

LEA means:

Load Effective Address
Enter fullscreen mode Exit fullscreen mode

So:

lea rdx, message
Enter fullscreen mode Exit fullscreen mode

places the address of message into RDX.

Conceptually:

RDX -> address of message string
Enter fullscreen mode Exit fullscreen mode

The Windows x64 Stack

Windows x64 also has another important rule.

Before calling a function, the caller reserves 32 bytes of shadow space.

This is sometimes also called:

home space
Enter fullscreen mode Exit fullscreen mode

That is why the example contains:

sub rsp, 28h
Enter fullscreen mode Exit fullscreen mode

The extra space also helps maintain the required stack alignment before making Windows API calls.

This becomes especially important when you begin writing larger assembly programs.

Step 2: Assemble the Program

Run:

ml64 /c hello.asm
Enter fullscreen mode Exit fullscreen mode

MASM translates:

hello.asm
Enter fullscreen mode Exit fullscreen mode

into:

hello.obj
Enter fullscreen mode Exit fullscreen mode

The process looks like this:

hello.asm
    |
    v
Assembler
    |
    v
hello.obj
Enter fullscreen mode Exit fullscreen mode

An .obj file is not yet an executable.

It contains machine code and metadata that the linker can combine with other libraries.

Step 3: Link the Program

Now run:

link hello.obj user32.lib kernel32.lib /SUBSYSTEM:WINDOWS /ENTRY:main /OUT:hello.exe
Enter fullscreen mode Exit fullscreen mode

This produces:

hello.exe
Enter fullscreen mode Exit fullscreen mode

We link against:

user32.lib
kernel32.lib
Enter fullscreen mode Exit fullscreen mode

because our program uses functions provided by those Windows libraries.

MessageBoxA comes from:

user32.dll
Enter fullscreen mode Exit fullscreen mode

and ExitProcess comes from:

kernel32.dll
Enter fullscreen mode Exit fullscreen mode

The .lib files allow the linker to construct the necessary import information inside our executable.

Step 4: Run It

Run:

hello.exe
Enter fullscreen mode Exit fullscreen mode

Windows should display a message box containing:

Hello from Windows x64 Assembly!
Enter fullscreen mode Exit fullscreen mode

Congratulations.

You have now created a Windows GUI program without C, C++, C#, Python, or another high-level language.

The application talks directly to Windows through the Win32 API.

From Message Box to a Real Window

A message box is useful for learning, but a real Windows application normally has its own window.

The structure of a traditional Win32 GUI application looks approximately like this:

Program Start
     |
     v
GetModuleHandle
     |
     v
Register Window Class
     |
     v
CreateWindowEx
     |
     v
ShowWindow
     |
     v
Message Loop
     |
     +---- GetMessage
     |
     +---- TranslateMessage
     |
     +---- DispatchMessage
                 |
                 v
              WndProc
                 |
                 +---- WM_CREATE
                 |
                 +---- WM_PAINT
                 |
                 +---- WM_KEYDOWN
                 |
                 +---- WM_MOUSEMOVE
                 |
                 +---- WM_DESTROY
Enter fullscreen mode Exit fullscreen mode

The central concept is the message loop.

Windows applications are usually event driven.

Your program waits for Windows to send events such as:

mouse movement
keyboard input
window resize
paint request
button click
window close
Enter fullscreen mode Exit fullscreen mode

Each event is represented by a Windows message.

A Typical Message Loop

In C, you might see:

while (GetMessage(&msg, NULL, 0, 0)) {
    TranslateMessage(&msg);
    DispatchMessage(&msg);
}
Enter fullscreen mode Exit fullscreen mode

In assembly, you manually place those function arguments into the correct registers and call the Win32 functions.

That is where assembly programming becomes extremely educational.

You can see exactly what a high-level language normally hides.

Understanding the Windows ABI

One of the biggest lessons from Windows assembly programming is understanding the ABI.

ABI means:

Application Binary Interface
Enter fullscreen mode Exit fullscreen mode

It defines rules such as:

  • where function arguments go
  • which registers a function must preserve
  • where return values are stored
  • how the stack is aligned
  • how functions call each other

For Windows x64, remember:

RCX = argument 1
RDX = argument 2
R8  = argument 3
R9  = argument 4
Enter fullscreen mode Exit fullscreen mode

Additional arguments are normally passed using the stack.

The return value is usually placed in:

RAX
Enter fullscreen mode Exit fullscreen mode

So conceptually:

mov rcx, argument1
mov rdx, argument2
mov r8,  argument3
mov r9,  argument4

call MyFunction
Enter fullscreen mode Exit fullscreen mode

corresponds to something like:

MyFunction(
    argument1,
    argument2,
    argument3,
    argument4
);
Enter fullscreen mode Exit fullscreen mode

Once this becomes familiar, reading compiler-generated assembly becomes much easier.

Assembly Is Not Isolated From the Operating System

One misconception about assembly is that everything must be implemented manually.

That is not necessarily true.

Your assembly program can still call operating-system libraries.

For example:

Assembly
   |
   +---- Windows API
   |
   +---- DLLs
   |
   +---- Graphics
   |
   +---- Files
   |
   +---- Networking
   |
   +---- Threads
   |
   +---- Memory management
Enter fullscreen mode Exit fullscreen mode

Assembly controls how you interact with these interfaces.

You could therefore build:

  • graphical applications
  • command-line utilities
  • DLLs
  • debuggers
  • emulators
  • development tools
  • system utilities
  • graphics demos
  • networking tools
  • experimental runtimes

entirely in assembly if you wanted to.

Windows Executables and the PE Format

When the linker creates:

hello.exe
Enter fullscreen mode Exit fullscreen mode

it normally creates a Windows PE executable.

PE means:

Portable Executable
Enter fullscreen mode Exit fullscreen mode

The PE format contains things such as:

PE Header
|
+-- Machine Code
|
+-- Data
|
+-- Import Table
|
+-- Export Table
|
+-- Relocations
|
+-- Resources
Enter fullscreen mode Exit fullscreen mode

Our program's import table tells Windows that the executable needs functions such as:

MessageBoxA
ExitProcess
Enter fullscreen mode Exit fullscreen mode

When Windows loads the program, the loader resolves these imports.

Understanding this process is extremely useful if you are studying operating systems or building your own low-level runtime.

A Good Learning Path

Do not try to build an enormous GUI application immediately.

Build progressively.

A useful progression is:

1. MessageBox
        |
        v
2. Console output
        |
        v
3. Basic Win32 window
        |
        v
4. Window message loop
        |
        v
5. Keyboard and mouse input
        |
        v
6. Buttons and controls
        |
        v
7. GDI graphics
        |
        v
8. File I/O
        |
        v
9. Threads
        |
        v
10. DLLs
        |
        v
11. Custom runtime
Enter fullscreen mode Exit fullscreen mode

At every step, you learn more about how Windows actually works underneath higher-level programming languages.

Why Learn This?

You probably would not write every modern Windows application entirely in assembly.

But learning to build one is extremely valuable.

It teaches you about:

CPU registers
      +
stack frames
      +
calling conventions
      +
ABI rules
      +
assemblers
      +
object files
      +
linkers
      +
DLL loading
      +
operating-system APIs
      +
executable formats
Enter fullscreen mode Exit fullscreen mode

Those ideas connect directly to areas such as:

  • operating-system development
  • compiler development
  • kernel development
  • reverse engineering
  • debugging
  • emulation
  • virtual machines
  • runtime development
  • systems programming

Once you understand assembly at this level, C stops looking quite so mysterious.

You start seeing what the compiler is actually doing for you.

Final Thoughts

Writing a Windows application in assembly sounds intimidating, but the first step can be surprisingly small.

At its core, our program simply does this:

prepare arguments
      |
      v
call Windows function
      |
      v
receive result
Enter fullscreen mode Exit fullscreen mode

Assembly just exposes all of the machinery normally hidden behind a high-level language.

Start with:

MessageBox
Enter fullscreen mode Exit fullscreen mode

Then build a window.

Then build a message loop.

Then add graphics, keyboard input, files, and eventually your own abstractions.

That journey takes you from simply using Windows APIs to understanding how programs actually communicate with an operating system.

And that is where assembly becomes really interesting.

Top comments (0)