DEV Community: Piyusha Akash

Windows Syscalls: Direct vs Indirect, and Why Malware Researchers Care

Piyusha Akash — Fri, 11 Sep 2026 13:21:39 +0000

When I started working more deeply with Windows internals, one thing became obvious very quickly:

If you only understand Win32 APIs, you are still looking at Windows from a fairly high level.

Functions such as VirtualAlloc, VirtualProtect, CreateThread, NtAllocateVirtualMemory, and NtProtectVirtualMemory eventually lead into the Windows kernel. The interesting part is what happens between the application and the kernel.

That is where syscalls come in.

For malware research, reverse engineering, EDR research, and offensive security, understanding this boundary is important because modern security products do not simply look at whether a process called a suspicious API.

They look at what the process is actually doing.

And this is also where the difference between direct and indirect syscalls becomes interesting.

First: AV is not the same as EDR

I see these two terms mixed together quite often.

Antivirus and EDR overlap, but they are not the same thing.

Traditional antivirus has historically focused heavily on identifying malicious files, signatures, known patterns, suspicious behavior, and other indicators.

Modern AV products do considerably more than simple signature matching, but the basic distinction is still useful.

EDR is much more focused on visibility and behavioral telemetry.

An EDR can observe things such as:

Process creation
Parent-child process relationships
Image loading
Memory allocation
Memory protection changes
Thread creation
Handle operations
Registry activity
File activity
Network connections
Authentication activity
Suspicious sequences of behavior

The important part is that an EDR does not necessarily care only about the API name.

For example, imagine a process eventually causes executable memory to exist inside another process.

Changing the exact API used does not magically make that behavior legitimate.

The telemetry can still exist somewhere else.

This is why syscall research should not be viewed simply as:

"How do I bypass EDR?"

A better question is:

"Where does Windows transition from user mode into the kernel, and what can security software observe before and after that transition?"

That is the interesting part.

What is a syscall?

A syscall is the mechanism used by user-mode software to request services from the Windows kernel.

Applications normally execute in user mode.

The Windows kernel executes in kernel mode.

These are different privilege levels.

User-mode applications cannot simply execute arbitrary kernel operations whenever they want. Instead, they request kernel services through defined interfaces.

A simplified view looks like this:

Application
    |
    v
Win32 API
    |
    v
ntdll.dll
    |
    v
System Call
    |
    v
Windows Kernel
    |
    v
Kernel operation

For example, an application might call:

VirtualAlloc()

That eventually reaches lower-level Windows functionality.

At the Native API layer you may encounter functions such as:

NtAllocateVirtualMemory()

Eventually the execution reaches the system-call boundary.

The exact internal path depends on the operation and Windows version, but conceptually:

User Mode
------------------------------------------------
Application
    |
    | Win32 API
    v
Kernel32 / KernelBase
    |
    v
ntdll.dll
    |
    | syscall transition
    v
------------------------------------------------
Kernel Mode
    |
    v
System service handling
    |
    v
Kernel implementation

The important boundary is the transition between user mode and kernel mode.

What is inside ntdll?

ntdll.dll is extremely important when studying Windows internals.

It contains a large portion of the Native API exposed to user-mode applications.

You will encounter functions with names such as:

NtOpenProcess
NtAllocateVirtualMemory
NtProtectVirtualMemory
NtCreateThreadEx
NtReadVirtualMemory
NtWriteVirtualMemory
NtQueryInformationProcess

These functions are not equivalent to the entire kernel implementation.

They are user-mode entry points into the Native API.

A simplified example is:

Application
    |
    v
NtAllocateVirtualMemory
    |
    v
syscall
    |
    v
Kernel

The interesting part is that the small user-mode stub eventually performs the transition into kernel mode.

Why does the syscall number matter?

Windows needs a way to identify which kernel service the caller is requesting.

This is where the system service number comes into the picture.

Conceptually:

Syscall instruction
        |
        +---- System Service Number
        |
        v
Kernel system-service dispatcher
        |
        v
Requested kernel service

You will often see researchers referring to this as the SSN, or System Service Number.

The important detail is that syscall numbers are not something you should treat as permanent constants across Windows versions.

They can change between builds.

That is one reason serious syscall research normally involves resolving the appropriate information for the running system rather than blindly assuming that a number from one Windows build will work everywhere.

What is a direct syscall?

A direct syscall means that code in the process invokes the system-call transition without first calling the normal higher-level API path.

Conceptually:

Application
    |
    v
Custom code
    |
    v
syscall
    |
    v
Kernel

Instead of:

Application
    |
    v
Win32 API
    |
    v
ntdll
    |
    v
syscall
    |
    v
Kernel

The distinction is important when studying user-mode monitoring.

Security software can place instrumentation or hooks at different locations in user mode.

For example, a security product may monitor certain functions inside ntdll.dll.

A traditional API call could therefore look like:

Application
     |
     v
ntdll!NtSomething
     |
     v
EDR instrumentation / hook
     |
     v
syscall
     |
     v
Kernel

A direct syscall changes the user-mode path:

Application
     |
     v
custom syscall stub
     |
     v
syscall
     |
     v
Kernel

This is why direct syscalls became interesting in malware research.

They can avoid certain user-mode interception points.

But this does not mean:

"Direct syscalls bypass EDR."

That statement is too simplistic.

They can bypass a particular user-mode hook or instrumentation point.

They do not make the underlying operation invisible.

Direct syscalls do not make malware invisible

This is one of the most important things to understand.

Suppose a program performs suspicious memory operations.

Changing:

NtAllocateVirtualMemory()

to a custom syscall does not change the fact that memory was allocated.

Likewise, changing the user-mode entry path does not automatically erase:

Kernel telemetry
ETW-related visibility
Process activity
Memory state
Thread activity
Image information
Handle activity
Behavioral correlations
Security callbacks
Other EDR sensors

Modern endpoint security is not dependent on a single hook.

This is why I don't consider syscall techniques a magic "EDR bypass."

They are better understood as a way of changing the execution path and reducing dependence on particular user-mode instrumentation points.

Then what is an indirect syscall?

The terminology can be confusing.

An indirect syscall generally refers to performing the syscall transition through a syscall instruction located in an existing legitimate ntdll.dll syscall stub rather than placing the syscall instruction directly inside custom code.

Conceptually:

Custom code
    |
    v
Resolve legitimate ntdll syscall stub
    |
    v
syscall instruction
    |
    v
Kernel

Compare that with a direct approach:

Custom code
    |
    v
custom syscall stub
    |
    v
syscall
    |
    v
Kernel

The distinction is primarily about where the actual syscall transition occurs.

This becomes interesting because the return address and surrounding user-mode execution context can look different from a completely custom syscall stub.

Why are indirect syscalls interesting to EDR research?

Imagine an EDR is monitoring a normal Native API function.

The normal path might be:

Application
    |
    v
ntdll!NtAllocateVirtualMemory
    |
    v
hook
    |
    v
syscall

If the process instead reaches a legitimate syscall instruction elsewhere in ntdll, the execution path through the monitored function can be different.

This is one reason indirect syscall techniques became popular in offensive security research.

But again, this does not mean:

Indirect syscall = invisible

It means:

Different user-mode execution path

That distinction matters.

Why do malware developers care about this?

Malware is an excellent environment for understanding defensive technology because malware authors constantly try to understand where their activity becomes observable.

For example, consider process injection.

At a high level, the operation may involve:

Open target process
        |
        v
Allocate memory
        |
        v
Write data
        |
        v
Change memory protection
        |
        v
Create/execute a thread

The API names can change.

The syscall path can change.

The implementation can change.

But the underlying behavior remains.

That is exactly why studying the syscall layer is useful.

It teaches you to separate:

API

from:

operation

and from:

observable behavior

Those are three different things.

User-mode hooks

One of the first concepts you encounter when studying EDR internals is user-mode hooking.

A simplified example:

Normal:

Application
    |
    v
NtFunction
    |
    v
syscall
    |
    v
Kernel

With instrumentation:

Application
    |
    v
NtFunction
    |
    v
EDR hook
    |
    v
original function
    |
    v
syscall

The hook gives the security product an opportunity to inspect the operation.

Depending on the product and technique, it can collect information about:

Calling process
Arguments
Call stack
Target process
Memory addresses
Requested permissions
Thread information
Other contextual information

This is extremely useful for behavioral detection.

Why malware researchers study clean ntdll

When researching syscalls, one of the things researchers often examine is the syscall stub itself.

A simplified Native API stub can conceptually look like:

NtFunction:
    move system service number
    prepare registers
    syscall
    return

The exact instructions depend on the Windows architecture and version.

Researchers can compare the in-memory implementation of ntdll.dll with a known clean copy.

Why?

Because security products may instrument user-mode functions.

This is useful for both sides.

An offensive researcher wants to understand:

Where is the interception?

A defensive researcher wants to understand:

How can I detect when the normal execution path has been manipulated?

The same technical knowledge applies to both.

Syscalls and call stacks

Another important concept is the call stack.

Consider:

Application
    |
    v
SomeFunction()
    |
    v
NtAllocateVirtualMemory()
    |
    v
syscall

An EDR can potentially use stack information as part of its detection logic.

If a syscall originates from an unusual memory region or an unexpected module, that itself can become suspicious.

This is one reason simply replacing the normal API call with a syscall instruction does not automatically solve the problem.

Security software can look at more than the immediate function.

It can correlate context.

The kernel changes the game

Once execution reaches kernel mode, the security model becomes very different.

The kernel has access to information that user-mode code does not have.

It can observe or participate in mechanisms surrounding:

Processes
Threads
Virtual memory
Handles
Drivers
File systems
Networking
Security tokens
System configuration

This is why modern EDR architecture is not simply:

Hook API -> detect malware

It is closer to:

        Process activity
              |
     +--------+--------+
     |        |        |
     v        v        v
 User-mode  Kernel   ETW / other
 telemetry  telemetry  telemetry
     |        |        |
     +--------+--------+
              |
              v
       Correlation engine
              |
              v
       Detection / response

The exact architecture differs between vendors, but the general idea is important.

Direct syscall vs indirect syscall

A simple comparison:

Technique	User-mode path	Main research interest
Normal API	Win32 -> ntdll -> syscall	Standard Windows execution
Direct syscall	Custom code -> syscall	Avoid specific user-mode interception points
Indirect syscall	Custom code -> legitimate syscall stub -> syscall	Different syscall execution path

The important thing is not which one is "better."

The important question is:

What part of the security visibility model changes when the execution path changes?

That is the question I care about when researching this topic.

Syscalls are not the same thing as Native APIs

These terms are also frequently mixed together.

Native API:

NtCreateFile
NtOpenProcess
NtQueryInformationProcess

Syscall:

The transition from user mode into kernel mode

They are related, but they are not identical concepts.

You can think about it like this:

NtFunction
    |
    | user-mode interface
    v
syscall instruction
    |
    | privilege transition
    v
Windows kernel

The Native API function is the user-mode interface.

The syscall is the transition mechanism.

Why I study this from a malware perspective

I don't study syscalls just because they are commonly associated with malware.

I study them because malware forces you to understand Windows at a deeper level.

If I want to understand why a process injection technique is detected, I need to understand:

What API was called?
        |
        v
What Native API was reached?
        |
        v
What syscall occurred?
        |
        v
What happened inside the kernel?
        |
        v
What telemetry was generated?
        |
        v
What behavior did the EDR correlate?

That is much more useful than memorizing:

"Use this API to do X."

The second approach teaches you a technique.

The first teaches you the system.

The bigger lesson

Direct and indirect syscalls are interesting because they expose an important weakness in simplistic security thinking.

Security is not determined by the name of a function.

Changing:

VirtualAlloc

to:

NtAllocateVirtualMemory

doesn't fundamentally change what the system is doing.

Changing that again to a custom syscall does not fundamentally change the resulting kernel operation either.

The execution path changes.

The observability can change.

The telemetry can change.

But the underlying operation still exists.

That is why modern EDR research is much more interesting than simply learning API hooking.

You need to understand the entire chain:

Application
      |
      v
Win32
      |
      v
Native API
      |
      v
Syscall
      |
      v
Kernel
      |
      v
System activity
      |
      v
Telemetry
      |
      v
EDR correlation
      |
      v
Detection / response

Once you understand that chain, direct and indirect syscalls stop looking like mysterious malware tricks.

They become what they really are:

different ways of reaching the same operating-system boundary while changing parts of the user-mode execution path.

And that is exactly why they are worth studying.

Final thoughts

For me, syscall research is less about finding a magical way around an EDR and more about understanding where security visibility actually comes from.

If an EDR hooks a user-mode function, I want to understand the hook.

If malware avoids that hook, I want to understand the new execution path.

If the EDR still detects the activity, I want to know which telemetry exposed it.

If detection happens in the kernel, I want to understand the kernel-side visibility.

That mindset turns syscall research into Windows internals research.

And once you start looking at Windows this way, things like ntdll, Native APIs, SSNs, system-service dispatching, user/kernel transitions, EPROCESS, threads, handles, memory management, and EDR telemetry all start connecting together.

That is where the real learning begins.

What is API Hooking in a EDR?

Piyusha Akash — Fri, 11 Sep 2026 13:16:10 +0000

So you know there is a big different between Traditional Anti Virus and EDR (Endpoint Detection and Response) system.

Traditional AV was mostly about finding known bad files, signatures, hashes and doing scans. But EDR is more about what is happening inside the system.

For example, a process is running.

It opens a file.

It allocates memory.

It creates another process.

It loads a DLL.

It starts a thread.

It connects to a remote system.

For an EDR, all these things can become useful information.

But how does EDR actually see these things?

One of the techniques used for user mode visibility is API hooking.

What is API Hooking?

API hooking basically means changing the normal execution path of a function so something else can observe or handle the call before the original function continues.

Let's take a simple Windows API.

CreateFileW(...)

Normally your program calls CreateFileW, and Windows handles the request.

Conceptually:

Application
    |
    v
CreateFileW()
    |
    v
Windows
    |
    v
File System

With a hook installed, the execution path can look more like this:

Application
    |
    v
CreateFileW()
    |
    v
EDR Hook
    |
    +----> Collect information
    |
    +----> Analyze the operation
    |
    v
Original CreateFileW()
    |
    v
Windows

The important part here is that the EDR is not necessarily replacing the entire Windows API.

It can intercept the call, collect information about it, and then allow the original function to continue.

Why would an EDR hook an API?

Think about a process doing this:

CreateFileW(
    L"C:\\Users\\Public\\test.exe",
    ...
);

The EDR may be interested in things like:

Which process made the call?
Which file is being accessed?
What operation is being performed?
What is the process doing before and after this?
Does this behavior look suspicious?

One API call alone usually does not tell the whole story.

That is important.

If a normal application opens a file, that does not mean the application is malicious.

But imagine a process doing something like:

Open suspicious file
        |
Allocate executable memory
        |
Write data into memory
        |
Change memory permissions
        |
Create a thread
        |
Execute the memory

Now the EDR has a much more interesting sequence of events.

This is one of the main ideas behind behavioral detection.

Where does the hook actually exist?

This is where things become more interesting.

On Windows, many applications interact with the operating system through user mode APIs.

For example:

Application
    |
    v
kernel32.dll
    |
    v
kernelbase.dll
    |
    v
ntdll.dll
    |
    v
System Call
    |
    v
Windows Kernel

The exact path depends on the API and Windows version, but ntdll.dll is especially important because it contains the user mode system call stubs for many Native API functions.

For example:

NtCreateFile
NtOpenProcess
NtAllocateVirtualMemory
NtProtectVirtualMemory
NtWriteVirtualMemory
NtCreateThreadEx

An EDR operating in user mode can place hooks at different points in this path.

For example:

Application
    |
    v
WinAPI
    |
    v
EDR instrumentation
    |
    v
Native API
    |
    v
System call

This gives the EDR visibility into operations performed by applications.

Inline Hooking

One common type of API hook is an inline hook.

The basic idea is simple.

The beginning of a function is modified so execution is redirected somewhere else.

For example, imagine a function starts like this:

Function:
    instruction 1
    instruction 2
    instruction 3
    instruction 4

A hook can change the beginning so it becomes conceptually:

Function:
    JMP EDR_HANDLER
    instruction 2
    instruction 3
    instruction 4

Now when the program calls the function, execution first goes to the handler.

Something like:

Application
     |
     v
Target Function
     |
     v
JMP
     |
     v
EDR Handler
     |
     v
Original Instructions

Of course, real implementations are more complicated than this.

On x64 Windows, instruction boundaries, relative addressing, register state, calling conventions and executable memory all matter.

You cannot just overwrite random bytes and expect everything to work.

What happens inside the hook?

The hook can collect information about the current execution.

For example:

Process ID
Thread ID
Function being called
Arguments
Return value
Call stack
Loaded modules
Process information

Depending on the EDR architecture, this information can be sent to another component for further processing.

Conceptually:

Process
   |
   v
Hook
   |
   v
Telemetry
   |
   v
EDR Sensor
   |
   v
Detection Engine

The interesting part is the telemetry.

The EDR is trying to build a picture of what the process is doing.

API Hooking is not the same thing as Kernel Monitoring

This is a very important distinction.

An EDR can have user mode components and kernel mode components.

User mode hooking gives visibility into operations happening through hooked user mode functions.

Kernel components can provide visibility much closer to the operating system itself.

For example:

                 EDR
                  |
        +---------+---------+
        |                   |
    User Mode           Kernel Mode
        |                   |
     API Hooks          Kernel callbacks
        |                   |
        +---------+---------+
                  |
             Telemetry

This is one reason modern EDR products are much more complicated than simply putting hooks inside kernel32.dll.

They usually combine multiple visibility mechanisms.

Why API Hooking can be useful for EDR research

If you want to understand EDR internals, API hooking is a good place to start because it forces you to understand what happens between an application and the Windows kernel.

You start with something simple:

CreateFileW()

Then you start asking questions.

Where does this function actually live?

What DLL exports it?

Does it call another function?

Does it eventually reach ntdll.dll?

What does the assembly look like?

Where does the system call happen?

What happens to the arguments?

What happens when the function returns?

You can investigate these questions with tools like WinDbg, System Informer and a debugger.

For example:

WinAPI
  |
  +-- kernel32.dll
  |
  +-- kernelbase.dll
  |
  +-- ntdll.dll
          |
          +-- Nt*
          |
          +-- syscall

Once you understand this path, EDR internals start becoming much easier to understand.

How do you identify a hook?

This is where reverse engineering becomes useful.

Suppose you expect the beginning of a function to look like normal Windows code.

Instead, the first instructions look unusual.

For example:

Expected:

mov r10, rcx
mov eax, <SSN>
syscall
ret

But the memory you are looking at contains a jump to another address.

That is something worth investigating.

The destination may belong to another module or another executable memory region.

You can inspect:

Module name
Memory address
Memory protection
Instruction bytes
Call target
Call stack

A debugger can help you trace the execution.

The goal is not simply:

"I found a JMP."

The interesting question is:

"Why is execution being redirected here?"

That is the difference between looking at bytes and actually doing reverse engineering.

What about malware and API hooking?

Malware authors also care about API hooks.

If a security product hooks a function inside a process, modifying or bypassing that hook can potentially change what the security product observes.

This is one reason API hooking becomes an important subject when studying EDR internals.

But there is another side to this.

From the defender's point of view, attempts to interfere with security instrumentation can themselves become useful telemetry.

So you can end up with something like:

EDR installs instrumentation
          |
          v
Process executes
          |
          v
Suspicious behavior
          |
          v
Process interacts with instrumentation
          |
          v
EDR observes the activity

This is why modern EDR research should not be reduced to "find the hook and remove it."

The real question is:

What visibility does the EDR have, where does that visibility come from, and what happens when one visibility mechanism is no longer available?

API Hooking is only one piece

This is probably the most important thing to understand.

A modern EDR is not just:

API Hooks = EDR

It is much closer to:

                 EDR
                  |
       +----------+----------+
       |          |          |
    User Mode  Kernel     Network
       |        Mode         |
       |          |          |
    API hooks  Callbacks   Traffic
       |          |          |
       +----------+----------+
                  |
              Telemetry
                  |
                  v
          Detection Engine
                  |
                  v
               Alert

Different products use different combinations of technologies.

Some visibility can happen in user mode.

Some can happen in the kernel.

Some information can come from Windows event sources, ETW, network telemetry, memory inspection and other sensors.

That is why learning EDR internals requires learning Windows internals first.

What I am studying next

For me, API hooking is not really the final topic.

It is the beginning of understanding how an EDR sees a process.

The next questions become much more interesting:

How are hooks installed?

How are hooked functions detected?

How does the EDR capture arguments?

How does it collect call stacks?

How does it correlate multiple events?

What happens when a process does not use the expected API?

How does kernel telemetry complement user mode telemetry?

How does the EDR detect behavior instead of individual API calls?

And eventually:

Process
   |
   v
User Mode
   |
   +---- API
   |
   +---- Native API
   |
   +---- Syscall
   |
   v
Kernel
   |
   v
EDR Telemetry
   |
   v
Detection

Understanding this entire path is much more useful than simply memorizing API names.

That is basically where I am starting my EDR internals research.

API hooking is just one part of the bigger picture.