When I started working more deeply with Windows internals, one thing became obvious very quickly:
If you only understand Win32 APIs, you are still looking at Windows from a fairly high level.
Functions such as VirtualAlloc, VirtualProtect, CreateThread, NtAllocateVirtualMemory, and NtProtectVirtualMemory eventually lead into the Windows kernel. The interesting part is what happens between the application and the kernel.
That is where syscalls come in.
For malware research, reverse engineering, EDR research, and offensive security, understanding this boundary is important because modern security products do not simply look at whether a process called a suspicious API.
They look at what the process is actually doing.
And this is also where the difference between direct and indirect syscalls becomes interesting.
First: AV is not the same as EDR
I see these two terms mixed together quite often.
Antivirus and EDR overlap, but they are not the same thing.
Traditional antivirus has historically focused heavily on identifying malicious files, signatures, known patterns, suspicious behavior, and other indicators.
Modern AV products do considerably more than simple signature matching, but the basic distinction is still useful.
EDR is much more focused on visibility and behavioral telemetry.
An EDR can observe things such as:
- Process creation
- Parent-child process relationships
- Image loading
- Memory allocation
- Memory protection changes
- Thread creation
- Handle operations
- Registry activity
- File activity
- Network connections
- Authentication activity
- Suspicious sequences of behavior
The important part is that an EDR does not necessarily care only about the API name.
For example, imagine a process eventually causes executable memory to exist inside another process.
Changing the exact API used does not magically make that behavior legitimate.
The telemetry can still exist somewhere else.
This is why syscall research should not be viewed simply as:
"How do I bypass EDR?"
A better question is:
"Where does Windows transition from user mode into the kernel, and what can security software observe before and after that transition?"
That is the interesting part.
What is a syscall?
A syscall is the mechanism used by user-mode software to request services from the Windows kernel.
Applications normally execute in user mode.
The Windows kernel executes in kernel mode.
These are different privilege levels.
User-mode applications cannot simply execute arbitrary kernel operations whenever they want. Instead, they request kernel services through defined interfaces.
A simplified view looks like this:
Application
|
v
Win32 API
|
v
ntdll.dll
|
v
System Call
|
v
Windows Kernel
|
v
Kernel operation
For example, an application might call:
VirtualAlloc()
That eventually reaches lower-level Windows functionality.
At the Native API layer you may encounter functions such as:
NtAllocateVirtualMemory()
Eventually the execution reaches the system-call boundary.
The exact internal path depends on the operation and Windows version, but conceptually:
User Mode
------------------------------------------------
Application
|
| Win32 API
v
Kernel32 / KernelBase
|
v
ntdll.dll
|
| syscall transition
v
------------------------------------------------
Kernel Mode
|
v
System service handling
|
v
Kernel implementation
The important boundary is the transition between user mode and kernel mode.
What is inside ntdll?
ntdll.dll is extremely important when studying Windows internals.
It contains a large portion of the Native API exposed to user-mode applications.
You will encounter functions with names such as:
NtOpenProcess
NtAllocateVirtualMemory
NtProtectVirtualMemory
NtCreateThreadEx
NtReadVirtualMemory
NtWriteVirtualMemory
NtQueryInformationProcess
These functions are not equivalent to the entire kernel implementation.
They are user-mode entry points into the Native API.
A simplified example is:
Application
|
v
NtAllocateVirtualMemory
|
v
syscall
|
v
Kernel
The interesting part is that the small user-mode stub eventually performs the transition into kernel mode.
Why does the syscall number matter?
Windows needs a way to identify which kernel service the caller is requesting.
This is where the system service number comes into the picture.
Conceptually:
Syscall instruction
|
+---- System Service Number
|
v
Kernel system-service dispatcher
|
v
Requested kernel service
You will often see researchers referring to this as the SSN, or System Service Number.
The important detail is that syscall numbers are not something you should treat as permanent constants across Windows versions.
They can change between builds.
That is one reason serious syscall research normally involves resolving the appropriate information for the running system rather than blindly assuming that a number from one Windows build will work everywhere.
What is a direct syscall?
A direct syscall means that code in the process invokes the system-call transition without first calling the normal higher-level API path.
Conceptually:
Application
|
v
Custom code
|
v
syscall
|
v
Kernel
Instead of:
Application
|
v
Win32 API
|
v
ntdll
|
v
syscall
|
v
Kernel
The distinction is important when studying user-mode monitoring.
Security software can place instrumentation or hooks at different locations in user mode.
For example, a security product may monitor certain functions inside ntdll.dll.
A traditional API call could therefore look like:
Application
|
v
ntdll!NtSomething
|
v
EDR instrumentation / hook
|
v
syscall
|
v
Kernel
A direct syscall changes the user-mode path:
Application
|
v
custom syscall stub
|
v
syscall
|
v
Kernel
This is why direct syscalls became interesting in malware research.
They can avoid certain user-mode interception points.
But this does not mean:
"Direct syscalls bypass EDR."
That statement is too simplistic.
They can bypass a particular user-mode hook or instrumentation point.
They do not make the underlying operation invisible.
Direct syscalls do not make malware invisible
This is one of the most important things to understand.
Suppose a program performs suspicious memory operations.
Changing:
NtAllocateVirtualMemory()
to a custom syscall does not change the fact that memory was allocated.
Likewise, changing the user-mode entry path does not automatically erase:
- Kernel telemetry
- ETW-related visibility
- Process activity
- Memory state
- Thread activity
- Image information
- Handle activity
- Behavioral correlations
- Security callbacks
- Other EDR sensors
Modern endpoint security is not dependent on a single hook.
This is why I don't consider syscall techniques a magic "EDR bypass."
They are better understood as a way of changing the execution path and reducing dependence on particular user-mode instrumentation points.
Then what is an indirect syscall?
The terminology can be confusing.
An indirect syscall generally refers to performing the syscall transition through a syscall instruction located in an existing legitimate ntdll.dll syscall stub rather than placing the syscall instruction directly inside custom code.
Conceptually:
Custom code
|
v
Resolve legitimate ntdll syscall stub
|
v
syscall instruction
|
v
Kernel
Compare that with a direct approach:
Custom code
|
v
custom syscall stub
|
v
syscall
|
v
Kernel
The distinction is primarily about where the actual syscall transition occurs.
This becomes interesting because the return address and surrounding user-mode execution context can look different from a completely custom syscall stub.
Why are indirect syscalls interesting to EDR research?
Imagine an EDR is monitoring a normal Native API function.
The normal path might be:
Application
|
v
ntdll!NtAllocateVirtualMemory
|
v
hook
|
v
syscall
If the process instead reaches a legitimate syscall instruction elsewhere in ntdll, the execution path through the monitored function can be different.
This is one reason indirect syscall techniques became popular in offensive security research.
But again, this does not mean:
Indirect syscall = invisible
It means:
Different user-mode execution path
That distinction matters.
Why do malware developers care about this?
Malware is an excellent environment for understanding defensive technology because malware authors constantly try to understand where their activity becomes observable.
For example, consider process injection.
At a high level, the operation may involve:
Open target process
|
v
Allocate memory
|
v
Write data
|
v
Change memory protection
|
v
Create/execute a thread
The API names can change.
The syscall path can change.
The implementation can change.
But the underlying behavior remains.
That is exactly why studying the syscall layer is useful.
It teaches you to separate:
API
from:
operation
and from:
observable behavior
Those are three different things.
User-mode hooks
One of the first concepts you encounter when studying EDR internals is user-mode hooking.
A simplified example:
Normal:
Application
|
v
NtFunction
|
v
syscall
|
v
Kernel
With instrumentation:
Application
|
v
NtFunction
|
v
EDR hook
|
v
original function
|
v
syscall
The hook gives the security product an opportunity to inspect the operation.
Depending on the product and technique, it can collect information about:
- Calling process
- Arguments
- Call stack
- Target process
- Memory addresses
- Requested permissions
- Thread information
- Other contextual information
This is extremely useful for behavioral detection.
Why malware researchers study clean ntdll
When researching syscalls, one of the things researchers often examine is the syscall stub itself.
A simplified Native API stub can conceptually look like:
NtFunction:
move system service number
prepare registers
syscall
return
The exact instructions depend on the Windows architecture and version.
Researchers can compare the in-memory implementation of ntdll.dll with a known clean copy.
Why?
Because security products may instrument user-mode functions.
This is useful for both sides.
An offensive researcher wants to understand:
Where is the interception?
A defensive researcher wants to understand:
How can I detect when the normal execution path has been manipulated?
The same technical knowledge applies to both.
Syscalls and call stacks
Another important concept is the call stack.
Consider:
Application
|
v
SomeFunction()
|
v
NtAllocateVirtualMemory()
|
v
syscall
An EDR can potentially use stack information as part of its detection logic.
If a syscall originates from an unusual memory region or an unexpected module, that itself can become suspicious.
This is one reason simply replacing the normal API call with a syscall instruction does not automatically solve the problem.
Security software can look at more than the immediate function.
It can correlate context.
The kernel changes the game
Once execution reaches kernel mode, the security model becomes very different.
The kernel has access to information that user-mode code does not have.
It can observe or participate in mechanisms surrounding:
- Processes
- Threads
- Virtual memory
- Handles
- Drivers
- File systems
- Networking
- Security tokens
- System configuration
This is why modern EDR architecture is not simply:
Hook API -> detect malware
It is closer to:
Process activity
|
+--------+--------+
| | |
v v v
User-mode Kernel ETW / other
telemetry telemetry telemetry
| | |
+--------+--------+
|
v
Correlation engine
|
v
Detection / response
The exact architecture differs between vendors, but the general idea is important.
Direct syscall vs indirect syscall
A simple comparison:
| Technique | User-mode path | Main research interest |
|---|---|---|
| Normal API | Win32 -> ntdll -> syscall | Standard Windows execution |
| Direct syscall | Custom code -> syscall | Avoid specific user-mode interception points |
| Indirect syscall | Custom code -> legitimate syscall stub -> syscall | Different syscall execution path |
The important thing is not which one is "better."
The important question is:
What part of the security visibility model changes when the execution path changes?
That is the question I care about when researching this topic.
Syscalls are not the same thing as Native APIs
These terms are also frequently mixed together.
Native API:
NtCreateFile
NtOpenProcess
NtQueryInformationProcess
Syscall:
The transition from user mode into kernel mode
They are related, but they are not identical concepts.
You can think about it like this:
NtFunction
|
| user-mode interface
v
syscall instruction
|
| privilege transition
v
Windows kernel
The Native API function is the user-mode interface.
The syscall is the transition mechanism.
Why I study this from a malware perspective
I don't study syscalls just because they are commonly associated with malware.
I study them because malware forces you to understand Windows at a deeper level.
If I want to understand why a process injection technique is detected, I need to understand:
What API was called?
|
v
What Native API was reached?
|
v
What syscall occurred?
|
v
What happened inside the kernel?
|
v
What telemetry was generated?
|
v
What behavior did the EDR correlate?
That is much more useful than memorizing:
"Use this API to do X."
The second approach teaches you a technique.
The first teaches you the system.
The bigger lesson
Direct and indirect syscalls are interesting because they expose an important weakness in simplistic security thinking.
Security is not determined by the name of a function.
Changing:
VirtualAlloc
to:
NtAllocateVirtualMemory
doesn't fundamentally change what the system is doing.
Changing that again to a custom syscall does not fundamentally change the resulting kernel operation either.
The execution path changes.
The observability can change.
The telemetry can change.
But the underlying operation still exists.
That is why modern EDR research is much more interesting than simply learning API hooking.
You need to understand the entire chain:
Application
|
v
Win32
|
v
Native API
|
v
Syscall
|
v
Kernel
|
v
System activity
|
v
Telemetry
|
v
EDR correlation
|
v
Detection / response
Once you understand that chain, direct and indirect syscalls stop looking like mysterious malware tricks.
They become what they really are:
different ways of reaching the same operating-system boundary while changing parts of the user-mode execution path.
And that is exactly why they are worth studying.
Final thoughts
For me, syscall research is less about finding a magical way around an EDR and more about understanding where security visibility actually comes from.
If an EDR hooks a user-mode function, I want to understand the hook.
If malware avoids that hook, I want to understand the new execution path.
If the EDR still detects the activity, I want to know which telemetry exposed it.
If detection happens in the kernel, I want to understand the kernel-side visibility.
That mindset turns syscall research into Windows internals research.
And once you start looking at Windows this way, things like ntdll, Native APIs, SSNs, system-service dispatching, user/kernel transitions, EPROCESS, threads, handles, memory management, and EDR telemetry all start connecting together.
That is where the real learning begins.
Top comments (0)