So you know there is a big different between Traditional Anti Virus and EDR (Endpoint Detection and Response) system.
Traditional AV was mostly about finding known bad files, signatures, hashes and doing scans. But EDR is more about what is happening inside the system.
For example, a process is running.
It opens a file.
It allocates memory.
It creates another process.
It loads a DLL.
It starts a thread.
It connects to a remote system.
For an EDR, all these things can become useful information.
But how does EDR actually see these things?
One of the techniques used for user mode visibility is API hooking.
What is API Hooking?
API hooking basically means changing the normal execution path of a function so something else can observe or handle the call before the original function continues.
Let's take a simple Windows API.
CreateFileW(...)
Normally your program calls CreateFileW, and Windows handles the request.
Conceptually:
Application
|
v
CreateFileW()
|
v
Windows
|
v
File System
With a hook installed, the execution path can look more like this:
Application
|
v
CreateFileW()
|
v
EDR Hook
|
+----> Collect information
|
+----> Analyze the operation
|
v
Original CreateFileW()
|
v
Windows
The important part here is that the EDR is not necessarily replacing the entire Windows API.
It can intercept the call, collect information about it, and then allow the original function to continue.
Why would an EDR hook an API?
Think about a process doing this:
CreateFileW(
L"C:\\Users\\Public\\test.exe",
...
);
The EDR may be interested in things like:
Which process made the call?
Which file is being accessed?
What operation is being performed?
What is the process doing before and after this?
Does this behavior look suspicious?
One API call alone usually does not tell the whole story.
That is important.
If a normal application opens a file, that does not mean the application is malicious.
But imagine a process doing something like:
Open suspicious file
|
Allocate executable memory
|
Write data into memory
|
Change memory permissions
|
Create a thread
|
Execute the memory
Now the EDR has a much more interesting sequence of events.
This is one of the main ideas behind behavioral detection.
Where does the hook actually exist?
This is where things become more interesting.
On Windows, many applications interact with the operating system through user mode APIs.
For example:
Application
|
v
kernel32.dll
|
v
kernelbase.dll
|
v
ntdll.dll
|
v
System Call
|
v
Windows Kernel
The exact path depends on the API and Windows version, but ntdll.dll is especially important because it contains the user mode system call stubs for many Native API functions.
For example:
NtCreateFile
NtOpenProcess
NtAllocateVirtualMemory
NtProtectVirtualMemory
NtWriteVirtualMemory
NtCreateThreadEx
An EDR operating in user mode can place hooks at different points in this path.
For example:
Application
|
v
WinAPI
|
v
EDR instrumentation
|
v
Native API
|
v
System call
This gives the EDR visibility into operations performed by applications.
Inline Hooking
One common type of API hook is an inline hook.
The basic idea is simple.
The beginning of a function is modified so execution is redirected somewhere else.
For example, imagine a function starts like this:
Function:
instruction 1
instruction 2
instruction 3
instruction 4
A hook can change the beginning so it becomes conceptually:
Function:
JMP EDR_HANDLER
instruction 2
instruction 3
instruction 4
Now when the program calls the function, execution first goes to the handler.
Something like:
Application
|
v
Target Function
|
v
JMP
|
v
EDR Handler
|
v
Original Instructions
Of course, real implementations are more complicated than this.
On x64 Windows, instruction boundaries, relative addressing, register state, calling conventions and executable memory all matter.
You cannot just overwrite random bytes and expect everything to work.
What happens inside the hook?
The hook can collect information about the current execution.
For example:
Process ID
Thread ID
Function being called
Arguments
Return value
Call stack
Loaded modules
Process information
Depending on the EDR architecture, this information can be sent to another component for further processing.
Conceptually:
Process
|
v
Hook
|
v
Telemetry
|
v
EDR Sensor
|
v
Detection Engine
The interesting part is the telemetry.
The EDR is trying to build a picture of what the process is doing.
API Hooking is not the same thing as Kernel Monitoring
This is a very important distinction.
An EDR can have user mode components and kernel mode components.
User mode hooking gives visibility into operations happening through hooked user mode functions.
Kernel components can provide visibility much closer to the operating system itself.
For example:
EDR
|
+---------+---------+
| |
User Mode Kernel Mode
| |
API Hooks Kernel callbacks
| |
+---------+---------+
|
Telemetry
This is one reason modern EDR products are much more complicated than simply putting hooks inside kernel32.dll.
They usually combine multiple visibility mechanisms.
Why API Hooking can be useful for EDR research
If you want to understand EDR internals, API hooking is a good place to start because it forces you to understand what happens between an application and the Windows kernel.
You start with something simple:
CreateFileW()
Then you start asking questions.
Where does this function actually live?
What DLL exports it?
Does it call another function?
Does it eventually reach ntdll.dll?
What does the assembly look like?
Where does the system call happen?
What happens to the arguments?
What happens when the function returns?
You can investigate these questions with tools like WinDbg, System Informer and a debugger.
For example:
WinAPI
|
+-- kernel32.dll
|
+-- kernelbase.dll
|
+-- ntdll.dll
|
+-- Nt*
|
+-- syscall
Once you understand this path, EDR internals start becoming much easier to understand.
How do you identify a hook?
This is where reverse engineering becomes useful.
Suppose you expect the beginning of a function to look like normal Windows code.
Instead, the first instructions look unusual.
For example:
Expected:
mov r10, rcx
mov eax, <SSN>
syscall
ret
But the memory you are looking at contains a jump to another address.
That is something worth investigating.
The destination may belong to another module or another executable memory region.
You can inspect:
Module name
Memory address
Memory protection
Instruction bytes
Call target
Call stack
A debugger can help you trace the execution.
The goal is not simply:
"I found a JMP."
The interesting question is:
"Why is execution being redirected here?"
That is the difference between looking at bytes and actually doing reverse engineering.
What about malware and API hooking?
Malware authors also care about API hooks.
If a security product hooks a function inside a process, modifying or bypassing that hook can potentially change what the security product observes.
This is one reason API hooking becomes an important subject when studying EDR internals.
But there is another side to this.
From the defender's point of view, attempts to interfere with security instrumentation can themselves become useful telemetry.
So you can end up with something like:
EDR installs instrumentation
|
v
Process executes
|
v
Suspicious behavior
|
v
Process interacts with instrumentation
|
v
EDR observes the activity
This is why modern EDR research should not be reduced to "find the hook and remove it."
The real question is:
What visibility does the EDR have, where does that visibility come from, and what happens when one visibility mechanism is no longer available?
API Hooking is only one piece
This is probably the most important thing to understand.
A modern EDR is not just:
API Hooks = EDR
It is much closer to:
EDR
|
+----------+----------+
| | |
User Mode Kernel Network
| Mode |
| | |
API hooks Callbacks Traffic
| | |
+----------+----------+
|
Telemetry
|
v
Detection Engine
|
v
Alert
Different products use different combinations of technologies.
Some visibility can happen in user mode.
Some can happen in the kernel.
Some information can come from Windows event sources, ETW, network telemetry, memory inspection and other sensors.
That is why learning EDR internals requires learning Windows internals first.
What I am studying next
For me, API hooking is not really the final topic.
It is the beginning of understanding how an EDR sees a process.
The next questions become much more interesting:
How are hooks installed?
How are hooked functions detected?
How does the EDR capture arguments?
How does it collect call stacks?
How does it correlate multiple events?
What happens when a process does not use the expected API?
How does kernel telemetry complement user mode telemetry?
How does the EDR detect behavior instead of individual API calls?
And eventually:
Process
|
v
User Mode
|
+---- API
|
+---- Native API
|
+---- Syscall
|
v
Kernel
|
v
EDR Telemetry
|
v
Detection
Understanding this entire path is much more useful than simply memorizing API names.
That is basically where I am starting my EDR internals research.
API hooking is just one part of the bigger picture.
Top comments (0)