Malware Unpacking & Anti-Analysis Bypass: A Deep Dive into Real-World Techniques

Malware authors don't make our job easy. Every time we think we've figured out their tricks, they layer on another obfuscation technique, another anti-debugging check, another sandbox evasion. Over the past few weeks, I've been deep in the trenches with some particularly stubborn samples — the kind that detect your debugger, hide their strings behind XOR encoding, and hollow out legitimate processes to hide their payload.

This article walks through my hands-on exploration of these techniques. We'll look at how malware detects analysis tools, how it obfuscates its strings, how it unpacks itself in memory, and most importantly — how we can bypass these defenses to see what the malware is actually trying to do.

The tools we'll use:

• x64dbg/x32dbg for dynamic analysis and patching
• IDA Pro for static disassembly
• REMnux (Linux toolkit) for string deobfuscation
• FLOSS, XORSearch, bbcrack for automated string decoding
• Scylla & OllyDumpEx for dumping unpacked payloads
• Process Hacker for memory forensics

---

Problem Statement

Modern malware is rarely "what you see is what you get." A single executable might be:

• Packed — the actual malicious code is compressed/encrypted and only revealed at runtime
• Anti-debug aware — it checks for debuggers and changes behavior or terminates
• Sandbox-aware — it detects virtualized environments and refuses to execute its payload
• String-obfuscated — URLs, registry keys, and IOCs are encoded to evade signature detection
• Process-injecting — it hollows out a legitimate process (like explorer.exe) and runs its code there

Our goal: peel back these layers and extract the real payload for analysis.

---

Exercise 1: Bypassing Debugger Detection in getdown.exe

What I Found

The first sample, getdown.exe, refused to show any network activity when run inside a debugger. Outside the debugger, it connected to 1.234.27.146:80. Classic anti-debugging behavior.

The Detection Mechanism

Using x64dbg, I searched for intermodular calls and immediately spotted IsDebuggerPresent at the top of the list:

call qword ptr ds:[<&IsDebuggerPresent>]
test eax, eax
jne getdown.140001216  ; jumps away if debugger detected

When IsDebuggerPresent returns 1 (debugger present), the JNE jumps to code that terminates the process. When it returns 0, execution continues normally.

The Bypass

I set a breakpoint on the TEST EAX, EAX instruction after the call. When hit, RAX contained 1 — confirming debugger detection. The fix was simple: patch the conditional jump to unconditional NOPs.

; Before (at 14000102C):
jne getdown.140001216

; After patching:
nop
nop
nop
nop
nop
nop

In x64dbg: select the JNE instruction → press Space → type NOP → enable "Fill with NOP's" → OK. Now the malware executes its payload regardless of what IsDebuggerPresent returns.

---

Exercise 2: Deobfuscating Encoded Strings

Malware hides its strings using simple encoding algorithms. I experimented with several tools to decode them.

Tool 1: XORSearch

Searching for the known IP 1.234.27. inside getdown.exe:

xorsearch -i -s getdown.exe 1.234.27.

Output revealed an XOR key of 0x83. The -s flag generated getdown.exe.XOR.83 — a fully decoded file. Extracting strings from it:

strings getdown.exe.XOR.83 | more

This surfaced not just the C2 URL, but also an affiliate ID string — a nice unexpected find.

Tool 2: brxor.py & bbcrack.py

On hubert.dll, brxor.py automatically found XOR key 0x5 and decoded English-language strings:

brxor.py hubert.dll

The output suggested this sample was a fake antivirus tool, complete with registry key paths and URL patterns — solid IOCs for further hunting.

For deeper analysis, bbcrack.py with single-level transformations:

bbcrack.py -l 1 hubert.dll

This generated hubert_xor05.dll. Running strings on it confirmed the decoded content matched and extended what brxor.py found.

Tool 3: Manual Stack String Decoding in IDA

Some malware builds strings character-by-character on the stack at runtime. In 9.exe, between offsets 40133D and 4013B8, I found a block of MOV instructions pushing single bytes:

mov byte ptr [ebp+var_4], 5Ch  ; ''
mov byte ptr [ebp+var_3], 50h  ; 'P'
; ... etc

In IDA, highlighting each hex value and pressing R converts it to ASCII. Doing this for the entire block revealed the string:

\Program Files\Common Files\

Tool 4: Automated Stack String Extraction

For 9.exe, strdeob.pl on REMnux decoded these automatically:

strdeob.pl 9.exe | more

This found more strings than FLOSS managed to extract, showing the value of combining multiple tools:

floss 9.exe > 9-floss.txt

---

Exercise 3: Unpacking drtg.exe Using RtlDecompressBuffer

Static Recon with FLOSS

Before touching the debugger, I ran FLOSS on drtg.exe:

floss drtg.exe > drtg-floss.txt

FLOSS decoded 27 obfuscated strings, including:

• NtAllocateVirtualMemory
• ZwProtectVirtualMemory
• ZwWriteVirtualMemory
• RtlDecompressBuffer

These APIs scream "unpacking and injection." The presence of RtlDecompressBuffer was particularly interesting — it decompresses a buffer using a known compression algorithm.

Dynamic Unpacking in x32dbg

1. Enable ScyllaHide (Plugins → Scylla Hide → Options → check all first-column boxes) to cloak the debugger.

2. Set breakpoint on RtlDecompressBuffer:
   

   SetBPX RtlDecompressBuffer

1. Run (F9) until the breakpoint hits inside ntdll.dll.

2. Follow the UncompressedBuffer parameter (second arg, [esp+8]) in the Dump panel. Initially all zeros.

3. Set a return breakpoint at 402D66 (where execution returns after RtlDecompressBuffer):

   • Go to 402D66 (Ctrl+G)
   • Toggle breakpoint (F2)
   • Run (F9)

4. After returning, the Dump panel shows decompressed data. Step over (F8) nine times until offset 4022CC. The dump now shows:
   

4D 5A 90 00 03 00 00 00 04 00 00 00 FF FF 00 00  MZ.ÿÿ..
B8 00 00 00 00 00 00 00 40 00 00 00 00 00 00 00  ¸.......@.......

That MZ header and "This program cannot be run in DOS mode" confirms a valid PE file was decompressed into memory.

1. Dump the memory region: Right-click Dump → Follow in Memory Map → right-click region → Dump Memory to File → save as drtg-dumped.exe.

Verification

Loaded drtg-dumped.exe in PeStudio — parsed successfully. Checked imports and found NtQueryInformationProcess — another anti-debug API.

Bonus: Finding the Anti-Debug Check

In IDA, at offset 4054B7:

push 7                    ; ProcessInformationClass = 7 (debug port)
lea eax, [ebp-4]
push eax                  ; ProcessInformation
push 4                    ; ProcessInformationLength
call GetCurrentProcess
push eax                  ; ProcessHandle
call NtQueryInformationProcess
cmp dword ptr [ebp-4], 0
jnz short loc_4058C9      ; jumps to ExitProcess if debugged

The value 7 for ProcessInformationClass retrieves the debugger port number. Non-zero means a debugger is attached. To bypass: patch the JNZ at 4054C1 to JZ — invert the logic.

---

Exercise 4: Behavioral Analysis of WinHost32.exe

Before unpacking, I wanted to see what this sample actually does when it runs.

Setup

• REMnux: Wireshark capturing
• Windows REM Workstation: Process Hacker + Process Monitor running

Infection

Ran WinHost32.exe for ~30 seconds, then terminated it.

Network Findings (Wireshark)

DNS queries attempting to resolve google.com. No direct C2 connections in the short window — likely a connectivity check before revealing true behavior.

Process Monitor → ProcDOT

Exported the ProcMon log as CSV, loaded it into ProcDOT:

• Selected the initial WinHost32.exe PID as the launcher
• Generated the graph

The graph revealed process hollowing: the initial process spawned a child process with the same name. The child process showed registry activity (likely false positives, but worth noting).

---

Exercise 5: Unpacking WinHost32.exe (Process Hollowing)

Static Analysis in IDA

At offset 4021AE, CreateProcessA is called with dwCreationFlags = 4:

push 4                    ; CREATE_SUSPENDED
push ...
call CreateProcessA

CREATE_SUSPENDED is a red flag — the process starts with its main thread frozen, ready for manipulation.

Following the code:

• 40220D: VirtualAllocEx with flProtect = 0x40 (PAGE_EXECUTE_READWRITE)
• 402237: WriteProcessMemory — writing into the suspended process
• 4021F7: call edi where EDI = NtUnmapViewOfSection

This is the process hollowing pattern: unmap the legitimate executable's memory, allocate new RWX memory, write the malicious payload, then resume the thread.

Dynamic Extraction in x32dbg

1. Load WinHost32.exe, go to 402237 (Ctrl+G)
2. Set breakpoint on the call ebp (WriteProcessMemory)
3. Run (F9) until breakpoint
4. The lpBuffer parameter (3rd arg, [esp+8]) points to the payload. Follow in Dump
5. Dump shows MZ header — the payload is a full PE
6. Follow in Memory Map → Dump Memory to File → winhost32-dumped.exe

Verification

PeStudio confirmed it's a valid PE. Strings revealed:

• Multiple suspicious URLs
• google.com (the connectivity check we saw in Wireshark)
• Many more plaintext strings than the packed version

---

Exercise 6: Anti-Sandbox via Mouse Hooks (vbprop.exe)

Some malware refuses to execute until a real user interacts with it. This sample uses SetWindowsHookExA to wait for mouse events.

The Hook Setup (IDA)

At 40103E:

push 0                    ; hmod = 0 (current process)
push offset fn            ; lpfn = 4010B0 (our hook function)
push 0Eh                  ; idHook = WH_MOUSE (0x0E)
call SetWindowsHookExA

The hook function at 4010B0 checks wParam:

• 0x200 (WM_MOUSEMOVE) → pass to next hook
• 0x201 (WM_LBUTTONDOWN) → pass to next hook
• 0x202 (WM_LBUTTONUP) → unhook and execute malicious payload (sub_401170)

The malware only reveals itself when you release the left mouse button. Automated sandboxes that don't simulate real mouse clicks will never see the payload execute.

---

Exercise 7: Sandbox Evasion via Connectivity Check (winhost32-dumped.exe)

The unpacked WinHost32.exe checks for internet connectivity before executing its payload. It doesn't just check if the network is up — it downloads http://google.com and verifies the response starts with <!do (the beginning of <!doctype html>).

The Logic Flow

sub_401A90():
    download http://google.com
    check if buffer[0:4] == "<!do"
    return 1 if match, 0 otherwise

Caller at 4023B0:
    call sub_401A90
    test eax, eax
    jnz loc_4023C6        ; success path
    push 0EA60h           ; 60000 ms = 1 minute
    call Sleep
    jmp loop_start        ; try again forever

If no Google connectivity: sleep 60 seconds, retry indefinitely. The payload never executes.

The Bypass

At offset 4023B7, change JNE to JMP — unconditional jump over the Sleep:

; Before:
jne short loc_4023C6

; After:
jmp short loc_4023C6

In x32dbg: select instruction → Space → type jmp → OK. Now the malware executes its payload even when offline.

After patching, running the sample with fakedns on REMnux revealed DNS queries for:

• callereb.com
• supketwron.ru

These were completely hidden before bypassing the connectivity check.

---

Exercise 8: Toolkit Detection in raas.exe

This sample was the most paranoid I've seen. It checks for:

• AVG (avghookx.dll via GetModuleHandleW)
• OllyDbg (window title "OLLYDBG" via FindWindowW)
• WinDbg, Immunity, Zeta, Rock, Obsidian debuggers (same technique)
• Kernel debugger (reads KdDebuggerEnabled flag at 7FFE02D4)
• Process names (ProcMon, ProcExp, IDA, WinDbg via CreateToolhelp32Snapshot loop)
• VMware (registry key HARDWARE\DEVICEMAP\Scsi\Scsi Port 0 checks for "VMware" identifier)

The Detection Function (402BD6)

This function returns 1 if any toolkit is detected, 0 otherwise. The caller at 401263:

call sub_402BD6
test eax, eax
jne loc_4013BE          ; jumps to ExitProcess if detected

Bypass Strategy

Multiple options:

1. Modify EAX after the call — set to 0 manually in registers panel
2. Patch JNE at 401265 to NOPs — never jump to ExitProcess
3. Use ScyllaHide — cloaks many of these checks automatically

I also had to patch a secondary check at 40123F that looked for dwmapi.dll (present when running patched code for some reason).

---

Exercise 9 & 10: SEH Abuse and Unpacking windowsxp2.exe

SEH as a Red Herring

windowsxp2.exe sets up a Structured Exception Handler early:

push offset handler_519870
push dword ptr fs:[0]
mov dword ptr fs:[0], esp

Then it deliberately causes an access violation. The "handler" at 519870 isn't handling an error — it's the real entry point of the unpacked code. The exception is intentional misdirection.

Stack Breakpoint Method for Unpacking

Instead of tracing through the SEH maze, I used a stack breakpoint to catch the unpacking near the Original Entry Point (OEP):

1. Execute first two instructions (writes to stack)
2. Set hardware breakpoint on the top of stack (Hardware → Access → Dword)
3. Run (F9) — let it hit the breakpoint 5 times
4. At 51993D: jmp eax — potential OEP jump

Verification

Search for string references in the current region — suddenly dozens of URLs appear, confirming unpacked code.

Dumping and Fixing IAT

1. Step into jmp eax (F7) — lands at 4028E8 (OEP)
2. OllyDumpEx: Plugins → OllyDumpEx → Dump process → save as windowsxp2_dump.exe
3. Scylla: Plugins → Scylla → IAT Autosearch → Get Imports → Fix Dump → select windowsxp2_dump.exe
4. Generates windowsxp2_dump_SCY.exe with reconstructed imports

Confirmation

Ran the dumped file — showed a fake error dialog "Requerido Windows NT Server." Process Hacker confirmed it runs as windowsxp2.exe. BinText revealed far more strings than the packed version.

---

Exercise 11: TLS Callbacks and Advanced Anti-Debug (lansrv.exe)

TLS Callback Discovery

Thread Local Storage (TLS) callbacks run before the main entry point. Malware uses them to execute anti-debug checks early.

pescanner.py lansrv.exe | more

Output showed TLS callback at 401031.

Debugging the Callback

1. In x32dbg: Options → Preferences → Events → enable System Breakpoint
2. Restart (Ctrl+F2) — pauses at system breakpoint before TLS runs
3. Go to 401031 (Ctrl+G) → set breakpoint → Run (F9)

At 4011BA, the unpacked code calls IsDebuggerPresent. But here's the twist: it doesn't act on the result immediately. It stores it for later:

call eax                  ; IsDebuggerPresent
jne short loc_4011C4      ; skip ADD if no debugger
add eax, 0Ah              ; if debugger: 1 + A = B (hex)
loc_4011C4:
mov ds:[ecx], eax         ; store result at 401015

If debugger detected: stores 0xB at 401015. If not: stores 0.

The SEH Swap

Later, at 401289, the code does something weird:

mov gs, ax                ; make GS behave like FS
mov eax, dword ptr gs:[0] ; read SEH chain head
; ... manipulations ...
mov dword ptr ds:[edx], eax  ; overwrite handler with 403FEF

It swaps the SEH handler to point to 403FEF — the malicious handler.

The Division by Zero Trap

At 4012C8:

mov ecx, dword ptr ds:[esi+10]  ; loads value from 401015
; ...
div ecx                           ; divide by ECX

If debugger was detected earlier, ECX = 0xB → division succeeds → execution continues down the wrong path (crash).

If no debugger, ECX = 0 → divide by zero exception → Windows calls the swapped SEH handler at 403FEF → true execution path.

To force the correct path in the debugger, I manually set ECX = 0 before the DIV instruction. The exception fired, and the breakpoint at 403FEF caught the real payload unpacking.

---

Exercise 12: Unpacking yep.exe with pe_unmapper

IAT Analysis

PeStudio showed only these DLLs in the IAT: user32.dll, kernel32.dll, comctl32.dll, shell32.dll. Any LoadLibraryA call loading something else (like msvcrt.dll) suggests dynamic resolution — common in packed malware.

Breakpoint Strategy

SetBPX LoadLibraryA

Ran and hit the breakpoint multiple times. On the 5th hit, loading msvcrt.dll — not in the IAT. This is our signal.

Finding VirtualProtect

After returning from LoadLibraryA, scrolled down 114 instructions to find VirtualProtect. The first parameter pointed to a memory region containing a full PE file (MZ header visible in Dump).

Second VirtualProtect Call

35 instructions below, another VirtualProtect call with:

• Address: 401000
• Protection: 0x20 (PAGE_EXECUTE_READ)

This is making code executable — the payload is about to run.

Caching the Execution

Set a hardware execution breakpoint on 401000. Ran (F9) — hit! The unpacked code is now executing.

Extraction and Fixing

1. Dump from Memory Map: yep-dumped.exe
2. Fix with pe_unmapper:

   pe_unmapper yep-dumped.exe 400000 yep-dumped-fixed.exe

1. Fix IAT with Scylla: IAT Autosearch → Get Imports → Fix Dump → select yep-dumped-fixed.exe
2. Output: yep-dumped-fixed_SCY.exe — fully functional unpacked binary

---

Exercise 13 & 14: Code Injection Forensics (known.exe)

Memory Forensics with Volatility

On a memory dump (known.vmem):

export VOLATILITY_PROFILE=Win10x86
vol.py -f known.vmem malfind -D /tmp > known-malfind.txt

The malfind plugin found injected code in multiple processes, including explorer.exe. The extracted files were 132K each — same size across 21 processes, suggesting identical payload injected everywhere.

Live Extraction with Process Hacker

Infected the system with known.exe, then:

1. Opened Process Hacker — two explorer.exe processes appeared
2. The child process (no children of its own) was the hollowed one
3. Properties → Memory tab → sort by Protection → look for RWX regions
4. Found a 132K RWX region — matches the Volatility finding
5. Double-clicked to view contents — strings included .onion domains
6. Right-click → Save → explorer-dumped.exe

Fixing the Dump

pe_unmapper explorer-dumped.exe 0x3560000 explorer-dumped-fixed.exe

PeStudio now showed complete section details, IAT, and strings — ready for deeper static analysis in IDA.

Catching Injection Live with x32dbg

For real-time analysis of the injection:

1. Instance 1 (known.exe): Breakpoints on CreateProcessA and CreateProcessW
2. Ran — hit CreateProcessW launching C:\Windows\SysWOW64\explorer.exe
3. dwCreationFlags = 0x80000004 → CREATE_SUSPENDED confirmed
4. Instance 2 (new x32dbg): File → Attach → select the suspended explorer.exe
5. Options → Preferences → Events → enable Thread Entry
6. Back in Instance 1: Run (F9) — known.exe injects and resumes explorer.exe
7. In Instance 2: Run (F9) — pauses at first thread entry (legitimate DLL thread)
8. Run again — pauses at second thread entry. Title bar shows no DLL name — this is our injected code
9. String references in this region revealed malicious strings, confirming the injection point

---

How to Verify Your Unpacked Payload

Check	Tool	What to Look For
Valid PE structure	PeStudio	Sections, IAT, headers parse correctly
Executable runs	Direct execution	Should show behavior (even if fake error)
Strings exposed	BinText / FLOSS	URLs, registry keys, C2 domains visible
Imports resolved	Scylla / IDA	No "x" marks in Scylla, valid API names
MZ header present	x32dbg Dump	4D 5A at offset 0, DOS stub string
No packer signatures	PEiD / Exeinfo	Shows "Not packed" or compiler name

---

What I Learned

1. Anti-debugging is layered — it's never just IsDebuggerPresent. Modern samples check kernel flags, window titles, loaded DLLs, process lists, registry keys, and timing.

2. Unpacking is about anticipation — set breakpoints on APIs that appear near the end of unpacking (VirtualProtect, WriteProcessMemory, RtlDecompressBuffer, LoadLibraryA for unexpected DLLs).

3. String obfuscation is simple but effective — XOR with single-byte keys, stack strings built at runtime. FLOSS and strdeob.pl automate a lot, but manual IDA inspection still catches what automation misses.

4. Process hollowing follows a pattern — CreateProcess (suspended) → NtUnmapViewOfSection → VirtualAllocEx (RWX) → WriteProcessMemory → resume thread. Recognize the pattern, intercept at WriteProcessMemory.

5. SEH and TLS are misdirection tools — malware uses exception handlers and pre-main callbacks to execute code before you think execution has started. Always check for TLS callbacks with pescanner.py.

6. Sandbox evasion is behavioral — mouse hooks, connectivity checks, sleep loops. These aren't code-level defenses; they're environment-aware defenses. Patch the environment check (or the code) to proceed.

7. Memory forensics complements live debugging — Volatility's malfind finds injected code in dumps; Process Hacker extracts it live. Both approaches validate each other.

---

Common Mistakes

Mistake	Why It Happens	How to Avoid
Forgetting ScyllaHide	Debugger detected immediately, sample terminates before unpacking	Enable ALL first-column options before running
Patching at wrong offset	Changing the comparison instead of the jump	Trace the full logic: where is the result used? Patch the action, not the check
Dumping too early	MZ header not fully formed, incomplete payload	Wait for second VirtualProtect or until code execution begins
Not fixing IAT	Dumped file crashes on launch, missing imports	Always use Scylla/OllyDumpEx to reconstruct imports
Ignoring TLS callbacks	Anti-debug runs before you set breakpoints	Enable System Breakpoint in x32dbg preferences
Forgetting pe_unmapper	Dumped file has wrong base addresses, won't analyze properly	Note the memory base address from Memory Map, pass to pe_unmapper
Single-tool reliance	FLOSS misses stack strings, XORSearch misses non-XOR encoding	Use multiple tools: FLOSS + strdeob.pl + bbcrack + manual IDA

---

Conclusion

Unpacking malware is less about following a rigid recipe and more about recognizing patterns and knowing where to intervene. The samples I worked with each had their own personality: getdown.exe was straightforward with its IsDebuggerPresent check; lansrv.exe was a maze of TLS callbacks and SEH manipulation; raas.exe was paranoid, checking for every tool in the book.

The common thread? They all follow predictable patterns. Once you know what VirtualAllocEx + WriteProcessMemory looks like in assembly, once you know that CREATE_SUSPENDED means process hollowing, once you know that RtlDecompressBuffer signals a packed payload — you know where to set your breakpoints.

The best part is that these techniques transfer. Whether you're looking at a commodity info-stealer or a targeted APT payload, the unpacking patterns are the same. The defenses get more elaborate, but so do our tools and our intuition.

If you're getting into malware analysis, my advice is: build a lab, break things, and document everything. The act of writing down what you did — why you set that breakpoint, what that register value meant — is what turns a one-off success into repeatable skill.