CTF Lab Writeup: Binary Instrumentation 3

1. Executive Summary

Challenge Description:

"The executable was designed to write the flag but it seems like I messed up a few things on the way. Can you find a way to get it to work?"

What This Challenge Is Really About:

This challenge is a multi-layered reverse engineering problem that tests your ability to:

• Recognize and extract LZMA-compressed payloads hidden inside a custom PE section
• Understand PE manual mapping (reflective loading) — a technique heavily used in real-world malware
• Pivot from dynamic to static analysis when runtime instrumentation proves impractical
• Recover Base64-encoded flag data embedded as C++ static string literals

Primary Vulnerabilities / Techniques Used:

Technique	Description
LZMA Payload Extraction	Hidden inner PE compressed in .ATOM section
PE Manual Mapper Analysis	Outer binary loads inner PE reflectively
Static String Recovery	Flag parts stored as std::string literals in .rdata
Base64 Decoding	Flag parts concatenated and base64-decoded

The Flag: picoCTF{411_4r3_4p15_n07h1n9_3l53_0cfabbfe}

---

2. Reconnaissance & Static Analysis

2.1 — First Contact: What Are We Dealing With?

Before touching any tool, always ask: "What kind of file is this?" Never assume a file extension tells the truth in a CTF.

file bin-ins.exe

Output:

bin-ins.exe: PE32+ executable (console) x86-64, for MS Windows, 7 sections

What This Tells Us:

• PE32+ — This is a 64-bit Windows Portable Executable
• (console) — It runs in a terminal, not as a GUI application
• x86-64 — We are dealing with 64-bit assembly
• 7 sections — A standard Windows PE has ~5 sections. 7 sections is a hint that something non-standard exists

💡 Beginner Tip: The file command reads the magic bytes at the start of a file (the first few bytes), not the file extension. A file renamed .txt will still be correctly identified as a PE executable by file.

---

2.2 — String Analysis: The Cheapest Recon

Before disassembling anything, always run strings. It costs nothing and sometimes gives you everything.

strings bin-ins.exe | grep -iE "flag|pico|ctf|secret|write"

Output: (nothing)

strings -el bin-ins.exe  # -e l = little-endian 16-bit (UTF-16LE, used by Windows)

Output:

\KnownDlls\
NTDLL.DLL

Why We Tried Both:

Windows applications store strings in two formats:

• ASCII (1 byte per character) — found by strings (default)
• UTF-16LE (2 bytes per character, with null padding) — found by strings -el

The nearly empty results from both string searches are a major signal: the binary either:

1. Is packed/compressed, OR
2. Stores its interesting data in a non-obvious form

This immediately elevated our suspicion that something is hidden inside this binary.

---

2.3 — Import Table Analysis: What Does It Call?

A binary's imported functions reveal its capabilities. We used objdump -p to inspect the Import Address Table (IAT):

objdump -p bin-ins.exe | grep -A 20 "DLL Name"

Key Findings:

DLL Name: KERNEL32.dll
    HeapAlloc, HeapFree, HeapSize, GetProcessHeap
    UnhandledExceptionFilter, SetUnhandledExceptionFilter
    ReleaseSRWLockExclusive, TryAcquireSRWLockExclusive

DLL Name: USER32.dll
    MessageBoxA, MessageBoxW

What's Missing (The Road Not Taken):

A normal Windows application that "writes a flag to disk" would import:

• CreateFileA/W — to open a file
• WriteFile — to write data
• ReadFile — to read data

None of these are imported. This is critical. It tells us:

The outer binary does NOT write the flag itself. It must dynamically resolve these functions at runtime, or delegate to an inner executable that contains the real logic.

This is a hallmark of PE manual mapping — loading a second executable from memory without using the Windows loader, specifically to avoid importing obvious functions.

---

2.4 — Section Analysis: Finding the Anomaly

objdump -h bin-ins.exe

Output:

Idx Name    Size      VMA               File off
  0 .text   000046f0  0000000140001000  00000400
  1 .rdata  0000099c  0000000140006000  00004c00
  2 .data   00000200  0000000140007000  00005600
  3 .pdata  000002ac  0000000140008000  00005800
  4 .rsrc   000001e0  0000000140009000  00005c00
  5 .reloc  00000010  000000014000a000  00005e00
  6 .ATOM   0006fbff  000000014000b000  00006000  ← !!

The .ATOM section is 457,727 bytes (~446KB).

Everything else is tiny in comparison. This unnamed custom section has no standard purpose in a normal PE — it exists purely to hold a payload.

💡 Beginner Tip: Standard PE sections and their purposes:

• .text — executable code
• .rdata — read-only data (strings, constants)
• .data — writable global variables
• .rsrc — resources (icons, dialogs)
• .reloc — relocation table

Any section with a non-standard name like .ATOM is immediately suspicious.

---

2.5 — Running the Binary: Confirm the Bug

wine ./bin-ins.exe

Output (abbreviated):

Unhandled exception: page fault on write access to 0x0000000000400000
...
0x000001400010fe bin-ins+0x10fe: mov %al, (%rcx,%r10,1)

What This Crash Tells Us:

The crash occurs at a byte-copy instruction trying to write to address 0x400000.

Looking at the register state:

• RDI = 0x400000 (destination — the preferred ImageBase of the inner PE)
• RCX = 0 (a NULL pointer — the allocated memory address)
• R10 = 0x400000

The Bug: The outer binary calls VirtualAlloc asking for memory at address 0x400000. In Wine, this fails and returns NULL. The code doesn't check for NULL before writing — it tries to write to address 0x0 + 0x400000 = 0x400000, which is not writable.

Root Cause: The outer binary is a PE manual mapper (reflective loader) that:

1. Reads the inner PE from the .ATOM section
2. Allocates memory at the inner PE's preferred ImageBase
3. Fails silently when VirtualAlloc returns NULL
4. Crashes trying to copy the PE sections

---

3. Payload Extraction: Getting the Inner Binary

3.1 — Identifying the Compression Format

We need to extract whatever is in the .ATOM section. Let's look at its raw bytes:

python3 -c "
data = open('bin-ins.exe', 'rb').read()
atom = data[0x6000:]  # File offset of .ATOM section
print('First 64 bytes:')
for i in range(0, 64, 16):
    hex_str = ' '.join(f'{b:02x}' for b in atom[i:i+16])
    ascii_str = ''.join(chr(b) if 32 <= b < 127 else '.' for b in atom[i:i+16])
    print(f'  {i:04x}: {hex_str}  {ascii_str}')
"

Output:

0000: 5d 00 00 10 00 7f 5b 28 00 00 00 00 ...

Identifying 5d 00 00 10 00 as LZMA:

Magic Bytes	Format
1f 8b	gzip
fd 37 7a 58 5a 00	XZ/LZMA2
50 4b 03 04	ZIP
5d 00 00 10 00	LZMA "Alone" (LZMA1) ← This!

The LZMA Alone format has a specific 13-byte header:

• Byte 0: Properties byte (encodes lc, lp, pb parameters)
• Bytes 1–4: Dictionary size (little-endian 32-bit)
• Bytes 5–12: Uncompressed size (little-endian 64-bit, or 0xFFFFFFFFFFFFFFFF if unknown)

---

3.2 — Decompressing the Payload

python3 - <<'PY'
import lzma

d = open('bin-ins.exe', 'rb').read()[0x6000:]  # Start of .ATOM

# Parse the LZMA-Alone header manually
props = d[0]                                    # Properties byte
dict_size = int.from_bytes(d[1:5], 'little')   # Dictionary size
usize = int.from_bytes(d[5:13], 'little')       # Uncompressed size

# Decode the properties byte:
# props = lc + lp*9 + pb*9*5
lc = props % 9
rest = props // 9
lp = rest % 5
pb = rest // 5

# Use RAW LZMA1 decompressor with explicit filter parameters
dec = lzma.LZMADecompressor(format=lzma.FORMAT_RAW, filters=[{
    'id': lzma.FILTER_LZMA1,
    'dict_size': dict_size,
    'lc': lc,
    'lp': lp,
    'pb': pb
}])

out = dec.decompress(d[13:])  # Skip the 13-byte header
open('atom.dec', 'wb').write(out)
print(f'Decoded {len(out)} bytes. EOF={dec.eof}')
print(out[:16])  # Should start with MZ
PY

Output:

props=0x5d lc=3 lp=0 pb=2 dict=0x100000 usize=2644863
Decoded 2644863 bytes. EOF=True
b'MZ\x90\x00\x03\x00\x00\x00...'

Why We Used FORMAT_RAW Instead of FORMAT_ALONE:

The high-level lzma.decompress(data, format=lzma.FORMAT_ALONE) failed with:

LZMAError: Compressed data ended before the end-of-stream marker was reached

This is because the embedded LZMA stream omits the end-of-stream marker — a common optimization when the uncompressed size is known. By using FORMAT_RAW and supplying the parameters manually from the header, we bypass this requirement.

💡 Beginner Tip: LZMA has multiple container formats:

• LZMA Alone (.lzma) — Simple header + raw LZMA1 stream
• XZ (.xz) — More robust, adds checksums, block structure
• LZMA2 — Improved LZMA used inside XZ

When Python's high-level API fails, always try the raw decompressor with manually parsed parameters.

file atom.dec

atom.dec: PE32+ executable (console) x86-64, for MS Windows, 17 sections

We now have the inner executable with 17 sections — a full MinGW-compiled C++ program.

---

4. Inner Binary Analysis: Understanding the "Messed Up" Logic

4.1 — Running the Inner Binary Directly

wine ./atom.dec

Output:

[+] Let me get started!
[!] I didn't work!

Progress! The inner binary runs without crashing. The outer binary's only job was to load this inner PE — by running it directly, we bypassed the broken manual mapper entirely.

---

4.2 — String Analysis on the Inner Binary

Now that we have the real executable, string analysis is far more productive:

strings atom.dec | grep -iE "work|flag|fail|error|start|success|wrong|check|pico|ctf"

Key Findings:

[+] Let me get started!
C:\random\output_flag.txt       ← The flag output path
[!] Failed to open output file.
cmd.exe /c echo testing if redirection works  ← Prerequisite check
[!] Failed
[!] I didn't work!
[+] I think I worked!           ← The success message we want to reach
_ZL9flagParts                   ← C++ mangled name: "flagParts" static array!

Decoding _ZL9flagParts:

This is a C++ mangled symbol name. The mangling convention:

• _Z — C++ mangled name prefix
• L — local linkage (the variable is static)
• 9 — the name is 9 characters long
• flagParts — the actual variable name

So _ZL9flagParts = static flagParts — a static array that holds the flag pieces!

---

4.3 — The Critical Bug: Creating the Missing Directory

The string C:\random\output_flag.txt tells us the binary tries to write to a directory that doesn't exist in Wine. Let's fix that:

mkdir -p ~/.wine/drive_c/random && wine ./atom.dec

Output:

[+] Let me get started!
[!] I didn't work!

Still failing. The directory exists now, so something else is wrong.

---

4.4 — Disassembling main: Finding All the Bugs

This is where deep static analysis begins. We located main via the symbol table:

objdump -t atom.dec | grep " main"
# main is at VMA 0x401560

objdump -Mintel -d atom.dec --start-address=0x401560 --stop-address=0x401800

Annotated Program Flow:

; Step 1: Print "[+] Let me get started!"
lea rdx, [0x4b3001]         ; Points to "[+] Let me get started!"
call cout<<

; Step 2: Open C:\random\output_flag.txt
lea rcx, [0x4b3019]         ; "C:\random\output_flag.txt"
mov edx, 0x40000000         ; GENERIC_WRITE access
mov r8d, 0x1                ; FILE_SHARE_READ
mov DWORD [rsp+0x28], 0x80  ; FILE_ATTRIBUTE_NORMAL
mov DWORD [rsp+0x20], 0x2   ; CREATE_ALWAYS
call [__imp_CreateFileA]
mov [rbp+0x100], rax        ; Save file handle

; Step 3: Check if CreateFileA failed (INVALID_HANDLE_VALUE = -1)
cmp QWORD [rbp+0x100], -1
jne success_path
; → Print "[!] Failed to open output file."

; Step 4: BUG! Compare file handle to 0xa (decimal 10)
mov QWORD [rbp+0x88], 0xa   ; Store the value 10
...
cmp QWORD [rbp+0x100], [rbp+0x88]  ; Is file handle == 10?
je continue_to_flag_writing
; → File handle is NOT 10, so print "[!] I didn't work!"

The Second Bug — The Intentional Sabotage:

After CreateFileA succeeds (the directory exists), the code compares the returned file handle value to 0xa (decimal 10).

Windows file handles are opaque integers assigned by the OS. There is no guarantee that the handle will be exactly 10. This comparison is the "messed up thing" — it's checking a meaningless condition that will almost never be true.

What the success path looks like:

• If handle == 10: Iterate flagParts, concatenate parts, run cmd.exe /c echo <flag> > output.txt
• Otherwise: Print [!] I didn't work!

---

4.5 — The Road Not Taken: Why We Didn't Try to Patch the Binary

At this point, we could have:

1. Patched the binary — Change cmp rax, 0xa to cmp rax, rax (always equal), or NOP the branch
2. Used Frida — Hook the comparison and force it to take the success path
3. Used GDB/x64dbg — Set a breakpoint, manually change registers

We didn't, because:

Frida on Wine had significant attachment issues (the process exits too fast, Wine path parsing conflicts). Binary patching would work but is slow and risky. Instead, we pivoted to pure static analysis — reading the flag directly from the binary without executing it.

💡 Key Insight: Dynamic analysis is not always the right tool. When a process is too fast, too protected, or too environment-dependent, static analysis of what the code would do is often faster and more reliable.

---

5. Flag Recovery: Static Extraction

5.1 — Tracing flagParts to Its Source

We found _ZL9flagParts is initialized in a C++ static initializer function called before main:

objdump -t atom.dec | grep "GLOBAL__sub_I_main"
# Located at 0x401a87

objdump -Mintel -d atom.dec --start-address=0x401a87 --stop-address=0x401c7f

The initializer does this 5 times:

lea rdx, [rip + 0xb15d8]    ; → 0x4b30ca (pointer to string literal in .rdata)
mov rcx, rsi                 ; rsi = &flagParts[i]
call std::string::string(const char*, const allocator&)
add rsi, 0x20                ; Move to next element (sizeof std::string = 32 bytes)

What We Learned:

• flagParts is an array of 5 std::string objects, each 32 bytes in size
• Each std::string is initialized from a string literal in the .rdata section
• The literal addresses are: 0x4b30ca, 0x4b30d7, 0x4b30e4, 0x4b30f1, 0x4b30fe

Additionally, main at 0x40187e uses a prefix string at 0x4b30a3 before the flag parts.

---

5.2 — Reading the Strings Directly from .rdata

python3 -c "
data = open('atom.dec', 'rb').read()
rdata_vma  = 0x4b3000
rdata_foff = 0xb1e00

# All interesting string addresses from our disassembly
addrs = [0x4b30a3, 0x4b30b4, 0x4b30ca, 0x4b30d7, 0x4b30e4, 0x4b30f1, 0x4b30fe]

for a in addrs:
    # Convert VMA to file offset: file_offset = VMA - section_VMA + section_file_offset
    foff = a - rdata_vma + rdata_foff
    end = foff
    while end < len(data) and data[end] != 0:
        end += 1
    s = data[foff:end].decode('ascii', errors='replace')
    print(f'0x{a:x}: \"{s}\"')
"

Output:

0x4b30a3: "cmd.exe /c echo "
0x4b30b4: "[+] I think I worked!"
0x4b30ca: "cGljb0NURns0"
0x4b30d7: "MTFfNHIzXzRw"
0x4b30e4: "MTVfbjA3aDFu"
0x4b30f1: "OV8zbDUzXzBj"
0x4b30fe: "ZmFiYmZlfQo="

Understanding the VMA-to-File-Offset Conversion:

This is a fundamental PE concept:

File Offset = (Virtual Address) - (Section VMA) + (Section File Offset)

For 0x4b30ca in .rdata:
  File Offset = 0x4b30ca - 0x4b3000 + 0xb1e00
              = 0x00ca   + 0xb1e00
              = 0xb1eca

The program would have run: cmd.exe /c echo cGljb0NURns0MTFfNHIzXzRwMTVfbjA3aDFuOV8zbDUzXzBjZmFiYmZlfQo= >> C:\random\output_flag.txt

---

5.3 — Decoding the Flag

The 5 flag parts are clearly Base64 encoded (alphanumeric characters + = padding):

echo "cGljb0NURns0MTFfNHIzXzRwMTVfbjA3aDFuOV8zbDUzXzBjZmFiYmZlfQo=" | base64 -d

Output:

picoCTF{411_4r3_4p15_n07h1n9_3l53_0cfabbfe}

💡 Beginner Tip: The = at the end of a string is a Base64 padding character. Base64 encodes 3 bytes into 4 characters. If the input isn't a multiple of 3 bytes, = signs are added as padding. Seeing = at the end of a string is one of the strongest indicators that something is Base64 encoded.

---

6. Lessons Learned & Mitigation

6.1 — Attack Chain Summary

bin-ins.exe (outer PE)
    └── .ATOM section (457KB)
            └── LZMA-compressed payload
                    └── atom.dec (inner PE)
                            └── flagParts[] in .rdata
                                    └── Base64 strings
                                            └── picoCTF{...}

6.2 — Key Lessons for CTF Players

Lesson	Detail
Section names are hints	Non-standard sections like .ATOM almost always hide payloads
Empty strings = hidden data	No readable strings → packing/compression/encryption
Missing imports = dynamic resolution	No WriteFile in IAT → manual API resolution or inner loader
Dynamic ≠ always better	Static analysis recovered the flag faster than any runtime hook
Know your compression signatures	5d 00 00 10 00 = LZMA Alone. Memorize common magic bytes
Read the symbols	Mangled C++ names like _ZL9flagParts are goldmines

6.3 — Blue Team Mitigations

If you were a defender trying to detect this technique:

1. Detection — Anomalous PE Sections:

   • Monitor for PE files with non-standard section names (.ATOM, .data2, .pack, etc.)
   • Tools like YARA can rule-match against unusual section entropy (high entropy = compressed/encrypted)
   • Example YARA rule concept:

    rule SuspiciousCustomSection {
        condition:
            pe.number_of_sections > 6 and
            for any section in pe.sections: (
                not (section.name matches /\.(text|rdata|data|rsrc|reloc|pdata|idata)/)
            )
    }
   

2. Detection — Manual PE Mapping:

   • Manual mappers almost always call VirtualAlloc with MEM_COMMIT | MEM_RESERVE followed by byte-by-byte copying
   • EDR solutions (CrowdStrike, SentinelOne) specifically watch for executable memory allocation (VirtualAlloc with PAGE_EXECUTE_READWRITE) not backed by a file on disk — this is a hallmark of reflective loading and process injection

3. Prevention — Application Whitelisting:

   • Use Windows Defender Application Control (WDAC) or AppLocker to prevent unknown executables from running
   • The inner PE (atom.dec) extracted from memory would have no file on disk and no code signing certificate — both are red flags modern EDRs catch

---

Writeup completed. Total techniques demonstrated: PE analysis, section entropy analysis, LZMA decompression, PE manual mapper identification, C++ symbol analysis, VMA-to-file-offset conversion, static string recovery, Base64 decoding.