Adapting Ghidra for Reverse Engineering Undocumented Binary Architectures

1. Language Architecture in Ghidra

When Ghidra loads an architecture (such as the MOS 6502), it parses the .ldefs manifest file, which declares metadata and binds three foundational specification pillars:

• The .pspec (Processor Specification):
  Defines the processor’s hardware context. It declares special-purpose registers (e.g., stack pointer SP, status/flags registers), default memory maps (RAM, ROM, I/O), and hardware interrupt vectors.

• The .cspec (Compiler Specification):
  Defines the ABI and calling conventions (e.g., parameter passing mechanisms), stack alignment rules, and return value handling. This is the critical building block enabling the decompiler to reconstruct assembly into readable C code.

• The .sla / .slaspec (SLEIGH Specification):

  • .slaspec: The human-readable source file describing the instruction set architecture (opcodes, instruction formats, and p-code semantics).
  • .sinc (SLEIGH Include): Modular inclusion files (typically used to split complex architectures like ARM or x86, or isolate instruction subsets like Thumb). Given the simplicity of the 6502, everything is defined directly within the .slaspec file.
  • .sla: The compiled binary version of the .slaspec (generated by the Sleigh compiler). Ghidra loads this compiled .sla file into memory at runtime for optimal performance.

---

2. The Challenges of Reverse Engineering Undocumented Binaries

When dealing with a binary compiled for an undocumented processor, Ghidra's default paradigm faces major limitations:

• The .slaspec file is unavailable.
• Ghidra attempts to aggressively disassemble everything.
• Analyzing an undocumented target requires a strict two-phase approach.

---

3. Missing .slaspec File

Without a valid .slaspec definition, Ghidra renders ?? for every opcode. The primary objective when tackling an unknown CPU is precisely to reconstruct this missing .slaspec specification.

---

4. Overcoming Ghidra's Aggressive Disassembly

By default, Ghidra (like most disassemblers) employs an exhaustive strategy (using linear sweep or recursive control-flow traversal). Upon encountering unknown data, instruction set switches, or embedded data structures (data-in-code), it attempts to interpret those bytes as executable code anyway. Consequently, the disassembler loses synchronization, creating cascading "garbage code" that corrupts the entire analysis.

To tackle an unknown architecture, we must invert Ghidra’s core logic: shifting from an "aggressive/exhaustive" mindset to an "opportunistic/conservative" model.

A. Why Prevent Full Disassembly?

On an undocumented architecture, instruction alignment, variable opcode boundaries, and the exact demarcation between data and executable code are initially unknown.

Forcing Ghidra to halt rather than guess provides major technical advantages:

• Prevents Memory Contamination: A single misidentified instruction can corrupt register tracking, stack depth calculations, and Control Flow Graphs (CFG) across thousands of subsequent bytes.
• Isolates "Islands of Certainty": Instead of a 100% flawed disassembly, we isolate distinct, 100% reliable execution blocks (e.g., function prologues/epilogues, branch instructions, or jump tables).
• Enables Guided Analysis: Ghidra's analysis engine operates strictly where we (or our heuristic algorithms) explicitly grant permission.

B. Patching Ghidra Under the Hood

Ghidra is modular, written primarily in Java with its decompilation engine in C++. Adapting its disassembly engine can be achieved across three integration levels:

a. Disabling Auto-Analysis Passes

Ghidra executes background automated analyzers upon loading a binary (e.g., Disassemble Entry Points, Subroutine-Direct Call Analyzer).

• Action: Disable all automatic disassembly analyzers.
• Result: Ghidra loads the binary as raw, unparsed bytes (undefined) without attempting speculative decoding.

b. Overriding DissectedFlow & SLEIGH Rules (Architectural Level)

SLEIGH governs instruction semantics and control-flow parsing. For an unknown architecture, we can construct a strict, minimalist SLEIGH module:

• If a byte sequence fails to match a known pattern with 100% confidence, it defaults to UNDEFINED rather than throwing an exception or attempting partial decoding.
• Unresolved control-flow instructions (unknown JMP/CALL) act as hard terminators, stopping the disassembler from blindly parsing subsequent bytes.

c. Modifying the Java Disassembler Engine (Disassembler.java)

This involves modifying Ghidra's core exploration algorithms (ghidra.app.plugin.core.disass).

• By default, given an entry point, Ghidra recursively queues and processes all jump targets.
• Patch: Enforce traversal depth limits or inject confidence metrics (e.g., based on byte entropy, alignment, or pattern matching). If confidence drops below a threshold, the disassembler halts the branch and flags the location as an "uncertainty node".

---

5. Two-Phase Analysis for Undocumented CISC Processors

The challenge extends beyond disabling the disassembly engine; variable-length (CISC-like) architectures require a two-phase analysis pipeline:

1. Instruction Boundary Detection: Identifying length and boundaries.
2. Opcode Semantic Mapping: Decoding underlying instruction logic.

Ghidra lacks an intermediate representation layer for instruction boundaries; a byte sequence is either a fully formed instruction or nothing at all.

• Step 1: Chunking (Instruction Boundary Detection): Through statistical entropy, alignment, or flow analysis, we determine that bytes 0xFA 0x12 0x88 constitute a single instruction unit, without yet knowing its underlying semantics.
• Step 2: Decoding (Opcode Mapping): Mapping that sequence to an opcode, operands, and p-code semantics (e.g., MOV R1, [R2+8]).

Because Ghidra bridges raw bytes directly to fully decoded SLEIGH instructions, failure at Step 2 (due to an unknown opcode) breaks the pipeline entirely. Ghidra fails to record the boundary discovered in Step 1 and falls back to an undefined byte, triggering cascading misalignment across subsequent variable-length bytes.

Bridging the Abstraction Gap in Ghidra

To introduce this missing "Boundary/Chunk" layer without re-architecting Ghidra's core engine, two surgical workarounds can be applied:

Option A: Dummy / Proxy Instructions (SLEIGH Abstraction)

Define generic parsing rules within the SLEIGH specification:

• Instead of failing on unknown opcodes, SLEIGH instantiates a "Proxy Instruction" matching the precise length identified by your boundary analyzer.
• Ghidra Display Example: UNK_LEN3 0xFA, 0x12, 0x88 (a 3-byte unresolved instruction).
• Benefit: Ghidra treats this chunk as a valid instruction unit, advancing the Program Counter (PC) by exactly 3 bytes. Control Flow Graphs remain intact, memory alignment is preserved, and phase-1 boundaries are stored directly within Ghidra’s database.

Option B: Data Types & Marker Engines (Ghidra Java API)

Instead of instantiating fake instructions, Phase 1 applies custom data structures across byte ranges via the Ghidra API:

• Define a custom low-level Data Type (e.g., InstructionChunk of size N).
• Apply it to byte sequences validated by Phase 1.
• Ghidra locks these bytes as an indivisible unit. As Phase 2 algorithms progressively resolve opcode semantics, scripts replace InstructionChunk instances with fully decoded instructions.

---

6. Conclusion

While Ghidra is natively unequipped for out-of-the-box analysis of undocumented processors, rebuilding a reverse engineering framework from scratch would be a mistake. Leveraging and adapting Ghidra's existing infrastructure to handle unknown architectures is significantly more efficient than reinventing the wheel.

My upcoming articles will detail experimental results, patches, and the final integration solutions chosen for this adaptation.