Creating a disassembler for an undocumented or novel architecture is a notoriously complex challenge. It requires a deep understanding of how fundamental software constructs—such as variable initialization, function calls, control loops, and return statements—are translated into binary patterns across diverse instruction set architectures (ISAs).
Since the number of existing processors, microcontrollers and chips is huge, knowing all the different cases is impossible for a human being.
That's why working on this subject necessitates a two-phase pipeline:
- Filling a pattern matching database
- Using the pattern matching database to guess which series of bytes corresponds to which assembly instructions using a Sudoku-like algorithm
1. Filling a pattern matching database (Semantic P-Code Mapping)
During this offline phase, a massive corpus of C source code is compiled across multiple known architectures (ARM, MIPS, RISC-V, x86, etc.). The resulting binaries are parsed using Ghidra's SLEIGH engine to extract their P-Code intermediate representation. By mapping high-level C constructs (variable initialization, loop counters, frame setups) directly to their corresponding P-Code execution graphs, we build an architecture-agnostic database of semantic patterns.
2. Guessing the bytes
A. Entropy Analysis & Decompression / Decryption Detection
Before attempting disassembly, the baseline entropy of the target binary must be evaluated to detect encryption or compression routines. If an encryption or compression layer is identified, one can exploit the hypothesis that the stub or routine resides within the initial boot instructions (e.g., the first 300 to 500 instructions). Decompression and decryption routines leave distinctive algorithmic signatures; identifying them provides critical semantic clues about memory access patterns and execution flow.
B. Finding the reset vector of the processor.
The simplest way to do it is to start by using already known reset vectors.
C. Constraint Propagation and Hypothesis Scoring
This is where the core analysis takes place. The engine scans the unknown binary to match byte sequences against the P-Code semantic patterns stored in our database. When a match is suspected, the candidate sequence is simulated through Ghidra's P-Code engine to check local structural validity (e.g., control flow, stack alignment, and register bounds).
To evaluate ambiguous paths or edge cases, the Hypothesis Evaluator queries an LLM Service to provide high-level semantic scoring based on software idioms and code structure. If an hypothesis leads to impossible execution states or invalid memory access, it is pruned.
Unlike traditional Sudoku puzzles, where constraints naturally collapse the search space quickly, binary disassembly on unknown ISAs faces a massive combinatorial explosion—making this hybrid P-Code simulation and LLM evaluation loop essential.
3. Conclusion
While reverse-engineering unknown architectures remains a formidable challenge due to state-space explosion, combining P-Code constraint propagation with LLM-driven semantic evaluation provides a structured, scalable approach. Beyond its practical applications in hardware security and legacy system recovery, building such a pipeline offers deep insights into the fundamental mechanics of compilation and instruction set semantics.
Top comments (0)