When facing a binary for which Ghidra lacks the SLEIGH specifications required to disassemble it, you hit a problem that the vast majority of reverse engineers will never encounter.
Out of an estimated 25,000 reverse engineers in the United States, only about 500 (~2%) work on this specific domain.
The Reality of Undocumented ISAs
A processor without a documented instruction set is never completely exotic. Recreating an Instruction Set Architecture (ISA) from scratch is far too expensive.
Consequently, almost all undocumented processors are derivatives of major existing families:
- Base architectures with stripped-down instructions
- Custom instructions added for dedicated workloads
- Scrambled or non-standard opcodes due to the confidentiality of underlying operations
To master this niche, hands-on experience is everything. CTFs with custom VMs are a good start, but nothing beats generating your own custom processors and sharpening your tools against custom-built challenges.
The Pitfalls of Processor Generation
Building a custom processor generator sounds straightforward, until you run into data extraction issues: retrieving valid opcode/instruction tables for major architectures.
Note on LLMs: For ancient 8-bit architectures with reduced instruction sets, LLMs can extract tables flawlessly (see my previous research on dev.to/ddupard). However, as soon as you target modern, heavy ISAs, LLMs break down.
The Reliable Approach:
The most robust solution consists of extracting, cleaning, and parsing the xxx-dis.c files (such as arm-dis.c) directly from binutils-gdb.
The Custom Processor Toolchain
To generate a custom processor, you need a core (match, mask, instruction) triplet.
Example in ARM 32-bit:
0x04400000 | 0x0c500010 | strb%t%c\t%12-15R, %a
This triplet determines the possible permutations applicable to both opcode and operand bits, as well as the exact search space. Once you possess a clean table, the process becomes a pure pipeline problem:
- Create a Transcoding Table: Map native instructions to your custom ISA (optionally dropping specific instructions).
- Transcode the Target Binary: Process a standard binary compiled for the native ISA through your map.
- Inject Custom Instructions: Insert new logic while recalculating jump, branch, and call offsets.
- Fix Binary Headers: Re-align sections (e.g., within ELF structures).
- Re-assemble & Output: Regenerate a valid, clean custom binary.
You now have a fully functional custom binary ready for analysis practice.
Taking It to the Hardware Level (FPGA Obfuscation)
This software approach also maps directly to hardware logic. You can easily derive a Verilog file from your pipeline to deploy an FPGA dedicated to this new processor.
If your goal is to make analysis virtually impossible, you can add layers of defense:
- Two-Layer Encryption: One static key combined with one dynamic key.
- LUT Geometry Alteration: Tweak Lookup Table geometry to break standard ASIC/FPGA reversing.
- Redundancy & Voting Systems: Implement hardware voting modules.
- Anti-Tamper Physical Actions: Remove JTAG access and blow physical security fuses.
By removing hardware debug interfaces, standard chip-attack techniques fail — leaving the analyst with pure, unassisted combinatorial complexity.
Top comments (0)