Building a Decompiler Pipeline in Rust: Why Fission Separates NIR and HIR

Atariboy — Tue, 21 Jul 2026 12:45:58 +0000

Building a Decompiler Pipeline in Rust: Why Fission Separates NIR and HIR

Decompiler output often looks simple from the outside.

A binary goes in. Pseudocode comes out.

But between those two points, a decompiler must recover several different kinds of information:

instruction semantics
register and memory effects
control flow
stack variables
calling conventions
data types
expressions
loops and conditionals
readable source-like structure

Trying to represent all of this in one intermediate representation quickly becomes difficult.

While building Fission, a reverse-engineering and binary decompilation workspace written primarily in Rust, I decided to separate the decompiler pipeline into two main intermediate representations:

NIR, a lower-level representation intended to preserve machine semantics
HIR, a higher-level representation intended to express recovered, human-readable program structure

This article explains why that separation exists, what each representation owns, and why it makes decompiler development easier to reason about.

Correctness and readability want different things

A decompiler has at least two responsibilities.

First, it must preserve the behavior of the original machine code.

Second, it must produce output that a human can understand.

Those goals overlap, but they are not identical.

Consider a simplified fragment of machine-level behavior:

tmp0 = RAX
tmp1 = tmp0 + 1
RAX = tmp1
flags = update_flags(tmp0, 1, tmp1)

A human reader may prefer to see:

rax++;

The concise form is easier to read, but it omits details that may still matter elsewhere in the pipeline.

The flags update could affect a later conditional branch. The operation width may matter. The source and destination could alias. The operation may have originated from an instruction with additional side effects.

If the decompiler converts everything into source-like syntax too early, it becomes easy to discard evidence.

If it keeps everything at machine level until the final rendering stage, the output remains explicit but difficult to understand.

That tension is the main reason Fission separates NIR and HIR.

The Fission pipeline

At a high level, the pipeline looks like this:

Binary bytes
    ↓
Binary loader and static facts
    ↓
Sleigh decoding
    ↓
Raw P-Code
    ↓
Fission NIR
    ↓
Normalization and semantic recovery
    ↓
Fission HIR
    ↓
Control-flow structuring
    ↓
Pseudocode rendering

Fission uses Sleigh language definitions for instruction decoding and P-Code semantics.

The later semantic pipeline, however, is owned by Fission itself.

That includes:

NIR and HIR
normalization passes
type and calling-convention recovery
control-flow analysis
structural recovery
pseudocode rendering
telemetry
regression testing

This ownership boundary is important.

Sleigh describes what an instruction does.

Fission decides how that behavior is represented, analyzed, transformed, structured, and presented.

What NIR is for

NIR is the representation closest to lifted machine semantics.

Its primary responsibility is not readability. Its responsibility is preserving enough information to support reliable analysis and transformation.

NIR should make it possible to reason about:

explicit data flow
register and memory operations
address spaces
operation widths
branch targets
side effects
stack behavior
control-flow edges
lifted instruction provenance

A simplified NIR sequence might look like this:

v0 = load64(stack_pointer + 8)
v1 = int_add(v0, 1)
store64(stack_pointer + 8, v1)
branch_if(v1 < 10, block_4)

This is already easier to analyze than raw P-Code, but it remains explicit.

The representation should not claim more than the analysis has proven.

For example:

store64(stack_pointer + 8, v1)

should not immediately become:

counter = v1;

unless the system has enough evidence that the stack location represents a stable local variable and that counter is an appropriate recovered identity.

NIR is therefore an evidence-preserving layer.

What HIR is for

HIR represents higher-level concepts that are useful for transformation and human-readable output.

It may contain constructs such as:

Assignment
Variable
Call
Return
If
While
Loop
Break
Continue
BinaryExpression
UnaryExpression
Cast

An HIR fragment could look like this:

while (counter < 10) {
    counter = counter + 1;
}

HIR is not merely prettier NIR.

It has a different responsibility.

NIR models behavior in a form suitable for analysis.

HIR models recovered intent and program structure in a form suitable for normalization, structuring, and rendering.

This distinction helps prevent the renderer from becoming responsible for decompiler semantics.

The renderer should consume an already meaningful structure. It should not be the component that discovers loops, invents variables, repairs malformed conditions, or guesses expression relationships.

Why a single IR becomes problematic

A single representation initially appears simpler.

There are fewer types, fewer conversion steps, and less code.

Over time, however, one IR often accumulates incompatible responsibilities.

A single node may be expected to represent both:

INT_ADD(register_space[0x20], constant(1))

and:

counter++;

These are related, but they do not have the same invariants.

The lower-level operation cares about:

bit width
signedness evidence
address spaces
overflow behavior
exact operands
provenance

The higher-level expression cares about:

variable identity
expression simplification
source-like syntax
type presentation
readability

Combining both levels usually leads to one of two outcomes.

The IR remains too low-level

The renderer becomes full of special cases:

If this store follows this addition,
and both access the same stack location,
and the increment is one,
render it as x++.

This is dangerous because the renderer begins to perform semantic analysis.

The IR becomes too high-level too early

Low-level facts disappear before the analysis is complete.

When output is incorrect, it becomes difficult to determine whether the problem came from:

instruction lifting
NIR normalization
variable recovery
type inference
control-flow structuring
rendering

Separating the layers gives each stage a clearer contract.

Ownership boundaries matter

One of the main design principles in Fission is that a bug should be fixed in the layer that owns the behavior.

For example:

Incorrect binary metadata belongs in the loader or static-analysis layer.
Incorrect instruction semantics belong in the lifting layer.
Broken data flow belongs in NIR analysis or normalization.
Incorrect variable recovery belongs in the semantic-recovery layer.
Broken loop reconstruction belongs in the structuring layer.
Formatting problems belong in the renderer.

This sounds straightforward, but decompiler development makes it very tempting to patch the final output.

Suppose the decompiler prints:

if (x) {
    goto label_1;
}

when the expected structure is:

while (x) {
    ...
}

It may be possible to add a renderer heuristic that recognizes the pattern.

But the actual problem may be that:

the CFG was classified incorrectly
a dominance relation became stale
a back edge was not recognized
a region was collapsed before its proof was complete
a branch polarity was reversed
normalization removed required evidence

A renderer-level patch may improve one binary while hiding a structural defect that affects many others.

The NIR/HIR separation makes these ownership mistakes easier to detect.

Structuring is not formatting

Control-flow structuring deserves its own stage.

A CFG containing basic blocks and edges does not automatically become an if, while, or switch.

The structurer must reason about:

dominators
post-dominators
strongly connected components
loop headers
back edges
exit regions
irreducible control flow
region legality

In Fission, the structuring path is intended to be deterministic and evidence-driven.

A region should only be collapsed when the system has enough evidence that the resulting high-level construct preserves the original behavior.

When the proof is incomplete, explicit fallback output is preferable to deceptively clean pseudocode.

For example:

label_1:
    ...

    if (condition) {
        goto label_1;
    }

This may be less readable than a reconstructed loop.

But it is still better than emitting a clean-looking while statement whose semantics are wrong.

Readability matters.

Incorrect readability is worse than honest low-level output.

Why Rust helps

Rust is not automatically the best language for every decompiler.

It introduces real costs:

longer compile times
complex generic types
ownership friction in graph-heavy transformations
difficult mutation patterns
a learning curve around lifetimes and traits

Still, Rust has been useful for Fission because the system contains many ownership boundaries.

Different parts of the workspace own:

binary metadata
static facts
lifted semantics
NIR transformations
HIR normalization
CFG structures
analysis caches
emulator state
telemetry
user-facing output

Rust makes it harder to blur those boundaries accidentally.

Immutable inputs and explicit transformation outputs make questions like these easier to answer:

Which pass changed this function?
Is this graph analysis still valid after mutation?
Does the renderer own any semantic decisions?
Can two analysis components mutate the same state?
Is this result derived from canonical telemetry or from a parallel counter?

The Rust type system does not prove the decompiler correct.

It can, however, make architectural mistakes more visible.

Determinism is a correctness property

A decompiler should produce the same output for the same binary across separate runs.

This is more subtle than it appears.

An algorithm may accidentally rely on the iteration order of a hash map or hash set.

For example:

let candidate = nodes
    .iter()
    .find(|node| is_valid(node));

If nodes is backed by a randomized hash collection, the selected candidate may differ between processes.

That can alter:

region-collapse order
variable naming
expression ordering
block layout
final pseudocode

For a compiler or decompiler, this is not merely cosmetic.

Nondeterminism makes regressions difficult to reproduce and quality measurements unreliable.

Fission therefore treats deterministic output as an explicit correctness requirement.

Transformations that choose among multiple candidates must use stable ordering or an explicitly defined ranking strategy.

Measuring decompiler quality

A major unresolved question is how to measure whether decompiler output is actually improving.

Readable output is subjective.

A change may make one function prettier while making another less accurate.

Potential quality signals include:

semantic equivalence on controlled fixtures
structural similarity against known source
reduction in unnecessary goto statements
successful loop and conditional recovery
stable variable identity
type-recovery confidence
deterministic output across runs
regression rates across real binaries
preservation of side effects and branch behavior

No single metric is enough.

A practical decompiler quality system probably needs to combine:

mechanical checks
real-binary regression corpora
telemetry
differential testing
human review

The important distinction is between output that merely looks better and output that is structurally and semantically better.

Current status

Fission is still a research-heavy, early-stage project.

Most current work is focused on:

x86 and x86-64 correctness
NIR and HIR normalization
control-flow structuring
deterministic output
readable pseudocode
regression testing against real binaries

The repository also contains experimental work on:

P-Code emulation
deterministic execution recording and replay
taint tracking
concolic path exploration
symbolic reasoning in Rust

Some of these areas remain incomplete.

I prefer to state those limitations clearly rather than present them as production-ready features.

Conclusion

The separation between NIR and HIR is not only an implementation detail.

It is a way to preserve low-level evidence while still allowing the decompiler to recover readable, source-like structure.

NIR asks:

What does the machine appear to do?

HIR asks:

What higher-level structure is supported by the available evidence?

Keeping those questions separate helps make transformations easier to review, bugs easier to locate, and final output less dependent on renderer-specific heuristics.

Fission is open source, and feedback is welcome from people working on:

compiler IR design
reverse engineering
decompilation
control-flow structuring
symbolic execution
Rust architecture

Repository:

https://github.com/sjkim1127/Fission

DEV Community: Atariboy

Building a Decompiler Pipeline in Rust: Why Fission Separates NIR and HIR

Building a Decompiler Pipeline in Rust: Why Fission Separates NIR and HIR

Correctness and readability want different things

The Fission pipeline

What NIR is for

What HIR is for

Why a single IR becomes problematic

The IR remains too low-level

The IR becomes too high-level too early

Ownership boundaries matter

Structuring is not formatting

Why Rust helps

Determinism is a correctness property

Measuring decompiler quality

Current status

Conclusion