DEV Community

Cover image for The Butterfly Effect of a Compiler: Hunting an aarch64-only JVM SIGSEGV Down to the Source Line
jvmind
jvmind

Posted on

The Butterfly Effect of a Compiler: Hunting an aarch64-only JVM SIGSEGV Down to the Source Line

#debugging #aarch64 #native #java

A 22-kilobyte truncated ZIP file crashes an entire JVM — but only on aarch64, only with the Maven artifact, and only when built by gcc 4.9.4. A journey from a hs_err log down to the exact source line, through cross-compilation, byte-level reproduction, and the ghost of a ten-year-old compiler.

The Scene

It begins, as these stories often do, with a SIGSEGV that kills the entire Java process. A service running on an aarch64 machine calls into SevenZipJBinding — the popular JNI binding around the 7-Zip/p7zip C++ library — to open a ZIP archive. For a well-formed archive, everything is fine. But feed it a truncated ZIP — one with local file headers but no End Of Central Directory (EOCD) record — and the JVM vanishes in a puff of native code:

# SIGSEGV (0xb) at pc=0x...2038, pid=..., tid=...
# Problematic frame:
# C  [lib7-Zip-JBinding.so+0x102038]
#    Java_net_sf_sevenzipjbinding_SevenZip_nativeOpenArchive+0x9b4
#
# Core dump written. Default location: /work/core or core.5856
Enter fullscreen mode Exit fullscreen mode

Three things make this case genuinely hard:

  1. It only happens on aarch64. The x86_64 artifact is unaffected.
  2. It only happens with the Maven-published artifact. Rebuilding the library from source — even with the same gcc version — does not reproduce it.
  3. The crash address is stable (lib7-Zip-JBinding.so+0x102038), but the binary is stripped, so addr2line returns ??:0.

What follows is the story of how we closed all three gaps — identifying the toolchain, byte-for-byte reproducing the crash, and finally resolving the faulting instruction to a single line of C++.

Step 1 — Reading the Wreckage

The HotSpot fatal error log (hs_err_pid*.log) gives us the crash register state and a fragment of disassembly:

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0

Registers:
R0=0x0053005797794eac    R29=0x00000055039fdb30   ...

Instructions: (pc=0x...2038)
0x...2028: e2 03 15 aa a9 80 ff 97 f4 03 00 aa a0 01 66 9e
0x...2038: 16 04 40 f9 56 23 00 b4 15 08 40 f9 e0 03 15 aa
Enter fullscreen mode Exit fullscreen mode

The crash is a null-deref (si_addr=0x0). Let's decode the faulting instruction. The bytes at 0x...2038 are 16 04 40 f9, which in little-endian is 0xf9400416 — an A64 LDR (immediate, unsigned offset):

ldr x22, [x0, #8]      ; load 8 bytes from address (x0 + 8)
Enter fullscreen mode Exit fullscreen mode

The instruction just before it (a0 01 66 9e = 0x9e6601a0) is:

fmov x0, d13          ; move FP register d13 into GP register x0
Enter fullscreen mode Exit fullscreen mode

So the faulting instruction reads [d13 + 8], and d13 holds an invalid value. This is unusual: a floating-point register is being used to hold a pointer. The compiler (gcc 4.9) has decided to spill a pointer into a callee-saved FP register (d8–d15) to relieve register pressure on the GP file.

Step 2 — Disassembling the Maven Artifact

We pull the .so out of the jar and disassemble around the crash site with a cross-architecture objdump:

$ unzip -p sevenzipjbinding-linux-arm64-16.02-2.01.jar Linux-arm64/lib7-Zip-JBinding.so \
  | objdump ... -d -C --start-address=0x101f00 --stop-address=0x102200
Enter fullscreen mode Exit fullscreen mode 
102014: bl   jni::JMethod::initMethodID
102018: ldr  x21, [x21, #24]
10201c: cbz  x21, 10268c
102020: mov  x0, x27               ; JNIEnv*
102024: mov  x1, x22               ; jclass
102028: mov  x2, x21               ; jmethodID
10202c: bl   JNIEnv_::NewObject    ; --- create Java object ---
102030: mov  x20, x0               ; save return value
102034: fmov x0, d13               ; x0 = d13 (this pointer)        ← ★
102038: ldr  x22, [x0, #8]         ; *** SEGV: read this->field@8 *** ← CRASH
10203c: cbz  x22, 1024a4           ; NULL check
Enter fullscreen mode Exit fullscreen mode

The crash is right after a NewObject call. The code loads a value from d13, then dereferences it. This looks like the textbook pattern of "forgot to check for a pending JNI exception" — and indeed, running with -Xcheck:jni had already warned us:

WARNING in native method: JNI call made without checking exceptions
when required to from CallStaticObjectMethodV
Enter fullscreen mode Exit fullscreen mode

But that tempting narrative turns out to be wrong. Notice the crash dereferences d13, not the NewObject return value (which was saved into x20 and never touched). So this is not a "used a NULL JNI return value" story. We need to know what d13 actually is.

Step 3 — Tracing d13 Back to Its Birth

Disassembling the whole nativeOpenArchive function from its start (0x101684) and grepping for d13 reveals its entire life:

101694: stp  d12, d13, [sp, #128]     ; prologue: save callee-saved d13
...
1017b8: bl   operator new(0x18)        ; new JNIEnvInstance (24 bytes)   ← ★
1017bc: add  x2, x29, #0xd0
1017cc: fmov d13, x2                  ; *** d13 = address of the new object ***
...
101df0: fmov x0, d13                   ; used (still valid here)
101dfc: fmov x0, d13                   ; used (still valid here)
102034: fmov x0, d13                   ; *** used again, now corrupted *** ← CRASH
Enter fullscreen mode Exit fullscreen mode

d13 is assigned exactly once, at 0x1017cc, immediately after a new(0x18). It holds the address of a freshly allocated object — specifically, a JNIEnvInstance, the wrapper the library uses to scope JNI calls within this function. After the assignment, d13 is only read, never written again.

Here's the contradiction. d13 is callee-saved (AAPCS64 reserves the low 64 bits of d8–d15 as callee-saved). By the ABI, every bl we call between 0x1017cc and 0x102034 — including initMethodID and NewObject — must preserve it. Yet by 0x102034 it is garbage. Something along that call chain corrupted a callee-saved register that held a this pointer.

Step 4 — The Toolchain Fingerprint

The artifact is stripped, so addr2line gives us nothing:

$ addr2line -e lib7-Zip-JBinding.so -f -C 0x102038
?? ??:0
Enter fullscreen mode Exit fullscreen mode

We need to rebuild with debug info — but every rebuild we tried (gcc 11, gcc 15, even Linaro gcc 4.9.4) did not crash. The bug refuses to be reborn from source. So our first question became: what exactly built the Maven artifact?

The ELF .comment section answers. It's written by gcc and usually survives stripping:

$ readelf -p .comment lib7-Zip-JBinding.so
String dump of section '.comment':
  [     0]  GCC: (crosstool-NG ) 4.9.4
Enter fullscreen mode Exit fullscreen mode

gcc 4.9.4, via crosstool-NG. gcc 4.9 was the first stable gcc to support aarch64, and its backend was notoriously immature — aggressive about spilling pointers into FP callee-saved registers being exactly the kind of thing it did. Later gcc rewrote the aarch64 backend; that's why every modern rebuild dodges the crash.

Step 5 — Why "Same gcc" Still Doesn't Reproduce

We tried rebuilding with Linaro's prebuilt gcc 4.9.4 aarch64 toolchain. Same gcc version, same -O3. It did not crash. Same compiler, different result — how?

Because crosstool-NG and Linaro ship different binutils, glibc, and patch sets. Code generation for UB is exquisitely sensitive to the exact binary layout, and a slightly different instruction stream shifts the register allocator's decisions. The crash lives in the intersection of "gcc 4.9.4's aarch64 backend" and "the specific crosstool-NG configuration Maven used." Reproducing it required reproducing the toolchain, not just the compiler.

Step 6 — Finding the Exact Toolchain

Where did crosstool-NG 4.9.4 come from? The project's README pointed to DockCross — cross-compilation toolchains shipped as Docker images. Digging into DockCross's git history, we found the smoking gun:

commit 37c54a3  Thu Apr 16 2020
  [linux-arm64] bump up the version of gcc to 8
  linux-arm64 is currently using gcc 4...
Enter fullscreen mode Exit fullscreen mode

And the config just before that commit:

CT_CC_GCC_VERSION="4.9.4"
CT_CC_GCC_V_4_9_4=y
Enter fullscreen mode Exit fullscreen mode

SevenZipJBinding 16.02-2.01 was released in January 2020 — three months before DockCross bumped arm64 to gcc 8. So the release was built with dockcross/linux-arm64 at gcc 4.9.4. The .comment string is an exact match. We had the toolchain.

Step 7 — Byte-for-Byte Reproduction

Old Docker Hub tags get garbage-collected, but the tag dockcross/linux-arm64:20200119-1c10fb2 was still pullable. Building SevenZipJBinding with it produced a .so whose .comment matched, and then — finally — the crash came back:

Maven artifact Our dockcross rebuild
Crashes on bad.zip? Yes Yes
Crash offset +0x102038 +0x102038
Faulting instruction bytes 9e6601a0 f9400416 9e6601a0 f9400416
.comment crosstool-NG 4.9.4 crosstool-NG 4.9.4

Byte-identical at the crash site. We now had a crashing binary whose source we controlled.

Step 8 — The Source Line

One more rebuild, this time adding -g3 -ggdb to the Release flags (keeping -O3 so the codegen didn't shift). The crash offset stayed at 0x102038, and addr2line finally spoke:

$ addr2line -e lib7-Zip-JBinding-dockcross-g.so -f -C -i -p 0x102038

JNIEnvInstance::exceptionCheck()
  at jbinding-cpp/JBindingTools.h:337
  (inlined by) JavaToCPPSevenZip.cpp:291
Enter fullscreen mode Exit fullscreen mode

The source confirms the disassembly story. In JavaToCPPSevenZip.cpp:

// line 290 - create the Java InArchiveImpl object
jobject inArchiveImplObject = jni::InArchiveImpl::_newInstance(env);  // -> NewObject
// line 291 - check whether NewObject threw
if (jniEnvInstance.exceptionCheck()) {                                  // *** CRASH ***
    archive->Close();
    ...
}
Enter fullscreen mode Exit fullscreen mode

And exceptionCheck() is inlined from JBindingTools.h:

bool exceptionCheck() {
    if (_jniNativeCallContext) {        // line 337 - reads this->_jniNativeCallContext
        return _jniNativeCallContext->exceptionCheck(_env);
    }
    return _jbindingSession.exceptionCheck(_env);
}
Enter fullscreen mode Exit fullscreen mode

The member access this->_jniNativeCallContext compiles to ldr x, [this, #8] — and this is d13. Everything lines up. The full causal chain:

  1. A truncated ZIP (no EOCD) is opened; the archive ends up in an inconsistent state.
  2. _newInstance(env) calls NewObject to create the Java InArchiveImpl.
  3. During that call, the JNIEnvInstance::this pointer — held in callee-saved FP register d13 — gets corrupted (the physical-hardware crash gives si_addr=0x24, so d13 became 0x1c).
  4. The inlined exceptionCheck() reads this->_jniNativeCallContext via ldr x22, [d13, #8], dereferences the corrupted this, and faults.

gcc 4.9.4's decision to keep this in d13 across a JNI call is the load-bearing mistake. Every modern gcc uses a GP callee-saved register instead, which is why the bug only shows up in this one binary.

The lesson: "Build with the latest compiler" is not just about new features — for UB-laden legacy code, it can be the difference between a flawless release and a JVM that vanishes on one architecture only. A compiler bug from 2014, frozen into a published artifact, can lie dormant for years until someone hands it the wrong 22 KB of input.

The Fix

For end users, an immediate, 100% effective workaround is to validate the ZIP before handing it to SevenZipJBinding — reject anything without an EOCD record. For the project itself, the clean fix is to rebuild the aarch64 artifact with a modern gcc (≥5); we confirmed that gcc 8 (current DockCross), gcc 11, gcc 15, and even Linaro 4.9.4 all avoid the crash.

Appendix — The Tools That Got Us There

  • hs_err_pid*.log — HotSpot's register dump and the raw faulting bytes were the foundation.
  • readelf -p .comment — the toolchain fingerprint that named gcc 4.9.4.
  • DockCross git history (git log -S "4.9" -- linux-arm64) — pinned the exact image used at release time.
  • dockcross/linux-arm64:20200119-1c10fb2 — the time machine that reproduced the binary.
  • addr2line -g3 — turned bytes into a source line.

Reproduce It Yourself

All scripts used in this investigation — the dockcross reproducer, the addr2line harness, and an A/B verification that proves the fix (gcc 4.9.4 crashes, gcc 12+ does not) — are open source:

🔗 github.com/jvmind/jvm-deep-dives → case-01-sevenzipjbinding-aarch64-sigsegv/

The scripts are Docker-based and run end-to-end on a regular x86_64 machine (they use QEMU user-mode to run the arm64 JVM; see the repo's README for the cross-architecture setup guide). Clone it and:

cd case-01-sevenzipjbinding-aarch64-sigsegv
./verify_fix.sh    # builds both toolchains, runs the crash test on each
Enter fullscreen mode Exit fullscreen mode

Written after a multi-day debugging session. All disassembly, register values, and reproduction steps are verbatim from the actual investigation.

Top comments (0)