DEV Community

Artemii Amelin
Artemii Amelin

Posted on

Our QEMU Release Gate Caught a Go 1.27 SIGBUS on MIPS64. Then GitHub Models Returned 410.

Two things happened in the shell.online repo since Friday that are worth writing down, because both are failures most release pipelines never surface. On September 4 a new QEMU gate caught a Go 1.27 runtime bug on MIPS64 before it reached a router. On September 6 the repo's freshly added AI maintainer workflows failed the post-merge smoke test with HTTP 410, because GitHub Models no longer exists.

v0.8.0 shipped 36 binaries, most of them for hardware nobody on the team owns

shell.online wraps a command in a PTY on your machine and hands you a browser link for the live terminal. Until Friday, Windows support was a set of stubs. v0.8.0 replaced them with native ConPTY execution, detached background startup, and working list, attach, and kill, then widened the release matrix to 36 targets across Windows, macOS, Linux, the four BSDs, and Solaris.

The Windows work lives in a handful of _windows.go files. Background sessions start with CREATE_NEW_PROCESS_GROUP | DETACHED_PROCESS. Local control runs over a named pipe whose security descriptor grants access only to SYSTEM, Administrators, and the SID of the user who started the session, so another account on the same box cannot attach to your terminal. State files get a protected DACL instead of Unix mode bits.

Fifteen of the 36 artifacts are Linux: x86, ARMv5 through ARM64, MIPS and MIPS64 in both byte orders, PowerPC64, RISC-V, s390x, and LoongArch64. The static MIPS64 builds are aimed at router hardware such as the Ubiquiti EdgeRouter. Cross-compiling all of that with CGO_ENABLED=0 is cheap. A static binary that builds clean and dies on its first epoll_wait is the worst kind of release.

The QEMU gate, and what it found

v0.8.1 followed the same evening. The change is a shell script in the repo that reads the release manifest, refuses to run unless the QEMU manifest covers every Linux artifact exactly once, and then for each of the 15 targets runs the complete Go test suite under qemu-user via go test -exec, builds the binary, and checks that --version and help render.

The suite includes a real PTY round trip: spawn /bin/sh, resize the terminal to 91 by 37, write a line, and confirm the shell reports the new size and echoes the line back. CI runs the targets in five parallel groups (x86, ARM, MIPS, POWER and mainframe, RISC-V and LoongArch), so a resize bug on s390x fails the build the same way an amd64 bug does.

The gate found a crash on MIPS64: SIGBUS in the Go runtime's netpoll, in code shell.online does not own. That is Go issue 80978, filed August 20, one day after Go 1.27.0 shipped. The EpollEvent struct in internal/runtime/syscall lacks 64-bit alignment on MIPS64, and the reporter hit it in a similar setup, cross-compiled tests run under QEMU user-mode emulation. A change syncing the struct with the kernel header was linked on August 22, the issue closed on August 25 on the Go 1.28 milestone, and as of a comment on September 2 the fix had not reached a 1.27 minor release.

One nuance from that thread: a commenter noted that on native mips64le hardware the Linux kernel emulates misaligned accesses, so the crash may only reproduce under emulation. That is not a reason to ship it. If the release gate cannot pass, nobody can reproduce a field report, and "the kernel probably papers over it" is not a property to rely on for a router binary.

So the release toolchain is pinned. The go.mod now reads go 1.26.8 with a two-line comment naming the issue, and the README says Go 1.27.x must not be used for MIPS64 release binaries. The Homebrew formula was bumped the same night. The README's platform table now separates targets where the full suite ran under emulation from targets that are only build-verified: Windows x86-64 is tested natively, and the Windows x86 and ARM64 binaries are build-verified only. QEMU proves an executable starts on its ISA and exercises networking, crypto, persistence, PTY I/O, and resize. It does not reproduce a vendor kernel or a CPU erratum, and the docs say so.

Then the AI workflows hit a 410

Sunday morning the repo gained three GitHub Actions workflows, for issue triage, pull request review, and release notes, backed by prompt files and a small moderation policy module with its own tests. The PR review job checks out the default branch for the policy code, then fetches the contributor's diff through the API as bounded JSON, so untrusted code is read but never executed. A PR can only be closed and locked as spam or unrelated if two separate model calls, an assessment and an independent verifier, both return confidence of at least 0.98, and the policy returns false outright for owners, members, and collaborators. Flawed but relevant changes get a labelled review. If GitHub denies a GraphQL issue deletion, the fallback is close and lock.

The post-merge smoke test failed with HTTP 410. GitHub Models, the inference API those actions were built on, was retired on July 30 for every customer, including ones with active usage. The second PR of the morning moved all three workflows to the Copilot inference path: Copilot CLI pinned at 1.0.83, actions/ai-inference v3, and the copilot-requests: write permission that since July 2 lets the built-in GITHUB_TOKEN authenticate without a stored PAT. One wrinkle: actionlint 1.7.12 does not know that permission scope yet, so CI carries an ignore for exactly that string and nothing else.

If you built anything on GitHub Models in the spring and have not run it since, check it before it matters.

Toolchain pins are release decisions, and the reason belongs in the file

The same week, our Pilot Protocol repo moved its go.mod the other way, up to 1.25.13, because govulncheck resolves its toolchain from that file and the old pin was scanning against a standard library with five disclosed vulnerabilities reachable through net/http, all fixed in 1.25.13. One repo pinned down to dodge a runtime regression, the other pinned up to clear stdlib advisories. In both cases the version string is doing real work that a reader can check against a public issue.

The same fail-closed posture runs through the Pilot Protocol app catalogue. It is served straight off main, pilotctl verifies catalogue.json.sig against an Ed25519 key compiled into the binary, and a bad signature rejects the whole catalogue rather than one entry. A routine catalogue change on September 1 could not merge until it was re-signed. That is the same idea as a QEMU manifest that refuses to run when it does not cover every Linux target: make the gate loud, and make it impossible to ship around.

Top comments (0)