Postmortem: How a Bun 1.2 Runtime Bug and Node.js 22 Interop Caused Our API to Return Incorrect JSON for 30 Minutes

On October 17, 2024, our team experienced a 30-minute incident where our production Payment API returned malformed JSON responses to downstream clients, causing widespread payment processing failures for our users. This postmortem details the timeline, root cause, resolution, and preventative measures taken after the incident.

Incident Timeline (All Times UTC)

• 14:45: Deployment of Payment API v2.1.0 to production, upgrading the runtime from Node.js 22.1.0 to Bun 1.2.0 to reduce cold start latency.
• 14:47: First customer reports of payment failures, citing "JSON parse error" in client logs.
• 14:50: On-call engineer confirms API responses are returning invalid JSON: ~12% of responses have missing required fields, 8% have extra unexpected properties.
• 14:55: Incident declared SEV-1, rollback to previous Node.js 22 based deployment initiated.
• 15:10: Rollback complete, API traffic shifted back to Node.js 22.1.0.
• 15:15: All API responses confirmed valid, incident resolved.

Root Cause Analysis

We traced the issue to a regression in Bun 1.2's CommonJS (CJS) to ES Module (ESM) interoperability layer, combined with our use of Node.js 22-specific transpiled utility modules.

Our Payment API uses custom Data Transfer Objects (DTOs) to structure API responses, with each DTO defining a toJSON() method to control serialization. These DTOs were defined in a shared library transpiled with @babel/preset-env targeting Node.js 22, which outputs CJS modules with specific property descriptors for methods to align with Node 22's runtime behavior.

Bun 1.2 introduced a change to how it handles property enumeration for CJS modules transpiled for Node.js 22: it incorrectly marked toJSON() methods as non-enumerable in certain cases, causing JSON.stringify() to skip invoking the custom serialization logic. For DTOs where toJSON() was not invoked, two failure modes occurred:

• Raw object serialization omitted required fields that were only added via the toJSON() method.
• Internal transpilation metadata properties (e.g., __esModule) were included in the serialized output, as they were incorrectly marked as enumerable by Bun 1.2's interop layer.

We confirmed this by testing the shared library's DTOs on both runtimes: Node.js 22 correctly invoked toJSON() for all DTO instances, while Bun 1.2 skipped the method for ~20% of instances, depending on how the DTO was imported.

Resolution

Our immediate fix was rolling back to the previous Node.js 22.1.0 based deployment, which restored correct JSON serialization within 5 minutes of starting the rollback. After the incident, we:

• Filed Bun Issue #12345 detailing the CJS interop regression, which was confirmed by Bun maintainers and patched in Bun 1.2.1.
• Audited all services using Bun runtimes for similar transpiled Node.js 22 module dependencies.

Preventative Measures

To avoid similar incidents in the future, we implemented the following:

• Added mandatory JSON schema validation tests for all API endpoints, run against both Node.js and Bun runtimes during CI/CD pipelines.
• Pinned runtime versions in all deployment manifests, with mandatory 24-hour canary periods for minor runtime version upgrades.
• Deployed real-time response validation sidecars for all production APIs, which alert on malformed JSON responses within 10 seconds of occurrence.
• Deprecated use of Node.js 22-specific transpiled modules in Bun-based services, opting for ESM-native shared libraries instead.

We apologize for the impact this incident caused to our users and partners, and are committed to improving our runtime upgrade testing processes to prevent future occurrences.