This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
We like to think of bugs as clean, isolated errors, a misplaced semicolon, a faulty if statement, or a poorly optimized SQL query. Fix the line of code, merge the PR, smash the bug. Job done.
But veteren developers know that some of the most catastrophic system failures don't stem from "bad code." Instead, they breed in the shadows of architectural complexity, thriving in environments where multiple factors collide to create a perfect storm. This is the story of how a critical Health Information Exchange (HIE) and Central Health Data Repository went completely nuts not because of a single broken function, but because its infrastructure was drowning in its own environment.
The System Landscape
To understand how it all went wrong, you have to look at the machinery under the hood. The system was built on a modern, highly distributed stack designed to handle massive volumes of sensitive medical records:
Orchestration: A 3-node Docker Swarm cluster handling container deployments.
Architecture: A microservices-driven framework.
Core Services: A data transformation and loading middleware, a centralized FHIR Server for healthcare data compliance, and an Elasticsearch cluster utilized for real-time analytics and reporting.
On paper, it was resilient, scalable, and modern. In reality, it became a playground for data anomalies.
The Symptoms
We knew we were in trouble when the system started displaying erratic behavior that couldn't be pinned down to a single service:
Data Inconsistencies: Patient records and clinical events were inexplicably dropping.
The Delta Split: A massive discrepancy emerged between the source of truth (the FHIR Server) and the analytics engine (Elasticsearch), leaving healthcare operators with conflicting data.
Systemic Instability: Severe performance degradation that routinely pushed nodes to their limits and threatened total system downtime.
In this post, we’re going to do a deep dive into how we looked past the code repository, put on our architectural detective hats, and tracked these bugs down to their messy, real-world environment.
Optimistic Concurrency Control
To maintain high throughput in a distributed environment, Elasticsearch avoids heavy database locking by using Optimistic Concurrency Control (OCC). Instead of locking a document during an update, Elasticsearch assigns every document a sequence number (_seq_no) and a primary term (_primary_term). When a client attempts to update or delete a record, it must submit the sequence numbers it originally read. If another process has modified the document in the interim, the numbers won't match, and Elasticsearch will reject the incoming write with an HTTP 409 Conflict status code, leaving it entirely up to the application layer to catch the error and retry.
In a fast-moving microservices architecture, this localized error handling can quickly cascade into full-blown data inconsistencies. When our middleware orchestrated the ingestion pipeline—streaming real-time patient data to the primary FHIR Server while concurrently fanning out analytics packets to Elasticsearch—it implicitly assumed both writes would succeed. Under heavy load, simultaneous updates to the same patient records triggered rapid-fire race conditions. Elasticsearch dutifully did its job, rejecting colliding updates with 409 conflicts. However, because our middleware failed to properly intercept these specific exceptions, re-fetch the latest sequence tokens, and replay the payload, those failed updates were silently dropped on the analytics side.
This behavioral blind spot created a severe "delta split" between our core datastores. The FHIR Server, acting as our transactional source of truth, successfully committed every medical event and reflected the absolute latest state of the data. Meanwhile, Elasticsearch fell further and further behind, stubbornly holding onto outdated document versions. As thousands of concurrent synchronization jobs hit these unhandled OCC rejections during peak operational hours, the discrepancy between the underlying clinical records and our analytics platform grew exponentially, resulting in deep, structural data discrepancies that couldn't be caught by standard code linters or unit tests.
Designing Data-Aware Kafka Topics
To prevent race conditions before they ever hit your datastores, your message broker cannot remain data-agnostic. In our initial setup, treating Kafka topics as simple, undifferentiated pipes meant that sequential updates for a single patient were scattered across different partitions. Because different consumer instances read from different partitions simultaneously, the exact chronological order of a patient's medical history was lost mid-transit. A fresh vitals update could easily be processed by Consumer A milliseconds before Consumer B finished processing an older historical record, causing the system to inadvertently overwrite newer data with stale information.
The fix requires structuring Kafka topics around the domain model—specifically, using patient-centric partitioning. By assigning the unique patient_id as the Kafka message key, Kafka guarantees that every single incoming FHIR bundle or event tied to that specific individual is routed to the exact same partition. Since Kafka strictly preserves chronological ordering within an individual partition, a single consumer thread handles that patient's entire timeline sequentially. This entirely eliminates cross-thread race conditions for the same resource.
By making our messaging layout data-aware, we shift the burden of serialization away from heavy database-level locking and onto Kafka’s inherent partitioning mechanics. Grouping streaming medical data per patient ensures that updates are processed in a predictable, linear fashion. This structural change drastically reduces the high-concurrency collisions that trigger Elasticsearch OCC conflicts in the first place, ensuring that both the FHIR Server and the analytics layer ingest data in the correct, synchronized sequence.
Finding the Breaking Point: The Fallacy of Infinite Scale
Every distributed architecture looks flawlessly scalable on a whiteboard, but in production, every system has its own distinct choke point. In our ecosystem, that choke point was the centralized FHIR Server. While our middleware could ingest and transform records at a breakneck pace, the FHIR Server had strict computational limits on its write throughput. When an overwhelming surge of concurrent medical data hit the pipeline, the FHIR Server simply couldn't keep up, resulting in timed-out requests and dropped connections. This triggered an aggressive retry loop from upstream clients. Instead of recovering, the system entered a death spiral: the avalanche of unthrottled retries led to massive data duplication, deeper inconsistencies, and hardware utilization that went completely through the roof.
The lesson here is that a system is only as fast as its slowest component. To fix this, we stopped guessing and isolated each service for rigorous benchmarking based on our actual 3-node Docker Swarm infrastructure. By stress-testing the middleware, the FHIR Server, and Elasticsearch independently, we mapped out the precise requests-per-second ($req/sec$) threshold that each layer could reliably sustain. We established a foundational rule: the maximum throughput of the entire data pipeline must be governed by the lowest common denominator—the bottleneck service.
With these benchmark metrics in hand, we implemented a robust rate-limiting strategy at the gateway level. By aligning our ingress thresholds with the proven capacity of our slowest service, we introduced strict traffic governance. This not only protected the FHIR Server from being flattened during peak usage hours but also established a fair-share policy for all incoming data clients. Embracing our system’s physical limits allowed us to eliminate the chaotic retry storms and stabilize our hardware footprint for good.
The Operational Nightmare: Network Flapping and Database Split-Brain
Just when we thought we had sorted out our application layer and traffic governance, the underlying infrastructure threw us its biggest curveball: intermittent network partitions. In a 3-node Docker Swarm cluster, a stable network fabric is non-negotiable. When brief network drops or "flapping" occurred, the nodes would momentarily lose sight of each other. This not only fractured our Swarm routing mesh but also triggered a nightmare scenario for our highly available PostgreSQL database cluster: a classic split-brain situation.
Because the nodes could no longer communicate, multiple database instances independently came to the conclusion that the master node was dead. Following standard failover procedures, they both elected themselves as the new master database. When you have two concurrent master databases accepting writes simultaneously in a healthcare repository, you enter a data integrity danger zone. Conflicting modifications to patient medical histories were committed on both sides, creating a tangled web of divergent data states that are nearly impossible to automatically reconcile.
In an ideal, enterprise world, the textbook resolution is geo-redundancy—spreading nodes across multiple availability zones and regions with dedicated, high-speed interconnects. However, operating within our constrained, low-resource environment meant we had to adapt. Rather than relying on fragile automated master elections that easily misfire during temporary network drops, we disabled dynamic multi-master promotions entirely. By configuring a strict, manual failover protocol and enforcing rigid node-majority requirements, we chose a temporary system read-only state over data corruption. In a health repository, ensuring data consistency must always take precedence over forced uptime.
Conclusion: Bugs Are Sometimes a Product of Their Environment
If there is one key takeaway from smashing these systemic anomalies, it’s that bugs do not exist in a vacuum. We could have spent months linting our code, refactoring microservices, and writing thousands of flawless unit tests, yet the system still would have broken. Why? Because the code wasn't the problem—the context was.
Just as biological bugs thrive in damp, untidy environments, digital bugs thrive in the messy friction points where infrastructure, network stability, and data architectural designs collide. When you push a complex health information network into production on restricted resources, the underlying environment behaves like a living organism. A tiny network hiccup cascades into a database split-brain; a minor data race condition blows up into a massive Elasticsearch concurrency failure; an unexpected surge in medical packets turns a reliable FHIR Server into a critical roadblock.
Fixing these issues required us to step away from the IDE and look at the bigger picture. True system resilience isn't just about writing elegant code; it’s about anticipating how your system will breathe, bend, and react under the pressure of its real-world surroundings. By enforcing data-aware messaging boundaries, respecting hard hardware throughput constraints, and building defensive, fail-safe protocols for unpredictable networks, we didn't just fix a set of broken features—we sanitized the environment so the bugs had nowhere left to grow.
Top comments (0)