Most Kafka tutorials in healthcare assume you are starting with a reasonable foundation: EHRs that speak HL7 FHIR natively, internal networks that behave predictably, and vendors who actually respond to API documentation requests. That is not the world most hospitals outside North America and Western Europe operate in. And honestly, it is not even the world many smaller hospitals inside those regions operate in.
I spent the better part of a year building a real-time event architecture across the HIS, LIS, and billing systems of a mid-size hospital in Venezuela. What I learned has less to do with Kafka configuration and more to do with what happens when the textbook version of event-driven architecture meets legacy software, intermittent connectivity, and vendors who consider a CSV export a perfectly adequate integration story.
This post is for the engineers who are building in that gap.
The Actual Problem Is Not the Broker
Every time I see a healthcare streaming post, the interesting part gets skipped. The author sets up Kafka, defines a few topics, maybe sketches a consumer group, and the architecture diagram looks clean. What the diagram never shows is how the data actually got into Kafka in the first place.
In our case, the HIS did not have a webhook. It did not have a REST API. It had a database we were technically allowed to read from, and a vendor who was enthusiastic about helping us as long as we were willing to pay for a custom integration module that would arrive in eight to twelve months.
So we did what you do: we built a change-data-capture layer directly on top of the database using Debezium, pointed it at the tables that represented patient admissions, discharges, and order creation events, and treated every row mutation as a candidate event. This is not elegant. It means your event schema is determined by whatever the HIS vendor decided to name their columns in 2009. But it works, and it works in real time.
The LIS situation was different but equally creative. The lab system could export results to a shared folder over SFTP. So we built a small polling service that watched that folder, parsed the output files, and published structured events to Kafka. Polling is not streaming, but with a 15-second interval and a well-designed idempotency key, it is close enough for the clinical use cases we were supporting.
HL7 FHIR as the Contract, Not the Source
Here is the part I think gets undersold in most interoperability writing. HL7 FHIR should not be the thing you wait for your vendors to support. It should be the contract your internal event layer enforces, regardless of what garbage the source system produces.
What this means in practice is that every event flowing through Kafka, regardless of where it originated, gets normalized into a FHIR resource representation before it reaches any downstream consumer. The HIS row mutation becomes a FHIR Encounter. The lab file row becomes a FHIR DiagnosticReport. The billing record becomes a FHIR Claim.
{
"resourceType": "DiagnosticReport",
"id": "lis-result-00847",
"status": "final",
"subject": {
"reference": "Patient/MRN-10234"
},
"effectiveDateTime": "2024-11-03T14:22:00-04:00",
"result": [
{
"reference": "Observation/glucose-fasting"
}
]
}
The transformation happens in a dedicated service that sits between raw ingestion topics and the canonical topics that downstream consumers actually subscribe to. This means downstream engineers never have to know that the lab system uses a pipe-delimited flat file. They get a FHIR resource. The mess is contained upstream.
This is where the real value of a streaming architecture shows up in constrained environments. You are not replacing your legacy systems. You are building an interoperability layer on top of them that decouples their internal quirks from every future integration you might want to build.
Connectivity Is a First-Class Concern
In an environment with unreliable connectivity, the default Kafka configuration assumptions fall apart quickly. Replication across availability zones is irrelevant when your bigger risk is the network link between the hospital's server room and the wing where the LIS lives going down for 40 minutes on a Tuesday.
We ended up treating the local Kafka cluster as the source of truth and designing producers to be aggressive about local buffering before acknowledging failures. Consumer groups were configured with longer session timeouts than you would ever use in a cloud environment. We also invested heavily in dead-letter topic infrastructure, because the question in our environment was never whether a message would fail to process, but how quickly we could surface and reprocess the failure.
If you are building in an environment with constrained infrastructure, spend more time on your failure handling than on your happy path. The happy path mostly works. The failure path is where the clinical consequences live.
The Gap Nobody Writes About
There is a common story told about hospital digital transformation: the hospital buys a new system, the system goes live, the transformation is complete. What this story skips is the five years of point-to-point integrations that accumulate afterward, each one a one-off script or a vendor connector or a manual export that someone's nephew set up in 2019 and nobody fully understands anymore.
Stream-based architecture does not eliminate that legacy. But it gives you somewhere to route everything toward. Every new system you integrate is another producer or consumer. The chaos gets channeled rather than compounded.
The concrete takeaway from a year of this work is straightforward: the hardest part of real-time healthcare architecture is not Kafka. It is accepting that your sources will be dirty, your network will misbehave, and your vendors will not help you, and then building an architecture that accounts for all three from the start rather than discovering them one by one as production incidents.
Top comments (0)