DEV Community: Budi Widhiyanto

FHIR R4 for AI Engineers: What Actually Lives in a Patient Record

Budi Widhiyanto — Mon, 07 Sep 2026 08:23:16 +0000

When I started building a grounded RAG system over FHIR records, I assumed the hard part would be retrieval and generation. Two years of working with FHIR had made the resource model familiar, but that familiarity turned out to be the easy half. The real question was which parts of the FHIR structure the pipeline had to preserve instead of flattening the record into a document.

A FHIR record is not a pile of prose. It is a graph of typed, coded, cross-referencing objects, where the meaning of any one node depends on nodes it points at. Bringing the document-RAG mental model to it builds something that technically runs and quietly answers the wrong questions.

FHIR R4 patient records contain resources, terminology codes, and references. For an engineer who knows embeddings but has not worked with a patient chart, those details determine the pipeline design.

A Bundle is not a document

The unit we get is a Bundle: a JSON envelope with an entry array. Each entry has a fullUrl and a resource. The resources are not sections of one document. They are independent records that happen to have been shipped together.

{
  "resourceType": "Bundle",
  "entry": [
    { "fullUrl": "urn:uuid:8f2c...", "resource": { "resourceType": "Patient", "id": "8f2c..." } },
    { "fullUrl": "urn:uuid:1a4d...", "resource": { "resourceType": "Observation", "id": "1a4d..." } }
  ]
}

That distinction matters immediately. There is no document order to preserve, no narrative flow to keep intact across a chunk boundary, and no reason to concatenate entries. Each resource stands alone, which is why my ingestion treats one resource as one chunk and stops there.

The types that carry clinical meaning

FHIR R4 defines over 140 resource types. For question answering over a patient record, ten of them do almost all the work:

# src/ingestion/fhir_parser.py
SUPPORTED_TYPES = [
    "Patient", "Condition", "Observation", "MedicationRequest",
    "Encounter", "AllergyIntolerance", "Procedure", "DiagnosticReport",
    "Immunization", "CarePlan",
]

Roughly: Condition is a diagnosis or problem. Observation is a measured value (a lab result, a vital sign, a BMI percentile). MedicationRequest is a prescription. Encounter is a visit. Procedure is something done to the patient. DiagnosticReport groups observations into a report. CarePlan is the plan of care.

In a real Synthea corpus, Observation outnumbers everything else by an order of magnitude. My 78-patient corpus indexed 41,837 observations against 2,791 conditions and 3,416 medication requests. That imbalance is not a quirk of synthetic data. Real records look like that too, and it means a naive top-k retrieval is heavily biased toward lab values unless the pipeline corrects for it.

The types I skip are just as informative: Provenance, Claim, ExplanationOfBenefit, Organization. They are billing, audit, and directory infrastructure. They are enormous, they are numerous, and no clinician asks questions of them.

CodeableConcept, and why `display` is not a label

This is the single data structure I had to internalise. Almost every clinically meaningful field in FHIR is a CodeableConcept: a set of codings from standard terminologies, optionally with free text.

"code": {
  "coding": [{
    "system": "http://snomed.info/sct",
    "code": "44054006",
    "display": "Diabetes mellitus type 2"
  }]
}

Three things are easy to get wrong.

First, coding is an array. The same concept can be coded in SNOMED CT and ICD-10 and a local system simultaneously. Taking coding[0] is a heuristic, not a rule.

Second, display is a convenience string, not an identifier. It is allowed to be absent, allowed to be a local translation, and allowed to disagree with what the code actually means. The code plus its system is the fact. The display is a hint.

Third, some fields are a single CodeableConcept and some are an array of them. This caused a real bug in my system. Condition.code is singular. Encounter.type is plural. My path walker, _first_coding, assumed singular, so its isinstance(value, dict) check failed and every Encounter in the database rendered as Encounter: unspecified. until I fixed it:

# src/ingestion/text_renderer.py
# Several FHIR fields (Encounter.type, AllergyIntolerance.reaction[].
# manifestation) are arrays of CodeableConcept rather than a single
# one; without this the walk yields "unspecified" for every Encounter.
if isinstance(value, list):
    value = next((item for item in value if isinstance(item, dict)), None)
codings = value.get("coding", []) if isinstance(value, dict) else []

Read the spec page for the specific field. There is no general rule.

The terminologies, briefly

Three code systems cover most of what I see, and knowing which is which tells me what a resource is about before I read it:

SNOMED CT for clinical findings, conditions, and procedures. http://snomed.info/sct.
LOINC for lab tests and measurements. http://loinc.org. LOINC 4548-4 is hemoglobin A1c.
RxNorm for medications. http://www.nlm.nih.gov/research/umls/rxnorm.

I flatten these into a queryable JSONB column at ingestion, walking the whole resource recursively rather than looking in specific places, because codings show up nested at several depths:

# src/ingestion/chunker.py
def extract_codes(resource_json: dict) -> list[dict]:
    """Extract all coded values (SNOMED, LOINC, RxNorm) from a resource."""
    codes: list[dict] = []

    def walk(value):
        if isinstance(value, dict):
            coding = value.get("coding")
            if isinstance(coding, list):
                for item in coding:
                    if isinstance(item, dict) and item.get("code"):
                        code = {"system": item.get("system", ""),
                                "code": item["code"],
                                "display": item.get("display", "")}
                        if code not in codes:
                            codes.append(code)
            for child in value.values():
                walk(child)
        elif isinstance(value, list):
            for child in value:
                walk(child)

    walk(resource_json)
    return codes

These codes are a retrieval signal that a text splitter throws away, and they are the difference between matching "diabetes" as a string and matching the concept regardless of how it was spelled.

References, and the `urn:uuid` trap

Resources point at each other through Reference objects, which contain a reference string:

{
  "resourceType": "MedicationRequest",
  "subject": { "reference": "Patient/123" },
  "encounter": { "reference": "Encounter/456" }
}

Inside a Bundle, this is more complicated than it looks. Synthea (and plenty of real systems) writes references as urn:uuid:... pointing at the entry's fullUrl, not as canonical ResourceType/id. If we store the reference string as it appears, the foreign keys point at nothing, and patient-scoped retrieval silently misses records.

The fix is to build a Bundle-local map from fullUrl to canonical id while parsing, then normalise every reference through it:

# src/ingestion/fhir_parser.py
def _reference_map(entries: list[Any]) -> dict[str, str]:
    """Map Bundle fullUrls to canonical FHIR resource IDs."""
    references: dict[str, str] = {}
    for entry in entries:
        if not isinstance(entry, dict):
            continue
        full_url = entry.get("fullUrl")
        resource = entry.get("resource")
        if not isinstance(full_url, str) or not isinstance(resource, dict):
            continue
        resource_type = resource.get("resourceType")
        if not isinstance(resource_type, str):
            continue
        resource_id = _resource_id(resource, resource_type)
        if resource_id:
            references[full_url] = resource_id
    return references

This is also why I parse Bundles as raw JSON instead of running them through a validating FHIR model library. The library discards fullUrl context by the time we have objects, whole-Bundle validation on large Synthea bundles is slow enough to stall ingestion outright, and some emitted fields do not match the installed model version. Strict validation belongs at API boundaries. For bulk ingestion I want tolerant parsing plus explicit checks.

The patient link is not one field

I expected every resource to reference its patient the same way. It does not. Most use subject, some use patient, and Patient itself is its own answer:

# src/ingestion/fhir_parser.py
def resolve_patient_ref(resource, resource_type, reference_map=None):
    if resource_type == "Patient":
        return _resource_id(resource, resource_type)
    # FHIR resources use either subject or patient for their patient link.
    for field in ("subject", "patient"):
        reference = _reference_value(resource.get(field))
        if reference:
            return _normalize_reference(reference, reference_map)
    return None

This handful of lines is essential. Patient scoping is the highest-value filter in the system. Without it, a question about one person can be answered from all 78 people in the corpus.

What Synthea gives us, and where it differs from real data

Synthea generates synthetic patients with realistic disease progression, and it is the right way to develop against FHIR without touching real data. It is also not real data, and the differences affect retrieval quality.

It emits administrative records with clinical resource types. One Medication review due Condition per visit, dozens per patient, all nearly identical. It records social determinants ("Unemployed", "Limited social contact") as active Conditions, which are often the most recent conditions, so ordering a problem list by date buries the actual diagnoses. And its coverage of clinical workflows is uneven: there is no continuous glucose monitor history, no insulin pump model.

None of that makes it useless. It makes it a corpus with known distortions, which is better for development than a corpus with unknown ones.

What this changes about the pipeline

The structure is not overhead to strip before embedding. It is signal we get for free, and every part of it maps to a design decision. Resource type becomes a filter. Codes become a second retrieval signal. References become a graph we can walk for context. Resource ids become citation targets.

The next article in the series makes the chunking argument in full, and shows what happens when we ignore all of that and reach for a recursive character splitter instead.

I work on clinical data systems at a research unit, mostly FHIR R4, HL7 v2, and HAPI FHIR. This series documents an open-source grounded RAG system over FHIR records: github.com/budityw23/fhir-rag.

FHIR R4 for ML Engineers: What Actually Lives in a Patient Record

Budi Widhiyanto — Mon, 07 Sep 2026 06:30:02 +0000

A Bundle is not a document

{
  "resourceType": "Bundle",
  "entry": [
    { "fullUrl": "urn:uuid:8f2c...", "resource": { "resourceType": "Patient", "id": "8f2c..." } },
    { "fullUrl": "urn:uuid:1a4d...", "resource": { "resourceType": "Observation", "id": "1a4d..." } }
  ]
}

The types that carry clinical meaning

FHIR R4 defines over 140 resource types. For question answering over a patient record, ten of them do almost all the work:

# src/ingestion/fhir_parser.py
SUPPORTED_TYPES = [
    "Patient", "Condition", "Observation", "MedicationRequest",
    "Encounter", "AllergyIntolerance", "Procedure", "DiagnosticReport",
    "Immunization", "CarePlan",
]

CodeableConcept, and why `display` is not a label

"code": {
  "coding": [{
    "system": "http://snomed.info/sct",
    "code": "44054006",
    "display": "Diabetes mellitus type 2"
  }]
}

Three things are easy to get wrong.

First, coding is an array. The same concept can be coded in SNOMED CT and ICD-10 and a local system simultaneously. Taking coding[0] is a heuristic, not a rule.

# src/ingestion/text_renderer.py
# Several FHIR fields (Encounter.type, AllergyIntolerance.reaction[].
# manifestation) are arrays of CodeableConcept rather than a single
# one; without this the walk yields "unspecified" for every Encounter.
if isinstance(value, list):
    value = next((item for item in value if isinstance(item, dict)), None)
codings = value.get("coding", []) if isinstance(value, dict) else []

Read the spec page for the specific field. There is no general rule.

The terminologies, briefly

Three code systems cover most of what I see, and knowing which is which tells me what a resource is about before I read it:

SNOMED CT for clinical findings, conditions, and procedures. http://snomed.info/sct.
LOINC for lab tests and measurements. http://loinc.org. LOINC 4548-4 is hemoglobin A1c.
RxNorm for medications. http://www.nlm.nih.gov/research/umls/rxnorm.

I flatten these into a queryable JSONB column at ingestion, walking the whole resource recursively rather than looking in specific places, because codings show up nested at several depths:

# src/ingestion/chunker.py
def extract_codes(resource_json: dict) -> list[dict]:
    """Extract all coded values (SNOMED, LOINC, RxNorm) from a resource."""
    codes: list[dict] = []

    def walk(value):
        if isinstance(value, dict):
            coding = value.get("coding")
            if isinstance(coding, list):
                for item in coding:
                    if isinstance(item, dict) and item.get("code"):
                        code = {"system": item.get("system", ""),
                                "code": item["code"],
                                "display": item.get("display", "")}
                        if code not in codes:
                            codes.append(code)
            for child in value.values():
                walk(child)
        elif isinstance(value, list):
            for child in value:
                walk(child)

    walk(resource_json)
    return codes

These codes are a retrieval signal that a text splitter throws away, and they are the difference between matching "diabetes" as a string and matching the concept regardless of how it was spelled.

References, and the `urn:uuid` trap

Resources point at each other through Reference objects, which contain a reference string:

{
  "resourceType": "MedicationRequest",
  "subject": { "reference": "Patient/123" },
  "encounter": { "reference": "Encounter/456" }
}

The fix is to build a Bundle-local map from fullUrl to canonical id while parsing, then normalise every reference through it:

# src/ingestion/fhir_parser.py
def _reference_map(entries: list[Any]) -> dict[str, str]:
    """Map Bundle fullUrls to canonical FHIR resource IDs."""
    references: dict[str, str] = {}
    for entry in entries:
        if not isinstance(entry, dict):
            continue
        full_url = entry.get("fullUrl")
        resource = entry.get("resource")
        if not isinstance(full_url, str) or not isinstance(resource, dict):
            continue
        resource_type = resource.get("resourceType")
        if not isinstance(resource_type, str):
            continue
        resource_id = _resource_id(resource, resource_type)
        if resource_id:
            references[full_url] = resource_id
    return references

The patient link is not one field

I expected every resource to reference its patient the same way. It does not. Most use subject, some use patient, and Patient itself is its own answer:

# src/ingestion/fhir_parser.py
def resolve_patient_ref(resource, resource_type, reference_map=None):
    if resource_type == "Patient":
        return _resource_id(resource, resource_type)
    # FHIR resources use either subject or patient for their patient link.
    for field in ("subject", "patient"):
        reference = _reference_value(resource.get(field))
        if reference:
            return _normalize_reference(reference, reference_map)
    return None

This handful of lines is essential. Patient scoping is the highest-value filter in the system. Without it, a question about one person can be answered from all 78 people in the corpus.

What Synthea gives us, and where it differs from real data

None of that makes it useless. It makes it a corpus with known distortions, which is better for development than a corpus with unknown ones.

What this changes about the pipeline

The next article in the series makes the chunking argument in full, and shows what happens when we ignore all of that and reach for a recursive character splitter instead.

I work on clinical data systems at a research unit, mostly FHIR R4, HL7 v2, and HAPI FHIR. This series documents an open-source grounded RAG system over FHIR records: github.com/budityw23/fhir-rag.

I Built a RAG System That Answers Questions From FHIR Records and Refuses to Guess

Budi Widhiyanto — Mon, 31 Aug 2026 12:58:41 +0000

Every job listing I saw this year mentioned RAG, and many of them specifically mentioned graph RAG. I work with FHIR R4 and clinical data daily, and one question kept coming back: FHIR already gives you structured, coded, individually-addressable resources for every clinical fact in a patient record, and those resources link to each other through explicit references (an Observation points to its Encounter, a MedicationRequest points to its Medication and Patient). The data is already a graph. If any data was a good fit for retrieval-augmented generation, this should be it.

I decided to build it and find out. The system answers clinical questions over FHIR R4 records, with one hard constraint: every claim in the answer has to point at the resource it came from. When the record cannot support an answer, the system has to say so instead of producing something plausible.

A question like "What is this patient's most recent HbA1c, and how has it trended?" touches a dozen Observation resources, at least one Condition, and the Encounter each result was drawn at. "What insulin is she on?" needs MedicationRequest resources plus enough context to know which ones are still active. Retrieving the right resources was the easier part. Getting the system to refuse when the data is not there was harder.

Why grounding, not just answering

The system could answer questions after the first week. The problem was that wrong answers looked exactly like right ones. A fabricated HbA1c value of 7.2% reads the same as a real one. There is nothing in the sentence that marks it as invented, and the reader has no way to check it without going back to the source record, which is the work the system was supposed to save.

In most RAG demos, a confidently wrong answer is embarrassing. In a clinical context it is the failure mode that matters. The design target became grounding: a machine-checkable link from every claim back to a specific FHIR resource, and an explicit refusal when that link cannot be made.

That constraint shaped every layer below it. The chunk boundary is one resource, one chunk, because a citation needs a stable id. The prompt enforces citation format. The API returns a grounding verdict alongside every answer.

The pipeline

The request path has seven stages:

flowchart TD
    Q[Question] --> Embed
    Embed --> Retrieve
    Retrieve --> Resolve[Follow References]
    Resolve --> Context[Build Context]
    Context --> Generate
    Generate --> Verify[Verify Citations]

The code reads the same way:

# src/api/routes.py
query_embedding = embedder.embed_query(query.question)
primary = await hybrid_search(
    pool,
    query_embedding,
    query_text=query.question,
    patient_ref=query.patient_ref,
    resource_types=query.resource_types,
    top_k=settings.top_k,
)
supplementary = await resolve_references(
    pool, primary, max_hops=settings.max_reference_hops
)
context = build_context(primary, supplementary)
system_prompt = _prompt_environment.get_template("clinical_qa.jinja2").render(
    context=context, question=query.question,
)
llm_response = await llm_client.generate(system_prompt, query.question)
grounded = map_citations(llm_response.content, query.question, primary + supplementary)

Embed the question. Retrieve candidate resources with hybrid search. Follow FHIR references to pull in supporting context. Assemble a citation-ready context block. Render the prompt. Generate. Then verify what came back.

Ingestion is the mirror image. Parse Synthea bundles into individual resources, render each one to human-readable text, embed that text, and store it with its metadata in Postgres.

One resource, one chunk

The common RAG approach is to split documents into overlapping token windows, but a FHIR resource is already a clinically meaningful unit with a stable identifier. Splitting one destroys the citation target. Merging several puts unrelated clinical facts in the same retrievable blob. The right chunk boundary is the one FHIR already defines.

The chunker does almost nothing, and that is intentional:

# src/ingestion/chunker.py
def chunk_resource(parsed: ParsedResource) -> FHIRChunk:
    """Convert a ParsedResource into a FHIRChunk with structured metadata."""
    return FHIRChunk(
        resource_id=parsed.resource_id,
        resource_type=parsed.resource_type,
        patient_ref=parsed.patient_ref or "",
        resource_date=extract_date(parsed.resource_json, parsed.resource_type),
        codes=extract_codes(parsed.resource_json),
        references=list(parsed.references),
        text_content=render_resource_text(parsed.resource_json, parsed.resource_type),
    )

The metadata carried alongside the text is what makes the rest of the system work. patient_ref scopes retrieval to one record. resource_date supports temporal filters. codes holds every SNOMED, LOINC, and RxNorm coding found in the resource. references holds outbound links, which is how the reference resolver walks the graph later.

Postgres does all four jobs

The system needs a vector index, a full-text index, relational filters, and joins. pgvector inside PostgreSQL gives me all four against the same rows. There is no second datastore to keep in sync and no network hop between "find similar" and "filter to this patient".

-- db/init.sql
CREATE TABLE IF NOT EXISTS fhir_chunks (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    resource_id TEXT NOT NULL UNIQUE,
    resource_type TEXT NOT NULL,
    patient_ref TEXT NOT NULL,
    resource_date TIMESTAMPTZ,
    codes JSONB DEFAULT '[]',
    "references" TEXT[] DEFAULT '{}',
    text_content TEXT NOT NULL,
    embedding vector(384) NOT NULL,
    text_search TSVECTOR GENERATED ALWAYS AS (to_tsvector('english', text_content)) STORED,
    created_at TIMESTAMPTZ DEFAULT now()
);

text_search is a generated column so the lexical index can never drift from the text it indexes, and re-ingestion needs no extra write path. Embeddings are 384-dimensional, from all-MiniLM-L6-v2 running on CPU.

Retrieval runs two arms over that table, a vector arm and a full-text arm, and fuses them with weighted Reciprocal Rank Fusion. That query went through the most debugging iterations of any part of the system, and it gets its own article later in this series.

Verification, not trust

The prompt asks for citations in a fixed format:

Rules:
1. Base your answer exclusively on the provided FHIR context.
2. Cite specific resource IDs using [ResourceType/id] format.
3. If the provided context does not contain enough information to answer, say:
   "Insufficient data in the available FHIR resources to answer this question."
4. Never fabricate clinical information, medications, diagnoses, or dates.

The model does not always follow those rules. Every citation the model emits is parsed back out and checked against the set of resources that were actually retrieved:

# src/generation/citation_mapper.py
matches = CITATION_PATTERN.findall(llm_response)
resources = {
    (r.resource_type.lower(), r.resource_id.split("/", 1)[-1].lower()): r
    for r in available_resources
}
...
if not matches or valid_count == 0:
    confidence = "ungrounded"
elif valid_count == len(matches):
    confidence = "grounded"
else:
    confidence = "partially_grounded"

A cited id that was never retrieved does not become a citation. It becomes evidence that the answer is not fully grounded, and the API says so. The UI shows that verdict next to the answer rather than hiding it.

Running it

Postgres with pgvector and the app both come up from one Compose file. Data comes from Synthea, which generates synthetic patient records that are realistic enough to be useful and carry no privacy risk at all.

java -jar synthea-with-dependencies.jar -p 70 --exporter.fhir.export=true
cp output/fhir/*.json data/synthea/
docker compose up -d
docker compose exec app python -m src.ingestion.ingest --data-dir data/synthea/

Then open http://localhost:8000, pick a patient, and ask. One thing from the README worth repeating: do not use Synthea's -m diabetes flag to build a diabetes cohort. -m filters loaded modules, not patients, and it will produce a corpus with no Observation or MedicationRequest resources at all. This cost me an afternoon of debugging before I checked the generated bundles.

What it scores

Against a 78-patient corpus, 52 evaluation questions across two cohorts (adults with type 2 diabetes, and a pediatric patient with the atopic march and no glucose data at all):

Metric	Score
Retrieval recall	0.971
Citation accuracy	0.957
Answer-keyword coverage	0.952
Grounded / partial / ungrounded	45 / 0 / 7
Latency (mean / max)	14.7 s / 27.3 s

Five of the seven ungrounded results are the negative-control questions, where refusing is the correct behaviour. Latency is entirely generation-bound; patient-scoped retrieval itself runs in 5 to 24 ms.

What it still gets wrong

Two known problems.

Near-duplicate resources compete for retrieval slots. Synthea emits one Medication review due Condition per visit, they embed to nearly identical vectors, and at one point twelve of them occupied every Condition slot in the top 25 for a generic "what are the active problems?" question. That is fixed in practice but not fixed properly. Diversity-aware retrieval is still unimplemented.

There is no authentication. Every endpoint trusts the caller-supplied patient_ref. That is fine for a local demo over synthetic data and disqualifying for anything else.

This is the first in a series. The next article covers what actually lives in a FHIR record, written for people who know embeddings but have never opened a patient chart. After that the series gets into the parts that broke: a hybrid search arm that silently matched nothing for weeks, a CTE that hid two indexes, and a re-embed that computed 77,466 vectors and saved none of them.

The full source is on GitHub. If you want to see the design decisions with their evidence attached, docs/notes/fhir-rag-debugging.md is the honest version of this article.

I work on clinical data systems at a research unit, mostly FHIR R4, HL7 v2, and HAPI FHIR. This series documents an open-source grounded RAG system over FHIR records: github.com/budityw23/fhir-rag.

Testing HL7-to-FHIR Pipelines Without a Hospital: Mocking HAPI FHIR with respx

Budi Widhiyanto — Sun, 09 Aug 2026 04:42:58 +0000

You can't spin up a maternity ward in CI. But you can get 274 tests and 90% coverage without one.

Some Background

I've been building an open-source Maternity HL7-to-FHIR Pipeline that converts legacy HL7 v2.5 messages into FHIR R4 resources with Australian and European FHIR profiles. The pipeline has three layers: Mirth Connect for HL7 ingestion, FastAPI for FHIR transformation, and HAPI FHIR Server for persistence.

If you want the full architecture story, the first article covers the end-to-end design, and the second article explains why I split responsibilities between Mirth and FastAPI.

This article is about the part that made me confident enough to publish the whole thing: the tests. Specifically, how to build a reliable test suite for a healthcare integration pipeline when you don't have access to a hospital system, a live HAPI FHIR server, or an MLLP connection.

The Testing Problem in Healthcare Integration

Healthcare integration pipelines are awkward to test. The input is a decades-old wire protocol (MLLP) carrying pipe-delimited messages with positional field semantics. The output is validated FHIR resources persisted to a server that enforces its own schema rules. Between input and output, there's transformation logic full of edge cases: merging blood pressure readings, mapping HL7 gender codes to FHIR valuesets, handling missing fields gracefully.

If you test the whole thing end-to-end, you need Mirth Connect, a HAPI FHIR server, and something pretending to be a hospital. That's a docker compose up before every test run. CI becomes slow. Debugging becomes painful. You get flaky tests because you're now dependent on container startup timing and network behavior.

The alternative is to separate what you're actually testing from what you're testing through. My FastAPI layer doesn't care about MLLP or Mirth. It receives flat JSON payloads, transforms them into FHIR resources, and sends them to HAPI over HTTP. Every one of those steps is testable in isolation if you mock the HTTP boundary to HAPI.

Test Architecture: Two Layers, Clear Boundaries

The test suite has 222 unit tests, 31 integration tests, and 21 end-to-end tests, organized like this:

tests/
|-- unit/                       # 222 tests, no network, no Docker
|   |-- test_patient_transformer.py
|   |-- test_condition_transformer.py
|   |-- test_encounter_transformer.py
|   |-- test_observation_transformer.py   # 54 tests incl. BP panel merging
|   |-- test_ips_composition.py
|   |-- test_consent.py
|   |-- test_eu_transformers.py
|   |-- test_au_profiles.py
|   |-- test_profile_registry.py
|   |-- test_profile_contamination.py
|   |-- test_mirth_channel_contract.py
|   |-- test_validate_resource.py
|   |-- test_errors.py
|   |-- test_logging.py
|   |-- test_middleware.py
|   `-- test_validation.py
|
|-- integration/                # 31 tests, full HTTP round-trip
|   |-- test_patient_endpoint.py
|   |-- test_encounter_endpoint.py
|   |-- test_observation_endpoint.py
|   |-- test_health_endpoint.py
|   |-- test_consent_endpoint.py
|   |-- test_ips_endpoint.py
|   |-- test_validate_endpoint.py
|   `-- test_eu_pipeline.py
|
`-- e2e/                        # 21 tests, live Docker stack required
    |-- test_au_pipeline.py
    `-- test_eu_pipeline.py

Unit tests exercise the transformation logic directly. They call transformer functions with Pydantic input models and assert on the FHIR resource output. No HTTP, no server, no Docker. They run in under 2 seconds.

Integration tests hit the FastAPI endpoints through httpx.AsyncClient with HAPI responses mocked by respx. They verify the full request lifecycle: payload validation, transformation, HAPI client calls, response formatting, and error handling. Still no Docker needed.

End-to-end tests run against a live Docker stack (Mirth + FastAPI + HAPI FHIR) and are skipped in CI. They verify the full pipeline from MLLP message to persisted FHIR resource.

The key insight: unit and integration layers use respx to mock HAPI FHIR, but they test different things.

Why respx (Not unittest.mock)

The FastAPI layer talks to HAPI through httpx, an async HTTP client. That means the natural mocking tool is respx, which intercepts httpx requests at the transport level.

Why not unittest.mock.patch? Because patching httpx.AsyncClient.put gives you a dumb mock that returns whatever you tell it to. You lose the ability to assert on:

The exact URL the client constructed (did it include the right resource type and identifier?)
The request body (is the FHIR resource actually valid JSON with the right structure?)
The HTTP method (was it a PUT for upserts vs POST for creates?)
Whether conditional headers were set correctly

With respx, you define route patterns and response fixtures. The mock behaves like a real HTTP endpoint. If your code sends a request to an unexpected URL or with an unexpected method, the test fails with a clear message about what was actually called.

Here's the pattern I use in the integration tests. HAPI routes are mocked with a helper that sets up all the StructureDefinition and resource endpoints:

import respx
from httpx import ASGITransport, AsyncClient, Response

def _hapi_mock():
    mock = respx.mock(base_url="http://localhost:8080/fhir", assert_all_called=False)

    # Mock StructureDefinition endpoints (profile registration on startup)
    mock.put("/StructureDefinition/au-patient").mock(
        return_value=Response(200, json={"resourceType": "StructureDefinition", "id": "au-patient"})
    )
    # ... other StructureDefinition mocks ...

    # Mock the HAPI FHIR conditional PUT for Patient
    mock.put("/Patient").mock(
        return_value=Response(
            201,
            json={"resourceType": "Patient", "id": "pat-1"},
            headers={"Location": "/Patient/pat-1/_history/1"},
        )
    )
    return mock

async def test_patient_upsert_sends_conditional_put():
    """Verify that patient creation uses conditional PUT with identifier query."""
    with _hapi_mock():
        async with app.router.lifespan_context(app):
            async with AsyncClient(
                transport=ASGITransport(app=app), base_url="http://testserver"
            ) as client:
                response = await client.post(
                    "/fhir/Patient",
                    json={
                        "correlationId": "test-001",
                        "mrn": "1234567",
                        "name": {"family": "TEST", "given": "PATIENT"},
                        "birthDate": "19920315",
                        "gender": "F",
                        "address": {"line": "14 SAMPLE ST", "city": "SYDNEY",
                                    "state": "NSW", "postalCode": "2000"},
                    },
                )
    assert response.status_code == 200
    body = response.json()
    assert body["patientId"] == "pat-1"

The HapiClient.upsert_resource method constructs a conditional PUT with If-None-Exist headers internally. By mocking at the respx transport level, the test verifies the full chain: payload parsing, FHIR resource construction, HTTP method selection, and response formatting. If the transformer produces a malformed resource, the test catches it at the HTTP boundary, not through a vague assertion error.

Testing Transformers: Input Model In, FHIR Resource Out

Each transformer function has the same shape: take a Pydantic input model and a ProfileConfig, return a FHIR resource. This makes them pure functions (aside from configuration), which makes them easy to test.

Here's a simplified example of testing the patient transformer:

from app.models.adt_payload import AdtPayload, NamePayload, AddressPayload
from app.profiles.au_profile import AU_PROFILE
from app.transformers.patient import build_patient

def test_patient_basic_fields():
    """ADT payload maps correctly to FHIR Patient resource."""
    payload = AdtPayload(
        correlationId="test-001",
        messageType="ADT^A01",
        mrn="1234567",
        ihi="8003608166690503",
        name=NamePayload(family="TEST", given="PATIENT", middle="MARY", prefix="MS"),
        birthDate="19920315",
        gender="F",
        address=AddressPayload(
            line="14 SAMPLE ST", city="SYDNEY", state="NSW",
            postalCode="2000", country="AU",
        ),
        phone="0412345678",
    )

    patient = build_patient(payload, AU_PROFILE)

    # Check identifier mapping
    mrn_id = patient.identifier[0]
    assert mrn_id.value == "1234567"
    assert mrn_id.system == "http://hospital.local/mrn"

    # Check HL7 gender code mapped to FHIR valueset
    assert patient.gender == "female"  # HL7 "F" -> FHIR "female"

    # Check name structure
    assert patient.name[0].family == "TEST"
    assert patient.name[0].given == ["PATIENT", "MARY"]
    assert patient.name[0].prefix == ["MS"]

Notice what's not here: no HTTP mocking, no server setup, no async/await. The transformer is a function. The test calls the function. The assertion checks the output. Each test runs in microseconds.

The ProfileConfig parameter is what makes the same transformer work for both AU and EU profiles - it carries profile URLs, terminology systems, and timezone offsets. Tests pass AU_PROFILE or EU_PROFILE directly.

This is where having fhir.resources as a dependency pays off. The build_patient function returns a Patient Pydantic model, not a raw dict. If the function accidentally sets gender to "F" instead of "female", the Pydantic model raises a validation error inside the transformer, before the test even gets to the assertions.

Testing Edge Cases: Where Healthcare Gets Interesting

General-purpose APIs have edge cases. Healthcare APIs have clinically significant edge cases. Here are the ones that taught me the most.

Blood Pressure Panel Merging

In HL7, blood pressure comes as two separate OBX segments: one for systolic (LOINC 8480-6) and one for diastolic (LOINC 8462-4). In FHIR, they should be a single Observation with panel code 85354-9 and two component[] entries.

The tricky part: they're only a panel if they appear as consecutive OBX segments. A systolic reading followed by a body weight followed by a diastolic reading is two separate observations, not a panel.

from app.models.oru_payload import ObservationPayload, OruPayload
from app.profiles.au_profile import AU_PROFILE
from app.transformers.observation import build_observations

def _obs(**overrides) -> ObservationPayload:
    defaults = {"setId": 1, "code": "29463-7", "display": "Body weight",
                "value": 68.5, "unitCode": "kg", "status": "F"}
    defaults.update(overrides)
    return ObservationPayload(**defaults)

def _payload(observations):
    return OruPayload(correlationId="test-004", mrn="1234567", observations=observations)

def test_bp_codes_merged_into_panel():
    """Systolic + diastolic OBX segments merge into one BP panel."""
    payload = _payload([
        _obs(code="8480-6", display="Systolic BP", value=120, unitCode="mm[Hg]"),
        _obs(code="8462-4", display="Diastolic BP", value=80, unitCode="mm[Hg]"),
    ])
    results = build_observations(payload, "Patient/2", None, AU_PROFILE)

    assert len(results) == 1  # One panel, not two observations
    assert results[0].code.coding[0].code == "85354-9"  # BP panel code

def test_mixed_bp_and_simple():
    """BP pair merges; non-BP observations remain individual."""
    payload = _payload([
        _obs(code="8480-6", display="Systolic BP", value=120, unitCode="mm[Hg]"),
        _obs(code="8462-4", display="Diastolic BP", value=80, unitCode="mm[Hg]"),
        _obs(code="29463-7", display="Body weight", value=68.5, unitCode="kg"),
        _obs(code="55283-6", display="Fetal heart rate", value=145, unitCode="/min"),
    ])
    results = build_observations(payload, "Patient/2", "Encounter/4", AU_PROFILE)

    assert len(results) == 3
    codes = [r.code.coding[0].code for r in results]
    assert "85354-9" in codes  # BP panel
    assert "29463-7" in codes  # Body weight
    assert "55283-6" in codes  # Fetal heart rate

def test_orphan_systolic_built_individually():
    """Systolic without diastolic -> individual observation."""
    payload = _payload([
        _obs(code="8480-6", display="Systolic BP", value=120, unitCode="mm[Hg]"),
    ])
    results = build_observations(payload, "Patient/2", None, AU_PROFILE)

    assert len(results) == 1
    assert results[0].code.coding[0].code == "8480-6"  # Not wrapped in panel

Three tests, three different behaviors from the same function. The build_observations function takes a full OruPayload (not a raw list), a patient reference, an optional encounter reference, and a ProfileConfig. The BP merging logic scans all ObservationPayload items for matching systolic + diastolic LOINC codes and merges them into a panel. Orphan readings stay standalone. The test suite covers additional scenarios: orphan diastolic, worst-case status interpretation (one preliminary + one final = panel is preliminary), and abnormal flag propagation.

Mirth Contract Tests: Catching Drift Without Running Mirth

The Mirth JavaScript transformer parses HL7 messages by field position and produces flat JSON for FastAPI. If the field positions in the Mirth transformer and the Pydantic models in FastAPI ever drift apart, the pipeline silently produces wrong FHIR resources.

I test the contract without running Mirth. The test file contains a minimal HL7 v2 parser (about 60 lines of Python) that extracts fields by the same positional rules as the Mirth JavaScript. It reads the actual synthetic .hl7 sample files from disk, builds the flat JSON payload, and validates it against the real Pydantic models:

from app.models.adt_payload import AdtPayload

def test_adt_maps_to_valid_patient_payload() -> None:
    segments = _load("adt_a01_normal_delivery.hl7")
    assert _message_type(segments) == "ADT"

    payload = build_patient_payload(segments, "test-adt")
    model = AdtPayload.model_validate(payload)  # raises if the contract is broken

    assert model.mrn == "1234567"
    assert model.ihi == "8003608166690503"
    assert model.name.family == "TEST"
    assert model.gender == "F"
    assert len(model.diagnoses) == 1
    assert model.diagnoses[0].code == "O80"

The build_patient_payload function mirrors Mirth's field extraction: PID-3.1 for MRN, PID-3.5 for identifier type, PID-5 for name components. If someone changes the Pydantic model to rename a field or make a previously optional field required, this test fails immediately - even though Mirth isn't running.

The contract tests cover all three message types (ADT, ORM, ORU) in both AU and EU formats, HL7 escape sequence decoding (\T\ → &, \S\ → ^), and invalid inputs (missing MRN triggers ValidationError). That's 11 tests total, all running without Docker. What they don't test is Mirth's E4X runtime - its toString() behavior and subcomponent drilling - which requires a live MLLP smoke test.

Profile Cross-Contamination: Zero-Leakage Tests

When the same transformer serves both AU and EU profiles, there's a risk that AU profile URLs leak into EU output, or EU terminology systems appear in AU resources. Per-profile tests won't catch this - each profile passes its own assertions. But the wrong profile URL breaks FHIR validation in the other jurisdiction.

The contamination tests run the same payload through both profiles and assert on the serialized JSON:

AU_MARKERS = [
    "au-patient",
    "au-condition",
    "au-vitalsigns-bloodpressure",
    "hl7.org.au/fhir/CodeSystem/icd-10-am",
    "ns.electronichealth.net.au",
]
EU_MARKERS = [
    "patient-eu",
    "condition-eu-core",
    "fhir/sid/icd-10",
    "fhir.nhs.uk/Id/nhs-number",
]

def test_au_output_contains_only_au_values() -> None:
    out = _all_output(AU_PROFILE)
    for marker in AU_MARKERS:
        assert marker in out, f"expected AU marker missing: {marker}"
    for marker in EU_MARKERS:
        assert marker not in out, f"AU output leaked EU marker: {marker}"

def test_eu_output_contains_only_eu_values() -> None:
    out = _all_output(build_eu_profile("uk"))
    for marker in EU_MARKERS:
        assert marker in out, f"expected EU marker missing: {marker}"
    for marker in ["hl7.org.au", "icd-10-am", "ns.electronichealth.net.au"]:
        assert marker not in out, f"EU output leaked AU marker: {marker}"

The _all_output helper runs every transformer (patient, conditions, observations) and concatenates the serialized FHIR JSON into one string. Then it checks for marker strings from the wrong region. If a refactoring accidentally hardcodes an AU profile URL instead of reading from ProfileConfig, this test catches it.

There's also a parametrized test for EU national identifier systems. UK uses NHS numbers, Netherlands uses BSN, Germany uses KVID-10, Ireland uses PPSN. Each country code must produce the correct identifier system in the output.

Gender Code Mapping

HL7 v2 uses single-character gender codes. FHIR uses full words. The mapping isn't one-to-one:

from app.valuesets.hl7_to_fhir_gender import map_gender

@pytest.mark.parametrize("hl7_code,fhir_code", [
    ("F", "female"),
    ("M", "male"),
    ("O", "other"),
    ("U", "unknown"),
    ("A", "other"),      # Ambiguous -> other
    ("N", "unknown"),    # Not applicable -> unknown
    ("", "unknown"),     # Empty -> unknown
])
def test_gender_mapping(hl7_code, fhir_code):
    """HL7 gender codes map correctly to FHIR Administrative Gender."""
    assert map_gender(hl7_code) == fhir_code

Parametrized tests are perfect for valueset mappings. Seven test cases, one function, zero ambiguity about what the transformer does with each input. If a hospital sends gender code "A" (Ambiguous), we know it maps to FHIR "other", not that it crashes or silently drops the field. The actual codebase tests gender mapping through individual unit tests (test_gender_female, test_gender_male, test_gender_unknown), but @pytest.mark.parametrize would be equally effective for this finite mapping table.

Missing and Empty Fields

Hospital systems are inconsistent about what they include. Some always send a middle name. Some never do. Some send an empty string where others send nothing at all.

from app.models.adt_payload import AdtPayload, NamePayload, AddressPayload
from app.profiles.au_profile import AU_PROFILE
from app.transformers.patient import build_patient
from app.transformers.condition import build_conditions

def _sample_payload(**overrides) -> AdtPayload:
    defaults = {
        "correlationId": "test-uuid-001", "mrn": "1234567",
        "name": NamePayload(family="TEST", given="PATIENT"),
        "birthDate": "19920315", "gender": "F",
        "address": AddressPayload(
            line="14 SAMPLE ST", city="SYDNEY", state="NSW", postalCode="2000"),
    }
    defaults.update(overrides)
    return AdtPayload(**defaults)

def test_patient_without_middle_name():
    """Patient resource builds correctly when middle name is absent."""
    patient = build_patient(_sample_payload(), AU_PROFILE)
    assert patient.name[0].given == ["PATIENT"]  # List with one entry, not crash

def test_patient_with_empty_diagnosis_list():
    """Patient with no diagnoses produces no Conditions."""
    payload = _sample_payload()
    conditions = build_conditions(payload, "Patient/1", AU_PROFILE)
    assert conditions == []

def test_patient_with_no_phone():
    """Missing phone number doesn't set telecom at all."""
    patient = build_patient(_sample_payload(phone=""), AU_PROFILE)
    assert patient.telecom is None  # Not an empty list, not a placeholder

These tests are not clever. They are simple and boring, and that is what I want. The build_patient and build_conditions are separate functions - the endpoint orchestrates both, but tests exercise them independently. Every "what if this field is empty" question has a documented, tested answer.

Integration Tests: The Full HTTP Round-Trip

Unit tests verify the transformation logic. Integration tests verify that the FastAPI endpoints wire everything together correctly: payload validation, transformer calls, HAPI client calls, response formatting.

The integration tests use httpx.AsyncClient with FastAPI's app directly (no server startup needed) and respx to mock HAPI. The pattern uses a context manager for the mock and FastAPI's lifespan context:

import pytest
import respx
from httpx import ASGITransport, AsyncClient, Response
from app.main import app

def _hapi_mock():
    mock = respx.mock(base_url="http://localhost:8080/fhir", assert_all_called=False)
    # Mock StructureDefinition PUTs (FastAPI registers profiles on startup)
    mock.put("/StructureDefinition/au-patient").mock(
        return_value=Response(200, json={"resourceType": "StructureDefinition", "id": "au-patient"})
    )
    # ... other StructureDefinition mocks ...
    mock.put("/Patient").mock(
        return_value=Response(201, json={"resourceType": "Patient", "id": "pat-1"},
                              headers={"Location": "/Patient/pat-1/_history/1"})
    )
    mock.post("/Condition").mock(
        return_value=Response(201, json={"resourceType": "Condition", "id": "cond-1"})
    )
    return mock

class TestPatientEndpoint:
    @pytest.mark.asyncio
    async def test_with_diagnosis_returns_condition_ids(self):
        """POST /fhir/Patient returns patientId, conditionIds, and correlationId."""
        with _hapi_mock():
            async with app.router.lifespan_context(app):
                async with AsyncClient(
                    transport=ASGITransport(app=app), base_url="http://testserver"
                ) as client:
                    response = await client.post(
                        "/fhir/Patient",
                        json={
                            "correlationId": "int-test-002",
                            "mrn": "1234567",
                            "name": {"family": "Smith", "given": "Jane"},
                            "birthDate": "19920315",
                            "gender": "F",
                            "address": {"line": "1 Test St", "city": "Sydney",
                                        "state": "NSW", "postalCode": "2000"},
                            "diagnoses": [{"code": "O80", "display": "Normal delivery"}],
                        },
                    )
        assert response.status_code == 200
        body = response.json()
        assert body["conditionIds"] == ["cond-1"]

This test verifies the entire chain: the endpoint accepts the payload, the transformer builds valid FHIR resources, the HAPI client sends them with the right HTTP methods, and the response is formatted correctly. All without Docker. The lifespan_context ensures FastAPI's startup hooks run (including profile registration with HAPI), which is why the StructureDefinition mocks are needed.

Testing Error Paths

The error tests are where integration tests become important. I want to verify that invalid payloads produce RFC 7807 problem+json responses:

@pytest.mark.asyncio
async def test_empty_mrn_returns_422(self):
    """Empty MRN rejected with problem+json response."""
    async with app.router.lifespan_context(app):
        async with AsyncClient(
            transport=ASGITransport(app=app), base_url="http://testserver"
        ) as client:
            response = await client.post(
                "/fhir/Patient",
                json={
                    "correlationId": "int-test-003",
                    "mrn": "",
                    "name": {"family": "Smith", "given": "Jane"},
                    "birthDate": "19920315",
                    "gender": "F",
                    "address": {"line": "1 Test St", "city": "Sydney",
                                "state": "NSW", "postalCode": "2000"},
                },
            )
    assert response.status_code == 422
    assert response.headers["content-type"] == "application/problem+json"

The unit test layer also exercises the error handling directly:

from app.errors import problem_response

def test_problem_response_status():
    resp = problem_response(422, "Validation Error", "Missing field")
    assert resp.status_code == 422

def test_problem_response_content_type():
    resp = problem_response(422, "Validation Error", "Missing field")
    assert resp.media_type == "application/problem+json"

Without respx, testing HAPI failure paths would require either making HAPI actually reject a resource (which means running HAPI) or patching internals so deep that the test tells you nothing about real behavior.

The CI Pipeline

All of this runs in GitHub Actions on every push:

name: CI
on:
  push:
    branches: [main, master]
  pull_request:
    branches: [main, master]

jobs:
  lint-and-test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        profile_region: [au, eu]
    env:
      PROFILE_REGION: ${{ matrix.profile_region }}
      PROFILE_COUNTRY: ${{ matrix.profile_region == 'eu' && 'uk' || '' }}

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        working-directory: ./fastapi
        run: pip install ".[dev]"

      - name: Lint app
        working-directory: ./fastapi
        run: ruff check app/

      - name: Lint tests
        run: ruff check tests/

      - name: Type check
        working-directory: ./fastapi
        run: mypy app/ --ignore-missing-imports

      - name: Run tests with coverage
        run: |
          python -m pytest tests/ -v \
            --cov=fastapi/app \
            --cov-report=term-missing \
            --cov-fail-under=80

No Docker in CI. No Mirth. No HAPI. The entire test suite runs in under 10 seconds. The matrix strategy runs the full suite twice - once with PROFILE_REGION=au and once with PROFILE_REGION=eu - so both AU and EU profile paths are exercised in CI. The coverage threshold is set at 80% (actual coverage is 90%), so if someone adds a new transformer without tests, CI catches it.

The lint steps run ruff check on both application code and tests (separately, since they live in different directories). The type check step (mypy) catches type errors at the function boundary level. Between the three checks, most bugs are caught before the code ever runs.

What I Don't Test (and Why)

Being honest about what you don't test matters as much as coverage numbers.

Mirth Connect's JavaScript runtime is not tested in CI. However, the contract between Mirth and FastAPI is tested: 11 unit tests in test_mirth_channel_contract.py parse the actual sample HL7 messages using the same field positions as the Mirth JavaScript transformer, build the flat JSON payload, and validate it against the real FastAPI Pydantic models. This covers AU and EU messages, HL7 escape sequences (\T\ → &), and invalid inputs (missing MRN → ValidationError). What's not tested is Mirth's E4X runtime semantics - its toString() behavior, subcomponent drilling, and actual MLLP framing -
which would require running Mirth in Docker. The live MLLP smoke test covers that gap manually.

HAPI FHIR profile validation is not fully tested. The validate_before_persist setting exists in the app config but the unit tests don't exercise HAPI's $validate endpoint against real profiles. The mock returns whatever I tell it to. This means I'm testing that my code sends a valid resource, not that HAPI accepts it. The end-to-end test suite (which runs against a live HAPI instance in Docker) does validate this path, but those tests are skipped in CI.

MLLP protocol handling is not tested. That's Mirth Connect's job, and Mirth has its own test suite. I test the boundary I own: the HTTP interface between Mirth and FastAPI.

Each of these gaps maps to a deliberate architectural decision. The pipeline's layers have clear boundaries specifically so each layer's tests don't need to boot the other layers.

Patterns Worth Stealing

If you're building a healthcare integration and want to set up a similar test suite, here's the minimal recipe:

Use respx for any httpx-based FHIR client. It mocks at the transport level, so your tests exercise the actual URL construction, header setting, and body serialization code. unittest.mock.patch skips all of that.

Make transformers pure functions. If build_patient(payload, profile) returns a FHIR resource without side effects, it's testable without any mocking at all. Push all IO (HTTP calls, file writes, logging) to the edges.

Parametrize valueset mappings. Gender codes, observation status codes, encounter class mappings, all of these have a finite set of valid inputs. A @pytest.mark.parametrize decorator turns the entire mapping table into test cases in one block.

Test the failure modes, not just the happy path. Healthcare data is messy. Missing fields, empty arrays, unexpected codes. Each one deserves a test that documents what the pipeline does rather than crashing with an unhandled KeyError.

Set a coverage floor, not a ceiling. My threshold is 80%. Actual coverage is 90%. The floor catches regressions without encouraging people to write meaningless tests just to hit a number.

Try It

git clone https://github.com/budityw23/maternity-hl7-to-fhir-pipeline.git
cd maternity-hl7-to-fhir-pipeline

# Install dev dependencies
cd fastapi && pip install ".[dev]" && cd ..

# Run AU profile tests
python -m pytest tests/ -v --cov=fastapi/app --cov-report=term-missing

# Run EU profile tests
PROFILE_REGION=eu PROFILE_COUNTRY=uk python -m pytest tests/ -v

No Docker required. The unit + integration suite runs in seconds. E2E tests (tests/e2e/) need the Docker stack running - see docker compose up.

Source code: github.com/budityw23/maternity-hl7-to-fhir-pipeline

This is the fourth article in my series on the Maternity HL7-to-FHIR Pipeline. The first article covers the full architecture, the second explains the Mirth + FastAPI split, and the third shows how we added EU FHIR profile support to the same codebase.

One Pipeline, Two Continents: Adding EU FHIR Profiles to an Australian Healthcare Integration

Budi Widhiyanto — Mon, 03 Aug 2026 16:56:42 +0000

The European Health Data Space (EHDS) regulation requires EHR systems operating in the EU to exchange health data using FHIR. This is driving adoption of HL7 Europe's FHIR Implementation Guides across member states.

I'd just finished building a Maternity HL7-to-FHIR Pipeline targeting the Australian market - AU Base profiles, IHI identifiers, ICD-10-AM diagnosis codes. It worked. It had a solid test suite with good coverage. It was a complete portfolio piece.

Then I started asking myself: could this same pipeline work for a European hospital?

The HL7 v2 messages coming from the hospital are identical - an ADT^A01 admission in Melbourne looks the same as one in Amsterdam. The clinical data is the same - blood pressure is blood pressure, a pregnancy diagnosis is a pregnancy diagnosis. What changes is the FHIR metadata layer: which profiles to declare, which identifier systems to use, which terminology editions to bind.

If the architecture was good, this should be a configuration change - not a rewrite. If it wasn't, well, that would be a good lesson too.

GitHub: budityw23/maternity-hl7-to-fhir-pipeline

What's Actually Different Between AU and EU FHIR?

Before touching any code, I needed to understand what "European FHIR" means. The EU ecosystem is more complex than Australia's, and more interesting.

The Profiling Layer

Australia has a relatively flat structure: AU Base profiles sit directly on top of base FHIR R4. You declare http://hl7.org.au/fhir/StructureDefinition/au-patient on your Patient resource and you're done.

Europe has a layered architecture. HL7 Europe publishes three foundational Implementation Guides:

HL7 Europe Base - flexible foundation profiles
HL7 Europe Core - essential constraints on top of Base
HL7 Europe Extensions - EU-specific extensions (nationality, citizenship, etc.)

On top of that sit scoped IGs - domain-specific guides like the European Patient Summary. And below all of that, individual countries can add their own national constraints (NHS England, Nictiz Netherlands, KBV Germany).

HL7 EU Extensions
       ↓
HL7 EU Base Profiles (flexible foundation)
       ↓
HL7 EU Core Profiles (essential constraints)
       ↓
Scoped HL7 EU IGs (e.g. European Patient Summary)
       ↓
National IGs (e.g. NHS England, Nictiz NL, KBV DE)

This means the EU pipeline needs to operate at the EU Base/Core level - general enough to work across countries, specific enough to satisfy EHDS requirements.

Identifiers

In Australia, every patient has an IHI (Individual Healthcare Identifier) - a single national identifier system with one URI: http://ns.electronichealth.net.au/id/hi/ihi/1.0.

Europe doesn't have a single identifier. Each country has its own:

Country	Identifier	System URI
Australia	IHI	`http://ns.electronichealth.net.au/id/hi/ihi/1.0`
UK	NHS Number	`https://fhir.nhs.uk/Id/nhs-number`
Netherlands	BSN	`http://fhir.nl/fhir/NamingSystem/bsn`
Germany	KVNR	`http://fhir.de/sid/gkv/kvid-10`
Ireland	PPS Number	`https://fhir.ie/sid/ppsn`

(These are the exact system URIs used in fastapi/app/profiles/eu_profile.py.)

The pipeline needs a configurable national identifier, not a hardcoded one.

Terminology

Australian maternity care uses ICD-10-AM (Australian Modification) for diagnosis coding. The system URI is http://hl7.org.au/fhir/CodeSystem/icd-10-am.

European systems use standard ICD-10 (WHO edition) with system URI http://hl7.org/fhir/sid/icd-10. Some countries use their own modifications (Germany has ICD-10-GM), but at the EU Base level, the WHO edition is the common ground.

SNOMED CT is used in both regions, but Australia binds to the AU refset while Europe uses the International Edition. The URI is the same (http://snomed.info/sct) - the difference is in which value sets and reference sets are expected.

LOINC and UCUM are universal. Blood pressure is still 85354-9. Weight is still 29463-7. Millimeters of mercury are still mm[Hg]. Some things, thankfully, don't need localization.

International Patient Summary

The IPS (International Patient Summary, ISO 27269) is the cross-border health document that EHDS uses for data exchange. When a Portuguese patient walks into a German emergency room, the IPS tells the German doctor what they need to know: allergies, medications, active conditions, recent vital signs, pregnancy status.

My pipeline already produces Patient, Condition, Observation, and Encounter resources. Wrapping those into an IPS Composition document is a natural extension.

GDPR

Australian healthcare has privacy regulations, but nothing that requires FHIR-level consent tracking. In the EU, GDPR Articles 6 and 9 create specific legal bases for processing health data. Modeling this as a FHIR Consent resource (recording which GDPR article authorizes the processing) isn't required by any FHIR IG, but European employers care about it.

My First Instinct (And Why It Was Wrong)

My initial approach was the obvious one: fork the repo, create a separate maternity-hl7-to-fhir-pipeline-eu project, and replace all the AU-specific values with EU ones.

Find and replace. au-patient → Patient-eu. IHI → NHS Number. ICD-10-AM → ICD-10. Ship it.

This would have worked for a demo. But it's the wrong architecture for several reasons:

Duplicated transformation logic. The mapping from PID-5.1 to Patient.name[0].family doesn't change between Australia and Europe. The blood pressure panel merging logic doesn't change. The encounter class mapping doesn't change. Forking the repo means maintaining two copies of all this logic - and if I fix a bug in the AU version, I have to remember to fix it in the EU version too.

No path to a third region. What happens when I want to add US Core profiles? Another fork? Now I have three repos with three copies of the same BP merging code. This is the kind of technical debt that turns portfolio projects into maintenance nightmares.

It hides the interesting design work. A fork says "I copied code and changed some strings." A configurable multi-profile architecture says "I designed a system that handles jurisdiction differences at the configuration layer, not the logic layer." One of these is impressive in an interview. The other isn't.

The right approach was to make the existing pipeline support multiple profiles - and to do it in a way that the transformation logic doesn't know or care which region it's operating in.

The Profile Configuration Pattern

The core design change was extracting all region-specific values into a profile configuration module:

fastapi/app/profiles/
├── __init__.py
├── base.py              # ProfileConfig dataclass
├── au_profile.py        # AU_PROFILE instance
├── eu_profile.py        # build_eu_profile(country) factory
└── registry.py          # PROFILE_REGION → config resolver

ProfileConfig is a frozen dataclass that captures everything that varies by jurisdiction:

# fastapi/app/profiles/base.py
@dataclass(frozen=True)
class ProfileConfig:
    region: str
    patient_profile_url: str
    condition_profile_url: str
    encounter_profile_url: str
    bp_observation_profile_url: str
    observation_profile_url: str
    national_id_system: str
    national_id_display: str
    diagnosis_code_system: str
    snomed_system: str
    timezone_offset: str
    default_country: str
    profile_definitions: list[dict[str, str]] = field(default_factory=list)

The AU configuration is a single frozen instance:

# fastapi/app/profiles/au_profile.py
AU_PROFILE = ProfileConfig(
    region="au",
    patient_profile_url="http://hl7.org.au/fhir/StructureDefinition/au-patient",
    condition_profile_url="http://hl7.org.au/fhir/StructureDefinition/au-condition",
    encounter_profile_url="http://hl7.org.au/fhir/StructureDefinition/au-encounter",
    bp_observation_profile_url="http://hl7.org.au/fhir/StructureDefinition/au-vitalsigns-bloodpressure",
    observation_profile_url="http://hl7.org/fhir/StructureDefinition/vitalsigns",
    national_id_system="http://ns.electronichealth.net.au/id/hi/ihi/1.0",
    national_id_display="IHI",
    diagnosis_code_system="http://hl7.org.au/fhir/CodeSystem/icd-10-am",
    snomed_system="http://snomed.info/sct",
    timezone_offset="+10:00",
    default_country="AU",
    profile_definitions=[
        {"id": "au-patient", "url": "http://hl7.org.au/fhir/StructureDefinition/au-patient",
         "name": "AUPatient", "type": "Patient"},
        {"id": "au-condition", "url": "http://hl7.org.au/fhir/StructureDefinition/au-condition",
         "name": "AUCondition", "type": "Condition"},
        # ... plus au-encounter, au-vitalsigns-bloodpressure
    ],
)

The EU side is a factory rather than a static instance, because the national identifier
depends on the country. PROFILE_COUNTRY selects it:

# fastapi/app/profiles/eu_profile.py
EU_NATIONAL_ID_SYSTEMS = {
    "uk": ("https://fhir.nhs.uk/Id/nhs-number", "NHS Number"),
    "nl": ("http://fhir.nl/fhir/NamingSystem/bsn", "BSN"),
    "de": ("http://fhir.de/sid/gkv/kvid-10", "KVNR"),
    "ie": ("https://fhir.ie/sid/ppsn", "PPS Number"),
}

EU_DEFAULT_ID_SYSTEM = "http://hl7.eu/fhir/base/NamingSystem/national-id"
EU_DEFAULT_ID_DISPLAY = "National ID"

def build_eu_profile(country: str = "") -> ProfileConfig:
    country_lower = country.lower().strip()
    if country_lower in EU_NATIONAL_ID_SYSTEMS:
        national_id_system, national_id_display = EU_NATIONAL_ID_SYSTEMS[country_lower]
    else:
        national_id_system = EU_DEFAULT_ID_SYSTEM
        national_id_display = EU_DEFAULT_ID_DISPLAY
    return ProfileConfig(
        region="eu",
        patient_profile_url="http://hl7.eu/fhir/base/StructureDefinition/patient-eu",
        condition_profile_url="http://hl7.eu/fhir/base/StructureDefinition/condition-eu-core",
        encounter_profile_url="http://hl7.org/fhir/StructureDefinition/Encounter",  # base FHIR
        bp_observation_profile_url="http://hl7.org/fhir/StructureDefinition/bp",     # FHIR core BP
        observation_profile_url="http://hl7.org/fhir/StructureDefinition/vitalsigns",
        national_id_system=national_id_system,
        national_id_display=national_id_display,
        diagnosis_code_system="http://hl7.org/fhir/sid/icd-10",
        snomed_system="http://snomed.info/sct",
        timezone_offset="+01:00",
        default_country=country_lower.upper() if country_lower else "EU",
        profile_definitions=[
            {"id": "patient-eu", "url": "http://hl7.eu/fhir/base/StructureDefinition/patient-eu",
             "name": "PatientEU", "type": "Patient"},
            {"id": "condition-eu-core", "url": "http://hl7.eu/fhir/base/StructureDefinition/condition-eu-core",
             "name": "ConditionEUCore", "type": "Condition"},
            # ... plus FHIR core BP
        ],
    )

Two environment variables control which profile is active:

# AU mode (default - backward compatible)
docker compose up --build

# EU mode, UK national identifier (NHS Number)
PROFILE_REGION=eu PROFILE_COUNTRY=uk docker compose up --build

The registry resolves the active config from those settings:

# fastapi/app/profiles/registry.py
def get_profile() -> ProfileConfig:
    if settings.profile_region.lower().strip() == "eu":
        return build_eu_profile(settings.profile_country)
    return AU_PROFILE

How the Transformers Changed

The transformers don't contain if/else logic for regions. Each route resolves the active ProfileConfig once via get_profile() (driven by the PROFILE_REGION / PROFILE_COUNTRY settings) and passes it straight into the transformer as an argument:

# fastapi/app/main.py — inside the /fhir/Patient handler
profile = get_profile()
patient = build_patient(payload, profile)

The transformer signature takes that config and uses it for every jurisdiction-specific value:

# fastapi/app/transformers/patient.py
def build_patient(payload: AdtPayload, profile: ProfileConfig) -> Patient:
    ...
    # Before (hardcoded AU):
    #   meta={"profile": ["http://hl7.org.au/fhir/StructureDefinition/au-patient"]}
    # After (profile-configured):
    patient = Patient(
        meta={"profile": [profile.patient_profile_url]},
        ...
        identifier=[..., Identifier(system=profile.national_id_system, value=payload.ihi)],
        address=[Address(..., country=payload.address.country or profile.default_country)],
    )

The transformation logic (mapping PID fields to Patient attributes, merging BP panels, resolving patient references by MRN) stays unchanged. It doesn't know whether it's producing an AU Patient or an EU Patient. It builds a valid FHIR Patient resource and applies whatever metadata the profile config provides. (It's plain function-argument passing, not FastAPI's Depends() injection — the profile is process-wide config, resolved once per request from the environment.)

Adding a third region is one new file. A US Core profile would need us_profile.py, one new entry in the registry, and zero changes to the transformers.

Same Message, Different FHIR

To see the impact, compare the same ADT^A01 admission message processed in both modes.

The HL7 v2 message is identical. A pregnant woman is admitted for a routine antenatal visit. The message carries her name, date of birth, identifier, and a pregnancy diagnosis.

AU mode produces (real transformer output, trimmed):

{
  "resourceType": "Patient",
  "meta": {
    "profile": ["http://hl7.org.au/fhir/StructureDefinition/au-patient"]
  },
  "identifier": [
    {
      "type": {"coding": [{"system": "http://terminology.hl7.org/CodeSystem/v2-0203", "code": "MR"}]},
      "system": "http://hospital.local/mrn",
      "value": "1234567"
    },
    {
      "system": "http://ns.electronichealth.net.au/id/hi/ihi/1.0",
      "value": "8003608166690503"
    }
  ],
  "name": [{"use": "official", "family": "TEST", "given": ["PATIENT", "MARY"]}],
  "gender": "female",
  "birthDate": "1992-03-15"
}

EU mode (PROFILE_REGION=eu PROFILE_COUNTRY=uk) produces:

{
  "resourceType": "Patient",
  "meta": {
    "profile": ["http://hl7.eu/fhir/base/StructureDefinition/patient-eu"]
  },
  "identifier": [
    {
      "type": {"coding": [{"system": "http://terminology.hl7.org/CodeSystem/v2-0203", "code": "MR"}]},
      "system": "http://hospital.local/mrn",
      "value": "1234567"
    },
    {
      "system": "https://fhir.nhs.uk/Id/nhs-number",
      "value": "9000000009"
    }
  ],
  "name": [{"use": "official", "family": "TEST", "given": ["PATIENT", "MARY"]}],
  "gender": "female",
  "birthDate": "1992-03-15"
}

Same patient, same clinical data, different regulatory metadata. The profile URL and the national identifier system changed (au-patient → patient-eu, IHI → NHS Number). Name,gender, birthDate, MRN (including its MR type coding) are identical.

The Condition resource follows the same pattern:

AU:  "system": "http://hl7.org.au/fhir/CodeSystem/icd-10-am", "code": "O80"
EU:  "system": "http://hl7.org/fhir/sid/icd-10",              "code": "O80"

Same ICD code, different system URI. The clinical meaning is identical; only the regulatory packaging shifts.

The IPS Endpoint

A pregnant woman from the Netherlands visits a hospital in Ireland. The Irish clinician needs her active conditions, recent vital signs, medications, and allergies, right now, in a format they can read. The IPS is that format, and it's what makes this pipeline relevant to EHDS.

The pipeline already had all the data. After processing ADT^A01 (Patient + Condition), ORM^O01 (Encounter), and ORU^R01 (Observations), HAPI FHIR contains a complete clinical picture for that patient. The IPS endpoint assembles those existing resources into a single FHIR Composition document.

POST /fhir/IPS
{
  "correlationId": "ips-001",
  "mrn": "1234567"
}

The endpoint resolves the Patient by MRN (422 if the patient hasn't been admitted yet), queries HAPI FHIR for that patient's resources, then builds an IPS Composition (LOINC 60591-5, "Patient summary Document") with six sections:

IPS Composition (LOINC 60591-5)  → wrapped in a Document Bundle
├── Allergies and Intolerances (48765-2)  → emptyReason "notasked"
├── Medications (10160-0)                  → emptyReason "notasked"
├── Problems (11450-4)                     → non-pregnancy Conditions
├── Results (30954-2)                      → non-vital-sign Observations
├── Vital Signs (8716-3)                   → vital-sign Observations (BP panel, weight, ...)
└── Pregnancy History (10162-6)            → pregnancy-related Conditions (by SNOMED/ICD-10)

The whole thing is wrapped in a FHIR Document Bundle (Bundle-uv-ips profile), so the result is a single, self-contained document any EHDS-compliant system can consume.

The Allergies and Medications sections use emptyReason (notasked) because the pipeline doesn't ingest those HL7 message types yet. Any section that ends up with no entries gets the same treatment. The IPS spec requires these sections but supports the "no information available"
pattern — an empty section with an emptyReason is more credible than fake data.

GDPR Consent

European healthcare data processing requires a legal basis under GDPR. For health data specifically, Article 9(2)(h) permits processing forhealthcare purposes. The pipeline models
this as a FHIR Consent resource:

POST /fhir/Consent
{
  "correlationId": "consent-001",
  "mrn": "1234567",
  "policyRule": "gdpr-art-9-2-h",
  "provisionType": "permit"
}

The policyRule field carries the GDPR basis; the pipeline recognises gdpr-art-6-1-a (explicit consent, the default), gdpr-art-9-2-h (health-data processing), and gdpr-art-6-1-e (public interest), mapping each to a human-readable display in the Consent's policyRule.coding.
provisionType is permit or deny, and optional periodStart / periodEnd set the validity window. On success it returns 200 with {"consentId": "...", "correlationId": "..."}.

This endpoint is EU-only: it returns 404 when PROFILE_REGION=au, because GDPR doesn't apply in Australia. The profile configuration controls not just metadata values but also which endpoints are available.

No FHIR IG requires this. But EU health IT companies want engineers who understand why health data processing needs a legal basis and how consent flows work, not just engineers who can write valid FHIR resources.

Testing Across Profiles

The existing AU tests must not break. The EU extension adds capability without changing existing behavior.

The CI pipeline now runs tests in a matrix:

# .github/workflows/ci.yml
strategy:
  matrix:
    profile_region: [au, eu]
env:
  PROFILE_REGION: ${{ matrix.profile_region }}
  PROFILE_COUNTRY: ${{ matrix.profile_region == 'eu' && 'uk' || '' }}

Every test runs twice - once with PROFILE_REGION=au and once with PROFILE_REGION=eu (PROFILE_COUNTRY=uk). The AU test suite is unchanged. The EU tests add EU-specific assertions:

EU Patient resources carry the patient-eu profile URL, AU ones carry au-patient
EU Conditions use ICD-10 (WHO), AU Conditions use ICD-10-AM
EU identifiers use the configured national system (NHS Number for uk), not IHI
The IPS endpoint produces a valid IPS Document Bundle
The Consent endpoint returns 404 in AU mode and 200 in EU mode

The suite is now 277 tests at 92% line coverage (unit + integration + a tests/e2e/ suite
that drives real HL7 over MLLP against a live Docker stack and auto-skips when it isn't running).
Adding EU capability did not change a single existing AU assertion.

EU Sample Messages

The samples/ directory now includes EU-specific HL7 messages:

samples/
├── adt_a01_normal_delivery.hl7       # AU sample
├── adt_a01_escaped_name.hl7          # AU sample with HL7 escape sequences
├── orm_o01_antenatal_28w.hl7         # AU sample
├── oru_r01_vitals.hl7                # AU sample
├── invalid/
│   └── adt_missing_mrn.hl7
└── eu/
    ├── adt_a01_normal_delivery.hl7   # EU sample (NHS Number, ICD-10)
    ├── orm_o01_antenatal_28w.hl7     # EU sample
    └── oru_r01_vitals.hl7            # EU sample

The EU samples use NHS Number identifiers (NHS...^^^NHS^NH), ICD-10 (WHO) diagnosis codes, and UK-style facility naming (e.g. ST_THOMAS, LONDON_TRUST). The HL7 v2 message structure is identical to the AU samples — the same segments, the same field positions. Only the content in the identifier and diagnosis fields differs.

What I Learned

Building the first version taught me about HL7, FHIR, and healthcare integration. Extending it to Europe taught me different things.

The clinical domain is more universal than the regulatory layer. Blood pressure merging, patient admission flows, observation linking: none of this changes between Australia and Europe. The HL7-to-FHIR mapping logic is universal. What changes is the metadata layer: profile URLs, identifier systems, terminology editions, timezone offsets. Separating these two layers is the design challenge.

The EU FHIR ecosystem is more layered than AU. Australia's approach is relatively flat: AU Base sits on FHIR R4 and that's mostly it. Europe has EU Base → EU Core → Scoped IGs → National IGs. Understanding where your pipeline sits in that stack matters. I targeted EU Base/Core because it's the broadest level - a Dutch hospital and a German hospital can both accept resources profiled at this level. Going deeper into country-specific profiles (Nictiz, KBV) is a natural next step, but the foundation needs to be right first.

If you're building a FHIR portfolio for Europe, implement IPS. EHDS is built around it, MyHealth@EU exchanges it, and European health IT companies are hiring for it. A pipeline that can both ingest HL7 v2 messages and produce an IPS document covers both sides of the interoperability problem.

Configuration-driven architecture pays compound interest. The ProfileConfig pattern took maybe two hours to design. When I was implementing the EU profile, I never had to think about "what does this transformer do?" Only "what values should the EU config provide?" Two hours of design saved days of implementation.

GDPR awareness is a soft skill encoded in code. European employers want engineers who understand why health data processing requires a legal basis, what Article 6 vs Article 9 means, and how consent flows work. A Consent resource in your repo shows this better than a resume bullet point.

Backward compatibility is a feature. The hardest part of the EU extension was ensuring the existing AU pipeline stayed unchanged. Every AU test passing without modification was non-negotiable. Adding features is easy. Adding features without breaking existing behavior takes discipline.

What's Next

The pipeline now supports two regulatory jurisdictions with a common transformation core. Next on the list:

US Core - a third PROFILE_REGION=us option targeting the US market (US Core IG, Argonaut profiles, USCDI)
Terminology validation - integrating with a terminology server to validate SNOMED CT and ICD-10 codes against the correct value sets for each region
Additional HL7 message types - adding allergies (ADT^A60) and medications (RDE^O11) to populate the empty IPS sections
Full IG package loading - replacing placeholder StructureDefinitions with the complete HL7 EU Base/Core IG package in HAPI for meaningful $validate results

The full source code, documentation, and Docker Compose setup are on GitHub: maternity-hl7-to-fhir-pipeline.

This is Part 2 of the Maternity HL7-to-FHIR Pipeline series. Part 1 covered the initial AU pipeline architecture. If you're working on FHIR interoperability in the European market - especially around EHDS compliance or IPS implementation - I'd like to hear what challenges you're facing.

Source code: github.com/budityw23/maternity-hl7-to-fhir-pipeline

Tags: #fhir #healthit #architecture #showdev

Why I Put Mirth Connect in Front of FastAPI Instead of Parsing HL7 in Python

Budi Widhiyanto — Sat, 25 Jul 2026 21:42:55 +0000

When I started building my Maternity HL7-to-FHIR Pipeline, my first instinct was to do everything in Python. Parse the HL7 message, map the fields, validate the FHIR resource, persist it, all in one FastAPI service. It was clean. It was simple. It was wrong.

Some Background

I've been building an open-source pipeline that takes HL7 v2.5 messages, the kind hospital maternity systems still send over raw TCP connections, and turns them into FHIR R4 resources with Australian and European FHIR profiles. Patient admissions, lab orders, vital signs. The sort of data that's been flowing between hospital systems in pipe-delimited text since the 1990s, now mapped into modern healthcare APIs.

If you're curious about the full project, I wrote a detailed walkthrough in Bridging Legacy Hospital Messages to Modern Healthcare APIs. That article covers the end-to-end AU pipeline architecture. The second article, One Pipeline, Two Continents, covers how I extended the pipeline to support EU FHIR profiles, IPS, and GDPR Consent. Here's the short version of the architecture:

Hospital Maternity System
  |
  | HL7 v2.5 messages (ADT, ORM, ORU)
  | over MLLP (raw TCP, port 6661)
  v
┌─────────────────────────┐
│   Mirth Connect 4.5     │  Receives MLLP, parses HL7,
│   (Integration Engine)  │  extracts fields, routes
└───────────┬─────────────┘  by message type
            |
            | Clean JSON over HTTP
            v
┌─────────────────────────┐
│  FastAPI + Python 3.11  │  Pydantic validation, FHIR R4
│  (Transformation Layer) │  mapping, AU Base profiles
└───────────┬─────────────┘
            |
            | FHIR R4 resources
            | (conditional PUT/POST)
            v
┌─────────────────────────┐
│  HAPI FHIR Server 7.0.3 │  Persistence + FHIR API
└─────────────────────────┘

Three containers, one docker compose up. Mirth handles the legacy protocol world, FastAPI handles the FHIR world, HAPI stores everything.

But this article isn't about the what. It's about one particular why: why the pipeline has two services doing the transformation work instead of one.

The "Just Parse It in Python" Phase

My initial architecture looked like this:

Hospital System --MLLP--> Python Script --> HAPI FHIR Server

I used python-hl7 to split messages on | and count field positions. For a single ADT^A01 (patient admission) message, it worked fine. I could pull the patient name from PID-5, the MRN from PID-3, the gender from PID-8, and build a FHIR Patient resource from it.

Then I tried a real-ish maternity workflow (an admission, an order, and a set of vitals) and things fell apart quickly.

Five Problems That Changed My Mind

1. MLLP Is Not HTTP

Hospital systems don't send HL7 over HTTP. They send it over MLLP (Minimum Lower Layer Protocol), which is a TCP socket protocol with specific framing bytes (\x0b at the start, \x1c\x0d at the end). The sender expects an ACK or NACK response in HL7 format, not an HTTP status code.

Building an MLLP listener in Python is possible. Libraries like aiohl7 exist. But you're now maintaining a custom TCP server alongside your HTTP API server, handling connection pooling, timeouts, and HL7 acknowledgment generation. That's a lot of infrastructure code that has nothing to do with your actual transformation logic.

Mirth Connect handles MLLP natively. You point it at a port, it listens, it parses, it ACKs. Done. One config screen, no custom code.

2. HL7 Parsing Is Messier Than It Looks

The pipe-delimited format looks simple:

PID|1||1234567^^^RPA^MR||TEST^PATIENT^MARY^^MS||19920315|F|||14 SAMPLE ST^^SYDNEY^NSW^2000^AUS

But consider:

Component separators: PID-5 is TEST^PATIENT^MARY^^MS, which is family, given, middle, suffix (empty), prefix. Miss the empty suffix and your prefix ends up as the suffix.
Repeating fields: PID-3 can contain multiple identifiers separated by ~. One might be the MRN, another the IHI (Individual Healthcare Identifier). You need to iterate and match by identifier type, not just grab the first one.
Escape characters: A patient named O'Brien might appear as O\T\Brien in HL7 (where \T\ is the subcomponent separator escape). Or it might not, depending on the sending system's configuration.
Encoding characters: The first field of MSH (MSH-1) is the field separator itself (|), and MSH-2 defines the component, repetition, escape, and subcomponent separators. Different hospitals can (and do) use different separators.

In my Python script, I was doing segment.split('|')[5].split('^')[0] to get a family name. One unexpected empty field, one unexpected repeating group, and the whole positional mapping shifted silently. No error, just wrong data in the FHIR resource.

Mirth Connect's HL7 parser handles all of this natively. In a Mirth transformer, I write:

var familyName = msg['PID']['PID.5']['PID.5.1'].toString();
var givenName  = msg['PID']['PID.5']['PID.5.2'].toString();

That's an E4X/XML path against a parsed HL7 tree, not string splitting. It handles repeating fields, component separators, and encoding characters correctly because that's what the parser is built for. Years of edge cases baked into a mature parser versus my three-day-old string splitter.

3. Message Routing Is a Separate Concern

A maternity workflow involves three message types:

Message	Trigger	What It Creates
`ADT^A01`	Patient admitted	Patient + Condition
`ORM^O01`	Order placed	Encounter
`ORU^R01`	Results available	Observation(s)

In the "everything in Python" design, my MLLP listener would receive a raw HL7 message, I'd parse MSH-9 to determine the message type, then route to the right handler function. That's a message router, and I'd be building one from scratch.

Mirth Connect is literally a message router. You can define a channel per message type, or (as I did) a single channel whose transformer routes by message type. Either way, each type is POSTed to a different FastAPI endpoint, and Mirth handles the dispatch. The FastAPI endpoints receive clean, typed JSON payloads. They don't even need to know HL7 exists.

Mirth Channel: Maternity Inbound HL7
  Source:       MLLP listener on port 6661 (HL7 v2.5, auto ACK/NACK)
  Preprocessor: generate a correlation-id (UUID) per message
  Destination:  HTTP Sender - a JavaScript transformer routes on
                msg['MSH']['MSH.9']['MSH.9.1'] and POSTs to:
                  ADT -> http://fastapi:8000/fhir/Patient
                  ORM -> http://fastapi:8000/fhir/Encounter
                  ORU -> http://fastapi:8000/fhir/Observation/bundle

The transformer extracts only the fields relevant to each message type, builds a flat JSON payload, and tags every request with an X-Correlation-ID header. FastAPI receives structured data with Pydantic validation on the input shape. Clean separation.

4. ACK/NACK Is Surprisingly Important

When a hospital system sends an HL7 message, it expects an acknowledgment. If it gets an ACK (MSA-1 = AA), it moves on. If it gets a NACK (MSA-1 = AE or AR), it may retry, queue the message, or alert an operator.

In my Python approach, I'd need to:

Parse the incoming message
Process it
Build an HL7 ACK message with the correct MSH fields mirrored back
Send it over the same TCP socket
Handle the case where processing succeeds but the ACK fails to send

Mirth Connect auto-generates ACK messages. You configure whether to ACK on receipt (before processing) or after successful processing. You can customise the ACK content if needed. Another solved problem.

5. I Was Rebuilding an Integration Engine, Badly

Step back and look at what I was building in Python:

A TCP/MLLP listener
An HL7 v2.5 parser
A message router
An ACK/NACK generator
Connection management and error handling

That's an integration engine. Mirth Connect, Rhapsody, InterSystems HealthShare. These are products that teams of engineers have built and maintained for years. I was reimplementing one in a weekend, pretending it was "simpler" because it was in Python.

The honest assessment: I was spending 60-70% of my effort on plumbing (protocol handling, parsing, routing) and 30-40% on the actual value: the FHIR transformation and validation logic.

The Redesigned Architecture

Here's what I landed on:

Mirth Connect does what it's good at: protocol handling, HL7 parsing, message routing, ACK generation. It turns messy, protocol-specific HL7 into clean HTTP + JSON.

FastAPI does what it's good at: receiving typed payloads, applying business logic (like merging systolic + diastolic OBX segments into a single FHIR Blood Pressure panel), validating FHIR resources with Pydantic models from fhir.resources, and persisting to the FHIR server.

The boundary between them is a simple HTTP POST with a JSON body. FastAPI doesn't know about MLLP, segment separators, or ACK messages. Mirth doesn't know about FHIR profiles, conditional PUT, or Pydantic validation. Each component is testable in isolation.

Why Not Do Everything in Mirth?

Fair question. Plenty of teams build entire HL7-to-FHIR transformations inside Mirth Connect's JavaScript engine. You can construct FHIR JSON in a Mirth transformer and POST it directly to HAPI. Why add FastAPI at all?

Three reasons:

Type-Safe FHIR Validation

The fhir.resources library gives me Pydantic models for every FHIR R4 resource type. When I build a Patient resource, the model enforces:

gender must be one of male, female, other, unknown, not F or M
name is a list of HumanName objects with specific structure
identifier requires both system and value
meta.profile must be a list of valid URIs

If I typo Observation.status as "done" instead of "final", Pydantic raises a validation error with the exact field path before the resource ever reaches HAPI. In Mirth's JavaScript, I'd be constructing a plain JSON object with no schema enforcement. Typos become silent bugs that surface as cryptic HAPI 400 errors.

Testability

My FastAPI transformers have 220+ unit tests, 30+ integration tests, and 20+ end-to-end tests at 80%+ coverage. Each transformer function takes a Pydantic input model and returns a FHIR resource. I can test edge cases (empty diagnoses arrays, missing middle names, orphan blood pressure readings) in milliseconds with pytest.

Testing Mirth channels is harder. You can write unit tests for Mirth's JavaScript functions, but testing the full channel behavior (source -> filter -> transformer -> destination) requires either Mirth's built-in test tools or a running Mirth instance. The feedback loop is slower, and CI integration is less straightforward.

Python Ecosystem

For the transformation logic, Python gives me things Mirth's JavaScript doesn't:

fhir.resources for FHIR R4 Pydantic models
httpx for async HTTP with connection pooling
pydantic-settings for typed configuration from environment variables
ruff for linting, mypy --strict for type checking
The entire pytest ecosystem for testing

The FHIR mapping is where I add value. I want the best tools for that specific job.

The Tradeoff I Accepted

This architecture has a cost: an extra network hop. Every message goes from Mirth to FastAPI over HTTP, adding latency. In a high-throughput production system processing thousands of messages per second, this matters.

For a maternity ward generating maybe 50-100 messages per hour? The latency is invisible. And the benefits (clean separation, type safety, testability, better error handling) far outweigh the cost.

If throughput became a bottleneck, I'd look at gRPC between Mirth and FastAPI, or batch endpoints that accept multiple messages per request. But I'd still keep the separation. The architectural benefit is worth more than saving a few milliseconds.

What This Taught Me

Use the right tool for each layer. Integration engines exist because protocol handling and message parsing are genuinely hard, well-understood problems. Application frameworks exist because business logic, validation, and API design are different problems. Trying to solve both in one layer means solving neither well.

The boundary between components should be boring. HTTP + JSON between Mirth and FastAPI is boring. That's the point. The interesting work (the FHIR mapping, the BP panel merging, the AU Base profiling) happens inside the components, not at the boundaries.

"Simpler" isn't always fewer components. My initial one-component Python approach looked simpler. In practice, it was a tangled mess of protocol code, parsing code, and transformation code all in one place. Three components with clear responsibilities turned out to be simpler to build, test, and debug.

The payoff shows up when you extend it. Adding EU FHIR profiles on top of the existing AU pipeline touched about 2,400 lines across 49 files. Roughly 20 of those lines were in the Mirth transformer: widening national ID extraction beyond AU's IHI to also accept NH, PN, and SS, and dropping a hardcoded country default so empty country codes fall through to the FastAPI profile. Everything else — the ProfileConfig registry, EU profile URLs, IPS Composition, GDPR Consent — landed in FastAPI. The transport layer barely noticed a second jurisdiction arriving.

What's Next

The separation between Mirth and FastAPI makes each of these extensions a clean addition rather than a risky refactor:

US Core profiles — a third PROFILE_REGION=us option. The profile configuration pattern from the EU extension means this is another ProfileConfig instance and a set of profile URLs — no transformer rewrites.
Terminology validation — integrating with a terminology server (like Ontoserver or the HAPI terminology service) to validate SNOMED CT and ICD-10 codes against region-specific value sets at transformation time.
Additional HL7 message types — allergies (ADT^A60) and medications (RDE^O11) to populate the currently empty IPS sections. Each one is a new Mirth routing rule and a new FastAPI transformer — the pattern scales without architectural changes.
Full IG package loading — replacing the placeholder StructureDefinitions with complete AU Base and EU Core IG packages in HAPI for meaningful $validate results across both regions.

If you're evaluating integration engine versus "parse it yourself" for a health IT project, I'd like to hear how you made that call. The first article in this series covers the full pipeline architecture, and the second article shows how the separation kept adding EU FHIR profiles almost entirely inside FastAPI.

Try It Yourself

Bring up the three containers:

docker compose up --build

Or with EU profiles:

PROFILE_REGION=eu docker compose up --build

Then deploy the Mirth channel (the stock Mirth image doesn't auto-load channels, so this imports and deploys it via the REST API):

./scripts/import_channels.sh

Now send a test HL7 message via MLLP:

python scripts/mllp_send.py samples/adt_a01_normal_delivery.hl7

Or skip Mirth entirely and hit FastAPI directly with JSON:

curl -s -X POST http://localhost:8000/fhir/Patient \
  -H "Content-Type: application/json" \
  -d '{ ... }'

Both paths produce the same validated, correctly profiled FHIR resources in HAPI.

Source code: github.com/budityw23/maternity-hl7-to-fhir-pipeline

This is the third article in my series on the Maternity HL7-to-FHIR Pipeline. The first article, Bridging Legacy Hospital Messages to Modern Healthcare APIs, covers the full AU pipeline architecture. The second article, One Pipeline, Two Continents, covers the EU FHIR profile extension.

Tags: #healthit #architecture #python #showdev

Maternity HL7-to-FHIR Pipeline: Bridging Legacy Hospital Messages to Modern Healthcare APIs

Budi Widhiyanto — Tue, 21 Jul 2026 02:07:48 +0000

How It Started

A few months ago, I was browsing remote FHIR engineer roles in Australia. Every single job description mentioned the same thing: experience integrating HL7 v2 with FHIR R4.

This makes sense. Australian hospitals - like most hospitals worldwide - still run on HL7 v2. Patient admissions, lab results, orders - they all flow as pipe-delimited messages over MLLP. But the direction is clear: the Australian Digital Health Agency (ADHA) is pushing hard toward FHIR R4 through the Sparked FHIR Accelerator and the AU Core Framework. Somebody has to build the bridge between these two worlds.

I decided to build one myself: a working pipeline that takes real HL7 v2 maternity messages and transforms them into valid, AU-profiled FHIR R4 resources. Not a toy. Not a tutorial. A real integration that I could show in interviews and say: "I built this. Here's how it works. Here's where it breaks."

GitHub: budityw23/maternity-hl7-to-fhir-pipeline

This article walks through how I designed it - including the mistakes I made along the way.

My Initial (Naive) Approach

My first instinct was simple. HL7 v2 comes in, I parse it in Python, I build a FHIR JSON, I POST it to a FHIR server. Done.

Hospital Maternity System
  |
  |  HL7 v2 messages (MLLP)
  v
Python Script
  |  -> parse HL7
  |  -> build FHIR JSON manually
  |  -> POST to FHIR server
  v
HAPI FHIR Server

I wrote a quick script. It worked for a single ADT^A01 (patient admission) message. I felt good about it. Then the problems started:

No MLLP handling: My script was reading messages from a file. Real hospital systems don't send files - they send HL7 over MLLP (a TCP-based protocol with specific framing). My script had no way to receive live messages.
Brittle parsing: I was splitting on | and counting positions. One unexpected empty field and the whole mapping shifted. A message with a maiden name in PID-6 but nothing in PID-5.4 (suffix) would silently put the wrong data in the wrong FHIR field.
No validation: I was constructing FHIR JSON by hand as Python dicts. Typo in a field name? Missing a required element? The script happily sent invalid resources to the FHIR server, which sometimes accepted them and sometimes threw cryptic errors.
No multi-resource linking: A maternity visit involves a Patient, an Encounter, Observations (blood pressure, weight, fetal heartbeat), and a Condition (pregnancy diagnosis). These resources need to reference each other. My script created them independently with no linking - the Observation had no idea which Encounter it belonged to.
No error trail: If the Patient was created successfully but the Observation failed, I had no record of what went wrong, no correlation between the original message and the failure, and no way to replay the failed message.

I was building for the happy path again - the same mistake I described in my vaccine appointment system article. One clean message in, one clean resource out. But healthcare data is never clean.

Rethinking the Architecture

The core insight was: this is not a single transformation. It's a pipeline with distinct responsibilities.

Receiving MLLP messages is a different concern from parsing HL7. Parsing is different from mapping. Mapping is different from FHIR validation. Validation is different from persistence. Each stage can fail independently, and each needs its own error handling.

I also realized that an integration engine like Mirth Connect already solves the hardest part - MLLP reception, HL7 parsing, and message routing. Fighting that battle in raw Python was wasting time on a solved problem.

Here's the redesigned architecture:

Hospital Maternity System
  |
  |  HL7 v2.5 messages over MLLP:
  |    ADT^A01  -> patient admitted for antenatal care
  |    ORM^O01  -> antenatal checkup ordered
  |    ORU^R01  -> checkup results (BP, weight, fetal heartbeat)
  v
Mirth Connect 4.5 (port 6661)
  |  -> receives MLLP, parses HL7 natively
  |  -> extracts key segments (PID, PV1, OBX, DG1, OBR)
  |  -> builds structured JSON payload
  |  -> routes by message type to correct endpoint
  v
Python 3.11 + FastAPI (port 8000)
  |  -> receives typed JSON from Mirth
  |  -> maps fields to FHIR R4 with AU Base profiles
  |  -> validates via fhir.resources (Pydantic models)
  |  -> conditional PUT/POST to HAPI FHIR (idempotent)
  |  -> structured JSON logging with correlation ID
  v
HAPI FHIR Server v7.0.3 (port 8080)
  |  -> stores Patient, Condition, Observation, Encounter
  |  -> supports $validate for profile conformance
  |  -> serves FHIR REST API
  v
Accessible via standard FHIR queries

The key difference: every component does one thing well, and failure at any stage doesn't corrupt the others.

The Improved Design

1. Mirth Connect: The HL7 Gatekeeper

Mirth Connect listens on MLLP port 6661 and handles the protocol-level complexity that I was trying to reinvent in Python. It natively understands HL7 v2.5 segment structure, so I can reference msg['PID']['PID.5']['PID.5.1'] directly in a JavaScript transformer - no string splitting, no position counting.

Mirth does three jobs:

Receives and parses: Handles MLLP framing (the \x0b header and \x1c\x0d trailer), parses segments, and sends ACK/NACK responses back to the sender.
Extracts and restructures: A JavaScript transformer pulls the fields I need and builds a clean JSON payload. This is where I handle HL7 structural quirks - pulling the MRN and IHI out of the repeating PID-3 field by identifier type (MR vs NI), taking the first PID-13 repetition for the phone number, and normalising the country code (AUS -> AU). One thing that tripped me up here: Mirth parses each field into components, so a field node's toString() returns XML markup, not the value - you have to drill to the .1 subcomponent (msg['PID']['PID.8']['PID.8.1']) to get F. Terminology mapping - F/M/U sex codes to female/male/unknown, ICD-10-AM to SNOMED - is deliberately left to the FastAPI layer, where the Pydantic models validate it.
Routes by message type: ADT^A01 goes to /fhir/Patient, ORM^O01 goes to /fhir/Encounter, ORU^R01 goes to /fhir/Observation/bundle. Each endpoint in FastAPI knows exactly what shape of data to expect.

Why not do everything in Mirth? You can - plenty of teams build entire FHIR transformations in Mirth's JavaScript engine. But I wanted the FHIR validation layer in Python using fhir.resources, which gives me Pydantic-based validation against the full FHIR R4 spec. If I accidentally set Observation.status to "done" instead of "final", Pydantic catches it before it ever reaches the FHIR server. You don't get that level of type safety in Mirth's JavaScript.

2. The Mapping Layer: Where the Real Work Happens

This is the heart of the pipeline. Each HL7 message type maps to one or more FHIR resources:

ADT^A01 (Patient Admission) produces:

Patient - from PID segment (name, DOB, sex, address, IHI identifier)
Condition - from DG1 segment (pregnancy diagnosis, ICD-10-AM coded)

ORM^O01 (Order) produces:

Encounter - from PV1 segment (visit number, class, admission date, location, attending doctor)
References back to existing Patient (looked up by MRN)

ORU^R01 (Observation Result) produces:

Observation - from OBX segments (vital signs: blood pressure, weight, fetal heartbeat)
References back to existing Patient (by MRN) and Encounter (by visit number)

Here's where the complexity lives. A simple mapping like PID-5.1 -> Patient.name[0].family is straightforward. But consider blood pressure. In HL7 v2, systolic and diastolic come as two separate OBX segments:

OBX|1|NM|8480-6^Systolic BP^LN||120|mm[Hg]|90-120|N|||F
OBX|2|NM|8462-4^Diastolic BP^LN||80|mm[Hg]|60-80|N|||F

In FHIR R4, blood pressure is a single Observation resource with LOINC code 85354-9 (Blood pressure panel) containing two component[] entries - one for systolic, one for diastolic. The transformer detects consecutive BP-related OBX segments (codes 8480-6 and 8462-4), merges them into a single panel Observation with the AU Base blood pressure profile (au-vitalsigns-bloodpressure), and applies the most conservative status and worst-case interpretation across both readings. Orphan systolic or diastolic readings (without their pair) fall back to individual Observations. This is the kind of domain logic that makes healthcare integration genuinely hard - it's not just field-to-field mapping.

3. Australian Localisation

Since this targets the AU market, the FHIR resources conform to AU Base profiles on all resource types:

Resource	Profile
Patient	`http://hl7.org.au/fhir/StructureDefinition/au-patient`
Condition	`http://hl7.org.au/fhir/StructureDefinition/au-condition`
Encounter	`http://hl7.org.au/fhir/StructureDefinition/au-encounter`
Observation (BP)	`http://hl7.org.au/fhir/StructureDefinition/au-vitalsigns-bloodpressure`

Additional AU-specific details:

Patient identifiers include the IHI (Individual Healthcare Identifier) with system URI http://ns.electronichealth.net.au/id/hi/ihi/1.0
Diagnosis coding uses ICD-10-AM (Australian Modification) rather than plain ICD-10
Clinical terminology uses SNOMED CT-AU where applicable
Addresses use 4-digit Australian postcodes and state codes (NSW, VIC, QLD, etc.)
All FHIR datetimes include the +10:00 AEST timezone offset

This isn't just cosmetic. AU Base profiles have specific cardinality and terminology constraints. An au-patient resource without a valid identifier type code will fail validation against the profile. Getting this right demonstrates that I understand the difference between generic FHIR and FHIR as it's actually used in Australian healthcare.

4. Idempotency: Handling Duplicate Messages

Hospital systems sometimes send the same message twice - network retries, interface engine restarts, manual resends. The pipeline handles this through conditional operations:

Patient: Conditional PUT using ?identifier=http://hospital.local/mrn|{MRN} - re-sending the same ADT^A01 updates the existing Patient rather than creating a duplicate.
Encounter: Conditional PUT using the visit number identifier - same ORM^O01 twice produces one Encounter.
Condition and Observation: Created via POST (new resource each time), but because they reference the idempotent Patient/Encounter, the referential links remain consistent.

This approach trades off perfect idempotency on Conditions/Observations for simplicity. In a production system, you might add conditional logic based on a combination of patient reference, code, and effective date. For a portfolio project, the Patient/Encounter idempotency demonstrates the pattern clearly.

5. Failure Scenarios and Observability

Just like with the vaccine system, I forced myself to think about what breaks:

Malformed HL7 message: Mirth rejects it at the protocol level and sends a NACK. The message never reaches FastAPI. Logged for review.
Missing required fields: FastAPI's Pydantic models catch empty MRNs, invalid gender codes, and missing observation codes at the payload boundary. Returns RFC 7807 application/problem+json with a 422 status.
FHIR validation failure: The fhir.resources Pydantic model rejects the resource - maybe Observation.status has an invalid value, or a required CodeableConcept is missing. The error includes exactly which field failed and why.
Patient not found for downstream messages: An ORM^O01 or ORU^R01 arrives before the ADT^A01. The pipeline looks up the Patient by MRN, fails to find it, and returns a 422: "Patient not found for MRN=X. Send ADT^A01 first."
HAPI FHIR rejects the resource: The FastAPI layer catches the HTTP error, writes the original payload to a dead-letter file in ./deadletter/ (with the correlation ID as filename prefix), and returns a 502 with the HAPI error details.

Correlation ID ties everything together. Every message gets a UUID (either from Mirth via X-Correlation-ID header, or auto-generated by middleware). The correlation ID appears in every structured JSON log line, in error responses, in dead-letter filenames, and in response headers. When something fails at 3am, you grep one ID and see the entire message lifecycle.

Structured JSON logging ensures every log line is machine-parseable - timestamp, level, logger name, message, and correlation ID in every entry. Noisy libraries (uvicorn access logs, httpx, httpcore) are suppressed. No PHI (names, dates of birth) in logs - only identifiers like MRN and correlation ID at INFO level.

6. Server-Side Validation

Client-side validation via fhir.resources catches structural errors - wrong field types, missing required elements, invalid enum values. But it can't validate against AU Base profile constraints. For that, the pipeline integrates with HAPI's $validate operation.

A standalone endpoint (POST /fhir/validate/{resource_type}) accepts any FHIR resource and returns the raw OperationOutcome from HAPI - useful for testing profile conformance during development.

For production use, a VALIDATE_BEFORE_PERSIST config flag enables pre-persist validation: every resource is validated via $validate before being written to HAPI. If validation finds errors or fatal issues, the persist is blocked and a 422 is returned with diagnostic messages. This is off by default (the demo seeds placeholder StructureDefinitions, not the full AU Base IG), but the plumbing is in place for when a real terminology server and IG package are loaded.

System Components

Here's the full component view:

+----------------------------------------------------------+
|                    Docker Compose                        |
|                    (maternity-net)                       |
|                                                          |
|  +-------------+    +--------------+    +------------+  |
|  |   Mirth      |    |   FastAPI     |    |  HAPI FHIR |  |
|  |  Connect     |--->|   Python      |--->|   Server   |  |
|  |  4.5         |    |   3.11+       |    |  v7.0.3    |  |
|  |              |    |              |    |            |  |
|  |  Port: 6661  |    |  Port: 8000  |    | Port: 8080 |  |
|  |  (MLLP)      |    |  (HTTP)      |    | (FHIR REST)|  |
|  +-------------+    +--------------+    +------------+  |
|                                                          |
|  Services:                                               |
|  - mirth       (nextgenhealthcare/connect:4.5)          |
|  - fastapi     (python:3.11-slim + FastAPI + uvicorn)   |
|  - hapi        (hapiproject/hapi:v7.0.3, H2 embedded)  |
+----------------------------------------------------------+

Mirth Connect 4.5 - The integration engine. Handles MLLP, parses HL7, routes by message type, sends structured JSON downstream.
Python FastAPI - The transformation and validation layer. Endpoints for Patient (/fhir/Patient), Encounter (/fhir/Encounter), Observations (/fhir/Observation/bundle), validation (/fhir/validate/{resource_type}), and health (/health). Dedicated transformer modules (patient.py, condition.py, observation.py, encounter.py). Uses fhir.resources for Pydantic-based FHIR R4 validation.
HAPI FHIR Server v7.0.3 - The FHIR repository. Stores resources, serves the REST API, supports $validate for profile conformance. Uses embedded H2 database for the demo (Postgres swap documented for production).

Everything runs with docker compose up. One command, full pipeline.

Repository Structure

maternity-hl7-to-fhir/
|-- docker-compose.yml
|-- fastapi/
|   |-- Dockerfile
|   |-- pyproject.toml
|   `-- app/
|       |-- main.py                   # FastAPI routes + lifespan
|       |-- config.py                 # pydantic-settings (env vars)
|       |-- errors.py                 # RFC 7807 problem+json
|       |-- logging_setup.py          # Structured JSON logging
|       |-- middleware.py             # X-Correlation-ID middleware
|       |-- models/                   # Pydantic payload models
|       |   |-- adt_payload.py        #   ADT^A01 input
|       |   |-- orm_payload.py        #   ORM^O01 input
|       |   `-- oru_payload.py        #   ORU^R01 input
|       |-- transformers/             # HL7 -> FHIR mapping
|       |   |-- patient.py            #   -> Patient resource
|       |   |-- condition.py          #   -> Condition resource
|       |   |-- encounter.py          #   -> Encounter resource
|       |   `-- observation.py        #   -> Observation (+ BP panel)
|       |-- clients/                  # HAPI FHIR client
|       |   |-- hapi_client.py        #   Upsert/create + $validate + profile seeding
|       |   |-- patient_resolver.py   #   MRN -> Patient ID lookup
|       |   `-- encounter_resolver.py #   Visit number -> Encounter ID lookup
|       `-- valuesets/                # Terminology mappings
|           |-- hl7_to_fhir_gender.py
|           |-- hl7_to_fhir_encounter.py
|           `-- hl7_to_fhir_observation.py
|-- hapi/
|   |-- Dockerfile
|   |-- application.yaml
|   `-- HealthCheck.java
|-- mirth/
|   |-- channels/
|   `-- code_templates/
|-- samples/                          # Synthetic HL7 messages
|-- scripts/
|   |-- mllp_send.py                  # MLLP test client
|   |-- demo.sh                       # Interactive walkthrough
|   `-- reset.sh                      # Full reset (down -v + clean)
|-- tests/
|   |-- unit/                         # 157 unit tests
|   `-- integration/                  # 20 integration tests
|-- deadletter/                       # Failed message store (gitignored)
|-- logs/                             # Runtime logs (gitignored)
`-- .github/workflows/ci.yml          # CI: lint + type check + test

What I Learned

Building this project taught me things I couldn't have learned from reading the FHIR spec alone.

HL7 v2 is deceptively simple. The pipe-delimited format looks easy to parse until you encounter repeating fields, component separators, escape characters, and the fact that different hospitals implement the same message type differently. An integration engine like Mirth saves enormous time here - it's purpose-built for this chaos.

FHIR validation is your safety net, not your enemy. My naive approach skipped validation because it felt like extra work. In practice, the Pydantic models from fhir.resources caught errors that would have taken hours to debug at the FHIR server level. A missing Observation.code.coding[0].system? Pydantic tells you immediately. HAPI FHIR gives you a generic 400. Adding server-side $validate on top catches profile conformance issues that client-side validation can't - like missing AU Base constraints.

Idempotency is not optional in healthcare integration. Hospital systems send duplicate messages routinely. Conditional PUT with identifier queries means the second submission of the same ADT^A01 updates the existing Patient instead of creating a duplicate. This is a simple pattern that eliminates an entire class of production incidents.

Correlation IDs are the single most useful observability investment. One UUID, generated at the entry point, propagated through every log line, every error response, every dead-letter file. When a message fails at 3am, you grep one string and see everything that happened. Without it, you're correlating timestamps across multiple services and hoping the clocks are synchronized.

The AU localisation is what separates a tutorial from a portfolio project. Anyone can map PID-5 to Patient.name. Knowing that Australian systems use IHI identifiers, ICD-10-AM coding, AU Base profile URLs, and SNOMED CT-AU - and encoding all of that correctly in the FHIR resources - shows real domain expertise. In interviews, this is what gets follow-up questions.

Start with failure scenarios, not the happy path. This is the same lesson from my vaccine appointment system design, and it applies everywhere. The first question to ask about any integration is not "how does it work?" but "what happens when it breaks?" RFC 7807 error responses, dead-letter queues, and structured logging aren't glamorous features - but they're the difference between "it works in a demo" and "I'd trust this in production."

Testing and Quality

The pipeline has 177 automated tests (157 unit + 20 integration) at 90% line coverage:

Unit tests cover every transformer, valueset mapping, payload validator, error handler, logging formatter, and middleware component in isolation.
Integration tests exercise the full FastAPI ASGI stack (routes, middleware, error handlers) with mocked HAPI responses using respx. Tests cover happy paths, validation failures, missing upstream resources, and correlation ID propagation.
CI pipeline runs ruff (linting), mypy --strict (type checking), and pytest with a coverage gate of 80% on every push.

What's Next

This pipeline covers 4 FHIR resources, 3 HL7 message types, and AU Base profile conformance - deliberately scoped to be buildable as a portfolio project while demonstrating real integration patterns. Extensions I'm considering:

Full AU Core IG loading - replacing the placeholder StructureDefinitions with the complete AU Base Implementation Guide package for meaningful $validate results
Terminology validation - integrating with a FHIR terminology server (like Ontoserver) to validate SNOMED CT-AU and LOINC codes at transformation time
Monitoring dashboard - a simple frontend showing message throughput, transformation success/failure rates, and recent errors
Additional message types - ORU^R30 (unsolicited lab), ADT^A08 (update patient), ADT^A03 (discharge)

The full source code, documentation, and Docker Compose setup are on GitHub: maternity-hl7-to-fhir-pipeline.

If you're working on HL7-to-FHIR integrations or preparing for health IT interviews, I'd like to hear what challenges you've run into. You might also find my earlier article on the National Vaccine Appointment & Administration System useful - it covers similar design thinking around failure handling and rollback patterns.

Source code: github.com/budityw23/maternity-hl7-to-fhir-pipeline

Tags: #fhir #healthit #hl7 #architecture

Our GCP daily cost dropped 80% in six weeks, here is every change we made

Budi Widhiyanto — Wed, 08 Jul 2026 14:09:43 +0000

When our project manager forwarded the May 2026 invoice, the number had tripled from where it was two months earlier. That was the monthly GCP bill for two regional deployments of the same platform. We had seen it climb through April, but this was the first time the daily cost chart showed no sign of flattening. The peak day had already hit 3x our normal baseline.

Nothing was broken. The converter was processing records and the dashboards were up. The system was doing exactly what it was built to do. Together with Mas @chiqors , our devops teammate, we pulled up the billing console and started figuring out whether the system actually needed to cost that much.

What we run

We operate a federated health data platform deployed in two districts in Indonesia. The platform collects data from twelve different source systems: electronic medical records from hospitals, nutrition tracking from community health workers, TB surveillance from the national system, immunization registries, midwife apps, and several others. All of that gets converted into FHIR R4 resources and stored in a central FHIR datastore per district.

Each district runs its own GCP project. The converter runs on Cloud Run alongside about 30 other microservices per project. BigQuery handles analytics and dashboard queries. Cloud Healthcare API is the FHIR datastore. Cloud Scheduler triggers the converter and about 50 scheduled queries across both projects. Then there is Secret Manager for credentials and checkpoints, Cloud SQL for application databases, a handful of Compute Engine VMs for data science and tooling, and Cloud Logging collecting output from everything.

At peak, we were processing over 500,000 records per day across 80 source tables. Two projects, two districts, running 24/7. That is a lot of moving parts generating cost, and before the spike, we had never looked carefully at which parts were generating how much.

Following the data to find the cost drivers

We did not start by guessing. We set up the billing export queries, pulled Cloud Run utilization data, and looked at INFORMATION_SCHEMA.JOBS_BY_PROJECT to understand where the money was actually going:

SELECT
  DATE(creation_time) AS dt,
  ROUND(SUM(total_bytes_billed) / POW(2, 40), 2) AS tib_billed
FROM `region-asia-southeast2.INFORMATION_SCHEMA.JOBS_BY_PROJECT`
WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
GROUP BY dt ORDER BY dt

This query gave us daily scan volume in TiB, which we could plot against the billing export to see exactly when costs started climbing and why. BigQuery was roughly 36% of the total bill. Cloud Run was another 21%. But the percentages alone did not explain the spike.

What the data showed was a compounding pattern. Several upstream data providers had pushed large historical backfills, which is a normal part of operating a health data platform. Migrations happen. New data sources come online. One upstream migration brought in 427,000 rows in one week (43x the normal volume). Another pushed 2.27 million rows in a single week.

The system handled it correctly. The converter processed the records, and the FHIR store grew. The main resource table went from 150 GiB to 201 GiB. But a deduplication scheduler was scanning the full FHIR store three times per hour. At 150 GiB, each scan cost very little. At 201 GiB and growing, the same scheduler became the single most expensive query in the system. The scan volume data on one project tells the story clearly:

Period	Daily BQ scan
Week 1	2.36 TiB
Week 4	4.0 TiB
Week 5	7.37 TiB
Week 6	9.26 TiB (peak)
After scheduler pauses	1.34 TiB
After all fixes	0.47 TiB

From 2.36 TiB to 9.26 TiB in five weeks, without anyone changing a single scheduler configuration. The data volume changed, and the cost followed. That same pattern was happening on the other project too, going from 1.1 TiB to 2.85 TiB daily, though the absolute numbers were smaller.

What the data told us to do

Once we had the billing and utilization data in front of us, we split the work across BigQuery, Cloud Run, Compute Engine, and the smaller services. Each fix started with a specific metric.

BigQuery: query analysis pointed to the converter polling pattern

We used INFORMATION_SCHEMA.JOBS_BY_PROJECT to rank queries by bytes billed. The top consumer was the converter's polling query, which checks for unprocessed records using an anti-join against a multi-million-row ledger table. Every poll for every table scanned the entire ledger.

The scan data told us a CLUSTER BY table_name on the ledger, combined with adding a table_name filter to the JOIN clause, would let BigQuery prune 90% of the scan. The change was small:

-- Before: scans entire ledger on every poll
LEFT JOIN processing_report AS B
  ON A.uuid = B.uuid

-- After: BigQuery prunes to just this table's slice (~94% fewer bytes)
LEFT JOIN processing_report AS B
  ON A.uuid = B.uuid AND B.table_name = 'source_table_name'

After deploying that change, per-query scan dropped from 531 MiB to 53 MiB.

We then looked at the pattern more carefully. The data showed that on a typical hourly run, most tables have zero new rows. We built a high-water mark routing layer: if a table has no backlog, use a cheap timestamp query instead of the full anti-join.

def get_unprocessed_data(table_name):
    """Routes to HWM or anti-join based on backlog check."""
    if table_name not in _HWM_READY_TABLES:
        backlog = _check_backlog(table_name)
        if backlog == 0:
            _HWM_READY_TABLES.add(table_name)
        else:
            return _get_unprocessed_antijoin(table_name)
    return _get_unprocessed_hwm(table_name)

The HWM path runs a simple WHERE timestamp > last_processed instead of joining millions of rows. This brought the converter's BigQuery cost down by over 95%.

BigQuery: evaluating Enterprise Edition after reducing query volume

Both projects were running BigQuery Enterprise Edition with autoscaling slots. Enterprise Edition pricing is based on slot usage, and when scan volume was high, the autoscaler was spinning up slots constantly. This was the single largest line item on the bill.

After deploying the clustering fix, the high-water mark routing, and pausing the expensive schedulers, our actual BigQuery compute usage dropped by over 90%. We looked at the numbers and asked whether Enterprise Edition still made sense. The autoscaling slots were now mostly idle, but we were still paying for the baseline. We deactivated Enterprise Edition on both projects and fell back to on-demand pricing.

The result was immediate. The projected BigQuery cost on on-demand is about 4% of what we were paying with Enterprise Edition. The query volume reductions we had already made meant that on-demand pricing was now far cheaper than even the minimum Enterprise Edition commitment. This single change, deactivating a pricing model that no longer matched our usage, was the largest cost reduction in the entire effort. There is a risk: if scan volume spikes again (another large backfill, or re-enabling the dedup scheduler), on-demand could become more expensive than Enterprise Edition was. We are monitoring on-demand costs weekly to catch that before it happens.

Scheduled query audit: frequency, dead joins, and dead code

We listed every scheduled query and its run frequency, then compared it against when the output data was actually consumed. The gaps were wide.

One dedup scheduler ran a UNION ALL full-scan of every FHIR resource table three times per hour. One scheduler, hundreds of dollars per month. We checked its downstream consumers and found it could safely be paused while we work on a better dedup approach.

We found schedulers with a dev- prefix running in the production project. One ran every 7 minutes, another ran every hour. They did not produce errors, so they never surfaced in monitoring. They only showed up when we audited the scheduler list against actual usage.

Export queries ran every 12 hours but were consumed once a day. Dashboard refresh queries ran every 5 hours but updated a table used in a daily report. We matched frequency to actual consumption patterns across both projects and paused or reduced 26+ schedulers.

Rewriting scheduled query SQL: the dead JOIN that ate 96% of slot-time

Frequency was not the only problem. Some scheduled queries were expensive because the SQL itself was inefficient.

Two scheduled queries that build a maternal health dashboard table started timing out at 6 hours. The SQL had not changed. The data had. Both queries contained a LEFT JOIN against a clinical orders table. The joined alias was referenced only in the ON clause, never in any SELECT or WHERE. It did nothing except multiply rows. The orders table stores one row per ordered lab test, so each visit record (full screening panel) fanned out 113 times. With 362,000 visit records, the intermediate result was 41 million rows before SELECT DISTINCT collapsed them back down.

This was harmless when the orders table was small. Then bulk conversion of two data sources landed 8 million new rows in one week. The fan-out blew past the 6-hour query timeout. When we queried INFORMATION_SCHEMA.JOBS_BY_PROJECT for slot usage, these two queries had consumed 96.1% of all project slot-time over the previous three days (62.6% and 33.5% respectively), leaving 3.9% for the 35,582 other queries.

The fix was one line: comment out the dead JOIN. Run time dropped from 6 hours (timeout) to 25 minutes. We also rewrote the partitioned update query to read from a 52 MB computed dashboard table instead of scanning 1 TB of raw FHIR tables directly. That cut its monthly cost from about USD 130 to USD 3.

The 4-billion-row table that should have been 12,000

One table had 4,040,397,903 rows. The actual data was 12,033 records.

A scheduled query had been running with WRITE_APPEND since February, duplicating every row on each run. The table grew to 764 GiB. This showed up when we sorted scheduled query outputs by storage size. We rebuilt the table from source, switching to WRITE_TRUNCATE. If we had not looked at storage sizes, this table would still be growing.

Cloud Run: utilization metrics vs. allocated resources

We pulled 30-day p99 CPU and memory utilization for every Cloud Run service across both projects. The gap between allocated and actual was large. One service had 4 CPU and 12 GiB allocated, with peak utilization at 12.3% CPU and 1.0% memory. Several services had similar profiles: provisioned for a load that never came, or provisioned for a peak that happened once and never returned.

Based on the utilization data, we downsized 44 services in one session, deleted 4 with zero traffic in 30 days, and switched 17 idle always-on services to request-based billing. The Cloud Run revision history shows it all happened in about 13 minutes. One project's daily compute cost dropped by 57% the next day.

We also right-sized the converter itself from 4 CPU / 16 GiB to 2 CPU / 4 GiB, and reduced the trigger frequency from every 30 minutes to hourly. During the backfill peak, the higher spec made sense. The converter was processing 571,000 records per day across 80 tables, running 20 worker threads with batches of 25 records. After the backfill cleared, utilization data showed the converter sitting well below capacity. We lowered workers and batch sizes to match the new steady state: 2 table workers, batches of 5, query limit down from 100,000 to 15,000.

Secret Manager: storage billing we were not tracking

We also went through every line item in the billing export, including services we assumed were cheap. Secret Manager was not cheap. Our patient deduplication service stores its checkpoint as a secret version, creating a new version on every run. Since late 2025, old versions were never destroyed. We found over 30,000 accumulated versions. The "Secret version replica storage" SKU was a top-10 line item on the bill. After cleanup, each checkpoint secret has 1 version. Cost dropped significantly.

Compute Engine: the GPU VM and the static IPs

Compute Engine was 6.6% of the bill. Most of that was expected: 10 VMs running various services (data science, form collection, OCR, migration tools). But one line item stood out.

A GPU VM with an NVIDIA L4 and 1 TB disk had been provisioned for OCR model fine-tuning. The fine-tuning work had finished weeks earlier. The VM was still running. It was the single most expensive Compute Engine resource. After confirming the fine-tuning was complete, we deleted it.

We also found unused static IP addresses being billed across both projects, some attached to services that no longer needed them. Small amounts individually, but they add up when nobody is watching.

Cloud SQL: fewer resources for the same workload

Cloud SQL was running multiple PostgreSQL instances 24/7. After reviewing actual query loads, we right-sized the instances. Total Cloud SQL savings: -36%.

Cloud Logging: paying for logs nobody reads

Cloud Logging on one project dropped 45% after reducing log levels and removing redundant logging. On the other project, logging actually increased 50% because the converter was processing more records after the backfill work. We still need to add exclusion filters there.

External API costs

We use LLM API calls for several features in the platform. One service makes an API call per record to validate field mappings during conversion. Another runs nightly to generate data quality summaries. A few others were experimental tools from earlier prototyping that were still hitting the API on a schedule. These API calls were costing more than Cloud Logging across both projects.

We reviewed each caller: how often it ran, how many API calls per run, and whether anyone was actually using the output. Two experimental tools had not had their output checked in over a month. The field validation service was running on every converter cycle, but we could batch the calls and reduce frequency without affecting quality. After removing the unused callers and batching the rest, the cost dropped 75%. The lesson here is the same as with schedulers: things that run automatically tend to keep running long after the reason for running them has passed.

Billing visibility: the foundation for everything else

We set up the GCP billing export to BigQuery in late April 2026. Before that, our only visibility was the monthly invoice and the GCP console billing dashboard. We could see the total was going up, but we could not see why. Once the billing export was in place, we could break down costs by SKU, by day, by service account. We could compare any two dates for a specific service and see exactly what changed. Every fix in this article started with a query against that billing export table. Without it, we would still be guessing.

The numbers

All numbers are from the GCP billing export. The daily cost is what matters, because it shows when each optimization took effect.

Daily cost trend (combined, both projects):

Period	Daily cost	What happened
Baseline	1x	Normal operations
Data volume growth	1.5x - 2x	Backfill volume hits BQ costs
Peak day	3x	Backfill volume + dedup cascade
BQ scans climbing	2x - 2.8x	Scheduler scans compounding
BQ fixes deployed	1.8x	Clustering + scheduler pauses
Cloud Run downsized	0.7x	Utilization-based downsizing
HWM deployed	0.6x	Converter BQ near-zero
Current	0.6x	Lowest point so far

From peak to current: an 80% reduction in daily cost, in six weeks.

Three optimization waves are visible in the data. First was BigQuery (clustering and scheduler pauses). Second was Cloud Run (utilization-based downsizing). Third was the high-water mark deployment.

Service-level changes (monthly):

Service	Change
BigQuery (Enterprise Edition)	-41%, then deactivated (Jul: ~0)
Cloud Run	-26%
Cloud SQL	-21%
Secret Manager	-26%
Cloud Healthcare API	+24% (data ingestion spike)
Cloud Logging (one project)	-45%
External API	-75%

Not everything went down, and that is worth understanding. Cloud Healthcare API increased 24%. The billing data explains why: upstream data providers were pushing large historical backfills during this period. One migration drove Healthcare API cost up 25x per day during the last two weeks of May. Another pushed daily Healthcare cost up 6x starting in early June. More records coming in means more FHIR writes, and that cost is proportional to actual data volume. It went up because the platform was doing what it was built to do.

Cloud Logging on one project increased 50% for a similar reason: more incoming data means more converter runs, more logs. We have not added exclusion filters there yet, so that is still on the list.

What we learned about the process

Start with the billing data. We did not guess which services were expensive. We exported billing data, queried INFORMATION_SCHEMA for scan volumes, and pulled utilization metrics from Cloud Run. Every optimization started with a number that told us where to look. The billing export showed us Secret Manager costs we would never have guessed. The utilization data showed us Cloud Run services at 1% capacity that had been running for months. Without the data, we would have started with the wrong services.

Schedulers are the silent cost multiplier. They get set up once and forgotten. We found scheduled queries running every 5 hours for dashboards refreshed once a day, and export queries running every 12 hours for data consumed once a week. The cost of each individual scheduler was small, but we had over 50 of them across both projects. Listing every scheduler alongside its actual consumer was the single most productive exercise we did.

The Cloud Run utilization review is a good example of how little effort some of these wins take. We spent about 30 minutes pulling p99 CPU and memory numbers for every service, sorted by gap between allocated and actual. That 30-minute exercise identified a large portion of total savings. The gap between "provisioned for the worst case" and "what actually happens" grows silently. We now plan to run this review monthly.

What is next

We still have optimization targets identified from the data. The FHIR export tables need partitioning by meta.lastUpdated so we can re-enable the dedup scheduler without recreating the scan cost problem. Cloud Logging on one project needs exclusion filters. Several Compute Engine VMs are running at low utilization. We also need to keep monitoring BigQuery on-demand costs to make sure they stay below what Enterprise Edition would have cost.

The daily cost is down 80% from peak. We are aiming to stabilize at the current level. The data will tell us when we get there.

Fixing 168K Failed FHIR Conversions with Parallel AI Agents and Git Worktrees

Budi Widhiyanto — Fri, 17 Apr 2026 16:30:19 +0000

One afternoon we ran a simple BigQuery query just to check how our FHIR converter was doing. Nothing special, we had been busy with new features for weeks and someone on the team just wanted to see the numbers.

The result was worse than we thought.

100.0% | tb_tracker_table_a          |  7,753 failures
 99.4% | nutrition_app_table_a       | 23,602 failures
 77.2% | hepatitis_app_table_a       | 22,437 failures
 43.2% | health_service_table_c      |  8,096 failures
 28.1% | health_service_table_a      | 168,713 failures

168,713 failed conversions on one table. Thirteen tables above 25% failure rate. We had a backlog that had been quietly growing while we were looking the other way.

If you have ever maintained a data pipeline, you know how this feels. The normal way to fix it is one table at a time. Pull some failed records, find the error, patch the template, run the tests, move on. Two or three hours per table when you know the codebase. Thirteen tables means weeks of focused work.

We had been experimenting with AI agents for this kind of maintenance work, and they were actually pretty good at it. The problem was that we could only run one agent at a time. Two agents on the same branch would step on each other almost immediately.

So we were running them one after another, which mostly defeated the point.

What finally unblocked us was a Git feature that had been sitting there for years and we had not really used: git worktree. This is the story of how we went from "we will fix this when we have time" to clearing the whole backlog in a few days.

a bit of context

Our team maintains a FHIR R4 converter for a national health platform. It pulls health data from local government systems and converts it into FHIR resources that flow into the central platform serving more than 2,000 healthcare facilities across two districts, from small community health posts to hospitals. Immunization records, maternal care, TB treatment, nutrition monitoring, community health screening. The data sources cover around 30 different table types between the two districts.

If you have not worked with FHIR before, the short version is this. FHIR R4 is the international standard for exchanging healthcare data. Every record has a strict structure, required fields, and value sets that the validator checks. A patient's gender cannot just be any string. It has to be one of the allowed codes. A date cannot be empty if the resource needs it. A coded value has to come from the right terminology system. If anything is wrong, the FHIR server rejects the resource and the record never enters the platform.

So when we say "conversion failure", we are not just saying the script crashed. We are saying clinical data from a local health facility never made it into the national system. Lab results, immunization shots, maternal visits. Gone, until we fix it.

Each data source has its own structure and its own quirks. One district had been getting less attention for a while. We did not ignore it on purpose. We always had to fix the most urgent problems first, and the rest kept piling up. You probably know how that goes.

So when we saw those failure numbers, the first reaction was honestly just tired. We knew this work was waiting. We just had not had the bandwidth to face it.

why one agent at a time was not enough

Before we get to worktrees, here is what the agent workflow actually looks like, because that is where the bottleneck became clear.

For every failing table, the steps are roughly:

Pull 100 failed UUIDs from BigQuery
Check data quality, fill rates, date format issues
Test UUIDs one at a time to find the actual error
Fix the template or the shared utility function
Run all 100, make sure at least 90% pass
Delete the failed rows from the report table so the system picks them up again

A note on the 90% threshold in step 5. We accept that the last few percent often come from genuinely bad source data that we cannot fix on our side. Records with corrupted fields, encoding issues from old systems, or data that should not have been entered in the first place. Chasing 100% on every batch means spending hours on records that are not really fixable. 90% is the threshold where we stop and move on.

This is the kind of work where an agent shines. It is not creative work. It is patient detective work that follows a clear pattern.

A concrete example. One table was failing because a date field was coming in as "2024-01-15 00:00:00" instead of "2024-01-15". Our convertStringToDate function expected %Y-%m-%d and returned empty when given the datetime format. Empty date meant a required FHIR field was missing. The validator rejected the resource. Record gone.

The agent found this in about 20 minutes. The third UUID hit the error, the agent walked back to the utility function, added a fallback for the extra time part, then ran the full test batch again to confirm. The actual code change was 4 lines, with a clear explanation of what was wrong.

Another table had a similar issue but in a different shape. A boolean field was arriving as the string "true". FHIR boolean fields expect an actual boolean primitive, so the validator rejected it with expected boolean: found "true". Same workflow. The agent found a function returning ["true"] instead of [True], fixed it, and checked that no other templates depended on the old behavior before merging.

So the per-table work was fine. The bottleneck was scale.

If we started one agent on nutrition_app_table_a and another on tb_tracker_table_a, they would conflict almost immediately. Both agents change tests/test_specific_uuids.py to set up their tests. Both might also touch shared functions in extra_logics/general.py if they find a common bug. On the same branch, in the same directory, they overwrite each other within minutes.

We tried the obvious workaround first. Separate clones of the repo, one per agent. It worked, but it was slow to set up, ate disk space, and made it annoying to share git history. Each clone had its own .git folder, its own remotes to configure, its own everything. We also forgot once to pull the latest main into one of the clones, and the agent fixed a bug that was already fixed on main. Wasted half an hour on that one.

Then we read the worktree docs properly and realized this was the tool we had been needing all along.

what worktrees actually do

If you are like us and have been using Git for years without touching worktrees, here is the short version.

A git worktree lets you check out multiple branches in separate folders at the same time. Each folder has its own files and its own changes. They all share the same .git folder.

git worktree add ../converter-fix-1 -b fix/nutrition-app-a
git worktree add ../converter-fix-2 -b fix/tb-tracker-a
git worktree add ../converter-fix-3 -b fix/hepatitis-app-a
git worktree add ../converter-fix-4 -b fix/health-service-a

Four folders, four branches, one repository. The extra storage is small because we are not copying the whole .git folder, only the working files.

We knew roughly what worktrees do. What we did not realize until recently is that this is exactly what we needed for parallel agents. Each agent gets its own folder. They never see each other's files. Agent 1 can be halfway through testing nutrition_app_table_a while Agent 2 is just starting on tb_tracker_table_a. No conflicts, no waiting for each other.

This is the kind of feature that is great to know about even if you do not need it today. The day you do need it, you will remember it is there.

how we organized the actual run

With 13 failing tables, we sorted them by how many failed conversions they had. We also checked which tables share the same template file, because fixing one template sometimes fixes several tables at once. Worth doing this step. We saved a lot of duplicate work this way.

The first batch had four worktrees running at the same time:

worktree	table	failed conversions
1	health_service_table_a	168,713
2	health_service_table_b	45,707
3	nutrition_app_table_b	28,977
4	hepatitis_app_table_a	22,437

Each agent got a prompt like this:

please create a new worktree for fixing health_service_table_a,
then run the fix-converter-failures workflow.

TABLE_NAME=health_service_table_a
CODE=health_service_code_a
BQ project=your-project

After the process is READY TO DEPLOY, merge back to the main branch,
commit and push, then delete the worktree.

The prompt looks short because fix-converter-failures is a workflow file we maintain in the repo. It defines the 6 steps above and the conventions the agent should follow, so the prompt itself only needs to say which table to run it on. Building this workflow file took us a few iterations, but once it was stable, kicking off a new fix became a one-line task.

The agents ran independently. While worktree 1 was checking fill rates on the patient service table, worktree 2 was already on its third UUID. When one agent found a bug in a shared utility function, it fixed the bug in its own branch, and the other agents kept going.

The first batch did not go perfectly. We hit a small problem on the second day. Two agents found the same bug in convertStringToDate at almost the same time, and both fixed it slightly differently. One added a fallback, the other rewrote the function to use dateutil. Not a real conflict, since each fix was on its own branch, but during merge we had to pick one and revert the other. We added a rule for the next batch: when an agent touches shared code, it has to flag that in the PR description so we know to check for parallel fixes.

After the first batch finished and merged, we started the next four. The whole experience felt more like reviewing work than doing it. Honestly, this took a moment to get used to. After a few rounds, the rhythm became natural. Open four terminals, kick off four agents, let them run, come back to review.

the other half: template coverage

Fixing failures was one problem. The other quality issue was something we did not even have a name for at first. We were calling it "silent data loss".

We have a coverage check that compares each template against its data dictionary and counts what percentage of the source fields are actually mapped to FHIR. If it is below 90%, the template needs more work.

With 15 templates to check, we used the same approach. Four worktrees, grouped by data source type:

worktree 1: maternal death reporting templates
worktree 2: nutrition monitoring (three templates)
worktree 3: community health screening (four templates)
worktree 4: immunization and TB treatment

Agents ran coverage analysis on each template. If it passed 90%, move on. If not, add the missing fields, test, validate against the FHIR server, merge, delete the worktree.

This step found a different kind of problem than the failure fixes, and honestly the more important one. A template can have a 0% failure rate and still be missing half the fields it should be mapping. The records were converting successfully and reaching the FHIR server clean. But blood glucose, cholesterol, abdominal circumference, lab results, risk factor flags, procedure details, all of it was being silently dropped because the template never mapped those fields.

The failure rate query does not show this. The FHIR server does not show this. From every monitoring view we had, the system looked healthy. But the clinical data we were supposed to be capturing was not actually arriving.

This part was uncomfortable to look at. We had been measuring success by "no errors" without checking if we were actually capturing the data correctly. Some templates were already above 90% and needed no changes. Others were well below.

One community health screening template was correctly mapping vital signs but was missing several fields that the screening program actually collects. Blood glucose, cholesterol, abdominal circumference. All present in the source data, all listed in the data dictionary, just never connected to FHIR Observation resources. The agent added them, ran the tests, validated against the FHIR server. That kind of work would normally sit in a backlog for weeks.

A small thing we did not expect. Some data dictionaries were split into multiple CSV files. One screening app reference came in four separate files with around 1,400 rows total. The agent had to combine them before running the analysis. Worth knowing before you start.

We also had one false positive. The agent reported 78% coverage on a template that was actually fine. The data dictionary listed fields that were deprecated and no longer used in production. We had to manually verify the missing fields before adding code for them. Coverage numbers are useful, but the data dictionary itself can be stale, and the agent cannot know that without help.

a few things worth knowing if you try this

Sharing what tripped us up so you do not have to learn the same way.

Worktrees share your git history and remotes. A commit you make in ../converter-fix-2 will show up in git log from your main folder. git push works normally from any worktree. This is convenient once you are used to it, but it can feel weird at first.

Config files that are not committed need to exist in each worktree folder separately. Credential files, .env, anything in .gitignore, each worktree needs its own copy. We learned this the hard way when the first agent could not connect to BigQuery and we spent 15 minutes thinking the credentials were wrong. We keep a short setup note for this now.

Name your worktrees after what they are doing, not just numbers. ../fix-nutrition-app is much easier to work with than ../worktree-2 when you have four terminals open at the same time. Trust us on this.

Cleanup is one command:

git worktree remove ../converter-fix-1

The folder is removed and the worktree reference is cleaned up. The branch and its commits stay in the repository, in case you need to revisit them later.

For us the real lesson from this work was not about worktrees, and not even really about agents. It was about what we were measuring.

For months we had a dashboard that said the converter was healthy. Failure rates were low on most tables, the FHIR server was accepting resources, alerts were quiet. The dashboard was not lying. It was just answering a different question from the one that mattered for healthcare data. "Is the conversion working" is not the same as "is the clinical data arriving correctly". We were tracking the first one and assuming it answered the second.

Worktrees and agents did not solve that. They just made it cheap enough to fix once we noticed.

If you are working with agents on tasks that are similar and do not depend on each other, the parallel worktree pattern is worth trying. The setup takes a few minutes. The harder part is changing how you think about the work. From one task at a time to groups of tasks running side by side. That feels strange at first and gets natural quickly.

For us, the result was clearing months of backlog in a few days, plus the coverage findings we did not know we were missing. The agents are not magic. We still review every change carefully, especially anything that touches shared code or terminology mappings. But letting them work in parallel changed what was actually possible for our team, on a system that more than 2,000 healthcare facilities depend on.

How I Built an AI Agent That Writes FHIR Templates in Hours Instead of Days

Budi Widhiyanto — Sat, 11 Apr 2026 15:49:03 +0000

3-4 days to write one JSON template. Sometimes a full week.

It was a Tuesday night and I had the FHIR specification open in twelve browser tabs. On my left monitor, a CSV data dictionary with 150 fields from a Kobo maternal health form, postnatal care, the kind that tracks five KF checkups for the mother and four KN visits for the newborn. On my right, a half-finished JSON template already 200 lines deep. I wasn't even past the Encounter resource yet.

Each visit block needed its own set of vital signs: blood pressure (systolic and diastolic, split from a combined "120/80" string), pulse rate, respiratory rate, temperature. Each needed clinical observations like uterine fundal height, lochia status, breastfeeding assessment. Each needed the right LOINC code, the right data type handler, the right reference chain connecting it back to the Patient and Encounter it belonged to. Five mother visits times eight to twelve observations each, plus four newborn visits with their own observation sets, plus the baby Patient resource with mother-baby linkage via NIK identifiers.

I did the math. This template would land somewhere around 320 resources and several thousand lines of JSON.

I'd done this before. Many times. Four days minimum, probably five. And somewhere around day two I'd lose my mental model of the reference graph, come back the next morning staring at field mappings I no longer remembered the reasoning behind, spend an hour just re-orienting before I could write another line.

That night I closed the FHIR spec tabs and started building the agent instead.

The Old World

Some context. I'm a software engineer working on health data interoperability for district health systems in Indonesia, specifically Purbalingga and Lombok Barat. We pull data from over a dozen source systems: ePuskesmas (community health center records), Kobo Toolbox (mobile data collection forms), SiGizi (nutrition surveillance), SITB (tuberculosis registry), SIHEPI (hepatitis tracking), SIMRS (hospital information systems), and more. Each system has its own data model, its own field names, its own quirks. Our job is to convert all of it into FHIR (Fast Healthcare Interoperability Resources), the international standard for healthcare data exchange.

FHIR, if you haven't worked with it, models healthcare data as discrete "resources." A patient is a Patient resource. A doctor visit is an Encounter. A blood pressure reading is an Observation with a specific LOINC code (85354-9), split into systolic and diastolic components, each coded separately. A diagnosis is a Condition. A vaccination is an Immunization with a CVX vaccine code and dose protocol. Every resource links to others through references: an Observation points to the Patient it belongs to, the Encounter it was recorded during, the Organization that performed it.

The conversion pipeline itself isn't the hard part. We have a Python engine that reads a JSON template, maps incoming data fields to FHIR resource structures, runs transformation functions, and POSTs the result as a FHIR Bundle to Indonesia's national FHIR server (SatuSehat). The hard part is writing the template.

A template in our system is a JSON file that describes every FHIR resource the converter should produce from a given data source. Here's what even a simple resource definition looks like. This is a real Patient resource from a pregnancy tracking form:

{
  "resourceType": "Patient",
  "fullUrl": "urn:uuid:patient",
  "keyField": "[nik_ibu]",
  "mergeWithExistingData": true,
  "requiredFields": {
    "id": "[generated/patient_res_id]",
    "identifier.0.system": "https://fhir.kemkes.go.id/id/nik",
    "identifier.0.value": "[generated/nik_adjusted]",
    "name.0.text": "[nama_ibu]",
    "birthDate": "[generated/birth_date]"
  },
  "extraLogics": [
    {
      "type": "getPatientIdWithFuzzyLogic",
      "input": ["", "[generated/nik_adjusted]",
               "[nama_ibu]", "[generated/birth_date]",
               "female", ""],
      "output": ["[patient_existing_id]"]
    }
  ]
}

The [square_bracket] notation maps to incoming data fields. The [generated/...] notation refers to values produced by transformation functions, called extraLogics, that run before the FHIR resource is assembled. In this example, getPatientIdWithFuzzyLogic queries the FHIR server to find if this patient already exists (matching on national ID, name, birth date, and gender).

That Patient resource is one of two resources in the simplest template in our system: 279 lines of JSON total. Our hospital templates run to 300+ resources and nearly 500KB. The postnatal care form that broke me that Tuesday night had 320 resources across about 5,000 lines.

Manual template creation took 3-4 days of wall-clock time. Sometimes a full week. And that's with the hardest intellectual work already done for me.

Before I write a single line of JSON, our health informatics team has already spent their own days (sometimes weeks) building the CSV data dictionary. They're the ones who decide that berat_badan maps to LOINC 29463-7 "Body weight," that tekanan_darah should become a blood pressure panel with code 85354-9, that this particular field is a SNOMED-coded Condition and not just a free-text Observation. That work requires clinical domain knowledge I don't have. They understand what these data points mean in a maternal health context, which FHIR resource types are clinically appropriate, which code systems are correct. I turn their decisions into working JSON. Without that CSV, I'd be guessing at clinical semantics, and guessing wrong.

So even with a carefully prepared data dictionary in hand, the template creation itself still took 3-4 days.

The first day was mapping. Read the CSV line by line. For each field (berat_badan, tekanan_darah, tinggi_fundus, denyut_jantung_janin) decide which FHIR resource type it becomes. Observation for vitals, Condition for diagnoses, Procedure for clinical actions. Pick the clinical code: LOINC 29463-7 for body weight, SNOMED 364589006 for fundal height measurement. Choose the data type handler. Numeric fields need handleNumericValueOrDataAbsentReason, which returns four outputs: the value (or null), plus the data-absent-reason system, code, and display for when the field is empty. String fields, dates, booleans, coded values all have their own handlers.

Day two was wiring. Connect everything. Every clinical resource needs a subject reference pointing to the Patient, an encounter reference pointing to the Encounter, NIK (national ID) traceability in subject.identifier, and a meta.source field for audit provenance. Every numeric Observation needs addValueIfNotEmpty guards on the unit triple (valueQuantity.unit, valueQuantity.code, valueQuantity.system) because if the value is absent, emitting an empty valueQuantity object alongside a dataAbsentReason violates FHIR's XOR constraint. I'd forgotten this guard at least twice in earlier templates and spent hours debugging why the FHIR server rejected my bundles.

Day three was debugging. Run the converter against real sample data. Watch it crash. Trace the error to a missing generateUUID extraLogic, or an output array that doesn't match the function's return count, or a reference to a resource that was conditionally skipped because its keyField was empty in the sample. Fix, re-run, fix, re-run. Each cycle takes a few minutes but there are dozens of them.

If it stretched to a week, day four was review and edge cases. Check that every field in the data dictionary actually made it into the template. Verify code systems (is it http://loinc.org or http://loinc.org/? The trailing slash matters). Handle the weird cases: blood pressure stored as a single "120/80" string that needs to be parsed into two separate Observations with component coding. Baby-mother linkage where the newborn's Patient resource needs a nik-ibu identifier pointing back to the mother.

The worst part wasn't any single step. It was the context loss. Building a 200-resource template across multiple days means losing your mental model every night. You come back the next morning, open the half-finished JSON, and spend the first hour asking yourself: "Why did I use handleCodeableConceptOrDataAbsentReason here instead of handleValueOrDataAbsentReason? Was that intentional or a mistake?" The template is too large to hold in your head. You're constantly re-deriving conclusions you already reached yesterday.

Why an Agent, Not a Script

I didn't set out to "automate FHIR mapping with AI." I set out to make this work fit in a single focused session.

A Python script was my first thought. But template creation isn't a mechanical transformation; it requires judgment. Is letak_janin (fetal position) a coded Observation or free-text? Depends on whether we have a SNOMED mapper for the values. Should komplikasi_persalinan be a Condition resource or an Observation? Depends on whether the field represents a diagnosis or just a documentation note. These decisions vary across data sources and there's no algorithm for them.

My second thought was a one-shot LLM prompt. "Here's a CSV, here's a sample, generate a FHIR template." I tried it. The output looked plausible but failed in subtle ways: wrong function signatures for extraLogics, missing addValueIfNotEmpty guards, LOINC codes that didn't exist, references to resources that were never generated. A FHIR template is a program that runs through our converter engine, and it has to be mechanically correct, not just structurally plausible.

An agent was the right shape. FHIR mapping requires judgment (which resource type? which code?) interleaved with lookup (does this LOINC code exist? does this extraLogic function exist in our codebase?) followed by validation against real data. That loop, think-look up-try-validate-fix, is exactly the kind of work agents handle well.

My honest motivation wasn't full automation. I wanted to compress the tedious parts, choosing codes and wiring references, so my judgment could focus on the edge cases and domain-specific decisions that require understanding what the data actually means in a clinical context.

Anatomy of the Agent

The agent is built as a set of Claude Code custom slash commands: markdown files that define prompts, tool access, and autonomy rules. They chain together into a pipeline. I went through two major iterations, an original 7-step pipeline and a streamlined 3-step version I use today.

The agent receives three files as input:

First, a reference CSV, which is a data dictionary mapping field names to FHIR resource types, clinical codes, and paths. Our health informatics team builds these, and they're the foundation everything else depends on. Each row represents a clinical judgment call: this field is a vital sign Observation, that one is a Condition, this code is LOINC, that one is SNOMED. A typical row: berat_badan | Body weight in kg | Observation | valueQuantity.value | LOINC | 29463-7 | Body weight | numeric. The agent can wire JSON and pick handlers, but it can't decide whether a field is clinically meaningful or which code system is appropriate. That's what the CSV encodes.

Second, a sample JSON record, a real data row from BigQuery showing actual field values and types. This tells the agent what the data looks like in practice, not just in theory. It reveals things the CSV doesn't: that tekanan_darah is actually a combined "120/80" string, that usia_kehamilan is an integer, that some fields are null in real data.

Third, a reference template, an existing structure/*.json template for a similar data source. The agent uses this to extract project-specific patterns: how we do patient fuzzy matching, which regional helper functions exist (like getPurbalinggaVillageId), what the EpisodeOfCare strategy looks like for this program type.

Step A: Plan and Resolve

The first step is 80% deterministic Python, 20% LLM for genuinely ambiguous cases. It parses the CSV and sample data, cross-references them, resolves code systems from friendly labels to canonical URLs ("LOINC" becomes "http://loinc.org"), detects the appropriate data-absent-reason handler per field based on its type, and identifies the unit of measure from the CSV description.

The thing that makes this step work is a defaults file, twelve pre-resolved design decisions that the agent must follow without asking me:

## Coded Fields Without an Existing Mapper

When a field has coded values but no mapper function exists:
Default: use handleValueOrDataAbsentReason, store as valueString.
Do NOT stop to ask. Do NOT invent a mapper.

## Baby / Neonate Patient Linking

When the template has a second Patient for the baby:
Default: always add mother-baby linkage without asking.

## Blood Pressure from Combined String

When tekanan_darah is a combined "120/80" string:
Default: two separate Observations (systolic + diastolic),
each with parseSystolicBP / parseDiastolicBP.

I wrote this file after running the first version of the agent and noticing it kept stopping to ask me the same twelve questions across different templates. "Should I use valueString or valueCodeableConcept for this field?" valueString, every time, unless we have a mapper. "Should I add baby-mother linking?" Yes, always. "What should period.end be when there's no discharge field?" Same as period.start. These aren't judgment calls. They're conventions. Encoding them as defaults lets the agent run with far less hand-holding.

The output is an enriched_mapping.json with every field fully typed, every code resolved, every handler assigned.

Step B: Generate

This is where the template gets written. One continuous pass, following a strict resource ordering: Patient first, then EpisodeOfCare, Encounter, Conditions, vital sign Observations, clinical assessments, numeric measurements, datetime Observations, obstetric history, boolean/coded Observations, Procedures, baby Patient with mother linking, baby Observations, QuestionnaireResponse for risk factors, and any remaining resources. Seventeen groups in total, generated without pausing.

The agent assembles each resource from validated skeleton patterns. Here's a simplified skeleton for a numeric Observation, the most common resource type in a typical template:

{
  "resourceType": "Observation",
  "keyField": "[generated/observation_datetime]",
  "extraLogics": [
    {"type": "generateUUID",
     "output": ["[obs_id]"]},
    {"type": "returnPatientReference",
     "input": ["[generated/patient_res_id]", "[nama]", "[nik]"],
     "output": ["[patient_ref]", "[patient_display]", "[patient_nik]"]},
    {"type": "handleNumericValueOrDataAbsentReason",
     "input": ["[berat_badan]"],
     "output": ["[weight_value]", "[weight_absent_system]",
               "[weight_absent_code]", "[weight_absent_display]"]},
    {"type": "addValueIfNotEmpty",
     "input": ["[generated/weight_value]", "kg"],
     "output": ["[weight_unit_code]"]}
  ],
  "requiredFields": {
    "code.coding.0.system": "http://loinc.org",
    "code.coding.0.code": "29463-7",
    "code.coding.0.display": "Body weight",
    "subject.reference": "[generated/patient_ref]",
    "subject.identifier.value": "[generated/patient_nik]"
  },
  "optionalFields": {
    "valueQuantity.value": "[generated/weight_value]",
    "valueQuantity.unit": "[generated/weight_unit_code]",
    "dataAbsentReason.coding.0.code": "[generated/weight_absent_code]"
  }
}

Notice the addValueIfNotEmpty guard on the unit field. Without it, when the numeric value is absent, the handler correctly sets dataAbsentReason, but the unit field still emits an empty valueQuantity object, violating FHIR's constraint that a resource cannot have both value[x] and dataAbsentReason. This bug bit me three times in manual templates before I documented it in failure-modes.md and baked the guard into every pattern.

The strongest autonomy rule, repeated in every step:

Never ask "ready to continue?" or "shall I proceed?". Just proceed.

Before finishing, the agent runs a coverage check, verifying that every field in the sample data is mapped somewhere in the template. Target: zero uncovered fields.

Step C: Validate and Fix

The final step runs the actual converter against real sample data, then passes the output through a 7-layer validator that checks: FHIR R4 schema compliance, reference closure (every referenced resource exists in the bundle), mandatory fields per resource type, NIK traceability and meta.source format, code system terminology (LOINC/SNOMED/ICD-10/CVX existence), value consistency against the sample data, and helper/handler coverage. The first six layers flag errors; the last one flags warnings.

If errors exist, the agent reads only the error report (not the full bundle, not the full template), applies surgical fixes to the specific failing resources, and re-runs. Maximum two retry cycles. If errors persist after two passes, it stops and surfaces the remaining issues for me to fix manually.

A completed run produces four things: structure/<name>.json (the production-ready template), updated settings.py and settings_local.py (registering the new data source), bundle.json (a test FHIR Bundle from the sample data), and validation_report.json (showing zero errors across all layers).

Where the Template Lives

The agent's output, a JSON template, plugs into a larger system.

BigQuery holds raw health data across 50+ tables, ingested from source systems across Purbalingga and Lombok Barat districts. The orchestrator queries for unprocessed records and farms them out in parallel batches (15 workers, batches of 25 records). For each record, the converter loads the matching template from structure/, runs the extraLogics transformation chain, maps fields to FHIR paths, resolves references between resources, and assembles a FHIR Bundle. The bundle goes to Indonesia's SatuSehat FHIR server via POST. Success and failure get logged back to BigQuery.

The core of the converter, where the template becomes a FHIR resource:

def merge_resources_and_build_references(fullUrl, json, resource,
                                         references_data):
    existing_json = {}
    if resource.get("mergeWithExistingData", False):
        if "id" in json:
            existing_json = get_existing_data_with_id(
                resource["resourceType"], json['id'])

    resource_json = merge_nested_dicts(existing_json, json)

    references = copy.deepcopy(resource["references"])
    if "dynamicReferences" in resource:
        for ref in resource["dynamicReferences"]:
            if ref["key"] not in references_data:
                continue
            # ... build conditional reference

    resource_json = merge_dicts_with_list_values(
        resource_json, references)
    return {
        "fullUrl": fullUrl,
        "request": {"method": "POST",
                    "url": resource["resourceType"]},
        "resource": resource_json
    }

The agent unblocks this pipeline. Everything upstream (data extraction) and downstream (conversion, FHIR submission, reporting) is mechanical. Before the agent, adding a new data source meant I couldn't take on other work for three to four days. Now it means an afternoon.

The Payoff

3-4 days of wall-clock time down to 3-6 hours.

About 30 minutes goes to setup: extracting a sample record from BigQuery, prepping the CSV data dictionary (our health informatics team builds these, but I sometimes need to clean up column names or add missing code system labels), picking a reference template close to the new data source.

1-3 hours for the agent pipeline and review. Run the three steps, then read through the generated template. Check code selections ("did it pick the right LOINC code for fundal height?"), verify the extraLogics chains make sense ("is it using the right date parser for this data source?"), spot-check the reference graph ("do all Observations point to the right Encounter?").

Then 2-3 hours of testing and manual fixes. Run the converter against real sample data. Fix the things the agent got wrong, usually a transformation function with the wrong number of arguments, or a reference to a field name that doesn't match the sample. Re-run until the validation report comes back clean.

I still read every template. But the review is different from building from scratch. When I built manually, I was making decisions and implementing them simultaneously: the cognitive load of "what should this be?" stacked on top of "how do I express that in our template format?" Now I'm only doing the first part. The agent handles the expression.

One moment that sticks with me: the agent generated a template for a nutrition surveillance form (sigizi_balita_dipantau_pmt) with about 60 Observations covering growth monitoring, nutrition status, and supplementary feeding data. In review, I noticed it had correctly applied addValueIfNotEmpty guards on the unit fields for every single numeric Observation. All sixty of them. That's a pattern I'd missed in at least two earlier manual templates, which led to hours of debugging FHIR constraint violations when the server rejected bundles where a patient had no weight recorded but the template still emitted an empty valueQuantity alongside the dataAbsentReason.

The agent got this right because I'd documented the bug in failure-modes.md after debugging it the hard way, and baked the fix into the skeleton patterns. My past pain became the agent's default behavior.

How It Evolved

The first version of the agent had seven steps: analyze inputs, generate a base template at ~40% coverage, evaluate coverage, implement missing fields, test structural integrity, validate quality, check data consistency. Each step was its own Claude Code command with its own system prompt.

It worked. Templates came out correct. I used it for weeks, and it was a real improvement over manual work. But after running it on maybe a dozen templates, I started noticing where the friction was.

Token cost added up. Seven LLM invocations, each re-reading the template, reference files, and sample data. The pipeline consumed roughly 107,000 tokens per run, about 4,773 lines of markdown across the seven command files. The generate-then-measure-then-patch loop (steps 2, 3, 4) felt especially redundant after a while: why generate an incomplete template at 40% coverage just to measure what's missing and patch it? I had enough experience by then to know the agent could do it in one pass.

Confirmation prompts wore me down. Despite instructions not to, the agent kept pausing between resource groups. "I've generated the vital signs Observations. Ready to continue with clinical assessments?" The first few times I didn't mind. After the twentieth template, pressing enter seventeen times per run felt like exactly the kind of tedious interruption I'd built the agent to eliminate.

Small quality issues compounded. The old data consistency validator (step 7) checked whether sample values appeared in the generated bundle using Python's in operator on the serialized JSON string: value in json.dumps(bundle). The value "1" matched every UUID containing the digit 1. Not a dealbreaker, but it meant an extra fix-and-retry cycle on most runs, and after a few weeks those extra cycles added up.

None of these were really wrong. The 7-step pipeline was a huge improvement over manual work. But once you've lived with a tool long enough, you start seeing the next version. I refactored it over a weekend, less out of frustration than the natural instinct to tighten something you use every day.

The 3-step version changed three things. One-pass generation replaced the generate-measure-patch loop: Step B produces the complete template in a single continuous pass. Pre-resolved defaults reduced confirmation prompts: the defaults.md file encodes every recurring design decision, and Claude still asks about some things (tool calls, file writes, ambiguous decisions outside the defaults) but the constant back-and-forth between resource groups is gone. Typed path-aware validation replaced substring matching: the new validator does type-aware comparison (float() cast for numerics, ISO normalization for dates, case-insensitive for strings) instead of value in json.dumps(bundle).

Token usage dropped by roughly 70%. Fix-and-retry cycles went from 2-3 typical to 0-1. It's not fully hands-off; I still sit with the agent, approve tool calls, and occasionally steer it when it goes down a wrong path. But the interruptions are meaningful now. Not mechanical.

When the Agent Gets It Wrong

The agent still makes mistakes. The most common: picking a transformation function that's incompatible with the actual data shape. Using handleNumericValueOrDataAbsentReason on a field that the CSV says is numeric but the sample data reveals is actually a string like "Normal" or "Tidak ada". The handler tries to cast it to a float, throws an exception, the converter crashes.

Another recurring issue is wrong argument counts in extraLogics. Our transformation functions have strict input/output contracts. getPatientIdWithFuzzyLogic takes exactly 6 inputs and returns 1 output. If the agent wires 5 inputs, the function throws an unexpected argument count error at runtime. Step C's test run always catches this, but it requires a manual fix to figure out which argument was omitted.

Sometimes the agent chooses a structure that looks right but doesn't fit how the converter actually processes the data. A field that should be a requiredField ends up in optionalFields, or vice versa, and the missing-value behavior changes in ways that only show up with certain input records. These are the subtlest bugs. They pass validation against the sample data but fail on edge cases in production.

These bugs are fast to fix because the error messages are specific and the template structure is already complete. Patching a single resource definition, not rebuilding from scratch. A 10-minute fix versus a 4-hour reconstruction.

The lesson from building two versions: the first agent you build optimizes for correctness. The second optimizes for cost and autonomy. Both matter, but you don't know which autonomy rules you need until you've run the first version enough times to feel the friction. Every line in defaults.md represents a specific moment where the agent's behavior cost me time, either by asking an unnecessary question or failing in a way I'd already seen before.

What Changed

I now support 60 template configurations across ten Indonesian health programs: hospital systems, community health centers, mobile data collection, nutrition surveillance, tuberculosis tracking, hepatitis monitoring. The templates range from a 2-resource pregnancy tracker (279 lines) to a 332-resource hospital ANC integration (490KB of JSON). Every one of them was built using the agent pipeline.

Before the agent, I had to defer new integrations. Each new data source meant committing to a multi-day block of focused work, and there was always something more urgent. A new nutrition monitoring form would sit in the backlog for weeks while I finished a hospital template. Now, when the health informatics team hands me a new data dictionary, I can have a working template the same day. That changed what integrations I was willing to take on, and how quickly new health programs could start reporting data through the national FHIR infrastructure.

My job shifted from writing thousands of lines of JSON by hand to designing the system that writes it and then reviewing the output with the kind of attention I couldn't sustain across a four-day manual effort.

The most valuable thing I built was defaults.md, twelve pre-resolved design decisions that let the agent run without constantly asking me questions. Writing that file forced me to articulate judgment calls I'd been making intuitively for months. Which EpisodeOfCare strategy for which program type. How to handle coded fields without an existing mapper. When to split blood pressure into components versus a single observation. These decisions were in my head, applied inconsistently, never written down.

But I want to be clear about something. The agent automates my part of the work: turning clinical mappings into working FHIR templates. It does not automate the clinical thinking that comes before it. Every template starts with a CSV that our health informatics team built, row by row, mapping raw field names to the right FHIR resource types and clinical codes. They're the ones who know that denyut_jantung_janin is LOINC 55283-6 and not some other fetal assessment code. They're the ones who decide when a field warrants its own Condition resource versus being an Observation. Without that upstream work, I'd have nothing to automate. The agent compressed my 3-4 days into an afternoon, but it didn't touch the clinical expertise that makes the output meaningful.

It turns out the hard part of building an AI agent isn't the AI. It's figuring out what you actually know, and writing it down clearly enough that something else can apply it. And sometimes, it's recognizing that the knowledge you depend on most isn't yours at all.

National Vaccine Appointment & Administration System

Budi Widhiyanto — Sat, 28 Feb 2026 09:30:45 +0000

🌱 How It Started

Few Years ago, I had a system design interview. The interviewer gave me this scenario:

"Design a national vaccine appointment booking system. Millions of citizens need to register and book slots. Clinics must administer the doses. The government needs audit logs and fraud prevention."

My first thought was simple just let people book a slot, check the stock, and confirm. I drew a basic flow on the whiteboard and felt pretty good about it. Then the interviewer started asking harder questions.

"What if two people try to book the last slot at the same time?"

"What if the clinic runs out of doses after the booking is already confirmed?"

"How do you undo things if eligibility check fails in the middle?"

I didn't have good answers. I only designed for the happy path.

That interview stuck in my mind. Months later, I was doing research on inventory reservation patterns for an internet credit purchase system, and I realized the same ideas could have helped me in that interview. So I went back to the problem and redesigned it. This is what I came up with.

⚡ My Initial (Naïve) Solution

Here's what I proposed during the interview:

Simple, right? But the problems come fast:

Race conditions: Two people click "Book" at the same time for the last slot. Both get confirmed. Now one citizen has no seat.
Stock mismatch: Slot is confirmed, but the clinic ran out of vaccine doses between booking day and appointment day.
Late eligibility failure: System confirms appointment first, then finds out the citizen doesn't meet age or insurance requirement. Now you need to undo everything, but stock is already allocated.
No rollback: If something fails in the middle, there's no way to release the slot or dose back to the pool.

These are the same problems I found later when designing the internet credit purchase system the happy path is not enough when you deal with limited resources and many users at the same time.

🔍 Rethinking the Flow

The main idea, which I learned from inventory reservation strategies in e-commerce, is: don't confirm anything until everything is verified. Use a multi-stage process temporary hold first, then verify, then confirm. If anything fails, rollback.

It's like buying concert tickets. When you select a seat, it's held for you while you pay. If you don't finish in time, the seat goes back. Same concept here.

Here's the full flow of the improved design:

🧩 The Improved Design

1. Reserve First (Temporary Hold)

When a citizen selects a clinic, time slot, and vaccine type, the system does not confirm right away. Instead:

It creates a temporary reservation in Redis with a TTL (time-to-live), for example 5 minutes.
Appointment status is set to PENDING.
Slot capacity and vaccine dose count are decreased temporarily other users will see less availability.

Why Redis? Because we need something fast and temporary. A relational database could work too, but you would need a separate scheduled job to clean up expired reservations. Redis handles this automatically with TTL when 5 minutes pass, the key just disappears. For a system that handles millions of bookings during a national vaccine campaign, this performance difference is important.

How to handle race condition on Redis? We use Redis DECR command on the slot counter. This is atomic meaning if two requests come at the same time, Redis processes them one by one. If the counter reaches zero, the next request is rejected. For extra safety, you can use a Lua script to make the check-and-decrement happen in one step.

2. Eligibility Verification

While the slot is held, the system runs eligibility checks:

Age requirement (e.g., some vaccines only for 60+).
Insurance verification through external API.
Medical history (allergies, previous doses).
Geographic check (is this citizen in the right region?).

If any check fails, the reservation is released Redis key is deleted, slot goes back to the pool. The citizen gets a clear message explaining why they are not eligible, not just "something went wrong."

3. Confirm Appointment

If all checks pass:

Slot capacity and vaccine stock are decreased permanently in the main database.
Appointment status changes from PENDING to CONFIRMED.
Redis reservation is cleared (not needed anymore).
Confirmation is sent to the citizen (SMS, email, or push notification).

This is the point of no return. Before this step, everything can be undone.

4. Administration (Vaccination Day)

When the citizen arrives at the clinic:

Clinic staff scans the citizen's QR code. The QR code contains the appointment ID and a verification hash. The hash is generated on the server using appointment ID + citizen ID + a secret key, so it cannot be faked.
System verifies the QR code against the appointment record.
Staff records the vaccine batch number and time of administration.
Appointment status changes to ADMINISTERED.
An event is sent to other systems analytics, government reporting, audit logs.

5. Failure & Rollback Scenarios

This is the part I completely missed in my interview. Here's how each failure is handled:

No-show: A scheduled job checks for CONFIRMED appointments that passed their time window. Status becomes NO_SHOW, stock is released back.
Citizen cancels: They can cancel through the portal. Stock is released right away.
Clinic cancels a slot (e.g., not enough staff): All affected appointments are flagged. Citizens get notified and can rebook with priority.
External API is down (e.g., insurance service): The system uses a circuit breaker pattern. After several failures in a row, the system stops calling that API temporarily. Meanwhile, the booking is either queued for retry (with increasing wait time between retries) or allowed provisionally with a flag for manual review later. The important thing is: one broken dependency should not block the whole flow.
Redis goes down: The system falls back to database-level reservations with a cleanup job. It's slower, but the booking still works.

🏗️ System Components

Here's the high-level architecture:

Frontend: Booking portal for citizens + Dashboard for clinic staff.
API Gateway: Authentication, rate limiting (very important during mass booking), and routing.
Core Services:
- Auth Service Login, national ID verification.
- Patient Service Medical records, vaccination history.
- Clinic Service Slot management, staff schedules, capacity.
- Inventory Service Vaccine stock per clinic, batch tracking.
- Appointment Service The main service. Manages reservations, confirmations, and status changes.
- Eligibility Service Rules engine + external API calls.
- Notification Service SMS, email, push. Retries if delivery fails.
- Audit Service Append-only logs for every status change. Required for government compliance.
Data Layer: PostgreSQL for permanent data, Redis for temporary reservations and caching.
Async Messaging: Kafka for events AppointmentReserved, AppointmentConfirmed, AppointmentAdministered, AppointmentCancelled. This keeps services separated and makes the system auditable by default.

🎯 What I Would Do Differently Now

Looking back at that interview, the biggest thing I missed was not about technology it was about mindset. I jumped to the happy path because it felt complete. But the interviewer was not testing if I can design a booking form. They were testing if I can think about what happens when things go wrong.

Here's what I learned from this experience:

Start with failure scenarios, not the happy path. Ask yourself "what can go wrong at each step?" before finalizing any design.
Temporary reservation is a pattern, not a hack. Whether it's concert tickets, flash sales, or vaccine slots if you have limited stock and many users, you need hold-then-confirm flow.
Don't be vague about rollbacks. "We'll handle errors" is not a design. Be specific what happens to the data, the stock, and the user when something fails.
External services will go down. Always have a plan for when the insurance API or notification service is not available. Circuit breakers and retry queues are not optional they are necessary.

If you're preparing for system design interviews, I recommend studying inventory reservation patterns. My earlier post on designing an internet credit purchase system covers these patterns with more detail and code examples. The core idea reserve first, verify, then commit appears in many systems once you start looking.

Thanks for reading. If you faced similar interview questions or have ideas to improve this design, I would like to hear about it in the comments.

Data Fetching Patterns Every Developer Should Know (And When to Actually Use Them)

Budi Widhiyanto — Sat, 28 Feb 2026 08:56:54 +0000

About a year ago, I was working on a payment app. Solid architecture, clean API design, decent frontend on paper, everything looked good. But a few months after launch, the ratings started tanking. Users were complaining about slow loads, failed transactions, and the whole thing falling apart on spotty connections.

I spent three months debugging those performance issues, and the fix wasn't some clever algorithm or a server upgrade. It was rethinking how we fetched data. That's it. Same features, same infrastructure, same design just smarter data fetching patterns. The app went from 3.2 stars to 4.7, and transaction volume jumped 30% within two months.

That experience a year ago changed how I think about data flow end-to-end. Most apps don't have a "feature" problem they have a "how we get data to the screen" problem. And the difference between a mediocre app and a great one often comes down to picking the right data fetching pattern for the right situation.

Here's everything I learned and wish I'd known sooner.

The Basics: Request-Response

This is where everyone starts, and for good reason. You ask the server for something, you wait, you get it back. It's the foundation of HTTP, and it handles the majority of use cases just fine.

const fetchUser = async (id: string) => {
  const response = await fetch(`/api/users/${id}`);
  return response.json();
};

Think of it like ordering at a counter you place your order, you wait, you get your food. Simple and predictable.

This works great for standard CRUD operations: loading a user profile, submitting a form, fetching account details on page load. Where it falls apart is when you start chaining multiple requests together. If your page needs data from five endpoints and each one takes 300ms, your user is staring at a spinner for 1.5 seconds. That adds up fast.

The key is recognizing when request-response stops being enough which brings us to everything else.

Polling: The "Are We There Yet?" Approach

Polling is exactly what it sounds like. You ask the server for updates on a regular interval. Every 5 seconds, every 30 seconds, whatever makes sense for your use case.

const pollForUpdates = () => {
  const intervalId = setInterval(async () => {
    try {
      const response = await fetch('/api/updates');
      const data = await response.json();
      updateUI(data);
    } catch (error) {
      console.error('Polling failed:', error);
    }
  }, 5000);

  // Don't forget cleanup
  return () => clearInterval(intervalId);
};

I've seen polling get a bad reputation, and honestly, sometimes it deserves it. Naive polling hammers your server with requests even when nothing has changed. On mobile, it eats battery life. And you'll always have that gap between intervals where updates get missed.

But here's the thing polling is dead simple to implement, works everywhere, and for many use cases (dashboards refreshing every 30 seconds, checking job status on a build pipeline, order tracking) it's perfectly fine. Not everything needs to be real-time. Sometimes "close enough" is the right engineering decision.

The smarter version is long polling, where the server holds the connection open until it actually has something to send back. It's a nice middle ground before committing to WebSockets.

WebSockets: When You Need Actual Real-Time

WebSockets maintain a persistent, two-way connection between the client and server. Unlike polling, neither side has to ask data flows both directions whenever either side has something to say.

const socket = new WebSocket('wss://example.com/socket');

socket.onopen = () => {
  console.log('Connected');
};

socket.onmessage = (event) => {
  const data = JSON.parse(event.data);
  updateUI(data);
};

socket.onclose = () => {
  // You'll want reconnection logic here connections drop
  console.log('Disconnected');
};

This is what powers chat apps, multiplayer games, collaborative editors like Google Docs, and trading platforms where milliseconds matter. If your users need to see changes the moment they happen, and especially if they need to send data back frequently, WebSockets are the right call.

The tradeoff is complexity. You need to handle reconnections (connections will drop). You need to think about scaling every connected user holds an open connection on your server. You need to deal with authentication differently than with regular HTTP. It's not hard, but it's more surface area than a simple fetch call.

My rule of thumb: if you're polling more than once every 5 seconds, it's probably time to consider WebSockets.

Server-Sent Events: Real-Time's Simpler Cousin

SSE is the pattern I wish more developers knew about. It's a one-way channel the server pushes updates to the client over a long-lived HTTP connection. No polling, no WebSocket complexity.

const eventSource = new EventSource('/api/stream');

eventSource.onmessage = (event) => {
  const data = JSON.parse(event.data);
  updateUI(data);
};

eventSource.onerror = () => {
  eventSource.close();
};

See how much simpler that is compared to WebSockets? And you get automatic reconnection for free the browser handles it.

SSE is perfect for notifications, live sports scores, progress bars for long-running tasks (think file processing or deployment pipelines), newsfeeds, and anything where the server is doing the talking and the client is just listening.

The limitation is right there in the name: server-sent. If your client needs to send data back frequently, SSE isn't enough. But for a surprising number of "real-time" features, one-way is all you need.

Caching: Making Your App Feel Instant

Caching is less of a fetching pattern and more of a fetching strategy that you layer on top of other patterns. The idea is simple: store data you've already fetched so you don't have to fetch it again.

import { useQuery } from 'react-query';

const { data, isLoading, error } = useQuery(
  'user',
  () => fetch('/api/user').then((res) => res.json()),
  {
    staleTime: 5 * 60 * 1000,
    cacheTime: 30 * 60 * 1000,
  }
);

Libraries like React Query and SWR have made caching dramatically easier. They handle stale-while-revalidate (show cached data immediately, then refresh in the background), cache invalidation, and deduplication of simultaneous requests.

The impact is hard to overstate. When a user navigates to a page they've already visited and the content appears immediately while a background refresh happens silently that's the kind of thing that makes an app feel native-quality.

The classic challenge is cache invalidation (there's a reason Phil Karlton called it one of the two hard things in computer science). You have to decide: how long is cached data acceptable? What events should invalidate the cache? What happens when two tabs have different cached versions? These are solvable problems, but they require deliberate thinking.

Lazy Loading: Don't Fetch What You Don't Need Yet

The fastest network request is the one you never make. Lazy loading defers fetching until the user actually needs the data typically triggered by scrolling, clicking a tab, or navigating to a new section.

const loadMore = async () => {
  if (isNearBottom()) {
    const response = await fetch(`/api/items?page=${nextPage}`);
    const newItems = await response.json();
    setItems(prev => [...prev, ...newItems]);
    setNextPage(prev => prev + 1);
  }
};

window.addEventListener('scroll', loadMore);

You see this everywhere: infinite scroll on social feeds, images loading as you scroll past them, tabs that only fetch their content when clicked. It makes initial page loads fast because you're only loading what's visible.

The gotchas are UX-related. Infinite scroll can make it impossible for users to reach the footer. Loading new content can cause layout shifts that make users lose their place. And for accessibility, you need to make sure screen readers can navigate lazy-loaded content properly.

For very large lists (thousands of items), pair lazy loading with virtualization only render the DOM elements that are visible in the viewport. Libraries like react-window or tanstack-virtual make this manageable.

Background Sync: Building for the Real World

This one is close to my heart because it solved the biggest pain point in that payment app I worked on last year. Background sync lets users take actions (send a message, submit a form, record a transaction) even when they're offline. The operations get queued and processed automatically when connectivity returns.

// Service Worker
self.addEventListener('sync', (event) => {
  if (event.tag === 'sync-transactions') {
    event.waitUntil(processQueuedTransactions());
  }
});

// Application code
async function recordTransaction(transaction) {
  await saveToLocalQueue(transaction);

  // Show the transaction in the UI immediately
  updateUIOptimistically(transaction);

  if ('serviceWorker' in navigator) {
    const registration = await navigator.serviceWorker.ready;
    await registration.sync.register('sync-transactions');
  }
}

This pattern is essential for mobile apps used in areas with unreliable connections field service apps, delivery tracking, healthcare in rural areas, anything where you can't assume a stable connection.

The complexity lives in conflict resolution. What happens if two offline users edit the same record? What if the server rejects a queued operation? You need clear strategies for these cases, and they're not always straightforward. But the user experience improvement is massive. Going from "you can't do anything without internet" to "everything just works, and syncs when it can" is a night-and-day difference.

Batch Fetching: One Trip Instead of Ten

If your page makes 8 separate API calls to render, something is probably wrong. Batch fetching combines multiple requests into a single network call.

// Instead of this:
const user = await fetch('/api/users/1');
const posts = await fetch('/api/users/1/posts');
const notifications = await fetch('/api/users/1/notifications');

// Do this:
const dashboard = await fetch('/api/dashboard?userId=1');
// Returns user, posts, and notifications in one response

The savings come from reducing HTTP overhead connection setup, headers, TLS handshakes. On mobile networks with high latency, the difference between one request and ten is very noticeable.

The downside is coupling. When you batch things together, you can't cache or invalidate them independently. If the notification data changes every 30 seconds but user profile data changes once a month, batching them means either over-fetching profile data or under-fetching notifications. You have to think about which data actually belongs together.

GraphQL: Ask for Exactly What You Need

GraphQL flips the traditional REST model. Instead of the server deciding what data each endpoint returns, the client specifies exactly what it needs.

const query = `
  query GetUser($id: ID!) {
    user(id: $id) {
      name
      email
      posts(last: 5) {
        title
        preview
      }
    }
  }
`;

const response = await fetch('/graphql', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ query, variables: { id: '123' } }),
});

With REST, a mobile app and a desktop app hitting the same /api/user endpoint get the same response even if the mobile app only needs the name and avatar while the desktop app needs the full profile. GraphQL eliminates that mismatch. Each client asks for exactly what it needs.

This matters most when you have multiple clients with different data requirements, deeply nested data relationships, or you're tired of creating one-off REST endpoints for every new screen design.

The investment is real, though. You need a GraphQL server, a schema, resolvers, and your team needs to learn a new paradigm. Caching is trickier than REST because everything goes through a single endpoint. And poorly written queries can cause serious performance issues on the backend (the N+1 query problem is very real with GraphQL).

For smaller apps with a single client, REST with good API design is usually simpler and sufficient.

Federated Fetching: Unifying Microservices

In microservice architectures, the data a single page needs might live across five different services. Federated fetching usually through a BFF (Backend-For-Frontend) layer or API gateway aggregates that data so the client makes one clean request.

// BFF endpoint
app.get('/api/dashboard/:userId', async (req, res) => {
  const [user, accounts, activity] = await Promise.all([
    fetch(`http://user-service/users/${req.params.userId}`),
    fetch(`http://account-service/accounts?userId=${req.params.userId}`),
    fetch(`http://activity-service/recent?userId=${req.params.userId}`)
  ]);

  res.json({
    user: await user.json(),
    accounts: await accounts.json(),
    recentActivity: (await activity.json()).slice(0, 5)
  });
});

The BFF pattern is a lifesaver in complex systems. Instead of the frontend knowing about every microservice and making separate calls to each, it talks to one unified API that handles the orchestration. The frontend stays clean, and you can tailor responses to what each client actually needs.

The downside is obvious you're adding another service to build, deploy, and maintain. And if your BFF goes down, everything goes down. It's a pattern that makes sense at a certain scale, but overkill for smaller applications.

Combining Patterns: Where It Gets Interesting

No real application uses just one pattern. The interesting decisions happen when you combine them. Here's what that looks like in practice:

A messaging app might use WebSockets for incoming messages, background sync for sending messages in poor connectivity, caching for conversation history, and lazy loading for scrolling through older messages.

An e-commerce app might use request-response for search, caching for product pages, SSE for inventory availability, and batch fetching for the cart summary.

A trading platform might use WebSockets for live prices, polling as a fallback, GraphQL for portfolio data, and caching for historical charts.

The point is to match each data need to the pattern that best serves it. Not every piece of data on a screen has the same freshness requirements, the same access patterns, or the same tolerance for latency.

Quick Reference

Pattern	Use When	Complexity	Offline Support
Request-Response	Standard CRUD, simple pages	Low	No
Polling	Periodic updates, status checks	Low	No
WebSockets	Two-way real-time (chat, collaboration)	High	No
Server-Sent Events	One-way real-time (notifications, feeds)	Medium	No
Caching	Repeated data access, speed matters	Medium	Partial
Lazy Loading	Large lists, heavy initial loads	Low	No
Background Sync	Offline-first, unreliable connections	High	Yes
Batch Fetching	Multiple related data needs	Low	No
GraphQL	Complex/varied data requirements	High	No
Federated Fetching	Microservices, unified APIs	High	No

Final Thoughts

A year ago, when that payment app went from 3.2 stars to 4.7, we didn't add a single new feature. We just changed how existing features got their data. Caching made it feel instant. Background sync made it work offline. WebSockets made payments confirm in real-time. Batch fetching cut load times by 80%.

Looking back, that project taught me something I keep coming back to: users don't care about your architecture. They care that things are fast, reliable, and don't waste their time. Data fetching patterns on both the backend and the frontend are how you deliver on that promise.

Pick the right pattern for each situation. Combine them thoughtfully. And when your app ratings start climbing, you'll know why.

DEV Community: Budi Widhiyanto

FHIR R4 for AI Engineers: What Actually Lives in a Patient Record

A Bundle is not a document

The types that carry clinical meaning

CodeableConcept, and why display is not a label

The terminologies, briefly

References, and the urn:uuid trap

The patient link is not one field

What Synthea gives us, and where it differs from real data

What this changes about the pipeline

FHIR R4 for ML Engineers: What Actually Lives in a Patient Record

A Bundle is not a document

The types that carry clinical meaning

CodeableConcept, and why display is not a label

The terminologies, briefly

References, and the urn:uuid trap

The patient link is not one field

What Synthea gives us, and where it differs from real data

What this changes about the pipeline

I Built a RAG System That Answers Questions From FHIR Records and Refuses to Guess

Why grounding, not just answering

The pipeline

One resource, one chunk

Postgres does all four jobs

Verification, not trust

Running it

What it scores

What it still gets wrong

Next

Testing HL7-to-FHIR Pipelines Without a Hospital: Mocking HAPI FHIR with respx

Some Background

The Testing Problem in Healthcare Integration

Test Architecture: Two Layers, Clear Boundaries

Why respx (Not unittest.mock)

Testing Transformers: Input Model In, FHIR Resource Out

Testing Edge Cases: Where Healthcare Gets Interesting

Blood Pressure Panel Merging

Mirth Contract Tests: Catching Drift Without Running Mirth

Profile Cross-Contamination: Zero-Leakage Tests

Gender Code Mapping

Missing and Empty Fields

Integration Tests: The Full HTTP Round-Trip

Testing Error Paths

The CI Pipeline

What I Don't Test (and Why)

Patterns Worth Stealing

Try It

One Pipeline, Two Continents: Adding EU FHIR Profiles to an Australian Healthcare Integration

What's Actually Different Between AU and EU FHIR?

The Profiling Layer

Identifiers

Terminology

International Patient Summary

GDPR

My First Instinct (And Why It Was Wrong)

The Profile Configuration Pattern

How the Transformers Changed

Same Message, Different FHIR

The IPS Endpoint

GDPR Consent

Testing Across Profiles

EU Sample Messages

What I Learned

What's Next

Why I Put Mirth Connect in Front of FastAPI Instead of Parsing HL7 in Python

Some Background

The "Just Parse It in Python" Phase

Five Problems That Changed My Mind

1. MLLP Is Not HTTP

2. HL7 Parsing Is Messier Than It Looks

3. Message Routing Is a Separate Concern

4. ACK/NACK Is Surprisingly Important

5. I Was Rebuilding an Integration Engine, Badly

The Redesigned Architecture

Why Not Do Everything in Mirth?

Type-Safe FHIR Validation

Testability

Python Ecosystem

The Tradeoff I Accepted

What This Taught Me

CodeableConcept, and why `display` is not a label

References, and the `urn:uuid` trap

CodeableConcept, and why `display` is not a label

References, and the `urn:uuid` trap