DEV Community: Budi Widhiyanto

One Pipeline, Two Continents: Adding EU FHIR Profiles to an Australian Healthcare Integration

Budi Widhiyanto — Mon, 03 Aug 2026 16:56:42 +0000

The European Health Data Space (EHDS) regulation requires EHR systems operating in the EU to exchange health data using FHIR. This is driving adoption of HL7 Europe's FHIR Implementation Guides across member states.

I'd just finished building a Maternity HL7-to-FHIR Pipeline targeting the Australian market - AU Base profiles, IHI identifiers, ICD-10-AM diagnosis codes. It worked. It had a solid test suite with good coverage. It was a complete portfolio piece.

Then I started asking myself: could this same pipeline work for a European hospital?

The HL7 v2 messages coming from the hospital are identical - an ADT^A01 admission in Melbourne looks the same as one in Amsterdam. The clinical data is the same - blood pressure is blood pressure, a pregnancy diagnosis is a pregnancy diagnosis. What changes is the FHIR metadata layer: which profiles to declare, which identifier systems to use, which terminology editions to bind.

If the architecture was good, this should be a configuration change - not a rewrite. If it wasn't, well, that would be a good lesson too.

GitHub: budityw23/maternity-hl7-to-fhir-pipeline

What's Actually Different Between AU and EU FHIR?

Before touching any code, I needed to understand what "European FHIR" means. The EU ecosystem is more complex than Australia's, and more interesting.

The Profiling Layer

Australia has a relatively flat structure: AU Base profiles sit directly on top of base FHIR R4. You declare http://hl7.org.au/fhir/StructureDefinition/au-patient on your Patient resource and you're done.

Europe has a layered architecture. HL7 Europe publishes three foundational Implementation Guides:

HL7 Europe Base - flexible foundation profiles
HL7 Europe Core - essential constraints on top of Base
HL7 Europe Extensions - EU-specific extensions (nationality, citizenship, etc.)

On top of that sit scoped IGs - domain-specific guides like the European Patient Summary. And below all of that, individual countries can add their own national constraints (NHS England, Nictiz Netherlands, KBV Germany).

HL7 EU Extensions
       ↓
HL7 EU Base Profiles (flexible foundation)
       ↓
HL7 EU Core Profiles (essential constraints)
       ↓
Scoped HL7 EU IGs (e.g. European Patient Summary)
       ↓
National IGs (e.g. NHS England, Nictiz NL, KBV DE)

This means the EU pipeline needs to operate at the EU Base/Core level - general enough to work across countries, specific enough to satisfy EHDS requirements.

Identifiers

In Australia, every patient has an IHI (Individual Healthcare Identifier) - a single national identifier system with one URI: http://ns.electronichealth.net.au/id/hi/ihi/1.0.

Europe doesn't have a single identifier. Each country has its own:

Country	Identifier	System URI
Australia	IHI	`http://ns.electronichealth.net.au/id/hi/ihi/1.0`
UK	NHS Number	`https://fhir.nhs.uk/Id/nhs-number`
Netherlands	BSN	`http://fhir.nl/fhir/NamingSystem/bsn`
Germany	KVNR	`http://fhir.de/sid/gkv/kvid-10`
Ireland	PPS Number	`https://fhir.ie/sid/ppsn`

(These are the exact system URIs used in fastapi/app/profiles/eu_profile.py.)

The pipeline needs a configurable national identifier, not a hardcoded one.

Terminology

Australian maternity care uses ICD-10-AM (Australian Modification) for diagnosis coding. The system URI is http://hl7.org.au/fhir/CodeSystem/icd-10-am.

European systems use standard ICD-10 (WHO edition) with system URI http://hl7.org/fhir/sid/icd-10. Some countries use their own modifications (Germany has ICD-10-GM), but at the EU Base level, the WHO edition is the common ground.

SNOMED CT is used in both regions, but Australia binds to the AU refset while Europe uses the International Edition. The URI is the same (http://snomed.info/sct) - the difference is in which value sets and reference sets are expected.

LOINC and UCUM are universal. Blood pressure is still 85354-9. Weight is still 29463-7. Millimeters of mercury are still mm[Hg]. Some things, thankfully, don't need localization.

International Patient Summary

The IPS (International Patient Summary, ISO 27269) is the cross-border health document that EHDS uses for data exchange. When a Portuguese patient walks into a German emergency room, the IPS tells the German doctor what they need to know: allergies, medications, active conditions, recent vital signs, pregnancy status.

My pipeline already produces Patient, Condition, Observation, and Encounter resources. Wrapping those into an IPS Composition document is a natural extension.

GDPR

Australian healthcare has privacy regulations, but nothing that requires FHIR-level consent tracking. In the EU, GDPR Articles 6 and 9 create specific legal bases for processing health data. Modeling this as a FHIR Consent resource (recording which GDPR article authorizes the processing) isn't required by any FHIR IG, but European employers care about it.

My First Instinct (And Why It Was Wrong)

My initial approach was the obvious one: fork the repo, create a separate maternity-hl7-to-fhir-pipeline-eu project, and replace all the AU-specific values with EU ones.

Find and replace. au-patient → Patient-eu. IHI → NHS Number. ICD-10-AM → ICD-10. Ship it.

This would have worked for a demo. But it's the wrong architecture for several reasons:

Duplicated transformation logic. The mapping from PID-5.1 to Patient.name[0].family doesn't change between Australia and Europe. The blood pressure panel merging logic doesn't change. The encounter class mapping doesn't change. Forking the repo means maintaining two copies of all this logic - and if I fix a bug in the AU version, I have to remember to fix it in the EU version too.

No path to a third region. What happens when I want to add US Core profiles? Another fork? Now I have three repos with three copies of the same BP merging code. This is the kind of technical debt that turns portfolio projects into maintenance nightmares.

It hides the interesting design work. A fork says "I copied code and changed some strings." A configurable multi-profile architecture says "I designed a system that handles jurisdiction differences at the configuration layer, not the logic layer." One of these is impressive in an interview. The other isn't.

The right approach was to make the existing pipeline support multiple profiles - and to do it in a way that the transformation logic doesn't know or care which region it's operating in.

The Profile Configuration Pattern

The core design change was extracting all region-specific values into a profile configuration module:

fastapi/app/profiles/
├── __init__.py
├── base.py              # ProfileConfig dataclass
├── au_profile.py        # AU_PROFILE instance
├── eu_profile.py        # build_eu_profile(country) factory
└── registry.py          # PROFILE_REGION → config resolver

ProfileConfig is a frozen dataclass that captures everything that varies by jurisdiction:

# fastapi/app/profiles/base.py
@dataclass(frozen=True)
class ProfileConfig:
    region: str
    patient_profile_url: str
    condition_profile_url: str
    encounter_profile_url: str
    bp_observation_profile_url: str
    observation_profile_url: str
    national_id_system: str
    national_id_display: str
    diagnosis_code_system: str
    snomed_system: str
    timezone_offset: str
    default_country: str
    profile_definitions: list[dict[str, str]] = field(default_factory=list)

The AU configuration is a single frozen instance:

# fastapi/app/profiles/au_profile.py
AU_PROFILE = ProfileConfig(
    region="au",
    patient_profile_url="http://hl7.org.au/fhir/StructureDefinition/au-patient",
    condition_profile_url="http://hl7.org.au/fhir/StructureDefinition/au-condition",
    encounter_profile_url="http://hl7.org.au/fhir/StructureDefinition/au-encounter",
    bp_observation_profile_url="http://hl7.org.au/fhir/StructureDefinition/au-vitalsigns-bloodpressure",
    observation_profile_url="http://hl7.org/fhir/StructureDefinition/vitalsigns",
    national_id_system="http://ns.electronichealth.net.au/id/hi/ihi/1.0",
    national_id_display="IHI",
    diagnosis_code_system="http://hl7.org.au/fhir/CodeSystem/icd-10-am",
    snomed_system="http://snomed.info/sct",
    timezone_offset="+10:00",
    default_country="AU",
    profile_definitions=[
        {"id": "au-patient", "url": "http://hl7.org.au/fhir/StructureDefinition/au-patient",
         "name": "AUPatient", "type": "Patient"},
        {"id": "au-condition", "url": "http://hl7.org.au/fhir/StructureDefinition/au-condition",
         "name": "AUCondition", "type": "Condition"},
        # ... plus au-encounter, au-vitalsigns-bloodpressure
    ],
)

The EU side is a factory rather than a static instance, because the national identifier
depends on the country. PROFILE_COUNTRY selects it:

# fastapi/app/profiles/eu_profile.py
EU_NATIONAL_ID_SYSTEMS = {
    "uk": ("https://fhir.nhs.uk/Id/nhs-number", "NHS Number"),
    "nl": ("http://fhir.nl/fhir/NamingSystem/bsn", "BSN"),
    "de": ("http://fhir.de/sid/gkv/kvid-10", "KVNR"),
    "ie": ("https://fhir.ie/sid/ppsn", "PPS Number"),
}

EU_DEFAULT_ID_SYSTEM = "http://hl7.eu/fhir/base/NamingSystem/national-id"
EU_DEFAULT_ID_DISPLAY = "National ID"

def build_eu_profile(country: str = "") -> ProfileConfig:
    country_lower = country.lower().strip()
    if country_lower in EU_NATIONAL_ID_SYSTEMS:
        national_id_system, national_id_display = EU_NATIONAL_ID_SYSTEMS[country_lower]
    else:
        national_id_system = EU_DEFAULT_ID_SYSTEM
        national_id_display = EU_DEFAULT_ID_DISPLAY
    return ProfileConfig(
        region="eu",
        patient_profile_url="http://hl7.eu/fhir/base/StructureDefinition/patient-eu",
        condition_profile_url="http://hl7.eu/fhir/base/StructureDefinition/condition-eu-core",
        encounter_profile_url="http://hl7.org/fhir/StructureDefinition/Encounter",  # base FHIR
        bp_observation_profile_url="http://hl7.org/fhir/StructureDefinition/bp",     # FHIR core BP
        observation_profile_url="http://hl7.org/fhir/StructureDefinition/vitalsigns",
        national_id_system=national_id_system,
        national_id_display=national_id_display,
        diagnosis_code_system="http://hl7.org/fhir/sid/icd-10",
        snomed_system="http://snomed.info/sct",
        timezone_offset="+01:00",
        default_country=country_lower.upper() if country_lower else "EU",
        profile_definitions=[
            {"id": "patient-eu", "url": "http://hl7.eu/fhir/base/StructureDefinition/patient-eu",
             "name": "PatientEU", "type": "Patient"},
            {"id": "condition-eu-core", "url": "http://hl7.eu/fhir/base/StructureDefinition/condition-eu-core",
             "name": "ConditionEUCore", "type": "Condition"},
            # ... plus FHIR core BP
        ],
    )

Two environment variables control which profile is active:

# AU mode (default - backward compatible)
docker compose up --build

# EU mode, UK national identifier (NHS Number)
PROFILE_REGION=eu PROFILE_COUNTRY=uk docker compose up --build

The registry resolves the active config from those settings:

# fastapi/app/profiles/registry.py
def get_profile() -> ProfileConfig:
    if settings.profile_region.lower().strip() == "eu":
        return build_eu_profile(settings.profile_country)
    return AU_PROFILE

How the Transformers Changed

The transformers don't contain if/else logic for regions. Each route resolves the active ProfileConfig once via get_profile() (driven by the PROFILE_REGION / PROFILE_COUNTRY settings) and passes it straight into the transformer as an argument:

# fastapi/app/main.py — inside the /fhir/Patient handler
profile = get_profile()
patient = build_patient(payload, profile)

The transformer signature takes that config and uses it for every jurisdiction-specific value:

# fastapi/app/transformers/patient.py
def build_patient(payload: AdtPayload, profile: ProfileConfig) -> Patient:
    ...
    # Before (hardcoded AU):
    #   meta={"profile": ["http://hl7.org.au/fhir/StructureDefinition/au-patient"]}
    # After (profile-configured):
    patient = Patient(
        meta={"profile": [profile.patient_profile_url]},
        ...
        identifier=[..., Identifier(system=profile.national_id_system, value=payload.ihi)],
        address=[Address(..., country=payload.address.country or profile.default_country)],
    )

The transformation logic (mapping PID fields to Patient attributes, merging BP panels, resolving patient references by MRN) stays unchanged. It doesn't know whether it's producing an AU Patient or an EU Patient. It builds a valid FHIR Patient resource and applies whatever metadata the profile config provides. (It's plain function-argument passing, not FastAPI's Depends() injection — the profile is process-wide config, resolved once per request from the environment.)

Adding a third region is one new file. A US Core profile would need us_profile.py, one new entry in the registry, and zero changes to the transformers.

Same Message, Different FHIR

To see the impact, compare the same ADT^A01 admission message processed in both modes.

The HL7 v2 message is identical. A pregnant woman is admitted for a routine antenatal visit. The message carries her name, date of birth, identifier, and a pregnancy diagnosis.

AU mode produces (real transformer output, trimmed):

{
  "resourceType": "Patient",
  "meta": {
    "profile": ["http://hl7.org.au/fhir/StructureDefinition/au-patient"]
  },
  "identifier": [
    {
      "type": {"coding": [{"system": "http://terminology.hl7.org/CodeSystem/v2-0203", "code": "MR"}]},
      "system": "http://hospital.local/mrn",
      "value": "1234567"
    },
    {
      "system": "http://ns.electronichealth.net.au/id/hi/ihi/1.0",
      "value": "8003608166690503"
    }
  ],
  "name": [{"use": "official", "family": "TEST", "given": ["PATIENT", "MARY"]}],
  "gender": "female",
  "birthDate": "1992-03-15"
}

EU mode (PROFILE_REGION=eu PROFILE_COUNTRY=uk) produces:

{
  "resourceType": "Patient",
  "meta": {
    "profile": ["http://hl7.eu/fhir/base/StructureDefinition/patient-eu"]
  },
  "identifier": [
    {
      "type": {"coding": [{"system": "http://terminology.hl7.org/CodeSystem/v2-0203", "code": "MR"}]},
      "system": "http://hospital.local/mrn",
      "value": "1234567"
    },
    {
      "system": "https://fhir.nhs.uk/Id/nhs-number",
      "value": "9000000009"
    }
  ],
  "name": [{"use": "official", "family": "TEST", "given": ["PATIENT", "MARY"]}],
  "gender": "female",
  "birthDate": "1992-03-15"
}

Same patient, same clinical data, different regulatory metadata. The profile URL and the national identifier system changed (au-patient → patient-eu, IHI → NHS Number). Name,gender, birthDate, MRN (including its MR type coding) are identical.

The Condition resource follows the same pattern:

AU:  "system": "http://hl7.org.au/fhir/CodeSystem/icd-10-am", "code": "O80"
EU:  "system": "http://hl7.org/fhir/sid/icd-10",              "code": "O80"

Same ICD code, different system URI. The clinical meaning is identical; only the regulatory packaging shifts.

The IPS Endpoint

A pregnant woman from the Netherlands visits a hospital in Ireland. The Irish clinician needs her active conditions, recent vital signs, medications, and allergies, right now, in a format they can read. The IPS is that format, and it's what makes this pipeline relevant to EHDS.

The pipeline already had all the data. After processing ADT^A01 (Patient + Condition), ORM^O01 (Encounter), and ORU^R01 (Observations), HAPI FHIR contains a complete clinical picture for that patient. The IPS endpoint assembles those existing resources into a single FHIR Composition document.

POST /fhir/IPS
{
  "correlationId": "ips-001",
  "mrn": "1234567"
}

The endpoint resolves the Patient by MRN (422 if the patient hasn't been admitted yet), queries HAPI FHIR for that patient's resources, then builds an IPS Composition (LOINC 60591-5, "Patient summary Document") with six sections:

IPS Composition (LOINC 60591-5)  → wrapped in a Document Bundle
├── Allergies and Intolerances (48765-2)  → emptyReason "notasked"
├── Medications (10160-0)                  → emptyReason "notasked"
├── Problems (11450-4)                     → non-pregnancy Conditions
├── Results (30954-2)                      → non-vital-sign Observations
├── Vital Signs (8716-3)                   → vital-sign Observations (BP panel, weight, ...)
└── Pregnancy History (10162-6)            → pregnancy-related Conditions (by SNOMED/ICD-10)

The whole thing is wrapped in a FHIR Document Bundle (Bundle-uv-ips profile), so the result is a single, self-contained document any EHDS-compliant system can consume.

The Allergies and Medications sections use emptyReason (notasked) because the pipeline doesn't ingest those HL7 message types yet. Any section that ends up with no entries gets the same treatment. The IPS spec requires these sections but supports the "no information available"
pattern — an empty section with an emptyReason is more credible than fake data.

GDPR Consent

European healthcare data processing requires a legal basis under GDPR. For health data specifically, Article 9(2)(h) permits processing forhealthcare purposes. The pipeline models
this as a FHIR Consent resource:

POST /fhir/Consent
{
  "correlationId": "consent-001",
  "mrn": "1234567",
  "policyRule": "gdpr-art-9-2-h",
  "provisionType": "permit"
}

The policyRule field carries the GDPR basis; the pipeline recognises gdpr-art-6-1-a (explicit consent, the default), gdpr-art-9-2-h (health-data processing), and gdpr-art-6-1-e (public interest), mapping each to a human-readable display in the Consent's policyRule.coding.
provisionType is permit or deny, and optional periodStart / periodEnd set the validity window. On success it returns 200 with {"consentId": "...", "correlationId": "..."}.

This endpoint is EU-only: it returns 404 when PROFILE_REGION=au, because GDPR doesn't apply in Australia. The profile configuration controls not just metadata values but also which endpoints are available.

No FHIR IG requires this. But EU health IT companies want engineers who understand why health data processing needs a legal basis and how consent flows work, not just engineers who can write valid FHIR resources.

Testing Across Profiles

The existing AU tests must not break. The EU extension adds capability without changing existing behavior.

The CI pipeline now runs tests in a matrix:

# .github/workflows/ci.yml
strategy:
  matrix:
    profile_region: [au, eu]
env:
  PROFILE_REGION: ${{ matrix.profile_region }}
  PROFILE_COUNTRY: ${{ matrix.profile_region == 'eu' && 'uk' || '' }}

Every test runs twice - once with PROFILE_REGION=au and once with PROFILE_REGION=eu (PROFILE_COUNTRY=uk). The AU test suite is unchanged. The EU tests add EU-specific assertions:

EU Patient resources carry the patient-eu profile URL, AU ones carry au-patient
EU Conditions use ICD-10 (WHO), AU Conditions use ICD-10-AM
EU identifiers use the configured national system (NHS Number for uk), not IHI
The IPS endpoint produces a valid IPS Document Bundle
The Consent endpoint returns 404 in AU mode and 200 in EU mode

The suite is now 277 tests at 92% line coverage (unit + integration + a tests/e2e/ suite
that drives real HL7 over MLLP against a live Docker stack and auto-skips when it isn't running).
Adding EU capability did not change a single existing AU assertion.

EU Sample Messages

The samples/ directory now includes EU-specific HL7 messages:

samples/
├── adt_a01_normal_delivery.hl7       # AU sample
├── adt_a01_escaped_name.hl7          # AU sample with HL7 escape sequences
├── orm_o01_antenatal_28w.hl7         # AU sample
├── oru_r01_vitals.hl7                # AU sample
├── invalid/
│   └── adt_missing_mrn.hl7
└── eu/
    ├── adt_a01_normal_delivery.hl7   # EU sample (NHS Number, ICD-10)
    ├── orm_o01_antenatal_28w.hl7     # EU sample
    └── oru_r01_vitals.hl7            # EU sample

The EU samples use NHS Number identifiers (NHS...^^^NHS^NH), ICD-10 (WHO) diagnosis codes, and UK-style facility naming (e.g. ST_THOMAS, LONDON_TRUST). The HL7 v2 message structure is identical to the AU samples — the same segments, the same field positions. Only the content in the identifier and diagnosis fields differs.

What I Learned

Building the first version taught me about HL7, FHIR, and healthcare integration. Extending it to Europe taught me different things.

The clinical domain is more universal than the regulatory layer. Blood pressure merging, patient admission flows, observation linking: none of this changes between Australia and Europe. The HL7-to-FHIR mapping logic is universal. What changes is the metadata layer: profile URLs, identifier systems, terminology editions, timezone offsets. Separating these two layers is the design challenge.

The EU FHIR ecosystem is more layered than AU. Australia's approach is relatively flat: AU Base sits on FHIR R4 and that's mostly it. Europe has EU Base → EU Core → Scoped IGs → National IGs. Understanding where your pipeline sits in that stack matters. I targeted EU Base/Core because it's the broadest level - a Dutch hospital and a German hospital can both accept resources profiled at this level. Going deeper into country-specific profiles (Nictiz, KBV) is a natural next step, but the foundation needs to be right first.

If you're building a FHIR portfolio for Europe, implement IPS. EHDS is built around it, MyHealth@EU exchanges it, and European health IT companies are hiring for it. A pipeline that can both ingest HL7 v2 messages and produce an IPS document covers both sides of the interoperability problem.

Configuration-driven architecture pays compound interest. The ProfileConfig pattern took maybe two hours to design. When I was implementing the EU profile, I never had to think about "what does this transformer do?" Only "what values should the EU config provide?" Two hours of design saved days of implementation.

GDPR awareness is a soft skill encoded in code. European employers want engineers who understand why health data processing requires a legal basis, what Article 6 vs Article 9 means, and how consent flows work. A Consent resource in your repo shows this better than a resume bullet point.

Backward compatibility is a feature. The hardest part of the EU extension was ensuring the existing AU pipeline stayed unchanged. Every AU test passing without modification was non-negotiable. Adding features is easy. Adding features without breaking existing behavior takes discipline.

What's Next

The pipeline now supports two regulatory jurisdictions with a common transformation core. Next on the list:

US Core - a third PROFILE_REGION=us option targeting the US market (US Core IG, Argonaut profiles, USCDI)
Terminology validation - integrating with a terminology server to validate SNOMED CT and ICD-10 codes against the correct value sets for each region
Additional HL7 message types - adding allergies (ADT^A60) and medications (RDE^O11) to populate the empty IPS sections
Full IG package loading - replacing placeholder StructureDefinitions with the complete HL7 EU Base/Core IG package in HAPI for meaningful $validate results

The full source code, documentation, and Docker Compose setup are on GitHub: maternity-hl7-to-fhir-pipeline.

This is Part 2 of the Maternity HL7-to-FHIR Pipeline series. Part 1 covered the initial AU pipeline architecture. If you're working on FHIR interoperability in the European market - especially around EHDS compliance or IPS implementation - I'd like to hear what challenges you're facing.

Source code: github.com/budityw23/maternity-hl7-to-fhir-pipeline

Tags: #fhir #healthit #architecture #showdev

Why I Put Mirth Connect in Front of FastAPI Instead of Parsing HL7 in Python

Budi Widhiyanto — Sat, 25 Jul 2026 21:42:55 +0000

When I started building my Maternity HL7-to-FHIR Pipeline, my first instinct was to do everything in Python. Parse the HL7 message, map the fields, validate the FHIR resource, persist it, all in one FastAPI service. It was clean. It was simple. It was wrong.

Some Background

I've been building an open-source pipeline that takes HL7 v2.5 messages, the kind hospital maternity systems still send over raw TCP connections, and turns them into FHIR R4 resources with Australian and European FHIR profiles. Patient admissions, lab orders, vital signs. The sort of data that's been flowing between hospital systems in pipe-delimited text since the 1990s, now mapped into modern healthcare APIs.

If you're curious about the full project, I wrote a detailed walkthrough in Bridging Legacy Hospital Messages to Modern Healthcare APIs. That article covers the end-to-end AU pipeline architecture. The second article, One Pipeline, Two Continents, covers how I extended the pipeline to support EU FHIR profiles, IPS, and GDPR Consent. Here's the short version of the architecture:

Hospital Maternity System
  |
  | HL7 v2.5 messages (ADT, ORM, ORU)
  | over MLLP (raw TCP, port 6661)
  v
┌─────────────────────────┐
│   Mirth Connect 4.5     │  Receives MLLP, parses HL7,
│   (Integration Engine)  │  extracts fields, routes
└───────────┬─────────────┘  by message type
            |
            | Clean JSON over HTTP
            v
┌─────────────────────────┐
│  FastAPI + Python 3.11  │  Pydantic validation, FHIR R4
│  (Transformation Layer) │  mapping, AU Base profiles
└───────────┬─────────────┘
            |
            | FHIR R4 resources
            | (conditional PUT/POST)
            v
┌─────────────────────────┐
│  HAPI FHIR Server 7.0.3 │  Persistence + FHIR API
└─────────────────────────┘

Three containers, one docker compose up. Mirth handles the legacy protocol world, FastAPI handles the FHIR world, HAPI stores everything.

But this article isn't about the what. It's about one particular why: why the pipeline has two services doing the transformation work instead of one.

The "Just Parse It in Python" Phase

My initial architecture looked like this:

Hospital System --MLLP--> Python Script --> HAPI FHIR Server

I used python-hl7 to split messages on | and count field positions. For a single ADT^A01 (patient admission) message, it worked fine. I could pull the patient name from PID-5, the MRN from PID-3, the gender from PID-8, and build a FHIR Patient resource from it.

Then I tried a real-ish maternity workflow (an admission, an order, and a set of vitals) and things fell apart quickly.

Five Problems That Changed My Mind

1. MLLP Is Not HTTP

Hospital systems don't send HL7 over HTTP. They send it over MLLP (Minimum Lower Layer Protocol), which is a TCP socket protocol with specific framing bytes (\x0b at the start, \x1c\x0d at the end). The sender expects an ACK or NACK response in HL7 format, not an HTTP status code.

Building an MLLP listener in Python is possible. Libraries like aiohl7 exist. But you're now maintaining a custom TCP server alongside your HTTP API server, handling connection pooling, timeouts, and HL7 acknowledgment generation. That's a lot of infrastructure code that has nothing to do with your actual transformation logic.

Mirth Connect handles MLLP natively. You point it at a port, it listens, it parses, it ACKs. Done. One config screen, no custom code.

2. HL7 Parsing Is Messier Than It Looks

The pipe-delimited format looks simple:

PID|1||1234567^^^RPA^MR||TEST^PATIENT^MARY^^MS||19920315|F|||14 SAMPLE ST^^SYDNEY^NSW^2000^AUS

But consider:

Component separators: PID-5 is TEST^PATIENT^MARY^^MS, which is family, given, middle, suffix (empty), prefix. Miss the empty suffix and your prefix ends up as the suffix.
Repeating fields: PID-3 can contain multiple identifiers separated by ~. One might be the MRN, another the IHI (Individual Healthcare Identifier). You need to iterate and match by identifier type, not just grab the first one.
Escape characters: A patient named O'Brien might appear as O\T\Brien in HL7 (where \T\ is the subcomponent separator escape). Or it might not, depending on the sending system's configuration.
Encoding characters: The first field of MSH (MSH-1) is the field separator itself (|), and MSH-2 defines the component, repetition, escape, and subcomponent separators. Different hospitals can (and do) use different separators.

In my Python script, I was doing segment.split('|')[5].split('^')[0] to get a family name. One unexpected empty field, one unexpected repeating group, and the whole positional mapping shifted silently. No error, just wrong data in the FHIR resource.

Mirth Connect's HL7 parser handles all of this natively. In a Mirth transformer, I write:

var familyName = msg['PID']['PID.5']['PID.5.1'].toString();
var givenName  = msg['PID']['PID.5']['PID.5.2'].toString();

That's an E4X/XML path against a parsed HL7 tree, not string splitting. It handles repeating fields, component separators, and encoding characters correctly because that's what the parser is built for. Years of edge cases baked into a mature parser versus my three-day-old string splitter.

3. Message Routing Is a Separate Concern

A maternity workflow involves three message types:

Message	Trigger	What It Creates
`ADT^A01`	Patient admitted	Patient + Condition
`ORM^O01`	Order placed	Encounter
`ORU^R01`	Results available	Observation(s)

In the "everything in Python" design, my MLLP listener would receive a raw HL7 message, I'd parse MSH-9 to determine the message type, then route to the right handler function. That's a message router, and I'd be building one from scratch.

Mirth Connect is literally a message router. You can define a channel per message type, or (as I did) a single channel whose transformer routes by message type. Either way, each type is POSTed to a different FastAPI endpoint, and Mirth handles the dispatch. The FastAPI endpoints receive clean, typed JSON payloads. They don't even need to know HL7 exists.

Mirth Channel: Maternity Inbound HL7
  Source:       MLLP listener on port 6661 (HL7 v2.5, auto ACK/NACK)
  Preprocessor: generate a correlation-id (UUID) per message
  Destination:  HTTP Sender - a JavaScript transformer routes on
                msg['MSH']['MSH.9']['MSH.9.1'] and POSTs to:
                  ADT -> http://fastapi:8000/fhir/Patient
                  ORM -> http://fastapi:8000/fhir/Encounter
                  ORU -> http://fastapi:8000/fhir/Observation/bundle

The transformer extracts only the fields relevant to each message type, builds a flat JSON payload, and tags every request with an X-Correlation-ID header. FastAPI receives structured data with Pydantic validation on the input shape. Clean separation.

4. ACK/NACK Is Surprisingly Important

When a hospital system sends an HL7 message, it expects an acknowledgment. If it gets an ACK (MSA-1 = AA), it moves on. If it gets a NACK (MSA-1 = AE or AR), it may retry, queue the message, or alert an operator.

In my Python approach, I'd need to:

Parse the incoming message
Process it
Build an HL7 ACK message with the correct MSH fields mirrored back
Send it over the same TCP socket
Handle the case where processing succeeds but the ACK fails to send

Mirth Connect auto-generates ACK messages. You configure whether to ACK on receipt (before processing) or after successful processing. You can customise the ACK content if needed. Another solved problem.

5. I Was Rebuilding an Integration Engine, Badly

Step back and look at what I was building in Python:

A TCP/MLLP listener
An HL7 v2.5 parser
A message router
An ACK/NACK generator
Connection management and error handling

That's an integration engine. Mirth Connect, Rhapsody, InterSystems HealthShare. These are products that teams of engineers have built and maintained for years. I was reimplementing one in a weekend, pretending it was "simpler" because it was in Python.

The honest assessment: I was spending 60-70% of my effort on plumbing (protocol handling, parsing, routing) and 30-40% on the actual value: the FHIR transformation and validation logic.

The Redesigned Architecture

Here's what I landed on:

Mirth Connect does what it's good at: protocol handling, HL7 parsing, message routing, ACK generation. It turns messy, protocol-specific HL7 into clean HTTP + JSON.

FastAPI does what it's good at: receiving typed payloads, applying business logic (like merging systolic + diastolic OBX segments into a single FHIR Blood Pressure panel), validating FHIR resources with Pydantic models from fhir.resources, and persisting to the FHIR server.

The boundary between them is a simple HTTP POST with a JSON body. FastAPI doesn't know about MLLP, segment separators, or ACK messages. Mirth doesn't know about FHIR profiles, conditional PUT, or Pydantic validation. Each component is testable in isolation.

Why Not Do Everything in Mirth?

Fair question. Plenty of teams build entire HL7-to-FHIR transformations inside Mirth Connect's JavaScript engine. You can construct FHIR JSON in a Mirth transformer and POST it directly to HAPI. Why add FastAPI at all?

Three reasons:

Type-Safe FHIR Validation

The fhir.resources library gives me Pydantic models for every FHIR R4 resource type. When I build a Patient resource, the model enforces:

gender must be one of male, female, other, unknown, not F or M
name is a list of HumanName objects with specific structure
identifier requires both system and value
meta.profile must be a list of valid URIs

If I typo Observation.status as "done" instead of "final", Pydantic raises a validation error with the exact field path before the resource ever reaches HAPI. In Mirth's JavaScript, I'd be constructing a plain JSON object with no schema enforcement. Typos become silent bugs that surface as cryptic HAPI 400 errors.

Testability

My FastAPI transformers have 220+ unit tests, 30+ integration tests, and 20+ end-to-end tests at 80%+ coverage. Each transformer function takes a Pydantic input model and returns a FHIR resource. I can test edge cases (empty diagnoses arrays, missing middle names, orphan blood pressure readings) in milliseconds with pytest.

Testing Mirth channels is harder. You can write unit tests for Mirth's JavaScript functions, but testing the full channel behavior (source -> filter -> transformer -> destination) requires either Mirth's built-in test tools or a running Mirth instance. The feedback loop is slower, and CI integration is less straightforward.

Python Ecosystem

For the transformation logic, Python gives me things Mirth's JavaScript doesn't:

fhir.resources for FHIR R4 Pydantic models
httpx for async HTTP with connection pooling
pydantic-settings for typed configuration from environment variables
ruff for linting, mypy --strict for type checking
The entire pytest ecosystem for testing

The FHIR mapping is where I add value. I want the best tools for that specific job.

The Tradeoff I Accepted

This architecture has a cost: an extra network hop. Every message goes from Mirth to FastAPI over HTTP, adding latency. In a high-throughput production system processing thousands of messages per second, this matters.

For a maternity ward generating maybe 50-100 messages per hour? The latency is invisible. And the benefits (clean separation, type safety, testability, better error handling) far outweigh the cost.

If throughput became a bottleneck, I'd look at gRPC between Mirth and FastAPI, or batch endpoints that accept multiple messages per request. But I'd still keep the separation. The architectural benefit is worth more than saving a few milliseconds.

What This Taught Me

Use the right tool for each layer. Integration engines exist because protocol handling and message parsing are genuinely hard, well-understood problems. Application frameworks exist because business logic, validation, and API design are different problems. Trying to solve both in one layer means solving neither well.

The boundary between components should be boring. HTTP + JSON between Mirth and FastAPI is boring. That's the point. The interesting work (the FHIR mapping, the BP panel merging, the AU Base profiling) happens inside the components, not at the boundaries.

"Simpler" isn't always fewer components. My initial one-component Python approach looked simpler. In practice, it was a tangled mess of protocol code, parsing code, and transformation code all in one place. Three components with clear responsibilities turned out to be simpler to build, test, and debug.

The payoff shows up when you extend it. Adding EU FHIR profiles on top of the existing AU pipeline touched about 2,400 lines across 49 files. Roughly 20 of those lines were in the Mirth transformer: widening national ID extraction beyond AU's IHI to also accept NH, PN, and SS, and dropping a hardcoded country default so empty country codes fall through to the FastAPI profile. Everything else — the ProfileConfig registry, EU profile URLs, IPS Composition, GDPR Consent — landed in FastAPI. The transport layer barely noticed a second jurisdiction arriving.

What's Next

The separation between Mirth and FastAPI makes each of these extensions a clean addition rather than a risky refactor:

US Core profiles — a third PROFILE_REGION=us option. The profile configuration pattern from the EU extension means this is another ProfileConfig instance and a set of profile URLs — no transformer rewrites.
Terminology validation — integrating with a terminology server (like Ontoserver or the HAPI terminology service) to validate SNOMED CT and ICD-10 codes against region-specific value sets at transformation time.
Additional HL7 message types — allergies (ADT^A60) and medications (RDE^O11) to populate the currently empty IPS sections. Each one is a new Mirth routing rule and a new FastAPI transformer — the pattern scales without architectural changes.
Full IG package loading — replacing the placeholder StructureDefinitions with complete AU Base and EU Core IG packages in HAPI for meaningful $validate results across both regions.

If you're evaluating integration engine versus "parse it yourself" for a health IT project, I'd like to hear how you made that call. The first article in this series covers the full pipeline architecture, and the second article shows how the separation kept adding EU FHIR profiles almost entirely inside FastAPI.

Try It Yourself

Bring up the three containers:

docker compose up --build

Or with EU profiles:

PROFILE_REGION=eu docker compose up --build

Then deploy the Mirth channel (the stock Mirth image doesn't auto-load channels, so this imports and deploys it via the REST API):

./scripts/import_channels.sh

Now send a test HL7 message via MLLP:

python scripts/mllp_send.py samples/adt_a01_normal_delivery.hl7

Or skip Mirth entirely and hit FastAPI directly with JSON:

curl -s -X POST http://localhost:8000/fhir/Patient \
  -H "Content-Type: application/json" \
  -d '{ ... }'

Both paths produce the same validated, correctly profiled FHIR resources in HAPI.

Source code: github.com/budityw23/maternity-hl7-to-fhir-pipeline

This is the third article in my series on the Maternity HL7-to-FHIR Pipeline. The first article, Bridging Legacy Hospital Messages to Modern Healthcare APIs, covers the full AU pipeline architecture. The second article, One Pipeline, Two Continents, covers the EU FHIR profile extension.

Tags: #healthit #architecture #python #showdev

Maternity HL7-to-FHIR Pipeline: Bridging Legacy Hospital Messages to Modern Healthcare APIs

Budi Widhiyanto — Tue, 21 Jul 2026 02:07:48 +0000

How It Started

A few months ago, I was browsing remote FHIR engineer roles in Australia. Every single job description mentioned the same thing: experience integrating HL7 v2 with FHIR R4.

This makes sense. Australian hospitals - like most hospitals worldwide - still run on HL7 v2. Patient admissions, lab results, orders - they all flow as pipe-delimited messages over MLLP. But the direction is clear: the Australian Digital Health Agency (ADHA) is pushing hard toward FHIR R4 through the Sparked FHIR Accelerator and the AU Core Framework. Somebody has to build the bridge between these two worlds.

I decided to build one myself: a working pipeline that takes real HL7 v2 maternity messages and transforms them into valid, AU-profiled FHIR R4 resources. Not a toy. Not a tutorial. A real integration that I could show in interviews and say: "I built this. Here's how it works. Here's where it breaks."

GitHub: budityw23/maternity-hl7-to-fhir-pipeline

This article walks through how I designed it - including the mistakes I made along the way.

My Initial (Naive) Approach

My first instinct was simple. HL7 v2 comes in, I parse it in Python, I build a FHIR JSON, I POST it to a FHIR server. Done.

Hospital Maternity System
  |
  |  HL7 v2 messages (MLLP)
  v
Python Script
  |  -> parse HL7
  |  -> build FHIR JSON manually
  |  -> POST to FHIR server
  v
HAPI FHIR Server

I wrote a quick script. It worked for a single ADT^A01 (patient admission) message. I felt good about it. Then the problems started:

No MLLP handling: My script was reading messages from a file. Real hospital systems don't send files - they send HL7 over MLLP (a TCP-based protocol with specific framing). My script had no way to receive live messages.
Brittle parsing: I was splitting on | and counting positions. One unexpected empty field and the whole mapping shifted. A message with a maiden name in PID-6 but nothing in PID-5.4 (suffix) would silently put the wrong data in the wrong FHIR field.
No validation: I was constructing FHIR JSON by hand as Python dicts. Typo in a field name? Missing a required element? The script happily sent invalid resources to the FHIR server, which sometimes accepted them and sometimes threw cryptic errors.
No multi-resource linking: A maternity visit involves a Patient, an Encounter, Observations (blood pressure, weight, fetal heartbeat), and a Condition (pregnancy diagnosis). These resources need to reference each other. My script created them independently with no linking - the Observation had no idea which Encounter it belonged to.
No error trail: If the Patient was created successfully but the Observation failed, I had no record of what went wrong, no correlation between the original message and the failure, and no way to replay the failed message.

I was building for the happy path again - the same mistake I described in my vaccine appointment system article. One clean message in, one clean resource out. But healthcare data is never clean.

Rethinking the Architecture

The core insight was: this is not a single transformation. It's a pipeline with distinct responsibilities.

Receiving MLLP messages is a different concern from parsing HL7. Parsing is different from mapping. Mapping is different from FHIR validation. Validation is different from persistence. Each stage can fail independently, and each needs its own error handling.

I also realized that an integration engine like Mirth Connect already solves the hardest part - MLLP reception, HL7 parsing, and message routing. Fighting that battle in raw Python was wasting time on a solved problem.

Here's the redesigned architecture:

Hospital Maternity System
  |
  |  HL7 v2.5 messages over MLLP:
  |    ADT^A01  -> patient admitted for antenatal care
  |    ORM^O01  -> antenatal checkup ordered
  |    ORU^R01  -> checkup results (BP, weight, fetal heartbeat)
  v
Mirth Connect 4.5 (port 6661)
  |  -> receives MLLP, parses HL7 natively
  |  -> extracts key segments (PID, PV1, OBX, DG1, OBR)
  |  -> builds structured JSON payload
  |  -> routes by message type to correct endpoint
  v
Python 3.11 + FastAPI (port 8000)
  |  -> receives typed JSON from Mirth
  |  -> maps fields to FHIR R4 with AU Base profiles
  |  -> validates via fhir.resources (Pydantic models)
  |  -> conditional PUT/POST to HAPI FHIR (idempotent)
  |  -> structured JSON logging with correlation ID
  v
HAPI FHIR Server v7.0.3 (port 8080)
  |  -> stores Patient, Condition, Observation, Encounter
  |  -> supports $validate for profile conformance
  |  -> serves FHIR REST API
  v
Accessible via standard FHIR queries

The key difference: every component does one thing well, and failure at any stage doesn't corrupt the others.

The Improved Design

1. Mirth Connect: The HL7 Gatekeeper

Mirth Connect listens on MLLP port 6661 and handles the protocol-level complexity that I was trying to reinvent in Python. It natively understands HL7 v2.5 segment structure, so I can reference msg['PID']['PID.5']['PID.5.1'] directly in a JavaScript transformer - no string splitting, no position counting.

Mirth does three jobs:

Receives and parses: Handles MLLP framing (the \x0b header and \x1c\x0d trailer), parses segments, and sends ACK/NACK responses back to the sender.
Extracts and restructures: A JavaScript transformer pulls the fields I need and builds a clean JSON payload. This is where I handle HL7 structural quirks - pulling the MRN and IHI out of the repeating PID-3 field by identifier type (MR vs NI), taking the first PID-13 repetition for the phone number, and normalising the country code (AUS -> AU). One thing that tripped me up here: Mirth parses each field into components, so a field node's toString() returns XML markup, not the value - you have to drill to the .1 subcomponent (msg['PID']['PID.8']['PID.8.1']) to get F. Terminology mapping - F/M/U sex codes to female/male/unknown, ICD-10-AM to SNOMED - is deliberately left to the FastAPI layer, where the Pydantic models validate it.
Routes by message type: ADT^A01 goes to /fhir/Patient, ORM^O01 goes to /fhir/Encounter, ORU^R01 goes to /fhir/Observation/bundle. Each endpoint in FastAPI knows exactly what shape of data to expect.

Why not do everything in Mirth? You can - plenty of teams build entire FHIR transformations in Mirth's JavaScript engine. But I wanted the FHIR validation layer in Python using fhir.resources, which gives me Pydantic-based validation against the full FHIR R4 spec. If I accidentally set Observation.status to "done" instead of "final", Pydantic catches it before it ever reaches the FHIR server. You don't get that level of type safety in Mirth's JavaScript.

2. The Mapping Layer: Where the Real Work Happens

This is the heart of the pipeline. Each HL7 message type maps to one or more FHIR resources:

ADT^A01 (Patient Admission) produces:

Patient - from PID segment (name, DOB, sex, address, IHI identifier)
Condition - from DG1 segment (pregnancy diagnosis, ICD-10-AM coded)

ORM^O01 (Order) produces:

Encounter - from PV1 segment (visit number, class, admission date, location, attending doctor)
References back to existing Patient (looked up by MRN)

ORU^R01 (Observation Result) produces:

Observation - from OBX segments (vital signs: blood pressure, weight, fetal heartbeat)
References back to existing Patient (by MRN) and Encounter (by visit number)

Here's where the complexity lives. A simple mapping like PID-5.1 -> Patient.name[0].family is straightforward. But consider blood pressure. In HL7 v2, systolic and diastolic come as two separate OBX segments:

OBX|1|NM|8480-6^Systolic BP^LN||120|mm[Hg]|90-120|N|||F
OBX|2|NM|8462-4^Diastolic BP^LN||80|mm[Hg]|60-80|N|||F

In FHIR R4, blood pressure is a single Observation resource with LOINC code 85354-9 (Blood pressure panel) containing two component[] entries - one for systolic, one for diastolic. The transformer detects consecutive BP-related OBX segments (codes 8480-6 and 8462-4), merges them into a single panel Observation with the AU Base blood pressure profile (au-vitalsigns-bloodpressure), and applies the most conservative status and worst-case interpretation across both readings. Orphan systolic or diastolic readings (without their pair) fall back to individual Observations. This is the kind of domain logic that makes healthcare integration genuinely hard - it's not just field-to-field mapping.

3. Australian Localisation

Since this targets the AU market, the FHIR resources conform to AU Base profiles on all resource types:

Resource	Profile
Patient	`http://hl7.org.au/fhir/StructureDefinition/au-patient`
Condition	`http://hl7.org.au/fhir/StructureDefinition/au-condition`
Encounter	`http://hl7.org.au/fhir/StructureDefinition/au-encounter`
Observation (BP)	`http://hl7.org.au/fhir/StructureDefinition/au-vitalsigns-bloodpressure`

Additional AU-specific details:

Patient identifiers include the IHI (Individual Healthcare Identifier) with system URI http://ns.electronichealth.net.au/id/hi/ihi/1.0
Diagnosis coding uses ICD-10-AM (Australian Modification) rather than plain ICD-10
Clinical terminology uses SNOMED CT-AU where applicable
Addresses use 4-digit Australian postcodes and state codes (NSW, VIC, QLD, etc.)
All FHIR datetimes include the +10:00 AEST timezone offset

This isn't just cosmetic. AU Base profiles have specific cardinality and terminology constraints. An au-patient resource without a valid identifier type code will fail validation against the profile. Getting this right demonstrates that I understand the difference between generic FHIR and FHIR as it's actually used in Australian healthcare.

4. Idempotency: Handling Duplicate Messages

Hospital systems sometimes send the same message twice - network retries, interface engine restarts, manual resends. The pipeline handles this through conditional operations:

Patient: Conditional PUT using ?identifier=http://hospital.local/mrn|{MRN} - re-sending the same ADT^A01 updates the existing Patient rather than creating a duplicate.
Encounter: Conditional PUT using the visit number identifier - same ORM^O01 twice produces one Encounter.
Condition and Observation: Created via POST (new resource each time), but because they reference the idempotent Patient/Encounter, the referential links remain consistent.

This approach trades off perfect idempotency on Conditions/Observations for simplicity. In a production system, you might add conditional logic based on a combination of patient reference, code, and effective date. For a portfolio project, the Patient/Encounter idempotency demonstrates the pattern clearly.

5. Failure Scenarios and Observability

Just like with the vaccine system, I forced myself to think about what breaks:

Malformed HL7 message: Mirth rejects it at the protocol level and sends a NACK. The message never reaches FastAPI. Logged for review.
Missing required fields: FastAPI's Pydantic models catch empty MRNs, invalid gender codes, and missing observation codes at the payload boundary. Returns RFC 7807 application/problem+json with a 422 status.
FHIR validation failure: The fhir.resources Pydantic model rejects the resource - maybe Observation.status has an invalid value, or a required CodeableConcept is missing. The error includes exactly which field failed and why.
Patient not found for downstream messages: An ORM^O01 or ORU^R01 arrives before the ADT^A01. The pipeline looks up the Patient by MRN, fails to find it, and returns a 422: "Patient not found for MRN=X. Send ADT^A01 first."
HAPI FHIR rejects the resource: The FastAPI layer catches the HTTP error, writes the original payload to a dead-letter file in ./deadletter/ (with the correlation ID as filename prefix), and returns a 502 with the HAPI error details.

Correlation ID ties everything together. Every message gets a UUID (either from Mirth via X-Correlation-ID header, or auto-generated by middleware). The correlation ID appears in every structured JSON log line, in error responses, in dead-letter filenames, and in response headers. When something fails at 3am, you grep one ID and see the entire message lifecycle.

Structured JSON logging ensures every log line is machine-parseable - timestamp, level, logger name, message, and correlation ID in every entry. Noisy libraries (uvicorn access logs, httpx, httpcore) are suppressed. No PHI (names, dates of birth) in logs - only identifiers like MRN and correlation ID at INFO level.

6. Server-Side Validation

Client-side validation via fhir.resources catches structural errors - wrong field types, missing required elements, invalid enum values. But it can't validate against AU Base profile constraints. For that, the pipeline integrates with HAPI's $validate operation.

A standalone endpoint (POST /fhir/validate/{resource_type}) accepts any FHIR resource and returns the raw OperationOutcome from HAPI - useful for testing profile conformance during development.

For production use, a VALIDATE_BEFORE_PERSIST config flag enables pre-persist validation: every resource is validated via $validate before being written to HAPI. If validation finds errors or fatal issues, the persist is blocked and a 422 is returned with diagnostic messages. This is off by default (the demo seeds placeholder StructureDefinitions, not the full AU Base IG), but the plumbing is in place for when a real terminology server and IG package are loaded.

System Components

Here's the full component view:

+----------------------------------------------------------+
|                    Docker Compose                        |
|                    (maternity-net)                       |
|                                                          |
|  +-------------+    +--------------+    +------------+  |
|  |   Mirth      |    |   FastAPI     |    |  HAPI FHIR |  |
|  |  Connect     |--->|   Python      |--->|   Server   |  |
|  |  4.5         |    |   3.11+       |    |  v7.0.3    |  |
|  |              |    |              |    |            |  |
|  |  Port: 6661  |    |  Port: 8000  |    | Port: 8080 |  |
|  |  (MLLP)      |    |  (HTTP)      |    | (FHIR REST)|  |
|  +-------------+    +--------------+    +------------+  |
|                                                          |
|  Services:                                               |
|  - mirth       (nextgenhealthcare/connect:4.5)          |
|  - fastapi     (python:3.11-slim + FastAPI + uvicorn)   |
|  - hapi        (hapiproject/hapi:v7.0.3, H2 embedded)  |
+----------------------------------------------------------+

Mirth Connect 4.5 - The integration engine. Handles MLLP, parses HL7, routes by message type, sends structured JSON downstream.
Python FastAPI - The transformation and validation layer. Endpoints for Patient (/fhir/Patient), Encounter (/fhir/Encounter), Observations (/fhir/Observation/bundle), validation (/fhir/validate/{resource_type}), and health (/health). Dedicated transformer modules (patient.py, condition.py, observation.py, encounter.py). Uses fhir.resources for Pydantic-based FHIR R4 validation.
HAPI FHIR Server v7.0.3 - The FHIR repository. Stores resources, serves the REST API, supports $validate for profile conformance. Uses embedded H2 database for the demo (Postgres swap documented for production).

Everything runs with docker compose up. One command, full pipeline.

Repository Structure

maternity-hl7-to-fhir/
|-- docker-compose.yml
|-- fastapi/
|   |-- Dockerfile
|   |-- pyproject.toml
|   `-- app/
|       |-- main.py                   # FastAPI routes + lifespan
|       |-- config.py                 # pydantic-settings (env vars)
|       |-- errors.py                 # RFC 7807 problem+json
|       |-- logging_setup.py          # Structured JSON logging
|       |-- middleware.py             # X-Correlation-ID middleware
|       |-- models/                   # Pydantic payload models
|       |   |-- adt_payload.py        #   ADT^A01 input
|       |   |-- orm_payload.py        #   ORM^O01 input
|       |   `-- oru_payload.py        #   ORU^R01 input
|       |-- transformers/             # HL7 -> FHIR mapping
|       |   |-- patient.py            #   -> Patient resource
|       |   |-- condition.py          #   -> Condition resource
|       |   |-- encounter.py          #   -> Encounter resource
|       |   `-- observation.py        #   -> Observation (+ BP panel)
|       |-- clients/                  # HAPI FHIR client
|       |   |-- hapi_client.py        #   Upsert/create + $validate + profile seeding
|       |   |-- patient_resolver.py   #   MRN -> Patient ID lookup
|       |   `-- encounter_resolver.py #   Visit number -> Encounter ID lookup
|       `-- valuesets/                # Terminology mappings
|           |-- hl7_to_fhir_gender.py
|           |-- hl7_to_fhir_encounter.py
|           `-- hl7_to_fhir_observation.py
|-- hapi/
|   |-- Dockerfile
|   |-- application.yaml
|   `-- HealthCheck.java
|-- mirth/
|   |-- channels/
|   `-- code_templates/
|-- samples/                          # Synthetic HL7 messages
|-- scripts/
|   |-- mllp_send.py                  # MLLP test client
|   |-- demo.sh                       # Interactive walkthrough
|   `-- reset.sh                      # Full reset (down -v + clean)
|-- tests/
|   |-- unit/                         # 157 unit tests
|   `-- integration/                  # 20 integration tests
|-- deadletter/                       # Failed message store (gitignored)
|-- logs/                             # Runtime logs (gitignored)
`-- .github/workflows/ci.yml          # CI: lint + type check + test

What I Learned

Building this project taught me things I couldn't have learned from reading the FHIR spec alone.

HL7 v2 is deceptively simple. The pipe-delimited format looks easy to parse until you encounter repeating fields, component separators, escape characters, and the fact that different hospitals implement the same message type differently. An integration engine like Mirth saves enormous time here - it's purpose-built for this chaos.

FHIR validation is your safety net, not your enemy. My naive approach skipped validation because it felt like extra work. In practice, the Pydantic models from fhir.resources caught errors that would have taken hours to debug at the FHIR server level. A missing Observation.code.coding[0].system? Pydantic tells you immediately. HAPI FHIR gives you a generic 400. Adding server-side $validate on top catches profile conformance issues that client-side validation can't - like missing AU Base constraints.

Idempotency is not optional in healthcare integration. Hospital systems send duplicate messages routinely. Conditional PUT with identifier queries means the second submission of the same ADT^A01 updates the existing Patient instead of creating a duplicate. This is a simple pattern that eliminates an entire class of production incidents.

Correlation IDs are the single most useful observability investment. One UUID, generated at the entry point, propagated through every log line, every error response, every dead-letter file. When a message fails at 3am, you grep one string and see everything that happened. Without it, you're correlating timestamps across multiple services and hoping the clocks are synchronized.

The AU localisation is what separates a tutorial from a portfolio project. Anyone can map PID-5 to Patient.name. Knowing that Australian systems use IHI identifiers, ICD-10-AM coding, AU Base profile URLs, and SNOMED CT-AU - and encoding all of that correctly in the FHIR resources - shows real domain expertise. In interviews, this is what gets follow-up questions.

Start with failure scenarios, not the happy path. This is the same lesson from my vaccine appointment system design, and it applies everywhere. The first question to ask about any integration is not "how does it work?" but "what happens when it breaks?" RFC 7807 error responses, dead-letter queues, and structured logging aren't glamorous features - but they're the difference between "it works in a demo" and "I'd trust this in production."

Testing and Quality

The pipeline has 177 automated tests (157 unit + 20 integration) at 90% line coverage:

Unit tests cover every transformer, valueset mapping, payload validator, error handler, logging formatter, and middleware component in isolation.
Integration tests exercise the full FastAPI ASGI stack (routes, middleware, error handlers) with mocked HAPI responses using respx. Tests cover happy paths, validation failures, missing upstream resources, and correlation ID propagation.
CI pipeline runs ruff (linting), mypy --strict (type checking), and pytest with a coverage gate of 80% on every push.

What's Next

This pipeline covers 4 FHIR resources, 3 HL7 message types, and AU Base profile conformance - deliberately scoped to be buildable as a portfolio project while demonstrating real integration patterns. Extensions I'm considering:

Full AU Core IG loading - replacing the placeholder StructureDefinitions with the complete AU Base Implementation Guide package for meaningful $validate results
Terminology validation - integrating with a FHIR terminology server (like Ontoserver) to validate SNOMED CT-AU and LOINC codes at transformation time
Monitoring dashboard - a simple frontend showing message throughput, transformation success/failure rates, and recent errors
Additional message types - ORU^R30 (unsolicited lab), ADT^A08 (update patient), ADT^A03 (discharge)

The full source code, documentation, and Docker Compose setup are on GitHub: maternity-hl7-to-fhir-pipeline.

If you're working on HL7-to-FHIR integrations or preparing for health IT interviews, I'd like to hear what challenges you've run into. You might also find my earlier article on the National Vaccine Appointment & Administration System useful - it covers similar design thinking around failure handling and rollback patterns.

Source code: github.com/budityw23/maternity-hl7-to-fhir-pipeline

Tags: #fhir #healthit #hl7 #architecture

Our GCP daily cost dropped 80% in six weeks, here is every change we made

Budi Widhiyanto — Wed, 08 Jul 2026 14:09:43 +0000

When our project manager forwarded the May 2026 invoice, the number had tripled from where it was two months earlier. That was the monthly GCP bill for two regional deployments of the same platform. We had seen it climb through April, but this was the first time the daily cost chart showed no sign of flattening. The peak day had already hit 3x our normal baseline.

Nothing was broken. The converter was processing records and the dashboards were up. The system was doing exactly what it was built to do. Together with Mas @chiqors , our devops teammate, we pulled up the billing console and started figuring out whether the system actually needed to cost that much.

What we run

We operate a federated health data platform deployed in two districts in Indonesia. The platform collects data from twelve different source systems: electronic medical records from hospitals, nutrition tracking from community health workers, TB surveillance from the national system, immunization registries, midwife apps, and several others. All of that gets converted into FHIR R4 resources and stored in a central FHIR datastore per district.

Each district runs its own GCP project. The converter runs on Cloud Run alongside about 30 other microservices per project. BigQuery handles analytics and dashboard queries. Cloud Healthcare API is the FHIR datastore. Cloud Scheduler triggers the converter and about 50 scheduled queries across both projects. Then there is Secret Manager for credentials and checkpoints, Cloud SQL for application databases, a handful of Compute Engine VMs for data science and tooling, and Cloud Logging collecting output from everything.

At peak, we were processing over 500,000 records per day across 80 source tables. Two projects, two districts, running 24/7. That is a lot of moving parts generating cost, and before the spike, we had never looked carefully at which parts were generating how much.

Following the data to find the cost drivers

We did not start by guessing. We set up the billing export queries, pulled Cloud Run utilization data, and looked at INFORMATION_SCHEMA.JOBS_BY_PROJECT to understand where the money was actually going:

SELECT
  DATE(creation_time) AS dt,
  ROUND(SUM(total_bytes_billed) / POW(2, 40), 2) AS tib_billed
FROM `region-asia-southeast2.INFORMATION_SCHEMA.JOBS_BY_PROJECT`
WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
GROUP BY dt ORDER BY dt

This query gave us daily scan volume in TiB, which we could plot against the billing export to see exactly when costs started climbing and why. BigQuery was roughly 36% of the total bill. Cloud Run was another 21%. But the percentages alone did not explain the spike.

What the data showed was a compounding pattern. Several upstream data providers had pushed large historical backfills, which is a normal part of operating a health data platform. Migrations happen. New data sources come online. One upstream migration brought in 427,000 rows in one week (43x the normal volume). Another pushed 2.27 million rows in a single week.

The system handled it correctly. The converter processed the records, and the FHIR store grew. The main resource table went from 150 GiB to 201 GiB. But a deduplication scheduler was scanning the full FHIR store three times per hour. At 150 GiB, each scan cost very little. At 201 GiB and growing, the same scheduler became the single most expensive query in the system. The scan volume data on one project tells the story clearly:

Period	Daily BQ scan
Week 1	2.36 TiB
Week 4	4.0 TiB
Week 5	7.37 TiB
Week 6	9.26 TiB (peak)
After scheduler pauses	1.34 TiB
After all fixes	0.47 TiB

From 2.36 TiB to 9.26 TiB in five weeks, without anyone changing a single scheduler configuration. The data volume changed, and the cost followed. That same pattern was happening on the other project too, going from 1.1 TiB to 2.85 TiB daily, though the absolute numbers were smaller.

What the data told us to do

Once we had the billing and utilization data in front of us, we split the work across BigQuery, Cloud Run, Compute Engine, and the smaller services. Each fix started with a specific metric.

BigQuery: query analysis pointed to the converter polling pattern

We used INFORMATION_SCHEMA.JOBS_BY_PROJECT to rank queries by bytes billed. The top consumer was the converter's polling query, which checks for unprocessed records using an anti-join against a multi-million-row ledger table. Every poll for every table scanned the entire ledger.

The scan data told us a CLUSTER BY table_name on the ledger, combined with adding a table_name filter to the JOIN clause, would let BigQuery prune 90% of the scan. The change was small:

-- Before: scans entire ledger on every poll
LEFT JOIN processing_report AS B
  ON A.uuid = B.uuid

-- After: BigQuery prunes to just this table's slice (~94% fewer bytes)
LEFT JOIN processing_report AS B
  ON A.uuid = B.uuid AND B.table_name = 'source_table_name'

After deploying that change, per-query scan dropped from 531 MiB to 53 MiB.

We then looked at the pattern more carefully. The data showed that on a typical hourly run, most tables have zero new rows. We built a high-water mark routing layer: if a table has no backlog, use a cheap timestamp query instead of the full anti-join.

def get_unprocessed_data(table_name):
    """Routes to HWM or anti-join based on backlog check."""
    if table_name not in _HWM_READY_TABLES:
        backlog = _check_backlog(table_name)
        if backlog == 0:
            _HWM_READY_TABLES.add(table_name)
        else:
            return _get_unprocessed_antijoin(table_name)
    return _get_unprocessed_hwm(table_name)

The HWM path runs a simple WHERE timestamp > last_processed instead of joining millions of rows. This brought the converter's BigQuery cost down by over 95%.

BigQuery: evaluating Enterprise Edition after reducing query volume

Both projects were running BigQuery Enterprise Edition with autoscaling slots. Enterprise Edition pricing is based on slot usage, and when scan volume was high, the autoscaler was spinning up slots constantly. This was the single largest line item on the bill.

After deploying the clustering fix, the high-water mark routing, and pausing the expensive schedulers, our actual BigQuery compute usage dropped by over 90%. We looked at the numbers and asked whether Enterprise Edition still made sense. The autoscaling slots were now mostly idle, but we were still paying for the baseline. We deactivated Enterprise Edition on both projects and fell back to on-demand pricing.

The result was immediate. The projected BigQuery cost on on-demand is about 4% of what we were paying with Enterprise Edition. The query volume reductions we had already made meant that on-demand pricing was now far cheaper than even the minimum Enterprise Edition commitment. This single change, deactivating a pricing model that no longer matched our usage, was the largest cost reduction in the entire effort. There is a risk: if scan volume spikes again (another large backfill, or re-enabling the dedup scheduler), on-demand could become more expensive than Enterprise Edition was. We are monitoring on-demand costs weekly to catch that before it happens.

Scheduled query audit: frequency, dead joins, and dead code

We listed every scheduled query and its run frequency, then compared it against when the output data was actually consumed. The gaps were wide.

One dedup scheduler ran a UNION ALL full-scan of every FHIR resource table three times per hour. One scheduler, hundreds of dollars per month. We checked its downstream consumers and found it could safely be paused while we work on a better dedup approach.

We found schedulers with a dev- prefix running in the production project. One ran every 7 minutes, another ran every hour. They did not produce errors, so they never surfaced in monitoring. They only showed up when we audited the scheduler list against actual usage.

Export queries ran every 12 hours but were consumed once a day. Dashboard refresh queries ran every 5 hours but updated a table used in a daily report. We matched frequency to actual consumption patterns across both projects and paused or reduced 26+ schedulers.

Rewriting scheduled query SQL: the dead JOIN that ate 96% of slot-time

Frequency was not the only problem. Some scheduled queries were expensive because the SQL itself was inefficient.

Two scheduled queries that build a maternal health dashboard table started timing out at 6 hours. The SQL had not changed. The data had. Both queries contained a LEFT JOIN against a clinical orders table. The joined alias was referenced only in the ON clause, never in any SELECT or WHERE. It did nothing except multiply rows. The orders table stores one row per ordered lab test, so each visit record (full screening panel) fanned out 113 times. With 362,000 visit records, the intermediate result was 41 million rows before SELECT DISTINCT collapsed them back down.

This was harmless when the orders table was small. Then bulk conversion of two data sources landed 8 million new rows in one week. The fan-out blew past the 6-hour query timeout. When we queried INFORMATION_SCHEMA.JOBS_BY_PROJECT for slot usage, these two queries had consumed 96.1% of all project slot-time over the previous three days (62.6% and 33.5% respectively), leaving 3.9% for the 35,582 other queries.

The fix was one line: comment out the dead JOIN. Run time dropped from 6 hours (timeout) to 25 minutes. We also rewrote the partitioned update query to read from a 52 MB computed dashboard table instead of scanning 1 TB of raw FHIR tables directly. That cut its monthly cost from about USD 130 to USD 3.

The 4-billion-row table that should have been 12,000

One table had 4,040,397,903 rows. The actual data was 12,033 records.

A scheduled query had been running with WRITE_APPEND since February, duplicating every row on each run. The table grew to 764 GiB. This showed up when we sorted scheduled query outputs by storage size. We rebuilt the table from source, switching to WRITE_TRUNCATE. If we had not looked at storage sizes, this table would still be growing.

Cloud Run: utilization metrics vs. allocated resources

We pulled 30-day p99 CPU and memory utilization for every Cloud Run service across both projects. The gap between allocated and actual was large. One service had 4 CPU and 12 GiB allocated, with peak utilization at 12.3% CPU and 1.0% memory. Several services had similar profiles: provisioned for a load that never came, or provisioned for a peak that happened once and never returned.

Based on the utilization data, we downsized 44 services in one session, deleted 4 with zero traffic in 30 days, and switched 17 idle always-on services to request-based billing. The Cloud Run revision history shows it all happened in about 13 minutes. One project's daily compute cost dropped by 57% the next day.

We also right-sized the converter itself from 4 CPU / 16 GiB to 2 CPU / 4 GiB, and reduced the trigger frequency from every 30 minutes to hourly. During the backfill peak, the higher spec made sense. The converter was processing 571,000 records per day across 80 tables, running 20 worker threads with batches of 25 records. After the backfill cleared, utilization data showed the converter sitting well below capacity. We lowered workers and batch sizes to match the new steady state: 2 table workers, batches of 5, query limit down from 100,000 to 15,000.

Secret Manager: storage billing we were not tracking

We also went through every line item in the billing export, including services we assumed were cheap. Secret Manager was not cheap. Our patient deduplication service stores its checkpoint as a secret version, creating a new version on every run. Since late 2025, old versions were never destroyed. We found over 30,000 accumulated versions. The "Secret version replica storage" SKU was a top-10 line item on the bill. After cleanup, each checkpoint secret has 1 version. Cost dropped significantly.

Compute Engine: the GPU VM and the static IPs

Compute Engine was 6.6% of the bill. Most of that was expected: 10 VMs running various services (data science, form collection, OCR, migration tools). But one line item stood out.

A GPU VM with an NVIDIA L4 and 1 TB disk had been provisioned for OCR model fine-tuning. The fine-tuning work had finished weeks earlier. The VM was still running. It was the single most expensive Compute Engine resource. After confirming the fine-tuning was complete, we deleted it.

We also found unused static IP addresses being billed across both projects, some attached to services that no longer needed them. Small amounts individually, but they add up when nobody is watching.

Cloud SQL: fewer resources for the same workload

Cloud SQL was running multiple PostgreSQL instances 24/7. After reviewing actual query loads, we right-sized the instances. Total Cloud SQL savings: -36%.

Cloud Logging: paying for logs nobody reads

Cloud Logging on one project dropped 45% after reducing log levels and removing redundant logging. On the other project, logging actually increased 50% because the converter was processing more records after the backfill work. We still need to add exclusion filters there.

External API costs

We use LLM API calls for several features in the platform. One service makes an API call per record to validate field mappings during conversion. Another runs nightly to generate data quality summaries. A few others were experimental tools from earlier prototyping that were still hitting the API on a schedule. These API calls were costing more than Cloud Logging across both projects.

We reviewed each caller: how often it ran, how many API calls per run, and whether anyone was actually using the output. Two experimental tools had not had their output checked in over a month. The field validation service was running on every converter cycle, but we could batch the calls and reduce frequency without affecting quality. After removing the unused callers and batching the rest, the cost dropped 75%. The lesson here is the same as with schedulers: things that run automatically tend to keep running long after the reason for running them has passed.

Billing visibility: the foundation for everything else

We set up the GCP billing export to BigQuery in late April 2026. Before that, our only visibility was the monthly invoice and the GCP console billing dashboard. We could see the total was going up, but we could not see why. Once the billing export was in place, we could break down costs by SKU, by day, by service account. We could compare any two dates for a specific service and see exactly what changed. Every fix in this article started with a query against that billing export table. Without it, we would still be guessing.

The numbers

All numbers are from the GCP billing export. The daily cost is what matters, because it shows when each optimization took effect.

Daily cost trend (combined, both projects):

Period	Daily cost	What happened
Baseline	1x	Normal operations
Data volume growth	1.5x - 2x	Backfill volume hits BQ costs
Peak day	3x	Backfill volume + dedup cascade
BQ scans climbing	2x - 2.8x	Scheduler scans compounding
BQ fixes deployed	1.8x	Clustering + scheduler pauses
Cloud Run downsized	0.7x	Utilization-based downsizing
HWM deployed	0.6x	Converter BQ near-zero
Current	0.6x	Lowest point so far

From peak to current: an 80% reduction in daily cost, in six weeks.

Three optimization waves are visible in the data. First was BigQuery (clustering and scheduler pauses). Second was Cloud Run (utilization-based downsizing). Third was the high-water mark deployment.

Service-level changes (monthly):

Service	Change
BigQuery (Enterprise Edition)	-41%, then deactivated (Jul: ~0)
Cloud Run	-26%
Cloud SQL	-21%
Secret Manager	-26%
Cloud Healthcare API	+24% (data ingestion spike)
Cloud Logging (one project)	-45%
External API	-75%

Not everything went down, and that is worth understanding. Cloud Healthcare API increased 24%. The billing data explains why: upstream data providers were pushing large historical backfills during this period. One migration drove Healthcare API cost up 25x per day during the last two weeks of May. Another pushed daily Healthcare cost up 6x starting in early June. More records coming in means more FHIR writes, and that cost is proportional to actual data volume. It went up because the platform was doing what it was built to do.

Cloud Logging on one project increased 50% for a similar reason: more incoming data means more converter runs, more logs. We have not added exclusion filters there yet, so that is still on the list.

What we learned about the process

Start with the billing data. We did not guess which services were expensive. We exported billing data, queried INFORMATION_SCHEMA for scan volumes, and pulled utilization metrics from Cloud Run. Every optimization started with a number that told us where to look. The billing export showed us Secret Manager costs we would never have guessed. The utilization data showed us Cloud Run services at 1% capacity that had been running for months. Without the data, we would have started with the wrong services.

Schedulers are the silent cost multiplier. They get set up once and forgotten. We found scheduled queries running every 5 hours for dashboards refreshed once a day, and export queries running every 12 hours for data consumed once a week. The cost of each individual scheduler was small, but we had over 50 of them across both projects. Listing every scheduler alongside its actual consumer was the single most productive exercise we did.

The Cloud Run utilization review is a good example of how little effort some of these wins take. We spent about 30 minutes pulling p99 CPU and memory numbers for every service, sorted by gap between allocated and actual. That 30-minute exercise identified a large portion of total savings. The gap between "provisioned for the worst case" and "what actually happens" grows silently. We now plan to run this review monthly.

What is next

We still have optimization targets identified from the data. The FHIR export tables need partitioning by meta.lastUpdated so we can re-enable the dedup scheduler without recreating the scan cost problem. Cloud Logging on one project needs exclusion filters. Several Compute Engine VMs are running at low utilization. We also need to keep monitoring BigQuery on-demand costs to make sure they stay below what Enterprise Edition would have cost.

The daily cost is down 80% from peak. We are aiming to stabilize at the current level. The data will tell us when we get there.

Fixing 168K Failed FHIR Conversions with Parallel AI Agents and Git Worktrees

Budi Widhiyanto — Fri, 17 Apr 2026 16:30:19 +0000

One afternoon we ran a simple BigQuery query just to check how our FHIR converter was doing. Nothing special, we had been busy with new features for weeks and someone on the team just wanted to see the numbers.

The result was worse than we thought.

100.0% | tb_tracker_table_a          |  7,753 failures
 99.4% | nutrition_app_table_a       | 23,602 failures
 77.2% | hepatitis_app_table_a       | 22,437 failures
 43.2% | health_service_table_c      |  8,096 failures
 28.1% | health_service_table_a      | 168,713 failures

168,713 failed conversions on one table. Thirteen tables above 25% failure rate. We had a backlog that had been quietly growing while we were looking the other way.

If you have ever maintained a data pipeline, you know how this feels. The normal way to fix it is one table at a time. Pull some failed records, find the error, patch the template, run the tests, move on. Two or three hours per table when you know the codebase. Thirteen tables means weeks of focused work.

We had been experimenting with AI agents for this kind of maintenance work, and they were actually pretty good at it. The problem was that we could only run one agent at a time. Two agents on the same branch would step on each other almost immediately.

So we were running them one after another, which mostly defeated the point.

What finally unblocked us was a Git feature that had been sitting there for years and we had not really used: git worktree. This is the story of how we went from "we will fix this when we have time" to clearing the whole backlog in a few days.

a bit of context

Our team maintains a FHIR R4 converter for a national health platform. It pulls health data from local government systems and converts it into FHIR resources that flow into the central platform serving more than 2,000 healthcare facilities across two districts, from small community health posts to hospitals. Immunization records, maternal care, TB treatment, nutrition monitoring, community health screening. The data sources cover around 30 different table types between the two districts.

If you have not worked with FHIR before, the short version is this. FHIR R4 is the international standard for exchanging healthcare data. Every record has a strict structure, required fields, and value sets that the validator checks. A patient's gender cannot just be any string. It has to be one of the allowed codes. A date cannot be empty if the resource needs it. A coded value has to come from the right terminology system. If anything is wrong, the FHIR server rejects the resource and the record never enters the platform.

So when we say "conversion failure", we are not just saying the script crashed. We are saying clinical data from a local health facility never made it into the national system. Lab results, immunization shots, maternal visits. Gone, until we fix it.

Each data source has its own structure and its own quirks. One district had been getting less attention for a while. We did not ignore it on purpose. We always had to fix the most urgent problems first, and the rest kept piling up. You probably know how that goes.

So when we saw those failure numbers, the first reaction was honestly just tired. We knew this work was waiting. We just had not had the bandwidth to face it.

why one agent at a time was not enough

Before we get to worktrees, here is what the agent workflow actually looks like, because that is where the bottleneck became clear.

For every failing table, the steps are roughly:

Pull 100 failed UUIDs from BigQuery
Check data quality, fill rates, date format issues
Test UUIDs one at a time to find the actual error
Fix the template or the shared utility function
Run all 100, make sure at least 90% pass
Delete the failed rows from the report table so the system picks them up again

A note on the 90% threshold in step 5. We accept that the last few percent often come from genuinely bad source data that we cannot fix on our side. Records with corrupted fields, encoding issues from old systems, or data that should not have been entered in the first place. Chasing 100% on every batch means spending hours on records that are not really fixable. 90% is the threshold where we stop and move on.

This is the kind of work where an agent shines. It is not creative work. It is patient detective work that follows a clear pattern.

A concrete example. One table was failing because a date field was coming in as "2024-01-15 00:00:00" instead of "2024-01-15". Our convertStringToDate function expected %Y-%m-%d and returned empty when given the datetime format. Empty date meant a required FHIR field was missing. The validator rejected the resource. Record gone.

The agent found this in about 20 minutes. The third UUID hit the error, the agent walked back to the utility function, added a fallback for the extra time part, then ran the full test batch again to confirm. The actual code change was 4 lines, with a clear explanation of what was wrong.

Another table had a similar issue but in a different shape. A boolean field was arriving as the string "true". FHIR boolean fields expect an actual boolean primitive, so the validator rejected it with expected boolean: found "true". Same workflow. The agent found a function returning ["true"] instead of [True], fixed it, and checked that no other templates depended on the old behavior before merging.

So the per-table work was fine. The bottleneck was scale.

If we started one agent on nutrition_app_table_a and another on tb_tracker_table_a, they would conflict almost immediately. Both agents change tests/test_specific_uuids.py to set up their tests. Both might also touch shared functions in extra_logics/general.py if they find a common bug. On the same branch, in the same directory, they overwrite each other within minutes.

We tried the obvious workaround first. Separate clones of the repo, one per agent. It worked, but it was slow to set up, ate disk space, and made it annoying to share git history. Each clone had its own .git folder, its own remotes to configure, its own everything. We also forgot once to pull the latest main into one of the clones, and the agent fixed a bug that was already fixed on main. Wasted half an hour on that one.

Then we read the worktree docs properly and realized this was the tool we had been needing all along.

what worktrees actually do

If you are like us and have been using Git for years without touching worktrees, here is the short version.

A git worktree lets you check out multiple branches in separate folders at the same time. Each folder has its own files and its own changes. They all share the same .git folder.

git worktree add ../converter-fix-1 -b fix/nutrition-app-a
git worktree add ../converter-fix-2 -b fix/tb-tracker-a
git worktree add ../converter-fix-3 -b fix/hepatitis-app-a
git worktree add ../converter-fix-4 -b fix/health-service-a

Four folders, four branches, one repository. The extra storage is small because we are not copying the whole .git folder, only the working files.

We knew roughly what worktrees do. What we did not realize until recently is that this is exactly what we needed for parallel agents. Each agent gets its own folder. They never see each other's files. Agent 1 can be halfway through testing nutrition_app_table_a while Agent 2 is just starting on tb_tracker_table_a. No conflicts, no waiting for each other.

This is the kind of feature that is great to know about even if you do not need it today. The day you do need it, you will remember it is there.

how we organized the actual run

With 13 failing tables, we sorted them by how many failed conversions they had. We also checked which tables share the same template file, because fixing one template sometimes fixes several tables at once. Worth doing this step. We saved a lot of duplicate work this way.

The first batch had four worktrees running at the same time:

worktree	table	failed conversions
1	health_service_table_a	168,713
2	health_service_table_b	45,707
3	nutrition_app_table_b	28,977
4	hepatitis_app_table_a	22,437

Each agent got a prompt like this:

please create a new worktree for fixing health_service_table_a,
then run the fix-converter-failures workflow.

TABLE_NAME=health_service_table_a
CODE=health_service_code_a
BQ project=your-project

After the process is READY TO DEPLOY, merge back to the main branch,
commit and push, then delete the worktree.

The prompt looks short because fix-converter-failures is a workflow file we maintain in the repo. It defines the 6 steps above and the conventions the agent should follow, so the prompt itself only needs to say which table to run it on. Building this workflow file took us a few iterations, but once it was stable, kicking off a new fix became a one-line task.

The agents ran independently. While worktree 1 was checking fill rates on the patient service table, worktree 2 was already on its third UUID. When one agent found a bug in a shared utility function, it fixed the bug in its own branch, and the other agents kept going.

The first batch did not go perfectly. We hit a small problem on the second day. Two agents found the same bug in convertStringToDate at almost the same time, and both fixed it slightly differently. One added a fallback, the other rewrote the function to use dateutil. Not a real conflict, since each fix was on its own branch, but during merge we had to pick one and revert the other. We added a rule for the next batch: when an agent touches shared code, it has to flag that in the PR description so we know to check for parallel fixes.

After the first batch finished and merged, we started the next four. The whole experience felt more like reviewing work than doing it. Honestly, this took a moment to get used to. After a few rounds, the rhythm became natural. Open four terminals, kick off four agents, let them run, come back to review.

the other half: template coverage

Fixing failures was one problem. The other quality issue was something we did not even have a name for at first. We were calling it "silent data loss".

We have a coverage check that compares each template against its data dictionary and counts what percentage of the source fields are actually mapped to FHIR. If it is below 90%, the template needs more work.

With 15 templates to check, we used the same approach. Four worktrees, grouped by data source type:

worktree 1: maternal death reporting templates
worktree 2: nutrition monitoring (three templates)
worktree 3: community health screening (four templates)
worktree 4: immunization and TB treatment

Agents ran coverage analysis on each template. If it passed 90%, move on. If not, add the missing fields, test, validate against the FHIR server, merge, delete the worktree.

This step found a different kind of problem than the failure fixes, and honestly the more important one. A template can have a 0% failure rate and still be missing half the fields it should be mapping. The records were converting successfully and reaching the FHIR server clean. But blood glucose, cholesterol, abdominal circumference, lab results, risk factor flags, procedure details, all of it was being silently dropped because the template never mapped those fields.

The failure rate query does not show this. The FHIR server does not show this. From every monitoring view we had, the system looked healthy. But the clinical data we were supposed to be capturing was not actually arriving.

This part was uncomfortable to look at. We had been measuring success by "no errors" without checking if we were actually capturing the data correctly. Some templates were already above 90% and needed no changes. Others were well below.

One community health screening template was correctly mapping vital signs but was missing several fields that the screening program actually collects. Blood glucose, cholesterol, abdominal circumference. All present in the source data, all listed in the data dictionary, just never connected to FHIR Observation resources. The agent added them, ran the tests, validated against the FHIR server. That kind of work would normally sit in a backlog for weeks.

A small thing we did not expect. Some data dictionaries were split into multiple CSV files. One screening app reference came in four separate files with around 1,400 rows total. The agent had to combine them before running the analysis. Worth knowing before you start.

We also had one false positive. The agent reported 78% coverage on a template that was actually fine. The data dictionary listed fields that were deprecated and no longer used in production. We had to manually verify the missing fields before adding code for them. Coverage numbers are useful, but the data dictionary itself can be stale, and the agent cannot know that without help.

a few things worth knowing if you try this

Sharing what tripped us up so you do not have to learn the same way.

Worktrees share your git history and remotes. A commit you make in ../converter-fix-2 will show up in git log from your main folder. git push works normally from any worktree. This is convenient once you are used to it, but it can feel weird at first.

Config files that are not committed need to exist in each worktree folder separately. Credential files, .env, anything in .gitignore, each worktree needs its own copy. We learned this the hard way when the first agent could not connect to BigQuery and we spent 15 minutes thinking the credentials were wrong. We keep a short setup note for this now.

Name your worktrees after what they are doing, not just numbers. ../fix-nutrition-app is much easier to work with than ../worktree-2 when you have four terminals open at the same time. Trust us on this.

Cleanup is one command:

git worktree remove ../converter-fix-1

The folder is removed and the worktree reference is cleaned up. The branch and its commits stay in the repository, in case you need to revisit them later.

For us the real lesson from this work was not about worktrees, and not even really about agents. It was about what we were measuring.

For months we had a dashboard that said the converter was healthy. Failure rates were low on most tables, the FHIR server was accepting resources, alerts were quiet. The dashboard was not lying. It was just answering a different question from the one that mattered for healthcare data. "Is the conversion working" is not the same as "is the clinical data arriving correctly". We were tracking the first one and assuming it answered the second.

Worktrees and agents did not solve that. They just made it cheap enough to fix once we noticed.

If you are working with agents on tasks that are similar and do not depend on each other, the parallel worktree pattern is worth trying. The setup takes a few minutes. The harder part is changing how you think about the work. From one task at a time to groups of tasks running side by side. That feels strange at first and gets natural quickly.

For us, the result was clearing months of backlog in a few days, plus the coverage findings we did not know we were missing. The agents are not magic. We still review every change carefully, especially anything that touches shared code or terminology mappings. But letting them work in parallel changed what was actually possible for our team, on a system that more than 2,000 healthcare facilities depend on.

How I Built an AI Agent That Writes FHIR Templates in Hours Instead of Days

Budi Widhiyanto — Sat, 11 Apr 2026 15:49:03 +0000

3-4 days to write one JSON template. Sometimes a full week.

It was a Tuesday night and I had the FHIR specification open in twelve browser tabs. On my left monitor, a CSV data dictionary with 150 fields from a Kobo maternal health form, postnatal care, the kind that tracks five KF checkups for the mother and four KN visits for the newborn. On my right, a half-finished JSON template already 200 lines deep. I wasn't even past the Encounter resource yet.

Each visit block needed its own set of vital signs: blood pressure (systolic and diastolic, split from a combined "120/80" string), pulse rate, respiratory rate, temperature. Each needed clinical observations like uterine fundal height, lochia status, breastfeeding assessment. Each needed the right LOINC code, the right data type handler, the right reference chain connecting it back to the Patient and Encounter it belonged to. Five mother visits times eight to twelve observations each, plus four newborn visits with their own observation sets, plus the baby Patient resource with mother-baby linkage via NIK identifiers.

I did the math. This template would land somewhere around 320 resources and several thousand lines of JSON.

I'd done this before. Many times. Four days minimum, probably five. And somewhere around day two I'd lose my mental model of the reference graph, come back the next morning staring at field mappings I no longer remembered the reasoning behind, spend an hour just re-orienting before I could write another line.

That night I closed the FHIR spec tabs and started building the agent instead.

The Old World

Some context. I'm a software engineer working on health data interoperability for district health systems in Indonesia, specifically Purbalingga and Lombok Barat. We pull data from over a dozen source systems: ePuskesmas (community health center records), Kobo Toolbox (mobile data collection forms), SiGizi (nutrition surveillance), SITB (tuberculosis registry), SIHEPI (hepatitis tracking), SIMRS (hospital information systems), and more. Each system has its own data model, its own field names, its own quirks. Our job is to convert all of it into FHIR (Fast Healthcare Interoperability Resources), the international standard for healthcare data exchange.

FHIR, if you haven't worked with it, models healthcare data as discrete "resources." A patient is a Patient resource. A doctor visit is an Encounter. A blood pressure reading is an Observation with a specific LOINC code (85354-9), split into systolic and diastolic components, each coded separately. A diagnosis is a Condition. A vaccination is an Immunization with a CVX vaccine code and dose protocol. Every resource links to others through references: an Observation points to the Patient it belongs to, the Encounter it was recorded during, the Organization that performed it.

The conversion pipeline itself isn't the hard part. We have a Python engine that reads a JSON template, maps incoming data fields to FHIR resource structures, runs transformation functions, and POSTs the result as a FHIR Bundle to Indonesia's national FHIR server (SatuSehat). The hard part is writing the template.

A template in our system is a JSON file that describes every FHIR resource the converter should produce from a given data source. Here's what even a simple resource definition looks like. This is a real Patient resource from a pregnancy tracking form:

{
  "resourceType": "Patient",
  "fullUrl": "urn:uuid:patient",
  "keyField": "[nik_ibu]",
  "mergeWithExistingData": true,
  "requiredFields": {
    "id": "[generated/patient_res_id]",
    "identifier.0.system": "https://fhir.kemkes.go.id/id/nik",
    "identifier.0.value": "[generated/nik_adjusted]",
    "name.0.text": "[nama_ibu]",
    "birthDate": "[generated/birth_date]"
  },
  "extraLogics": [
    {
      "type": "getPatientIdWithFuzzyLogic",
      "input": ["", "[generated/nik_adjusted]",
               "[nama_ibu]", "[generated/birth_date]",
               "female", ""],
      "output": ["[patient_existing_id]"]
    }
  ]
}

The [square_bracket] notation maps to incoming data fields. The [generated/...] notation refers to values produced by transformation functions, called extraLogics, that run before the FHIR resource is assembled. In this example, getPatientIdWithFuzzyLogic queries the FHIR server to find if this patient already exists (matching on national ID, name, birth date, and gender).

That Patient resource is one of two resources in the simplest template in our system: 279 lines of JSON total. Our hospital templates run to 300+ resources and nearly 500KB. The postnatal care form that broke me that Tuesday night had 320 resources across about 5,000 lines.

Manual template creation took 3-4 days of wall-clock time. Sometimes a full week. And that's with the hardest intellectual work already done for me.

Before I write a single line of JSON, our health informatics team has already spent their own days (sometimes weeks) building the CSV data dictionary. They're the ones who decide that berat_badan maps to LOINC 29463-7 "Body weight," that tekanan_darah should become a blood pressure panel with code 85354-9, that this particular field is a SNOMED-coded Condition and not just a free-text Observation. That work requires clinical domain knowledge I don't have. They understand what these data points mean in a maternal health context, which FHIR resource types are clinically appropriate, which code systems are correct. I turn their decisions into working JSON. Without that CSV, I'd be guessing at clinical semantics, and guessing wrong.

So even with a carefully prepared data dictionary in hand, the template creation itself still took 3-4 days.

The first day was mapping. Read the CSV line by line. For each field (berat_badan, tekanan_darah, tinggi_fundus, denyut_jantung_janin) decide which FHIR resource type it becomes. Observation for vitals, Condition for diagnoses, Procedure for clinical actions. Pick the clinical code: LOINC 29463-7 for body weight, SNOMED 364589006 for fundal height measurement. Choose the data type handler. Numeric fields need handleNumericValueOrDataAbsentReason, which returns four outputs: the value (or null), plus the data-absent-reason system, code, and display for when the field is empty. String fields, dates, booleans, coded values all have their own handlers.

Day two was wiring. Connect everything. Every clinical resource needs a subject reference pointing to the Patient, an encounter reference pointing to the Encounter, NIK (national ID) traceability in subject.identifier, and a meta.source field for audit provenance. Every numeric Observation needs addValueIfNotEmpty guards on the unit triple (valueQuantity.unit, valueQuantity.code, valueQuantity.system) because if the value is absent, emitting an empty valueQuantity object alongside a dataAbsentReason violates FHIR's XOR constraint. I'd forgotten this guard at least twice in earlier templates and spent hours debugging why the FHIR server rejected my bundles.

Day three was debugging. Run the converter against real sample data. Watch it crash. Trace the error to a missing generateUUID extraLogic, or an output array that doesn't match the function's return count, or a reference to a resource that was conditionally skipped because its keyField was empty in the sample. Fix, re-run, fix, re-run. Each cycle takes a few minutes but there are dozens of them.

If it stretched to a week, day four was review and edge cases. Check that every field in the data dictionary actually made it into the template. Verify code systems (is it http://loinc.org or http://loinc.org/? The trailing slash matters). Handle the weird cases: blood pressure stored as a single "120/80" string that needs to be parsed into two separate Observations with component coding. Baby-mother linkage where the newborn's Patient resource needs a nik-ibu identifier pointing back to the mother.

The worst part wasn't any single step. It was the context loss. Building a 200-resource template across multiple days means losing your mental model every night. You come back the next morning, open the half-finished JSON, and spend the first hour asking yourself: "Why did I use handleCodeableConceptOrDataAbsentReason here instead of handleValueOrDataAbsentReason? Was that intentional or a mistake?" The template is too large to hold in your head. You're constantly re-deriving conclusions you already reached yesterday.

Why an Agent, Not a Script

I didn't set out to "automate FHIR mapping with AI." I set out to make this work fit in a single focused session.

A Python script was my first thought. But template creation isn't a mechanical transformation; it requires judgment. Is letak_janin (fetal position) a coded Observation or free-text? Depends on whether we have a SNOMED mapper for the values. Should komplikasi_persalinan be a Condition resource or an Observation? Depends on whether the field represents a diagnosis or just a documentation note. These decisions vary across data sources and there's no algorithm for them.

My second thought was a one-shot LLM prompt. "Here's a CSV, here's a sample, generate a FHIR template." I tried it. The output looked plausible but failed in subtle ways: wrong function signatures for extraLogics, missing addValueIfNotEmpty guards, LOINC codes that didn't exist, references to resources that were never generated. A FHIR template is a program that runs through our converter engine, and it has to be mechanically correct, not just structurally plausible.

An agent was the right shape. FHIR mapping requires judgment (which resource type? which code?) interleaved with lookup (does this LOINC code exist? does this extraLogic function exist in our codebase?) followed by validation against real data. That loop, think-look up-try-validate-fix, is exactly the kind of work agents handle well.

My honest motivation wasn't full automation. I wanted to compress the tedious parts, choosing codes and wiring references, so my judgment could focus on the edge cases and domain-specific decisions that require understanding what the data actually means in a clinical context.

Anatomy of the Agent

The agent is built as a set of Claude Code custom slash commands: markdown files that define prompts, tool access, and autonomy rules. They chain together into a pipeline. I went through two major iterations, an original 7-step pipeline and a streamlined 3-step version I use today.

The agent receives three files as input:

First, a reference CSV, which is a data dictionary mapping field names to FHIR resource types, clinical codes, and paths. Our health informatics team builds these, and they're the foundation everything else depends on. Each row represents a clinical judgment call: this field is a vital sign Observation, that one is a Condition, this code is LOINC, that one is SNOMED. A typical row: berat_badan | Body weight in kg | Observation | valueQuantity.value | LOINC | 29463-7 | Body weight | numeric. The agent can wire JSON and pick handlers, but it can't decide whether a field is clinically meaningful or which code system is appropriate. That's what the CSV encodes.

Second, a sample JSON record, a real data row from BigQuery showing actual field values and types. This tells the agent what the data looks like in practice, not just in theory. It reveals things the CSV doesn't: that tekanan_darah is actually a combined "120/80" string, that usia_kehamilan is an integer, that some fields are null in real data.

Third, a reference template, an existing structure/*.json template for a similar data source. The agent uses this to extract project-specific patterns: how we do patient fuzzy matching, which regional helper functions exist (like getPurbalinggaVillageId), what the EpisodeOfCare strategy looks like for this program type.

Step A: Plan and Resolve

The first step is 80% deterministic Python, 20% LLM for genuinely ambiguous cases. It parses the CSV and sample data, cross-references them, resolves code systems from friendly labels to canonical URLs ("LOINC" becomes "http://loinc.org"), detects the appropriate data-absent-reason handler per field based on its type, and identifies the unit of measure from the CSV description.

The thing that makes this step work is a defaults file, twelve pre-resolved design decisions that the agent must follow without asking me:

## Coded Fields Without an Existing Mapper

When a field has coded values but no mapper function exists:
Default: use handleValueOrDataAbsentReason, store as valueString.
Do NOT stop to ask. Do NOT invent a mapper.

## Baby / Neonate Patient Linking

When the template has a second Patient for the baby:
Default: always add mother-baby linkage without asking.

## Blood Pressure from Combined String

When tekanan_darah is a combined "120/80" string:
Default: two separate Observations (systolic + diastolic),
each with parseSystolicBP / parseDiastolicBP.

I wrote this file after running the first version of the agent and noticing it kept stopping to ask me the same twelve questions across different templates. "Should I use valueString or valueCodeableConcept for this field?" valueString, every time, unless we have a mapper. "Should I add baby-mother linking?" Yes, always. "What should period.end be when there's no discharge field?" Same as period.start. These aren't judgment calls. They're conventions. Encoding them as defaults lets the agent run with far less hand-holding.

The output is an enriched_mapping.json with every field fully typed, every code resolved, every handler assigned.

Step B: Generate

This is where the template gets written. One continuous pass, following a strict resource ordering: Patient first, then EpisodeOfCare, Encounter, Conditions, vital sign Observations, clinical assessments, numeric measurements, datetime Observations, obstetric history, boolean/coded Observations, Procedures, baby Patient with mother linking, baby Observations, QuestionnaireResponse for risk factors, and any remaining resources. Seventeen groups in total, generated without pausing.

The agent assembles each resource from validated skeleton patterns. Here's a simplified skeleton for a numeric Observation, the most common resource type in a typical template:

{
  "resourceType": "Observation",
  "keyField": "[generated/observation_datetime]",
  "extraLogics": [
    {"type": "generateUUID",
     "output": ["[obs_id]"]},
    {"type": "returnPatientReference",
     "input": ["[generated/patient_res_id]", "[nama]", "[nik]"],
     "output": ["[patient_ref]", "[patient_display]", "[patient_nik]"]},
    {"type": "handleNumericValueOrDataAbsentReason",
     "input": ["[berat_badan]"],
     "output": ["[weight_value]", "[weight_absent_system]",
               "[weight_absent_code]", "[weight_absent_display]"]},
    {"type": "addValueIfNotEmpty",
     "input": ["[generated/weight_value]", "kg"],
     "output": ["[weight_unit_code]"]}
  ],
  "requiredFields": {
    "code.coding.0.system": "http://loinc.org",
    "code.coding.0.code": "29463-7",
    "code.coding.0.display": "Body weight",
    "subject.reference": "[generated/patient_ref]",
    "subject.identifier.value": "[generated/patient_nik]"
  },
  "optionalFields": {
    "valueQuantity.value": "[generated/weight_value]",
    "valueQuantity.unit": "[generated/weight_unit_code]",
    "dataAbsentReason.coding.0.code": "[generated/weight_absent_code]"
  }
}

Notice the addValueIfNotEmpty guard on the unit field. Without it, when the numeric value is absent, the handler correctly sets dataAbsentReason, but the unit field still emits an empty valueQuantity object, violating FHIR's constraint that a resource cannot have both value[x] and dataAbsentReason. This bug bit me three times in manual templates before I documented it in failure-modes.md and baked the guard into every pattern.

The strongest autonomy rule, repeated in every step:

Never ask "ready to continue?" or "shall I proceed?". Just proceed.

Before finishing, the agent runs a coverage check, verifying that every field in the sample data is mapped somewhere in the template. Target: zero uncovered fields.

Step C: Validate and Fix

The final step runs the actual converter against real sample data, then passes the output through a 7-layer validator that checks: FHIR R4 schema compliance, reference closure (every referenced resource exists in the bundle), mandatory fields per resource type, NIK traceability and meta.source format, code system terminology (LOINC/SNOMED/ICD-10/CVX existence), value consistency against the sample data, and helper/handler coverage. The first six layers flag errors; the last one flags warnings.

If errors exist, the agent reads only the error report (not the full bundle, not the full template), applies surgical fixes to the specific failing resources, and re-runs. Maximum two retry cycles. If errors persist after two passes, it stops and surfaces the remaining issues for me to fix manually.

A completed run produces four things: structure/<name>.json (the production-ready template), updated settings.py and settings_local.py (registering the new data source), bundle.json (a test FHIR Bundle from the sample data), and validation_report.json (showing zero errors across all layers).

Where the Template Lives

The agent's output, a JSON template, plugs into a larger system.

BigQuery holds raw health data across 50+ tables, ingested from source systems across Purbalingga and Lombok Barat districts. The orchestrator queries for unprocessed records and farms them out in parallel batches (15 workers, batches of 25 records). For each record, the converter loads the matching template from structure/, runs the extraLogics transformation chain, maps fields to FHIR paths, resolves references between resources, and assembles a FHIR Bundle. The bundle goes to Indonesia's SatuSehat FHIR server via POST. Success and failure get logged back to BigQuery.

The core of the converter, where the template becomes a FHIR resource:

def merge_resources_and_build_references(fullUrl, json, resource,
                                         references_data):
    existing_json = {}
    if resource.get("mergeWithExistingData", False):
        if "id" in json:
            existing_json = get_existing_data_with_id(
                resource["resourceType"], json['id'])

    resource_json = merge_nested_dicts(existing_json, json)

    references = copy.deepcopy(resource["references"])
    if "dynamicReferences" in resource:
        for ref in resource["dynamicReferences"]:
            if ref["key"] not in references_data:
                continue
            # ... build conditional reference

    resource_json = merge_dicts_with_list_values(
        resource_json, references)
    return {
        "fullUrl": fullUrl,
        "request": {"method": "POST",
                    "url": resource["resourceType"]},
        "resource": resource_json
    }

The agent unblocks this pipeline. Everything upstream (data extraction) and downstream (conversion, FHIR submission, reporting) is mechanical. Before the agent, adding a new data source meant I couldn't take on other work for three to four days. Now it means an afternoon.

The Payoff

3-4 days of wall-clock time down to 3-6 hours.

About 30 minutes goes to setup: extracting a sample record from BigQuery, prepping the CSV data dictionary (our health informatics team builds these, but I sometimes need to clean up column names or add missing code system labels), picking a reference template close to the new data source.

1-3 hours for the agent pipeline and review. Run the three steps, then read through the generated template. Check code selections ("did it pick the right LOINC code for fundal height?"), verify the extraLogics chains make sense ("is it using the right date parser for this data source?"), spot-check the reference graph ("do all Observations point to the right Encounter?").

Then 2-3 hours of testing and manual fixes. Run the converter against real sample data. Fix the things the agent got wrong, usually a transformation function with the wrong number of arguments, or a reference to a field name that doesn't match the sample. Re-run until the validation report comes back clean.

I still read every template. But the review is different from building from scratch. When I built manually, I was making decisions and implementing them simultaneously: the cognitive load of "what should this be?" stacked on top of "how do I express that in our template format?" Now I'm only doing the first part. The agent handles the expression.

One moment that sticks with me: the agent generated a template for a nutrition surveillance form (sigizi_balita_dipantau_pmt) with about 60 Observations covering growth monitoring, nutrition status, and supplementary feeding data. In review, I noticed it had correctly applied addValueIfNotEmpty guards on the unit fields for every single numeric Observation. All sixty of them. That's a pattern I'd missed in at least two earlier manual templates, which led to hours of debugging FHIR constraint violations when the server rejected bundles where a patient had no weight recorded but the template still emitted an empty valueQuantity alongside the dataAbsentReason.

The agent got this right because I'd documented the bug in failure-modes.md after debugging it the hard way, and baked the fix into the skeleton patterns. My past pain became the agent's default behavior.

How It Evolved

The first version of the agent had seven steps: analyze inputs, generate a base template at ~40% coverage, evaluate coverage, implement missing fields, test structural integrity, validate quality, check data consistency. Each step was its own Claude Code command with its own system prompt.

It worked. Templates came out correct. I used it for weeks, and it was a real improvement over manual work. But after running it on maybe a dozen templates, I started noticing where the friction was.

Token cost added up. Seven LLM invocations, each re-reading the template, reference files, and sample data. The pipeline consumed roughly 107,000 tokens per run, about 4,773 lines of markdown across the seven command files. The generate-then-measure-then-patch loop (steps 2, 3, 4) felt especially redundant after a while: why generate an incomplete template at 40% coverage just to measure what's missing and patch it? I had enough experience by then to know the agent could do it in one pass.

Confirmation prompts wore me down. Despite instructions not to, the agent kept pausing between resource groups. "I've generated the vital signs Observations. Ready to continue with clinical assessments?" The first few times I didn't mind. After the twentieth template, pressing enter seventeen times per run felt like exactly the kind of tedious interruption I'd built the agent to eliminate.

Small quality issues compounded. The old data consistency validator (step 7) checked whether sample values appeared in the generated bundle using Python's in operator on the serialized JSON string: value in json.dumps(bundle). The value "1" matched every UUID containing the digit 1. Not a dealbreaker, but it meant an extra fix-and-retry cycle on most runs, and after a few weeks those extra cycles added up.

None of these were really wrong. The 7-step pipeline was a huge improvement over manual work. But once you've lived with a tool long enough, you start seeing the next version. I refactored it over a weekend, less out of frustration than the natural instinct to tighten something you use every day.

The 3-step version changed three things. One-pass generation replaced the generate-measure-patch loop: Step B produces the complete template in a single continuous pass. Pre-resolved defaults reduced confirmation prompts: the defaults.md file encodes every recurring design decision, and Claude still asks about some things (tool calls, file writes, ambiguous decisions outside the defaults) but the constant back-and-forth between resource groups is gone. Typed path-aware validation replaced substring matching: the new validator does type-aware comparison (float() cast for numerics, ISO normalization for dates, case-insensitive for strings) instead of value in json.dumps(bundle).

Token usage dropped by roughly 70%. Fix-and-retry cycles went from 2-3 typical to 0-1. It's not fully hands-off; I still sit with the agent, approve tool calls, and occasionally steer it when it goes down a wrong path. But the interruptions are meaningful now. Not mechanical.

When the Agent Gets It Wrong

The agent still makes mistakes. The most common: picking a transformation function that's incompatible with the actual data shape. Using handleNumericValueOrDataAbsentReason on a field that the CSV says is numeric but the sample data reveals is actually a string like "Normal" or "Tidak ada". The handler tries to cast it to a float, throws an exception, the converter crashes.

Another recurring issue is wrong argument counts in extraLogics. Our transformation functions have strict input/output contracts. getPatientIdWithFuzzyLogic takes exactly 6 inputs and returns 1 output. If the agent wires 5 inputs, the function throws an unexpected argument count error at runtime. Step C's test run always catches this, but it requires a manual fix to figure out which argument was omitted.

Sometimes the agent chooses a structure that looks right but doesn't fit how the converter actually processes the data. A field that should be a requiredField ends up in optionalFields, or vice versa, and the missing-value behavior changes in ways that only show up with certain input records. These are the subtlest bugs. They pass validation against the sample data but fail on edge cases in production.

These bugs are fast to fix because the error messages are specific and the template structure is already complete. Patching a single resource definition, not rebuilding from scratch. A 10-minute fix versus a 4-hour reconstruction.

The lesson from building two versions: the first agent you build optimizes for correctness. The second optimizes for cost and autonomy. Both matter, but you don't know which autonomy rules you need until you've run the first version enough times to feel the friction. Every line in defaults.md represents a specific moment where the agent's behavior cost me time, either by asking an unnecessary question or failing in a way I'd already seen before.

What Changed

I now support 60 template configurations across ten Indonesian health programs: hospital systems, community health centers, mobile data collection, nutrition surveillance, tuberculosis tracking, hepatitis monitoring. The templates range from a 2-resource pregnancy tracker (279 lines) to a 332-resource hospital ANC integration (490KB of JSON). Every one of them was built using the agent pipeline.

Before the agent, I had to defer new integrations. Each new data source meant committing to a multi-day block of focused work, and there was always something more urgent. A new nutrition monitoring form would sit in the backlog for weeks while I finished a hospital template. Now, when the health informatics team hands me a new data dictionary, I can have a working template the same day. That changed what integrations I was willing to take on, and how quickly new health programs could start reporting data through the national FHIR infrastructure.

My job shifted from writing thousands of lines of JSON by hand to designing the system that writes it and then reviewing the output with the kind of attention I couldn't sustain across a four-day manual effort.

The most valuable thing I built was defaults.md, twelve pre-resolved design decisions that let the agent run without constantly asking me questions. Writing that file forced me to articulate judgment calls I'd been making intuitively for months. Which EpisodeOfCare strategy for which program type. How to handle coded fields without an existing mapper. When to split blood pressure into components versus a single observation. These decisions were in my head, applied inconsistently, never written down.

But I want to be clear about something. The agent automates my part of the work: turning clinical mappings into working FHIR templates. It does not automate the clinical thinking that comes before it. Every template starts with a CSV that our health informatics team built, row by row, mapping raw field names to the right FHIR resource types and clinical codes. They're the ones who know that denyut_jantung_janin is LOINC 55283-6 and not some other fetal assessment code. They're the ones who decide when a field warrants its own Condition resource versus being an Observation. Without that upstream work, I'd have nothing to automate. The agent compressed my 3-4 days into an afternoon, but it didn't touch the clinical expertise that makes the output meaningful.

It turns out the hard part of building an AI agent isn't the AI. It's figuring out what you actually know, and writing it down clearly enough that something else can apply it. And sometimes, it's recognizing that the knowledge you depend on most isn't yours at all.

National Vaccine Appointment & Administration System

Budi Widhiyanto — Sat, 28 Feb 2026 09:30:45 +0000

🌱 How It Started

Few Years ago, I had a system design interview. The interviewer gave me this scenario:

"Design a national vaccine appointment booking system. Millions of citizens need to register and book slots. Clinics must administer the doses. The government needs audit logs and fraud prevention."

My first thought was simple just let people book a slot, check the stock, and confirm. I drew a basic flow on the whiteboard and felt pretty good about it. Then the interviewer started asking harder questions.

"What if two people try to book the last slot at the same time?"

"What if the clinic runs out of doses after the booking is already confirmed?"

"How do you undo things if eligibility check fails in the middle?"

I didn't have good answers. I only designed for the happy path.

That interview stuck in my mind. Months later, I was doing research on inventory reservation patterns for an internet credit purchase system, and I realized the same ideas could have helped me in that interview. So I went back to the problem and redesigned it. This is what I came up with.

⚡ My Initial (Naïve) Solution

Here's what I proposed during the interview:

Simple, right? But the problems come fast:

Race conditions: Two people click "Book" at the same time for the last slot. Both get confirmed. Now one citizen has no seat.
Stock mismatch: Slot is confirmed, but the clinic ran out of vaccine doses between booking day and appointment day.
Late eligibility failure: System confirms appointment first, then finds out the citizen doesn't meet age or insurance requirement. Now you need to undo everything, but stock is already allocated.
No rollback: If something fails in the middle, there's no way to release the slot or dose back to the pool.

These are the same problems I found later when designing the internet credit purchase system the happy path is not enough when you deal with limited resources and many users at the same time.

🔍 Rethinking the Flow

The main idea, which I learned from inventory reservation strategies in e-commerce, is: don't confirm anything until everything is verified. Use a multi-stage process temporary hold first, then verify, then confirm. If anything fails, rollback.

It's like buying concert tickets. When you select a seat, it's held for you while you pay. If you don't finish in time, the seat goes back. Same concept here.

Here's the full flow of the improved design:

🧩 The Improved Design

1. Reserve First (Temporary Hold)

When a citizen selects a clinic, time slot, and vaccine type, the system does not confirm right away. Instead:

It creates a temporary reservation in Redis with a TTL (time-to-live), for example 5 minutes.
Appointment status is set to PENDING.
Slot capacity and vaccine dose count are decreased temporarily other users will see less availability.

Why Redis? Because we need something fast and temporary. A relational database could work too, but you would need a separate scheduled job to clean up expired reservations. Redis handles this automatically with TTL when 5 minutes pass, the key just disappears. For a system that handles millions of bookings during a national vaccine campaign, this performance difference is important.

How to handle race condition on Redis? We use Redis DECR command on the slot counter. This is atomic meaning if two requests come at the same time, Redis processes them one by one. If the counter reaches zero, the next request is rejected. For extra safety, you can use a Lua script to make the check-and-decrement happen in one step.

2. Eligibility Verification

While the slot is held, the system runs eligibility checks:

Age requirement (e.g., some vaccines only for 60+).
Insurance verification through external API.
Medical history (allergies, previous doses).
Geographic check (is this citizen in the right region?).

If any check fails, the reservation is released Redis key is deleted, slot goes back to the pool. The citizen gets a clear message explaining why they are not eligible, not just "something went wrong."

3. Confirm Appointment

If all checks pass:

Slot capacity and vaccine stock are decreased permanently in the main database.
Appointment status changes from PENDING to CONFIRMED.
Redis reservation is cleared (not needed anymore).
Confirmation is sent to the citizen (SMS, email, or push notification).

This is the point of no return. Before this step, everything can be undone.

4. Administration (Vaccination Day)

When the citizen arrives at the clinic:

Clinic staff scans the citizen's QR code. The QR code contains the appointment ID and a verification hash. The hash is generated on the server using appointment ID + citizen ID + a secret key, so it cannot be faked.
System verifies the QR code against the appointment record.
Staff records the vaccine batch number and time of administration.
Appointment status changes to ADMINISTERED.
An event is sent to other systems analytics, government reporting, audit logs.

5. Failure & Rollback Scenarios

This is the part I completely missed in my interview. Here's how each failure is handled:

No-show: A scheduled job checks for CONFIRMED appointments that passed their time window. Status becomes NO_SHOW, stock is released back.
Citizen cancels: They can cancel through the portal. Stock is released right away.
Clinic cancels a slot (e.g., not enough staff): All affected appointments are flagged. Citizens get notified and can rebook with priority.
External API is down (e.g., insurance service): The system uses a circuit breaker pattern. After several failures in a row, the system stops calling that API temporarily. Meanwhile, the booking is either queued for retry (with increasing wait time between retries) or allowed provisionally with a flag for manual review later. The important thing is: one broken dependency should not block the whole flow.
Redis goes down: The system falls back to database-level reservations with a cleanup job. It's slower, but the booking still works.

🏗️ System Components

Here's the high-level architecture:

Frontend: Booking portal for citizens + Dashboard for clinic staff.
API Gateway: Authentication, rate limiting (very important during mass booking), and routing.
Core Services:
- Auth Service Login, national ID verification.
- Patient Service Medical records, vaccination history.
- Clinic Service Slot management, staff schedules, capacity.
- Inventory Service Vaccine stock per clinic, batch tracking.
- Appointment Service The main service. Manages reservations, confirmations, and status changes.
- Eligibility Service Rules engine + external API calls.
- Notification Service SMS, email, push. Retries if delivery fails.
- Audit Service Append-only logs for every status change. Required for government compliance.
Data Layer: PostgreSQL for permanent data, Redis for temporary reservations and caching.
Async Messaging: Kafka for events AppointmentReserved, AppointmentConfirmed, AppointmentAdministered, AppointmentCancelled. This keeps services separated and makes the system auditable by default.

🎯 What I Would Do Differently Now

Looking back at that interview, the biggest thing I missed was not about technology it was about mindset. I jumped to the happy path because it felt complete. But the interviewer was not testing if I can design a booking form. They were testing if I can think about what happens when things go wrong.

Here's what I learned from this experience:

Start with failure scenarios, not the happy path. Ask yourself "what can go wrong at each step?" before finalizing any design.
Temporary reservation is a pattern, not a hack. Whether it's concert tickets, flash sales, or vaccine slots if you have limited stock and many users, you need hold-then-confirm flow.
Don't be vague about rollbacks. "We'll handle errors" is not a design. Be specific what happens to the data, the stock, and the user when something fails.
External services will go down. Always have a plan for when the insurance API or notification service is not available. Circuit breakers and retry queues are not optional they are necessary.

If you're preparing for system design interviews, I recommend studying inventory reservation patterns. My earlier post on designing an internet credit purchase system covers these patterns with more detail and code examples. The core idea reserve first, verify, then commit appears in many systems once you start looking.

Thanks for reading. If you faced similar interview questions or have ideas to improve this design, I would like to hear about it in the comments.

Data Fetching Patterns Every Developer Should Know (And When to Actually Use Them)

Budi Widhiyanto — Sat, 28 Feb 2026 08:56:54 +0000

About a year ago, I was working on a payment app. Solid architecture, clean API design, decent frontend on paper, everything looked good. But a few months after launch, the ratings started tanking. Users were complaining about slow loads, failed transactions, and the whole thing falling apart on spotty connections.

I spent three months debugging those performance issues, and the fix wasn't some clever algorithm or a server upgrade. It was rethinking how we fetched data. That's it. Same features, same infrastructure, same design just smarter data fetching patterns. The app went from 3.2 stars to 4.7, and transaction volume jumped 30% within two months.

That experience a year ago changed how I think about data flow end-to-end. Most apps don't have a "feature" problem they have a "how we get data to the screen" problem. And the difference between a mediocre app and a great one often comes down to picking the right data fetching pattern for the right situation.

Here's everything I learned and wish I'd known sooner.

The Basics: Request-Response

This is where everyone starts, and for good reason. You ask the server for something, you wait, you get it back. It's the foundation of HTTP, and it handles the majority of use cases just fine.

const fetchUser = async (id: string) => {
  const response = await fetch(`/api/users/${id}`);
  return response.json();
};

Think of it like ordering at a counter you place your order, you wait, you get your food. Simple and predictable.

This works great for standard CRUD operations: loading a user profile, submitting a form, fetching account details on page load. Where it falls apart is when you start chaining multiple requests together. If your page needs data from five endpoints and each one takes 300ms, your user is staring at a spinner for 1.5 seconds. That adds up fast.

The key is recognizing when request-response stops being enough which brings us to everything else.

Polling: The "Are We There Yet?" Approach

Polling is exactly what it sounds like. You ask the server for updates on a regular interval. Every 5 seconds, every 30 seconds, whatever makes sense for your use case.

const pollForUpdates = () => {
  const intervalId = setInterval(async () => {
    try {
      const response = await fetch('/api/updates');
      const data = await response.json();
      updateUI(data);
    } catch (error) {
      console.error('Polling failed:', error);
    }
  }, 5000);

  // Don't forget cleanup
  return () => clearInterval(intervalId);
};

I've seen polling get a bad reputation, and honestly, sometimes it deserves it. Naive polling hammers your server with requests even when nothing has changed. On mobile, it eats battery life. And you'll always have that gap between intervals where updates get missed.

But here's the thing polling is dead simple to implement, works everywhere, and for many use cases (dashboards refreshing every 30 seconds, checking job status on a build pipeline, order tracking) it's perfectly fine. Not everything needs to be real-time. Sometimes "close enough" is the right engineering decision.

The smarter version is long polling, where the server holds the connection open until it actually has something to send back. It's a nice middle ground before committing to WebSockets.

WebSockets: When You Need Actual Real-Time

WebSockets maintain a persistent, two-way connection between the client and server. Unlike polling, neither side has to ask data flows both directions whenever either side has something to say.

const socket = new WebSocket('wss://example.com/socket');

socket.onopen = () => {
  console.log('Connected');
};

socket.onmessage = (event) => {
  const data = JSON.parse(event.data);
  updateUI(data);
};

socket.onclose = () => {
  // You'll want reconnection logic here connections drop
  console.log('Disconnected');
};

This is what powers chat apps, multiplayer games, collaborative editors like Google Docs, and trading platforms where milliseconds matter. If your users need to see changes the moment they happen, and especially if they need to send data back frequently, WebSockets are the right call.

The tradeoff is complexity. You need to handle reconnections (connections will drop). You need to think about scaling every connected user holds an open connection on your server. You need to deal with authentication differently than with regular HTTP. It's not hard, but it's more surface area than a simple fetch call.

My rule of thumb: if you're polling more than once every 5 seconds, it's probably time to consider WebSockets.

Server-Sent Events: Real-Time's Simpler Cousin

SSE is the pattern I wish more developers knew about. It's a one-way channel the server pushes updates to the client over a long-lived HTTP connection. No polling, no WebSocket complexity.

const eventSource = new EventSource('/api/stream');

eventSource.onmessage = (event) => {
  const data = JSON.parse(event.data);
  updateUI(data);
};

eventSource.onerror = () => {
  eventSource.close();
};

See how much simpler that is compared to WebSockets? And you get automatic reconnection for free the browser handles it.

SSE is perfect for notifications, live sports scores, progress bars for long-running tasks (think file processing or deployment pipelines), newsfeeds, and anything where the server is doing the talking and the client is just listening.

The limitation is right there in the name: server-sent. If your client needs to send data back frequently, SSE isn't enough. But for a surprising number of "real-time" features, one-way is all you need.

Caching: Making Your App Feel Instant

Caching is less of a fetching pattern and more of a fetching strategy that you layer on top of other patterns. The idea is simple: store data you've already fetched so you don't have to fetch it again.

import { useQuery } from 'react-query';

const { data, isLoading, error } = useQuery(
  'user',
  () => fetch('/api/user').then((res) => res.json()),
  {
    staleTime: 5 * 60 * 1000,
    cacheTime: 30 * 60 * 1000,
  }
);

Libraries like React Query and SWR have made caching dramatically easier. They handle stale-while-revalidate (show cached data immediately, then refresh in the background), cache invalidation, and deduplication of simultaneous requests.

The impact is hard to overstate. When a user navigates to a page they've already visited and the content appears immediately while a background refresh happens silently that's the kind of thing that makes an app feel native-quality.

The classic challenge is cache invalidation (there's a reason Phil Karlton called it one of the two hard things in computer science). You have to decide: how long is cached data acceptable? What events should invalidate the cache? What happens when two tabs have different cached versions? These are solvable problems, but they require deliberate thinking.

Lazy Loading: Don't Fetch What You Don't Need Yet

The fastest network request is the one you never make. Lazy loading defers fetching until the user actually needs the data typically triggered by scrolling, clicking a tab, or navigating to a new section.

const loadMore = async () => {
  if (isNearBottom()) {
    const response = await fetch(`/api/items?page=${nextPage}`);
    const newItems = await response.json();
    setItems(prev => [...prev, ...newItems]);
    setNextPage(prev => prev + 1);
  }
};

window.addEventListener('scroll', loadMore);

You see this everywhere: infinite scroll on social feeds, images loading as you scroll past them, tabs that only fetch their content when clicked. It makes initial page loads fast because you're only loading what's visible.

The gotchas are UX-related. Infinite scroll can make it impossible for users to reach the footer. Loading new content can cause layout shifts that make users lose their place. And for accessibility, you need to make sure screen readers can navigate lazy-loaded content properly.

For very large lists (thousands of items), pair lazy loading with virtualization only render the DOM elements that are visible in the viewport. Libraries like react-window or tanstack-virtual make this manageable.

Background Sync: Building for the Real World

This one is close to my heart because it solved the biggest pain point in that payment app I worked on last year. Background sync lets users take actions (send a message, submit a form, record a transaction) even when they're offline. The operations get queued and processed automatically when connectivity returns.

// Service Worker
self.addEventListener('sync', (event) => {
  if (event.tag === 'sync-transactions') {
    event.waitUntil(processQueuedTransactions());
  }
});

// Application code
async function recordTransaction(transaction) {
  await saveToLocalQueue(transaction);

  // Show the transaction in the UI immediately
  updateUIOptimistically(transaction);

  if ('serviceWorker' in navigator) {
    const registration = await navigator.serviceWorker.ready;
    await registration.sync.register('sync-transactions');
  }
}

This pattern is essential for mobile apps used in areas with unreliable connections field service apps, delivery tracking, healthcare in rural areas, anything where you can't assume a stable connection.

The complexity lives in conflict resolution. What happens if two offline users edit the same record? What if the server rejects a queued operation? You need clear strategies for these cases, and they're not always straightforward. But the user experience improvement is massive. Going from "you can't do anything without internet" to "everything just works, and syncs when it can" is a night-and-day difference.

Batch Fetching: One Trip Instead of Ten

If your page makes 8 separate API calls to render, something is probably wrong. Batch fetching combines multiple requests into a single network call.

// Instead of this:
const user = await fetch('/api/users/1');
const posts = await fetch('/api/users/1/posts');
const notifications = await fetch('/api/users/1/notifications');

// Do this:
const dashboard = await fetch('/api/dashboard?userId=1');
// Returns user, posts, and notifications in one response

The savings come from reducing HTTP overhead connection setup, headers, TLS handshakes. On mobile networks with high latency, the difference between one request and ten is very noticeable.

The downside is coupling. When you batch things together, you can't cache or invalidate them independently. If the notification data changes every 30 seconds but user profile data changes once a month, batching them means either over-fetching profile data or under-fetching notifications. You have to think about which data actually belongs together.

GraphQL: Ask for Exactly What You Need

GraphQL flips the traditional REST model. Instead of the server deciding what data each endpoint returns, the client specifies exactly what it needs.

const query = `
  query GetUser($id: ID!) {
    user(id: $id) {
      name
      email
      posts(last: 5) {
        title
        preview
      }
    }
  }
`;

const response = await fetch('/graphql', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ query, variables: { id: '123' } }),
});

With REST, a mobile app and a desktop app hitting the same /api/user endpoint get the same response even if the mobile app only needs the name and avatar while the desktop app needs the full profile. GraphQL eliminates that mismatch. Each client asks for exactly what it needs.

This matters most when you have multiple clients with different data requirements, deeply nested data relationships, or you're tired of creating one-off REST endpoints for every new screen design.

The investment is real, though. You need a GraphQL server, a schema, resolvers, and your team needs to learn a new paradigm. Caching is trickier than REST because everything goes through a single endpoint. And poorly written queries can cause serious performance issues on the backend (the N+1 query problem is very real with GraphQL).

For smaller apps with a single client, REST with good API design is usually simpler and sufficient.

Federated Fetching: Unifying Microservices

In microservice architectures, the data a single page needs might live across five different services. Federated fetching usually through a BFF (Backend-For-Frontend) layer or API gateway aggregates that data so the client makes one clean request.

// BFF endpoint
app.get('/api/dashboard/:userId', async (req, res) => {
  const [user, accounts, activity] = await Promise.all([
    fetch(`http://user-service/users/${req.params.userId}`),
    fetch(`http://account-service/accounts?userId=${req.params.userId}`),
    fetch(`http://activity-service/recent?userId=${req.params.userId}`)
  ]);

  res.json({
    user: await user.json(),
    accounts: await accounts.json(),
    recentActivity: (await activity.json()).slice(0, 5)
  });
});

The BFF pattern is a lifesaver in complex systems. Instead of the frontend knowing about every microservice and making separate calls to each, it talks to one unified API that handles the orchestration. The frontend stays clean, and you can tailor responses to what each client actually needs.

The downside is obvious you're adding another service to build, deploy, and maintain. And if your BFF goes down, everything goes down. It's a pattern that makes sense at a certain scale, but overkill for smaller applications.

Combining Patterns: Where It Gets Interesting

No real application uses just one pattern. The interesting decisions happen when you combine them. Here's what that looks like in practice:

A messaging app might use WebSockets for incoming messages, background sync for sending messages in poor connectivity, caching for conversation history, and lazy loading for scrolling through older messages.

An e-commerce app might use request-response for search, caching for product pages, SSE for inventory availability, and batch fetching for the cart summary.

A trading platform might use WebSockets for live prices, polling as a fallback, GraphQL for portfolio data, and caching for historical charts.

The point is to match each data need to the pattern that best serves it. Not every piece of data on a screen has the same freshness requirements, the same access patterns, or the same tolerance for latency.

Quick Reference

Pattern	Use When	Complexity	Offline Support
Request-Response	Standard CRUD, simple pages	Low	No
Polling	Periodic updates, status checks	Low	No
WebSockets	Two-way real-time (chat, collaboration)	High	No
Server-Sent Events	One-way real-time (notifications, feeds)	Medium	No
Caching	Repeated data access, speed matters	Medium	Partial
Lazy Loading	Large lists, heavy initial loads	Low	No
Background Sync	Offline-first, unreliable connections	High	Yes
Batch Fetching	Multiple related data needs	Low	No
GraphQL	Complex/varied data requirements	High	No
Federated Fetching	Microservices, unified APIs	High	No

Final Thoughts

A year ago, when that payment app went from 3.2 stars to 4.7, we didn't add a single new feature. We just changed how existing features got their data. Caching made it feel instant. Background sync made it work offline. WebSockets made payments confirm in real-time. Batch fetching cut load times by 80%.

Looking back, that project taught me something I keep coming back to: users don't care about your architecture. They care that things are fast, reliable, and don't waste their time. Data fetching patterns on both the backend and the frontend are how you deliver on that promise.

Pick the right pattern for each situation. Combine them thoughtfully. And when your app ratings start climbing, you'll know why.

How a "Simple" QR Code Generator Ate All My RAM: A Tale of 50,000 QR Codes

Budi Widhiyanto — Tue, 24 Feb 2026 02:43:04 +0000

Sometimes the simplest tasks can become the biggest headaches. Here's how I learned that data size matters more than code complexity.

The Innocent Beginning

It started with a straightforward request: generate 50,000 unique QR codes for a project. "How hard could it be?" I thought. Python has excellent libraries for this. A quick script, a PDF output, done by lunch.

I was wrong. Very wrong.

What I didn't anticipate was that my "simple" script would consume every byte of RAM on my machine, freeze my computer, and teach me an important lesson about thinking at scale.

Let me walk you through what happened, how I fixed it, and what you can learn from my mistakes.

The Original Approach: Looks Good on Paper

Here's the approach I initially took. Generate all the QR codes first, cache them in memory, then write them to a PDF. It sounds logical, right? Pre-compute everything, then assemble the final output.

def generate_pdf(output_path: str, total: int = 50000):
    ids = generate_unique_ids(total)

    # Pre-generate ALL QR codes in parallel for "speed"
    print(f"Pre-generating {total} QR codes in parallel...")
    num_workers = cpu_count()

    # Split IDs into batches for parallel processing
    batch_size = max(1, total // (num_workers * 4))
    batches = [ids[i:i + batch_size] for i in range(0, len(ids), batch_size)]

    # Generate QR codes in parallel using multiprocessing
    qr_cache = {}
    with Pool(num_workers) as pool:
        results = list(tqdm(
            pool.imap(generate_qr_batch, batches),
            total=len(batches),
            desc="Generating QR codes"
        ))

    # Store ALL images in memory
    for batch_result in results:
        for uid, img_bytes in batch_result:
            buf = io.BytesIO(img_bytes)
            qr_cache[uid] = ImageReader(buf)

    # NOW create the PDF using cached images
    # ... PDF generation code ...

I was proud of this code. Multiprocessing! Parallel execution! Batch processing! All the buzzwords that make you feel like a "real" programmer.

Then I ran it.

The Disaster Unfolds

The script started running. Progress bars moved. CPU usage spiked to 100% across all cores. "Excellent," I thought, "parallel processing doing its thing."

Then I noticed my system getting sluggish. Browser tabs stopped responding. My IDE froze. I opened the system monitor and watched in horror as my RAM usage climbed:

2 GB...
4 GB...
8 GB...
12 GB...

My laptop has 16 GB of RAM. The script was devouring it all. Before I could react, the OOM (Out of Memory) killer struck. Process terminated. No PDF. Just a frozen computer and a lesson learned the hard way.

Understanding the Problem

After my system recovered, I sat down to analyze what went wrong. Let me break down the math:

Each QR code image:

Resolution: 400 × 400 pixels
Format: PNG in memory
Approximate size: 15-30 KB per image (compressed)
But in memory as a PIL Image object: ~500 KB - 1 MB

Scale it up:

50,000 QR codes × ~500 KB = ~25 GB of RAM

Even with the compressed PNG byte representation, we're looking at:

50,000 × 20 KB = ~1 GB just for the image bytes
Plus the ImageReader objects
Plus the BytesIO buffers
Plus Python's memory overhead
Plus multiprocessing duplicating data across workers

The actual memory consumption was somewhere between 2-4 GB, which was still way more than what should be acceptable for such a "simple" task.

The fundamental flaw in my approach was this: I was optimizing for speed when I should have been optimizing for resource consumption.

The Fix: Think Like a Stream, Not a Lake

The solution was embarrassingly simple once I understood the problem. Instead of loading all 50,000 QR codes into memory at once (a "lake" of data), I needed to process them as a stream—one page at a time.

Here's the key insight: A PDF with 50,000 QR codes has about 1,667 pages (30 QR codes per page). I only need to hold 30 QR codes in memory at any given time—the ones for the current page.

Here's the refactored approach:

def generate_pdf(output_path: str, total: int = 50000):
    ids = generate_unique_ids(total)
    total_pages = (total + PER_PAGE - 1) // PER_PAGE

    # Create PDF canvas
    c = canvas.Canvas(output_path, pagesize=A4)

    # Process ONE PAGE at a time
    for page_start in tqdm(range(0, total, PER_PAGE), desc="Generating PDF pages"):
        page_ids = ids[page_start : page_start + PER_PAGE]

        # Generate QR codes ONLY for this page
        page_qr_cache = {}
        for uid in page_ids:
            img = make_qr_image(uid)
            page_qr_cache[uid] = img_to_reader(img)

        # Draw this page
        for idx, uid in enumerate(page_ids):
            # ... draw QR code to PDF ...
            c.drawImage(page_qr_cache[uid], qr_x, qr_y, ...)

        c.showPage()

        # CRITICAL: Clear the cache after each page!
        page_qr_cache.clear()

    c.save()

The key changes:

Generate per-page: Only create QR codes for the 30 items on the current page
Clear after use: Explicitly clear the page cache after each page is written
No multiprocessing overhead: Removed the parallel processing that was duplicating data

The Trade-off: Speed vs. Safety

Let's be honest about the trade-offs:

Metric	Original (Parallel)	Optimized (Per-Page)
Memory Usage	2-4 GB	50-100 MB
Speed	Faster (theoretically)	Slower
Stability	Crashes on large datasets	Stable
Scalability	Limited by RAM	Limited by disk space

Yes, the optimized version is slower. Without parallel processing, we're generating QR codes sequentially. For 50,000 codes, the execution time went from "crash before completion" to "about 30-45 minutes of stable execution."

But here's the thing: a slow script that completes is infinitely faster than a fast script that crashes.

I ran the optimized version overnight. When I woke up, both PDF files (100,000 QR codes total) were sitting there, ready to use. My computer was fine. No crashes. No freezing. Just steady, predictable progress.

Lessons Learned

1. Data Size Changes Everything

A script that works perfectly for 100 items might explode at 10,000 items. Always ask yourself: "What happens when this scales 10x? 100x? 1000x?"

In my case, the script probably worked fine during testing with small batches. It was only at production scale that the memory issue became catastrophic.

2. Memory is Not Infinite

This sounds obvious, but it's easy to forget when you're writing code. Every object you create lives somewhere in memory. When you're dealing with images, those objects can be surprisingly large.

# This innocent-looking line...
qr_cache[uid] = ImageReader(buf)

# ...executed 50,000 times becomes a memory bomb

3. Parallel ≠ Better

Parallel processing is great for CPU-bound tasks where you have enough memory to support multiple workers. But when each worker is creating large objects, parallelism can actually make things worse by multiplying memory usage.

Sometimes, a simple sequential loop is the right answer.

4. Clear Your References

Python's garbage collector is good, but it's not magic. If you're holding references to large objects in a dictionary or list, that memory won't be freed until you explicitly remove those references.

# This single line saved gigabytes of RAM
page_qr_cache.clear()

5. Progress Bars Are Your Friend

When you're running long-executing tasks, always add progress bars. The tqdm library makes this trivially easy:

for page_start in tqdm(range(0, total, PER_PAGE), desc="Generating PDF pages"):
    # ... your code ...

Not only does this give you feedback on how long the task will take, but it also helps you identify when something is wrong. If the progress bar stalls, you know there's a problem.

The Bigger Picture: Thinking About Resources

This experience changed how I approach coding problems. Now, before I write any code that deals with data at scale, I ask myself three questions:

What's the memory footprint per item?
How many items will I process?
Can I process items one at a time instead of all at once?

This is especially important in scenarios like:

Image processing: Images are memory-hungry
Data pipelines: Processing large CSV/JSON files
API responses: Paginating through thousands of records
File operations: Reading/writing large files

The pattern is always the same: stream when you can, batch when you must, and never load everything into memory unless you absolutely have to.

Practical Tips for Your Own Projects

If you're working on a similar task—generating large numbers of images, processing big datasets, or handling any kind of bulk operation—here are some practical tips:

Use Generators Instead of Lists

# Bad: Creates a list of 50,000 items in memory
ids = [generate_id() for _ in range(50000)]

# Better: Generates one at a time
def id_generator(count):
    for _ in range(count):
        yield generate_id()

Process in Chunks

# Instead of processing all at once
for item in huge_list:
    process(item)

# Process in manageable chunks
chunk_size = 100
for i in range(0, len(huge_list), chunk_size):
    chunk = huge_list[i:i + chunk_size]
    for item in chunk:
        process(item)
    # Clean up after each chunk
    gc.collect()  # Force garbage collection if needed

Monitor Your Memory Usage

Add memory monitoring to long-running scripts:

import psutil
import os

def get_memory_usage():
    process = psutil.Process(os.getpid())
    return process.memory_info().rss / 1024 / 1024  # MB

# In your loop
for i, item in enumerate(items):
    process(item)
    if i % 1000 == 0:
        print(f"Processed {i} items, Memory: {get_memory_usage():.1f} MB")

Set Memory Limits

For critical scripts, you can set memory limits to prevent runaway consumption:

import resource

# Limit memory to 1GB
resource.setrlimit(resource.RLIMIT_AS, (1024 * 1024 * 1024, -1))

Conclusion

My "simple" QR code generator turned into a valuable lesson about resource management. The original code was clever—parallel processing, batch operations, caching. But clever code that doesn't work is worse than simple code that does.

The final version generates 100,000 QR codes across two PDF files. It takes about an hour to run. It uses less than 100 MB of RAM. And most importantly, it completes successfully every single time.

Sometimes the best optimization isn't making your code faster—it's making it actually work.

The next time you're writing code that processes data at scale, remember: think about memory first, speed second. A slow script that completes is infinitely more valuable than a fast script that crashes.

TL;DR: I tried to generate 50,000 QR codes by loading them all into memory at once. My computer ran out of RAM and crashed. The fix was simple: generate QR codes one page at a time (30 at a time instead of 50,000). It's slower, but it works. Always consider memory usage when working with data at scale.

Building a FHIR Patient Deduplication System: A Journey from Chaos to Performance

Budi Widhiyanto — Sat, 15 Nov 2025 01:32:38 +0000

I'm working on a national project to collect health data from legacy systems in two pilot districts in Indonesia. The goal is to create interoperability between different healthcare systems so we can make better healthcare decisions based on complete patient data. It's an important project, and it's been a challenging one.

One of the biggest challenges has been patient deduplication. We're collecting data from multiple legacy systems, and each one has its own way of storing patient information. When we convert all this data to FHIR R4 format, we end up with duplicate patient records—the same person appearing multiple times in our system because they exist in multiple source systems.

This is the story of how I built a patient deduplication system that processes thousands of records in minutes instead of hours, and the lessons I learned from approaches that didn't work. If you're working with FHIR data from multiple sources or dealing with patient deduplication in healthcare systems, I hope my experience helps you.

The Beginning: Starting with a Partner's Approach

We're working with a technology partner on this national interoperability project. They had already built systems for patient matching in their own applications, and they shared their approach with us. Their method seemed reasonable: when creating a new patient, first search for existing patients using gender and birthdate filters, then apply fuzzy matching on the patient's name. If you find a good match, use that patient ID. Otherwise, create a new patient. If gender or birthdate are missing, fall back to using NIK (Indonesian National Identity Number) for exact matching.

I implemented their approach in our FHIR converter. Here's what it looked like:

def getPatientIdWithFuzzyLogic(internal_id, nik, name, birthdate, gender, parent_name):
    # Strategy 1: Use gender and birthdate if available
    if gender and birthdate:
        # Get patients matching gender and birthdate
        params = {'gender': gender, 'birthdate': birthdate}
        candidates = get_patients_with_params(params)

        if len(candidates) > 0:
            # Apply fuzzy matching on names
            for patient in candidates:
                score = fuzz.token_sort_ratio(name, patient.name)
                if score >= FUZZY_THRESHOLD:
                    return [patient["id"]]

    # Strategy 2: Fall back to NIK if gender/birthdate didn't work
    if nik:
        nik_patients = get_patients_with_params({'identifier': nik})
        if len(nik_patients) > 0:
            return [nik_patients[0]["id"]]

    # No match found, create new patient
    return None

The logic seemed solid. Search by demographics first, verify with name matching, fall back to NIK if needed. I deployed it and started converting patient data from our legacy systems.

That's when the problems started appearing.

The Failed Solution: Why the Partner's Method Didn't Work

The partner's method worked well in their own internal systems, but it didn't work for our interoperability project. I discovered two fundamental problems that made their approach unsuitable for our needs.

Problem 1: Missing Data in Legacy Systems

The partner's method relied heavily on having gender and birthdate for every patient. But we had data quality issues. Many patient records were missing gender or birthdate fields. When that happened, the search by demographics would fail, and we'd fall back to NIK matching. But if NIK was also missing or inconsistent, we'd create a duplicate patient.

I started seeing duplicate patients in our FHIR server. The same person would appear multiple times because the legacy data from different sources had different levels of completeness. One source might have gender and birthdate, another might only have NIK, and a third might have partial information. The fuzzy matching couldn't handle this inconsistency reliably.

Problem 2: Pagination Limits

The bigger problem was the FHIR API's pagination limit. Our FHIR server returns a maximum of 100 records per search request. When I searched for patients by gender and birthdate, I'd get the first 100 results. If there were more than 100 patients matching those criteria (which is common for popular birthdates), I'd need to paginate through all the results to find the right patient.

But the partner's code didn't handle pagination. It only looked at the first page of results. If the patient I was looking for was on page 2 or page 3, the search would miss them, and the converter would create a duplicate.

I could have fixed the pagination issue by implementing proper page-through logic, but that would make every patient search much slower—potentially making multiple API calls just to check if a patient exists. For batch conversion of thousands of patients, this would be too slow.

The Real Problem

The partner's method was built for their internal systems, where they controlled the data quality and had different constraints. Our situation was different. We were collecting data from multiple independent legacy systems, each with its own data quality issues, and we needed to process it efficiently at scale.

I needed a different approach—one that worked with the data we actually had, not the data we wished we had.

Finding a Better Way

I went back to analyze what reliable data we did have. The answer was NIK—the Indonesian National Identity Number. Almost every patient in our system had a NIK, and it was consistent across different legacy systems. It's a 16-digit number, always formatted the same way, and it uniquely identifies a person.

Why was I treating NIK as a fallback? It should be the primary method. NIK is more reliable than gender or birthdate for identifying patients in Indonesia. Gender and birthdate can be missing or inconsistent, but NIK is designed to be unique.

Building the Reference System: NIK-First Strategy

I redesigned the patient matching system to use NIK as the primary identifier, with demographic matching as a fallback only when necessary. Here's the new approach:

def getPatientIdByNIK(nik):
    """Get patient ID by NIK with caching"""
    if nik in nik_cache:
        return nik_cache[nik]

    params = {'identifier': nik}
    patients = get_patients_with_params(params)

    if len(patients) > 0:
        patient_id = patients[0]["id"]
        nik_cache[nik] = patient_id  # Cache for future lookups
        return patient_id

    return None

def getPatientIdWithFuzzyLogic(nik, name, birthdate, gender, parent_name):
    # Strategy 1: Try NIK exact match first (most reliable)
    if nik:
        nik_patients = get_patients_with_params({'identifier': nik, 'active': True})

        if len(nik_patients) > 0:
            # Single match - verify with name fuzzy matching
            if len(nik_patients) == 1:
                patient = nik_patients[0]
                patient_name = get_full_name(patient)
                new_patient_name = name

                score = fuzz.token_sort_ratio(new_patient_name, patient_name)
                if score >= FUZZY_THRESHOLD:
                    return [patient["id"]]

            # Multiple NIK matches - use fuzzy matching to find best
            elif len(nik_patients) > 1:
                best_match = None
                best_score = 0

                for patient in nik_patients:
                    patient_name = get_full_name(patient)
                    score = fuzz.token_sort_ratio(name, patient_name)
                    if score > best_score:
                        best_score = score
                        best_match = patient

                if best_score >= FUZZY_THRESHOLD:
                    return [best_match["id"]]

    # Strategy 2: Fall back to demographic matching if NIK fails
    if gender and birthdate:
        params = {'gender': gender, 'birthdate': birthdate}
        candidates = get_patients_with_params(params)

        if len(candidates) > 0:
            for patient in candidates:
                score = fuzz.token_sort_ratio(name, get_full_name(patient))
                if score >= FUZZY_THRESHOLD:
                    return [patient["id"]]

    # No match found
    return None

This new system inverts the partner's approach. Instead of searching by demographics first and falling back to NIK, I search by NIK first and fall back to demographics. This solves both problems:

Missing data: NIK is more consistently available than gender/birthdate in our legacy systems
Pagination: Searching by NIK returns far fewer results (usually just one), so pagination isn't an issue

I also added a caching mechanism with getPatientIdByNIK. When converting thousands of patient records, many of them might be the same person (repeat visits, multiple encounters, etc.). By caching the NIK-to-patient-ID mapping, I avoid making redundant API calls for patients I've already looked up.

The fuzzy matching on names is still there as a safety check. Even when I find a patient by NIK, I verify that the name matches using fuzzy string comparison. This catches cases where NIK might have been entered incorrectly or where there might be data quality issues.

Testing the New System

When I tested the new NIK-first system with real data from our legacy systems, it worked much better. The converter found existing patients reliably, even when demographic data was missing. The pagination problem disappeared because NIK searches rarely return more than 100 results. And the caching made bulk conversion much faster.

I watched the logs during a test run converting 1,000 patient records: "Found patient by NIK... Found patient by NIK... Created new patient (no NIK match)... Found patient by NIK (cached)..." The system was working.

But I still had a problem. Before implementing this fix, the old system had already created duplicate patients in our FHIR server. Some NIKs had 2, 3, or even 10 duplicate patient records. While the new reference system prevented future duplicates, I needed to clean up the existing ones.

I needed a deduplication process.

The Deduplication Challenge: Two Versions

Building a system to deduplicate existing patients was a different challenge entirely. With the reference system, I was preventing new duplicates—a relatively simple task of checking before creating. With deduplication, I needed to find all existing duplicates, choose which one should be the "master," and then update potentially thousands of medical records to point to that master instead of the duplicates.

This was going to touch a lot of data. I needed to be careful.

Version 1: Sequential Processing - The Safe, Slow Way

For my first implementation, I chose the safest possible approach: sequential processing. I would handle one NIK at a time, processing each step completely before moving to the next. No parallelization, no batch operations, just simple, linear execution.

The algorithm was straightforward:

Find all patient IDs with the same NIK
Select which one should be the master (I chose the most recently updated)
Find all resources (observations, encounters, etc.) referencing the duplicate patient IDs
Update each resource to reference the master patient ID instead
Mark the duplicate patients as inactive

I wrote it as a simple loop:

for nik in nik_list:
    # Find duplicate patients
    patient_ids = find_patients_by_nik(nik)

    if len(patient_ids) <= 1:
        continue  # No duplicates, skip

    # Select master patient (most recently updated)
    master_id = select_master_patient(patient_ids)
    duplicate_ids = [pid for pid in patient_ids if pid != master_id]

    # Find all resources referencing duplicates
    for resource_type in RESOURCE_TYPES:
        for patient_id in duplicate_ids:
            resources = fetch_resources(resource_type, patient_id)

            # Update each resource
            for resource in resources:
                update_patient_reference(resource, master_id)
                put_resource(resource)

    # Mark duplicates inactive
    for dup_id in duplicate_ids:
        mark_patient_inactive(dup_id, master_id)

I tested it with a single NIK first. It worked. I checked the data afterward—all the medical records now pointed to the master patient, the duplicates were marked inactive with a replaced-by link to the master. Perfect.

Then I tried it with ten NIKs. It worked, but it took 15 minutes. Okay, that's not great, but acceptable for a cleanup operation, right?

Then I ran it on our actual list: 68 NIKs with known duplicates. I started the script, watched the logs for a few minutes, then went to get coffee. When I came back 30 minutes later, it had processed 3 NIKs. I did the math. 68 NIKs at 10 minutes each... over 11 hours.

I let it run overnight. The next morning, it had finished successfully. All the duplicates were cleaned up. But 11 hours was not acceptable. We had hundreds more NIKs to process in other datasets. At this rate, a full deduplication would take days, maybe weeks. And during that time, the script would be constantly hammering our FHIR server with API calls.

The problem was obvious: I was making way too many individual API calls. For each patient ID, I was searching for observations one at a time, then encounters one at a time, then medications, procedures, diagnostic reports—the list went on. And FHIR has a lot of resource types that can reference patients. Even though I filtered it down to the most common ones, I was still checking 27 different resource types. For each duplicate patient. Sequentially.

If a single NIK had 3 duplicate patients and each patient had 20 observations, that's 60 individual GET requests just for observations, plus 60 individual PUT requests to update them. Multiply that by all the other resource types, and you're talking about hundreds of API calls per NIK. No wonder it was slow.

I watched the script run for a while, looking at the logs. The server was responding quickly—each API call only took a few hundred milliseconds. But I was only making one call at a time. The network latency, the sequential execution, it all added up. I was wasting so much time just waiting.

That's when I remembered something. When I built the original FHIR converter, I had faced a similar problem. Converting thousands of patient records one at a time was slow. I had solved it by using batch operations and parallel processing. I could apply the same techniques here.

Version 2: Batch Processing & Parallelization - The Fast Way

The key insight was this: most of the steps in deduplication don't depend on each other. When I'm fetching observations for a patient, I don't need to wait for the encounters to be fetched first. When I'm updating resources, I don't need to update them one at a time—I can batch them together.

I redesigned the system with two major optimizations: parallel resource fetching and batch updates.

For parallel fetching, I used Python's ThreadPoolExecutor:

from concurrent.futures import ThreadPoolExecutor, as_completed

def find_all_references(self, patient_ids: List[str], master_id: str = None):
    """Find ALL resources that reference duplicate patient IDs (parallel)"""
    # Only search for duplicates, not master
    search_patient_ids = [pid for pid in patient_ids if pid != master_id]

    all_references = {}

    # Use ThreadPoolExecutor for parallel fetching
    with ThreadPoolExecutor(max_workers=self.max_workers) as executor:
        # Submit all resource type fetches in parallel
        future_to_type = {
            executor.submit(self._fetch_resources_for_type, resource_type, search_patient_ids): resource_type
            for resource_type in PATIENT_REFERENCING_RESOURCES
        }

        # Collect results as they complete
        for future in as_completed(future_to_type):
            resource_type = future_to_type[future]
            try:
                resource_type, resources = future.result()
                if resources:
                    all_references[resource_type] = resources
            except Exception as e:
                logger.error(f"Error fetching {resource_type}: {e}")

    return all_references

Instead of fetching observations, then encounters, then medications sequentially, I now launch all those searches in parallel. Five worker threads (configurable via MAX_WORKERS) simultaneously fetch different resource types. This alone cut the fetching time by about 80%.

But the real performance gain came from batch updates. FHIR supports bundle operations—instead of sending one resource update at a time, you can send a bundle of up to hundreds of updates in a single API call. I implemented this using FHIR batch bundles:

def _create_batch_bundle(self, resources_to_update: List[tuple]) -> dict:
    """Create a FHIR batch bundle for updating multiple resources"""
    bundle = {
        "resourceType": "Bundle",
        "type": "batch",
        "entry": []
    }

    for resource_type, resource in resources_to_update:
        bundle["entry"].append({
            "request": {
                "method": "PUT",
                "url": f"{resource_type}/{resource['id']}"
            },
            "resource": resource
        })

    return bundle

def update_all_references(self, all_references, duplicate_ids, master_id):
    """Update all references using batch bundles"""
    resources_to_update = []

    # Collect all resources that need updating
    for resource_type, resources in all_references.items():
        for resource in resources:
            if self.update_patient_references(resource, duplicate_ids, master_id):
                resources_to_update.append((resource_type, resource))

    # Split into batches of 100
    num_batches = math.ceil(len(resources_to_update) / self.batch_size)

    for i in range(num_batches):
        batch_resources = resources_to_update[i*self.batch_size:(i+1)*self.batch_size]

        # Create and execute batch bundle
        bundle = self._create_batch_bundle(batch_resources)
        result_bundle = self._execute_batch_bundle(bundle)

        # Check results
        for entry in result_bundle.get('entry', []):
            response = entry.get('response', {})
            if response.get('status', '').startswith('2'):
                self.stats['resources_updated'] += 1

Now instead of 60 separate PUT requests to update 60 observations, I send one request with a bundle containing all 60 updates. The FHIR server processes them efficiently on its end, and I get back a bundle with the results.

I also added a smart optimization: only fetch resources for duplicate patients, not the master. Resources already pointing to the master don't need to be fetched or updated. This simple check cut the amount of data I needed to process roughly in half:

# Only search for duplicates, not master (master resources don't need updating)
search_patient_ids = [pid for pid in patient_ids if pid != master_id]

I ran the new version on the same 68 NIKs that had taken 11 hours before. This time, I watched the progress in real-time. The parallel fetching worked beautifully—I could see all 27 resource types being queried simultaneously. The batch updates were lightning fast—bundles of 100 resources updated in seconds.

Twenty-three minutes later, it was done. The same operation that took 11 hours now took 23 minutes. That's roughly 30 times faster.

I ran it again on a larger dataset just to be sure. Same results. The system was consistently fast. The optimization worked.

Technical Deep Dives: Solving Real Problems

While the main architecture was solid, I ran into several challenges that required specific solutions. Let me share three that taught me the most.

Challenge 1: Pagination and URL Handling

One issue that cost me two hours of debugging was pagination. When fetching resources, FHIR servers often return results in pages. You get the first 100 results, plus a "next" link to get the next page. Simple enough, right?

Except the "next" link returned by our FHIR server had a trailing slash before the query parameters: /Patient/?_count=100&_page_token=abc. When I tried to fetch that URL, I got 404 errors. The server expected /Patient?_count=100 (no slash before the question mark).

I spent way too long staring at logs before I noticed that subtle difference. Once I saw it, the fix was simple:

def _get_next_link(self, bundle: dict) -> str:
    """Get next page URL from bundle"""
    for link in bundle.get('link', []):
        if link.get('relation') == 'next':
            next_url = link.get('url')
            if next_url and '/fhir/' in next_url:
                path_and_query = next_url.split('/fhir/', 1)[1]
                # Remove trailing slash before query string
                if '/?' in path_and_query:
                    path_and_query = path_and_query.replace('/?', '?')
                return f"{self.base_url}/{path_and_query}"
    return None

This taught me to always validate assumptions about external APIs. Just because something looks like a standard URL doesn't mean it will work exactly as you expect.

Challenge 2: Choosing the Right Master Patient

Initially, I just picked the most recently updated patient as the master. Simple logic: the newest record is probably the most complete. But then I realized this created a problem. If I ran the deduplication twice, I might choose a different master the second time (if one of the duplicates had been updated in between). This would cause unnecessary churn—moving all those resource references back and forth.

The solution was to implement stability: once a patient has been designated as master, it should stay the master. I did this using FHIR meta tags. When a patient becomes the master, I tag it with a "golden resource" tag that points to itself:

def _has_self_referencing_golden_tag(self, patient: dict) -> bool:
    """Check if patient has golden resource tag pointing to itself"""
    patient_id = patient.get('id')
    if not patient_id:
        return False

    tags = patient.get('meta', {}).get('tag', [])
    for tag in tags:
        if tag.get('system') == 'http://terminology.kemkes.go.id/sp-replaced-by':
            if tag.get('code') == patient_id:
                return True
    return False

def select_master_patient(self, patient_ids: List[str]) -> str:
    """Select which patient should be the master

    Priority:
    1. Active patient with golden resource tag (existing master)
    2. Most recently updated active patient
    """
    patients = [fetch_patient(pid) for pid in patient_ids]
    active_patients = [p for p in patients if p.get('active', True)]

    # Check for existing master first
    existing_masters = [p for p in active_patients
                       if self._has_self_referencing_golden_tag(p)]

    if existing_masters:
        return max(existing_masters,
                  key=lambda p: p.get('meta', {}).get('lastUpdated', ''))['id']
    else:
        return max(active_patients,
                  key=lambda p: p.get('meta', {}).get('lastUpdated', ''))['id']

Now when I run deduplication, it first checks if one of the patients is already marked as master. If so, use that one. If not, pick the most recently updated and mark it as master. This ensures consistency across multiple runs.

Challenge 3: Handling Batch Failures Gracefully

When I first implemented batch updates, I used FHIR transaction bundles (type: "transaction"). These are atomic—either all updates succeed, or they all fail. This seemed safe, but it had a major problem: if even one resource in the batch had an issue, the entire batch would fail, and none of the updates would be applied.

During testing, I had a batch of 100 observations to update. One of them had a validation issue (a missing required field from old data). The entire batch failed, and I had to figure out which one was problematic. This was frustrating and slow.

The solution was to switch to batch bundles (type: "batch") instead of transaction bundles. With batch bundles, each operation in the bundle succeeds or fails independently:

bundle = {
    "resourceType": "Bundle",
    "type": "batch",  # Independent operations, not atomic
    "entry": [...]
}

Now if one resource in a batch of 100 fails, the other 99 still get updated successfully. I log the failure, track it in my stats, but don't let it block the entire operation:

for idx, entry in enumerate(result_bundle.get('entry', [])):
    response = entry.get('response', {})
    status = response.get('status', '')

    if status.startswith('2'):  # Success (2xx status code)
        self.stats['resources_updated'] += 1
        self.stats['batch_successes'] += 1
    else:
        # Log failure but continue
        resource_type, resource = batch_resources[idx]
        error_msg = f"Failed to update {resource_type}/{resource['id']}: {status}"
        logger.warning(error_msg)
        self.stats['errors'].append(error_msg)
        self.stats['batch_failures'] += 1

This makes the system much more robust. Even with messy real-world data, the deduplication completes successfully for the vast majority of resources, and I have a clear log of anything that failed.

Making It Production-Ready: The API Layer

The command-line script worked great for batch deduplication, but for ongoing operations, I needed something more accessible. I built a FastAPI wrapper that exposes the deduplication functionality as a REST API.

The API has two main endpoints:

@app.post("/deduplicate")
async def deduplicate_single_nik(request: SingleNIKRequest):
    """Deduplicate patients for a single NIK"""
    start_time = time.time()

    deduplicator = FHIRPatientDeduplicator(
        base_url=FHIR_BASE_URL,
        nik_system=NIK_SYSTEM,
        api_key=API_KEY,
        batch_size=BATCH_SIZE,
        max_workers=MAX_WORKERS
    )

    patient_ids = deduplicator.find_patients_by_nik(request.nik)

    if len(patient_ids) > 1:
        deduplicator.deduplicate_by_nik(
            nik=request.nik,
            patient_ids=patient_ids,
            delete_duplicates=request.delete_duplicates
        )

    duration = time.time() - start_time

    return DeduplicationResponse(
        success=True,
        nik=request.nik,
        resources_found=deduplicator.stats['resources_found'],
        resources_updated=deduplicator.stats['resources_updated'],
        duration_seconds=round(duration, 2),
        timestamp=datetime.utcnow().isoformat()
    )

I added timing information so we can track how long each deduplication takes. This is useful for monitoring and capacity planning. I also added a batch endpoint that processes multiple NIKs in sequence, with per-NIK timing and summary statistics.

The API is deployed on Google Cloud Run, which handles scaling automatically. If we need to process a large batch of NIKs, we can send them to the batch endpoint and it processes them sequentially (to maintain data integrity) while still being fast thanks to the parallelization and batch updates happening under the hood.

The API also makes it easy for other teams to integrate deduplication into their workflows. They can call the endpoint whenever they import new data, and any duplicates get cleaned up automatically.

Reflection & Lessons Learned

Looking back on this project, I'm proud of what I built, but I'm also very aware of what I did wrong and what I'd do differently next time.

What Went Well

The NIK-based reference system is simple and reliable. By choosing the right unique identifier from the start, I avoided all the complexity of demographic matching. The system hasn't created a single duplicate patient since I deployed it.

The optimization from sequential to batch/parallel processing was a huge win. Going from 11 hours to 23 minutes isn't just about speed—it's about practicality. At 11 hours, running deduplication was something you'd do rarely, maybe once a month, as a special operation. At 23 minutes, it's something you can run weekly, or even daily if needed. That changes how useful the tool is.

The architectural decisions around resilience—using batch bundles instead of transactions, tracking errors but continuing, logging everything—have proven their value. The system handles real-world messy data gracefully. It doesn't fail catastrophically because one record has a problem.

What I'd Do Differently

I should have thought about deduplication from day one. If I had implemented the NIK check in the original converter, I wouldn't have created thousands of duplicates that needed cleaning up. This is a classic example of a small amount of foresight preventing a large amount of pain later.

I wasted a month trying to adapt the partner's solution. I should have analyzed our specific problem more carefully first. Their demographic matching system was sophisticated and well-built, but it was solving a different problem than ours. Understanding the problem deeply before jumping to solutions would have saved a lot of time.

I should have built the parallel/batch version first, or at least earlier. I learned more from building it than I would have from just thinking about it, but if I had started with "how do I make this fast?" instead of "how do I make this work?", I would have gotten to the good solution faster.

Technical Learnings

Batch operations are powerful. Reducing API calls from hundreds to dozens makes a massive difference. Whenever you're doing lots of similar operations, look for a way to batch them.

Parallelization works best when operations are independent. Fetching different resource types in parallel is perfect because they don't depend on each other. But I couldn't parallelize the actual deduplication of different NIKs because they might reference the same resources. Understanding these dependencies is crucial.

The FHIR standard is well-designed but implementations vary. Features like batch bundles, search parameters, and pagination work slightly differently on different servers. Always test against your actual FHIR server, not just against the spec.

Real-world data is messy. Invalid formats, missing fields, duplicate identifiers—they're all going to happen. Build your system to handle errors gracefully rather than assuming perfect data.

Future Improvements

If I were to continue improving this system, here's what I'd add:

More sophisticated master selection. Currently I use "most recently updated" as a tiebreaker. But there are other factors that could matter—which patient has the most complete data, which one has the most recent medical records, which one was verified most recently. A scoring system could help.

Automated detection of new duplicates. Right now someone has to identify that duplicates exist and call the API. I could build a background job that periodically scans for NIKs with multiple active patients and flags them for review or automatic deduplication.

Intelligent merging of patient demographic data. When deduplicating patients, I currently just pick one master patient and mark the others inactive. But sometimes the duplicate records have complementary information—one might have a phone number, another might have an address. I could merge the best available data from all duplicates into the master patient record before marking duplicates inactive. This would ensure no valuable information is lost during deduplication.

Conclusion

Building this patient deduplication system taught me that good software engineering isn't just about making things work—it's about making them work well, reliably, and efficiently. It's about thinking ahead, but also about being willing to rework things when your first approach doesn't scale.

I made mistakes. I spent time on solutions that didn't fit my problem. I built a slow version first when I could have built a fast one. But each of those mistakes taught me something valuable. Now I know to identify the right unique identifier before building a system around it. I know to batch operations whenever possible. I know to design for resilience, not just for the happy path.

Most importantly, I learned that performance optimization isn't just about making things faster—it's about making them useful. A tool that takes 11 hours to run gets used rarely. A tool that takes 23 minutes gets used regularly. Speed enables usefulness.

If you're building something similar—whether it's deduplication, data migration, or any kind of batch processing—I hope my journey helps you avoid some of the wrong turns I took. Think about deduplication early. Choose the right unique identifier. Build for resilience. Batch and parallelize when you can. And don't be afraid to throw away your first version if it doesn't scale.

The code is running in production now, quietly cleaning up duplicate patient records every week. It works. It's fast. And most importantly, it helps make sure that when a healthcare provider looks up a patient's medical history, they see the complete picture. That's what matters.

Relearning Microservices with a Weekend Mini eCommerce Build

Budi Widhiyanto — Wed, 24 Sep 2025 03:24:21 +0000

One rainy weekend I decided to refresh my microservices skills by building a small eCommerce platform from scratch. I wanted a playground that was close enough to real work to show the classic problems—clear boundaries, steady APIs, reliable deployments—without growing into a long project. This article is my field journal from that sprint: what I built, why I made certain choices, and how the code in this repo supports every decision.

Architecture at a Glance

Saturday morning started with a blank page and four simple boxes. I knew the weekend would stay calm only if every box owned one clear job and followed the same rules. The result is a Node.js monorepo with four deployable workspaces that live together but stay independent:

User Service handles registration, login, and profile lookups so the rest of the stack never has to guess who is calling.
Product Service manages the catalog and keeps price data clean.
Order Service turns carts into history by connecting users and products.
API Gateway sits on the edge and hides the backend layout from clients.

Each service gets its own Postgres database and REST API. To avoid copying the same setup again and again, every service depends on @mini/shared for logging, HTTP helpers, error classes, and configuration tools. From there the workflow stays simple on purpose: npm run compose:up brings the stack online with this Compose file driving the topology:

# docker-compose.yml
services:
  user-service:
    command: npm run dev --workspace services/user
    ports:
      - "3001:3001"
    depends_on: [user-db]

  product-service:
    command: npm run dev --workspace services/product
    ports:
      - "3002:3002"
    depends_on: [product-db]

  order-service:
    command: npm run dev --workspace services/order
    ports:
      - "3003:3003"
    depends_on: [order-db, user-service, product-service]

  api-gateway:
    command: npm run dev --workspace gateway
    ports:
      - "8080:8080"
    depends_on: [user-service, product-service, order-service]

volumes:
  user-db-data:
  product-db-data:
  order-db-data:

The manifests in k8s/ reproduce the same shape inside a Kubernetes cluster when I want to push things a little harder.

Shared Platform Capabilities

By midday I noticed the same pattern, service after service. Each one wanted identical Express plumbing, the same error classes, and the same .env routine. Rather than repeat myself, I moved those cross-cutting pieces into @mini/shared so the rest of the weekend could focus on business rules instead of setup.

The shared HTTP helper keeps every edge consistent by centralising the Express setup, wiring in JSON parsing, health checks, and error handling so every service exposes the same behaviour:

// shared/src/http.js
function createApp({ serviceName, logger, routes }) {
  if (!serviceName) throw new Error('serviceName is required');
  const app = express();
  app.disable('x-powered-by');
  app.use(express.json());

  app.get('/healthz', (_req, res) => {
    res.json({ service: serviceName, status: 'ok', uptime: process.uptime() });
  });

  if (typeof routes === 'function') {
    routes(app);
  }

  app.use((_req, _res, next) => next(new NotFoundError()));

  app.use((err, req, res, _next) => {
    const error = err instanceof AppError ? err : new AppError('Internal Server Error');
    logger?.error?.('request failed', { code: error.code, status: error.status, id: req.id });
    res.status(error.status).json({ error: { code: error.code, message: error.message } });
  });

  return app;
}

Error classes stay in one place, so every service can throw meaningful responses and map domain problems to HTTP status codes without duplicating boilerplate:

// shared/src/errors.js
class ValidationError extends AppError {
  constructor(message = 'Validation failed', details) {
    super(message, { status: 400, code: 'validation_error', details });
  }
}

class UnauthorizedError extends AppError {
  constructor(message = 'Unauthorized') {
    super(message, { status: 401, code: 'unauthorized' });
  }
}

Configuration loading is just as centralised, which means each service validates its environment variables before it starts and applies optional parsers or defaults in one predictable location:

// shared/src/env.js
function getConfig(schema) {
  return Object.entries(schema).reduce((acc, [key, options]) => {
    let value = process.env[key];
    const required = !!options?.required;
    const fallback = options?.default;
    const parser = options?.parser;

    if ((value === undefined || value === '') && fallback !== undefined) {
      value = typeof fallback === 'function' ? fallback() : fallback;
    }

    if ((value === undefined || value === '') && required) {
      throw new Error(`Missing required environment variable ${key}`);
    }

    acc[key] = typeof parser === 'function' && value !== undefined ? parser(value) : value;
    return acc;
  }, {});
}

Lastly, the shared logger stamps every log line with the service name, which makes cross-service debugging feel like reading a conversation instead of a jumble of anonymous messages:

// shared/src/logger.js
function createLogger(serviceName) {
  const prefix = serviceName ? `[${serviceName}]` : '[app]';
  const base = { info: console.log, error: console.error, warn: console.warn };

  return {
    info: (msg, meta) => base.info(prefix, msg, meta || ''),
    warn: (msg, meta) => base.warn(prefix, msg, meta || ''),
    error: (msg, meta) => base.error(prefix, msg, meta || ''),
  };
}

After that refactor each service file felt lighter. The interesting code stayed in front, and new features no longer meant reworking the foundations.

Service Deep Dive

User Service: Reestablishing Identity Basics

The first feature I added was identity. Past projects taught me that most bugs look like security bugs when the caller is unknown, so registerUser hashes the password, saves it, and issues a JWT in one short flow:

// services/user/src/service.js
async function registerUser({ username, password }) {
  if (!username || !password) {
    throw new ValidationError('username and password are required');
  }

  const existing = await findByUsername(username);
  if (existing) {
    throw new ValidationError('username already taken');
  }

  const passwordHash = await hashPassword(password);
  const user = await createUser({ username, passwordHash });
  const token = issueToken({ sub: user.id, username: user.username, role: user.role });
  return { user, token };
}

Startup logic seeds an admin account from environment variables because I have locked myself out of dashboards before; the database initializer keeps that safety net in place by creating the table and populating the admin row the moment the service boots:

// services/user/src/db.js
async function initDb(customPool = getPool()) {
  await customPool.query(`
    CREATE TABLE IF NOT EXISTS users (
      id TEXT PRIMARY KEY,
      username TEXT UNIQUE NOT NULL,
      password_hash TEXT NOT NULL,
      role TEXT NOT NULL DEFAULT 'user'
    );
  `);

  const { rows } = await customPool.query('SELECT id FROM users WHERE username = $1 LIMIT 1', [
    config.ADMIN_USERNAME,
  ]);

  if (rows.length === 0 && config.ADMIN_PASSWORD) {
    const passwordHash = await hashPassword(config.ADMIN_PASSWORD);
    await customPool.query(
      'INSERT INTO users (id, username, password_hash, role) VALUES ($1, $2, $3, $4)',
      [crypto.randomUUID(), config.ADMIN_USERNAME, passwordHash, 'admin'],
    );
  }
}

Authentication sits in a small middleware that checks Bearer tokens and attaches the decoded data to the request. The cryptography helpers stay in their own module so the rest of the code can trust req.user without drama, and so future changes to signing logic happen in one place:

// services/user/src/auth-middleware.js
function authRequired(req, _res, next) {
  const header = req.headers.authorization || '';
  const [, token] = header.split(' ');

  if (!token) {
    return next(new UnauthorizedError('Missing bearer token'));
  }

  try {
    const payload = verifyToken(token);
    req.user = { id: payload.sub, username: payload.username, role: payload.role };
    return next();
  } catch (error) {
    return next(new UnauthorizedError('Invalid token'));
  }
}

// services/user/src/security.js
function issueToken(payload) {
  return jwt.sign(payload, config.JWT_SECRET, { expiresIn: '1h' });
}

function verifyToken(token) {
  return jwt.verify(token, config.JWT_SECRET);
}

Product Service: Guarding the Catalog

With identity stable, I moved to the catalog. Public routes need to be friendly but safe, so they validate pagination settings before running a query to avoid accidental full-table scans or wasteful database calls:

// services/product/src/service.js
async function fetchProducts(query = {}) {
  if (query.limit !== undefined && isNaN(Number(query.limit))) {
    throw new ValidationError('limit must be numeric');
  }
  if (query.offset !== undefined && isNaN(Number(query.offset))) {
    throw new ValidationError('offset must be numeric');
  }
  return listProducts({ limit: query.limit, offset: query.offset });
}

Admin routes are stricter: the price parser stops invalid or negative numbers before they reach the database, and the admin middleware keeps write actions behind a trusted role so change control stays tight:

// services/product/src/service.js
function parsePrice(price) {
  if (price === undefined) return undefined;
  const value = Number(price);
  if (Number.isNaN(value) || value < 0) {
    throw new ValidationError('price must be a non-negative number');
  }
  return Math.round(value * 100) / 100;
}

async function createProductRecord({ name, description, price }) {
  if (!name || !description) {
    throw new ValidationError('name and description are required');
  }
  const parsedPrice = parsePrice(price);
  if (parsedPrice === undefined) {
    throw new ValidationError('price is required');
  }
  return createProduct({
    id: randomUUID(),
    name,
    description,
    price: parsedPrice,
  });
}

// services/product/src/admin-middleware.js
function adminOnly(req, _res, next) {
  if (!req.user) {
    return next(new UnauthorizedError('Auth required'));
  }

  if (req.user.role !== 'admin') {
    return next(new UnauthorizedError('Admin access required'));
  }

  next();
}

Each product receives a UUID when it is created and is stored in Postgres. That small step keeps tracking clear and makes later integrations easier if this prototype grows into something larger because every product ID stays unique across environments and migrations.

Order Service: Cross-Service Collaboration

Orders were the most satisfying part because they make the services work together and force the boundaries to prove themselves. The handler checks that both userId and productId exist, validates pagination options, and then calls the product service to confirm the item is still available:

// services/order/src/service.js
async function recordOrder({ userId, productId }) {
  if (!userId || !productId) {
    throw new ValidationError('userId and productId are required');
  }

  const product = await fetchProduct(productId);
  if (!product) {
    throw new ValidationError('product not found');
  }

  return createOrder({ id: randomUUID(), userId, productId });
}

That remote call lives in a small client that normalizes URLs, treats 404s as “not found,” and wraps other errors in a validation message so downstream consumers receive clean, human-readable results:

// services/order/src/clients/product-client.js
async function fetchProduct(productId) {
  const base = config.PRODUCT_SERVICE_URL.endsWith('/')
    ? config.PRODUCT_SERVICE_URL.slice(0, -1)
    : config.PRODUCT_SERVICE_URL;
  const res = await fetch(`${base}/products/${productId}`);

  if (res.status === 404) {
    return null;
  }

  if (!res.ok) {
    throw new ValidationError('product lookup failed');
  }

  const body = await res.json();
  return body.product;
}

The repository stays lean by saving only foreign keys. If the catalog changes later, the order history still reads well, and the service can rebuild richer views by fetching user and product details when needed, which keeps the storage footprint small and the coupling loose:

// services/order/src/repository.js
async function createOrder({ id, userId, productId }, pool = getPool()) {
  await pool.query(
    'INSERT INTO orders (id, user_id, product_id) VALUES ($1, $2, $3)',
    [id, userId, productId],
  );
  return mapOrder({ id, user_id: userId, product_id: productId, created_at: new Date() });
}

API Gateway and Service-to-Service Communication

From the start I wanted one door for clients. The gateway connects everything, and the proxyTo helper does the heavy lifting by taking an incoming request, rebuilding the destination URL, and streaming the response back without leaking hop-by-hop headers:

// gateway/src/index.js
function proxyTo(baseUrl) {
  const normalizedBase = baseUrl.endsWith('/') ? baseUrl.slice(0, -1) : baseUrl;

  return async (req, res, next) => {
    try {
      const targetUrl = new URL(req.originalUrl, `${normalizedBase}/`).toString();

      const headers = { ...req.headers };
      delete headers.host;

      const init = {
        method: req.method,
        headers,
      };

      if (req.method !== 'GET' && req.method !== 'HEAD') {
        init.body = req.body ? JSON.stringify(req.body) : undefined;
        init.headers['content-type'] = 'application/json';
      }

      const response = await fetch(targetUrl, init);
      const text = await response.text();
      res.status(response.status);
      try {
        const parsed = JSON.parse(text || '{}');
        res.json(parsed);
      } catch (_err) {
        res.send(text);
      }
    } catch (error) {
      next(error);
    }
  };
}

The routes mount each downstream service under a clean prefix, which keeps the public API steady even if I move services around inside the cluster and makes documentation easier for anyone consuming the gateway:

// gateway/src/index.js
router.use('/users', proxyTo(serviceConfig.userServiceUrl));
router.use('/products', proxyTo(serviceConfig.productServiceUrl));
router.use('/orders', proxyTo(serviceConfig.orderServiceUrl));

Inside the system, the order service calls the product service through the same HTTP endpoints. The approach is intentionally simple because it matches what many teams already run. Right now those calls trust the network and do not add extra authentication, so improving that handshake is near the top of my hardening list. When I explore rate limiting or service discovery, the gateway will be the natural place to add them.

Configuration, Security, and Secrets Management

One personal rule for the project was simple: avoid “works on my machine” bugs. Every service reads configuration through env.getConfig, which applies defaults, checks required values, and handles small type conversions before the app even starts:

// services/product/src/config.js
env.loadEnv({ files: [path.join(__dirname, '..', '.env')] });

const config = env.getConfig({
  PORT: { default: 3002, parser: Number },
  DATABASE_URL: { default: 'postgres://product_service:password@localhost:5434/product_db' },
  JWT_SECRET: { default: 'devsecret', required: true },
});

When the stack runs in Kubernetes, the JWT secret comes from a cluster secret instead of shipping inside the image, which means new secrets can be rotated without rebuilding containers:

# k8s/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: jwt-secret
  namespace: mini-ecommerce
stringData:
  value: devsecret

The user service issues tokens with that secret, the other services verify them locally, and role checks—like the admin filter in the product service—use the decoded payload to make decisions.

Local Development Workflow

Weekend hacking works only if the feedback loop stays short, so Docker Compose became the main control room:

Install dependencies once with npm install so every workspace shares the same node_modules tree.
Run npm run compose:up to launch the three services, the gateway, and their Postgres companions (using the compose file shown above) and let Docker wire the local network for you.
Send every request through http://localhost:8080 so the gateway path stays well traveled and the API surface mirrors production traffic.

Right now the services run with plain node processes, so I still restart them by hand when code changes. Hot reloaders are on the to-do list, but even without them the shared package keeps logs and errors consistent. Docker volumes remember the seeded catalog and test users between runs, so I can experiment, restart, and keep moving without rebuilding the database every time.

Deploying to Kubernetes

By Sunday afternoon curiosity won. I wanted to watch the system run inside a cluster, so the manifests in k8s/ mirror the Compose layout almost line for line.

The user service deployment is representative of the pattern:

# k8s/user-service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: user-service
          image: mini-ecommerce-user:latest
          env:
            - name: JWT_SECRET
              valueFrom:
                secretKeyRef:
                  name: jwt-secret
                  key: value
          readinessProbe:
            httpGet:
              path: /healthz
              port: 3001

The gateway pairs a deployment with an ingress so there is one public entry point:

# k8s/gateway.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
spec:
  template:
    spec:
      containers:
        - name: api-gateway
          image: mini-ecommerce-gateway:latest
          env:
            - name: USER_SERVICE_URL
              value: http://user-service
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-gateway
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api-gateway
                port:
                  number: 80

Dedicated Postgres deployments keep data siloed per service, honoring the “database per service” mantra without any shared state leaks.

With images tagged—think mini-ecommerce-user:latest—a kubectl apply -f k8s/ sets up the same architecture I run locally. Rolling updates and restarts behave the way I expect, which makes this repo a comfortable sandbox for practicing cluster operations. Secrets ship with kubectl apply -f k8s/secret.yaml, and the workload manifests read them as environment variables; config maps follow the same pattern for plain settings.

Observability, Testing, and Next Experiments

I kept observability light but friendly. The logger shown earlier prefixes every line with a service name, so one tail -f gives a clear picture of who is talking. Tests live next to the code inside each service’s __tests__ folder; they mix unit checks with small integration cases so I can change a function and still trust the boundaries, and they double as documentation because they show how the modules are meant to collaborate.

There is still plenty to explore. A message broker for order events, circuit breakers inside the product client, and rate limiting at the gateway are already on the list. The current setup leaves room for those ideas without tearing up the base.

What I Relearned

Clear domain boundaries keep ownership simple and give every rule a home.
A small shared toolkit (@mini/shared) stops the team—future me included—from rebuilding the same helpers.
The API gateway protects client URLs while backend services evolve in private.
Matching the local Compose setup inside Kubernetes lowers the stress when promoting changes.

The weekend build reminded me that microservices are less about counting repositories and more about choosing clear boundaries. Steady ownership, honest contracts, and repeatable operations beat shiny patterns every time. Now that this mini eCommerce system lives in the toolbox, I can reopen the code and the lessons whenever I need a quick refresher.

Scaling Healthcare Data Processing: Multi-Environment FHIR Patient Updates with Smart Batch Processing

Budi Widhiyanto — Tue, 23 Sep 2025 05:36:58 +0000

The request sounded simple: “Can we keep patient phone numbers up to date?”

At first we thought it was a quick operations chore. Then we traced the real data flow and saw the mess underneath. Phone numbers rolled in from WhatsApp, hospital front desks, and survey tools, each with its own format. Patients jumped between facilities, so their trails were often broken. The operations team lived in Google Sheets, and every region guarded its own FHIR server with different credentials, limits, and quirks.

Our first fix was a tiny script that looped through one row at a time. On a test file it worked fine, but once we aimed it at 10,000 rows the run dragged on for hours, chewed through hundreds of megabytes of memory, and could crash if a single record looked wrong.

This article is the story of how that fragile script became a production-ready workflow. The same 10,000-row load now finishes in about 10–12 minutes per region, using only 256Mi memory and 0.5 vCPU. More important, it stays steady, it survives bad data, and operations teams are happy to run it every day.

From One Region to a Platform

The moment a second region joined the queue, the to-do list grew fast. We suddenly needed to:

Serve multiple regions, such as Purbalingga and Lombok Barat, at the same time without stepping on each other.
Keep every environment on its own FHIR endpoint, spreadsheet, and credential set,no mixing, ever.
Give operators live feedback with status updates, readable logs, and a safe way to restart when things went wrong.

At that point the one-off script had nowhere to grow. We needed a real architecture that could respect those boundaries.

The Challenge: Scale, Isolation, and Real-World Limits

Running two regions side by side exposed the real limits:

Tens of thousands of records every day.
Updates that had to finish in under 15 minutes.
Tight resource limits (very small memory and CPU).
Zero tolerance for mixing environments.

We also needed more resilience: a single broken record could not freeze the run, and the FHIR servers deserved a gentle pace so they never tipped into overload.

Why the Simple Loop Fails

# Inefficient sequential approach
for record in all_records:
    patient = find_patient(record['identifier'])
    update_patient_phone(patient, record['phone_number'])

This loop looked fine in early tests. In production it fell apart,every row opened new network calls, memory crept upward, progress stayed invisible, and one exception could stop the whole job. That simple design hurt us later on.

The Solution: Smart Batching Across Multiple Environments

The turning point came when we stopped thinking about “a script that updates phones” and started thinking about “a pipeline that needs to stay healthy.” Stability, visibility, and consistency became the main goals.

Once we named those needs, the design almost wrote itself: batch the work, reuse connections, pace the requests, and send every result back into the spreadsheets everyone already trusted. On top of that, make sure each batch leaves a clear log trail so operators can watch the system move.

Architecture Overview

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│  Google Sheets   │    │   Flask Web      │    │    FHIR Server  │
│   (per region)   │───▶│   Application    │───▶│    (per region) │
│                  │    │                  │    │                 │
│ • Regional rows  │    │ • Batch engine   │    │ • Patient query │
│ • Status column  │    │ • Memory hygiene │    │ • Phone update  │
│ • Daily feeds    │    │ • Safe retries   │    │ • Rate limiting │
└─────────────────┘    └──────────────────┘    └─────────────────┘
         │                       │
         └───────────────┬───────┘
                         │
                ┌────────────────────┐
                │  Config & Secrets  │
                │  (per environment) │
                │  JSON + env vars   │
                └────────────────────┘

In practice, each region runs end to end on its own. Operators manage rows in a regional Google Sheet, the Flask app reads that sheet and processes batches, and every update goes to the matching FHIR server. The config and secrets layer supplies the right credentials and URLs per run, so requests stay isolated and nothing leaks across environments.

The Batch Engine (Built for Production)

def process_records():
    start_time = time.time()
    results = {
        'status': 'success',
        'total_records': len(records),
        'successful_updates': 0,
        'failed_updates': 0,
        'patients_not_found': 0,
        'processing_time_minutes': 0,
        'batches_processed': 0,
        'errors': []
    }

    try:
        session = setup_session()
        batch_size = 100
        logger.info(f"Starting batch processing with batch size: {batch_size}")

        for i in range(0, len(records), batch_size):
            batch = records[i:i+batch_size]
            batch_num = i // batch_size + 1
            total_batches = (len(records) - 1) // batch_size + 1
            logger.info(f"=== Processing batch {batch_num}/{total_batches} ({len(batch)} records) ===")

            for idx, record in enumerate(batch):
                try:
                    row_index   = record['row_index']
                    identifier  = record['identifier']
                    phone       = record['phone_number']
                    record_num  = i + idx + 1

                    if record_num % 500 == 0:
                        logger.info(f"=== PROGRESS: {record_num}/{len(records)} processed ===")

                    patients = find_patient(session, identifier, fhir_url, headers)
                    if not patients:
                        results['patients_not_found'] += 1
                        msg = f"Patient {identifier} not found"
                        results['errors'].append(msg)
                        update_worksheet_status(worksheet, row_index, "failed", msg)
                        continue

                    all_success = True
                    failed_msgs = []
                    success_count = 0

                    for patient in patients:
                        ok, message = update_patient_phone(
                            session, patient['id'], phone, fhir_url, headers
                        )
                        if ok:
                            success_count += 1
                        else:
                            all_success = False
                            failed_msgs.append(f"Patient {patient['id']}: {message}")

                    if all_success:
                        results['successful_updates'] += success_count
                        update_worksheet_status(worksheet, row_index, "success")
                    else:
                        results['failed_updates'] += (len(patients) - success_count)
                        msg = f"Failed {len(failed_msgs)}/{len(patients)}"
                        update_worksheet_status(worksheet, row_index, "failed", msg)

                    time.sleep(0.1)  # be kind to downstreams

                except Exception as e:
                    results['failed_updates'] += 1
                    msg = f"Error processing {record.get('identifier', 'unknown')}: {e}"
                    results['errors'].append(msg)
                    logger.exception(f"❌ {msg}")
                    continue

            # Keep memory flat
            del batch
            time.sleep(0.1)
            results['batches_processed'] += 1

        results['processing_time_minutes'] = round((time.time() - start_time) / 60, 2)
        logger.info(f"Processing completed in {results['processing_time_minutes']} minutes")

    except Exception as e:
        results['status'] = 'error'
        results['message'] = f'Processing failed: {e}'
        logger.exception("Fatal error")

    return results

The batch engine became the system’s heart. Inside the loop you can see how it slices the sheet into blocks of 100 rows, logs the batch number, and keeps track of every success or failure. That small pause after each batch and record keeps memory flat and slows the request rate so the FHIR servers never get hammered. Even when a row misbehaves, the error handler writes it down and the loop keeps going, which means operators still see steady progress instead of a half-finished run.

Practices That Made the System Work

1) Batch Size That Fits Reality

When we tried tiny batches the system spent more time setting up than doing real work. When we went too big, the process grabbed extra memory and slowed everything down. After a few trial runs, 100 records felt balanced,quick to process, light on resources, and easy to monitor in the logs and in the sheet.

2) Connection Pooling and Safe Retries

def setup_session():
    session = requests.Session()
    retry = Retry(
        total=3,
        backoff_factor=1,
        status_forcelist=[429, 500, 502, 503, 504]
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount("http://", adapter)
    session.mount("https://", adapter)
    return session

Pooling the HTTP session kept us from opening a fresh connection for every row, which trimmed latency and CPU spikes. The retry helper then waited a little longer after each failure, so short network hiccups cleared on their own instead of breaking the run. With those two pieces in place, the pipeline finished sooner and recovered smoothly from the usual internet noise.

3) Explicit Memory Hygiene

# keep memory flat
del batch
time.sleep(0.1)

By explicitly deleting each batch and pausing briefly, memory remained flat. No creeping leaks, no surprises during long runs.

4) Pacing to Protect Servers

time.sleep(0.1)  # ~10 ops/sec

A short delay between requests prevented FHIR servers from being overwhelmed. Paradoxically, slowing down slightly made the whole system finish faster, because retries and throttling were reduced.

Features That Build Trust

Real-Time Status in Sheets

def update_worksheet_status(worksheet, row_index, status, error_message=None):
    """
    Update the status_update_phone_number column (Column G)
    """
    try:
        status_column = 7  # Column G (1-indexed)
        value = "success" if status == "success" else (
            f"failed: {error_message}" if error_message else "failed"
        )
        worksheet.update_cell(row_index, status_column, value)
        logger.info(f"Row {row_index} (G): {value}")
    except Exception as e:
        logger.error(f"Failed to update status for row {row_index}: {e}")

Operators never asked for a new dashboard; they just wanted their sheet to tell them what happened. This helper does exactly that, dropping “success” or “failed” (with the reason) straight into Column G so they can watch the run move row by row.

Structured Batch Logs and Alerts

Every batch writes a compact log entry with the batch number, record count, and any failures. Those logs land in Cloud Logging and a small alerting rule pings the on-call channel when something looks off. If a row fails, the operator spots it in the sheet and can jump straight to the matching log line because the correlation ID is right there in the message.

Handle Multiple Patients per Identifier

def find_patient(session, identifier, fhir_url, headers):
    url = f"{fhir_url}/Patient"
    params = {'identifier': identifier}
    r = session.get(url, params=params, headers=headers, timeout=30)
    if r.status_code == 200:
        data = r.json()
        if data.get('total', 0) > 0:
            return [e['resource'] for e in data['entry']]
    return []

Some identifiers pointed to more than one Patient record. Rather than pretend the duplicates did not exist, the system updates each match so that all copies stay aligned,even when the source data is messy.

Phone Number Cleaning

def clean_phone_number(phone):
    if not phone:
        return None

    phone = str(phone).strip()

    # Sheets can turn big numbers into scientific notation
    if 'e+' in phone.lower() or 'e-' in phone.lower():
        try:
            phone = f"{float(phone):.0f}"
        except ValueError:
            logger.warning(f"Could not parse scientific notation: '{phone}'")

    # If multiple numbers, take the first
    if ',' in phone:
        phone = phone.split(',')[0].strip()

    # Keep only digits and plus
    phone = re.sub(r'[^\d+]', '', phone)

    # Indonesian heuristic: restore missing leading zero
    if not phone.startswith('+') and phone.startswith('8') and len(phone) in [11, 12]:
        phone = '0' + phone
        logger.info(f"Added missing leading zero: '{phone}'")

    # Basic sanity check
    if len(phone.replace('+', '')) < 8:
        return None

    return phone

Phone numbers are notoriously messy. Sheets loves to turn big numbers into scientific notation, people paste in two numbers separated by commas, and in Indonesia a missing leading zero can point to the wrong person. The cleaner walks through each of those cases so the final value is something we can safely send to FHIR.

Multi-Environment Deployment: One Region = One Tenant

JSON Config as Source of Truth

{
  "name": "environment-name",
  "project_id": "your-gcp-project-id",
  "service_name": "your-cloud-run-service-name",
  "region": "asia-southeast2",
  "platform": "managed",
  "memory": "256Mi",
  "cpu": "0.5",
  "timeout": "900",
  "max_instances": "1",
  "min_instances": "0",
  "concurrency": "1",
  "credential_file": "service-account-credentials.json",
  "env_vars": {
    "FHIR_SERVER_URL": "https://your-fhir-server.com/fhir",
    "FHIR_API_TOKEN": "your-api-token",
    "SPREADSHEET_ID": "your-google-sheets-id",
    "WORKSHEET_ID": "your-worksheet-name"
  }
}

Each environment,Purbalingga, Lombok Barat, and friends,gets its own JSON file. The application code stays the same, while the config file names the project, credentials, and spreadsheet for that region. That simple split keeps the runs isolated, makes audits easy, and lets us roll back a region without touching the others.

Cloud Run Profile Per Environment

Each region deploys to its own Cloud Run service with a lean profile: 256Mi memory, 0.5 vCPU, and a single instance. It keeps costs low, keeps performance predictable, and matches the steady pace we designed for.

Lessons Learned

When we stepped back after the first few successful runs, a handful of habits stood out.

Regional boundaries matter. Keeping configs and credentials per region meant every incident stayed where it started. If Lombok Barat hit a problem, Purbalingga kept running without even noticing.

100-record batches are the sweet spot. That size is big enough to move quickly but small enough to avoid memory spikes. It also lines up nicely with the logging we added, so operators can read progress in plain language.

The spreadsheet is still the source of truth. By writing results directly into Column G, we gave operators instant trust. They did not need to learn a new tool; their everyday sheet became the dashboard.

Polite clients make for calm servers. Gentle pacing and retries with backoff handled the usual internet noise. Instead of chasing flaky errors, we saw quiet logs and smooth throughput.

Clean data upfront saves pain later. Fixing phone numbers at the edge kept downstream systems clean. Once we did that, support tickets about wrong contacts dropped sharply.

Conclusion

The journey took us from a fragile script to a workflow the team can trust. We didn’t introduce exotic technology; we simply leaned on good habits,clear boundaries, careful batching, shared connections, steady pacing, and honest visibility.

Today the 10,000-row jobs finish in about 12 minutes per environment. Memory stays flat. Operators watch the spreadsheet fill with results while alerts stay quiet.

For us, that’s what healthcare data scaling looks like: not only faster runs, but calmer shifts, clearer feedback, and an architecture that can keep growing with the organization.