Pallavi for AptaCloud

Posted on Nov 2

The insurance edge case that breaks event-driven policy pipelines

#azure #architecture #python #eventdriven

If you're moving insurance policy data from a nightly batch to an event stream, the broker is the easy part. The hard part is that insurance transactions don't arrive in the order they take effect. An endorsement processed in May can be effective in January, and a consumer that simply applies events as they land will quietly compute the wrong policy state.

The fix is structural. Partition events by policy number, store every transaction instead of overwriting state, and rebuild the policy view in effective-date order each time something changes. Below is one way to build that on Azure, along with the failure modes worth planning for before anyone writes code.

Why the nightly batch hid this problem

Batch jobs recompute everything overnight from the system of record. If a backdated change came in at 4 p.m., the 2 a.m. run picked it up and re-rated the term from scratch. Nobody downstream ever saw the in-between state.

Streaming removes that safety net. Your consumer now sees each transaction once, in processing order, and has to get the answer right without a full recompute. Teams often find this out a few weeks into a pilot, usually when a premium report stops matching the policy administration system by a few hundred dollars and nobody can say why.

What an out-of-sequence endorsement looks like

Take a commercial auto policy with a term running January 1, 2026 to January 1, 2027.

Processed on	Transaction	Effective date
January 1	New business, 4 vehicles	January 1
February 10	Add a fifth vehicle	February 15
April 2	Garaging address change, backdated	January 20

The April transaction lands last but belongs second. Rating for January 20 through February 15 changes, and so does every period after it, because the fifth vehicle now has to be rated at the new garaging address too. Underwriting and billing teams call these out-of-sequence endorsements, and any policy book with real volume produces them every week.

A consumer that does state.update(event) gets this wrong silently. No exception, no alert, just a policy view that disagrees with the core system.

Start with an event contract that carries both dates

Every transaction event needs two clocks: when it takes effect and when it was processed. Add a per-policy sequence number so ties on effective date still have a deterministic order.

{
  "transaction_id": "3f9b2c71-8e4d-4a6b-9c1e-5d7f0a2b8e44",
  "policy_number": "CA-1048832",
  "term_id": "2026-2027",
  "transaction_type": "ENDORSEMENT",
  "effective_date": "2026-01-20",
  "processed_at": "2026-04-02T16:22:09Z",
  "sequence": 3,
  "changes": [
    { "path": "garaging_address.zip", "old": "75201", "new": "75080" }
  ],
  "schema_version": "1.2"
}

Two habits help later. Keep field names close to the ACORD data model where you can, because your distribution partners and downstream vendors probably already speak it. And register the schema (Azure Schema Registry in Event Hubs works with Avro) so version 1.3 doesn't break a consumer someone forgot about.

Partition by policy number

Azure Event Hubs guarantees ordering within a partition, not across partitions. Use the policy number as the partition key and every transaction for one policy goes to the same partition, in the order the producer sent it.

That gives you processing order per policy. It does not give you effective-date order, which is the whole point of the next section. Partitioning keeps a policy's events together; it can't fix a backdated change.

Store transactions, rebuild the view

Instead of mutating a stored policy record, append each transaction to a per-policy history and rebuild the current view from that history. A policy term rarely has more than a few dozen transactions, so the rebuild is cheap.

def build_policy_view(transactions):
    """Rebuild a policy term's current view from its full transaction history."""
    active = [t for t in transactions if not t.get("reversed_by")]
    ordered = sorted(active, key=lambda t: (t["effective_date"], t["sequence"]))

    view = {}
    for tx in ordered:
        for change in tx["changes"]:
            set_path(view, change["path"], change["new"])
    return view

Here is the consumer as a Python Azure Function on an Event Hubs trigger. The policy_store module stands in for whatever storage you use. Azure Cosmos DB fits well here, with the policy number as the partition key on that side too.

import json
import logging

import azure.functions as func

from policy_store import (
    already_processed,
    append_transaction,
    load_transactions,
    park_failed_event,
    save_policy_view,
)

app = func.FunctionApp()


@app.event_hub_message_trigger(
    arg_name="event",
    event_hub_name="policy-transactions",
    connection="EVENTHUB_CONNECTION",
)
def apply_policy_transaction(event: func.EventHubEvent):
    raw = event.get_body().decode("utf-8")
    try:
        tx = json.loads(raw)
        if already_processed(tx["transaction_id"]):
            return  # redelivered after a restart, safe to skip

        append_transaction(tx)
        history = load_transactions(tx["policy_number"], tx["term_id"])
        save_policy_view(tx["policy_number"], build_policy_view(history))
    except Exception:
        logging.exception("Could not apply policy transaction")
        park_failed_event(raw)  # somewhere a person actually looks

One caveat on scope. This rebuilds the policy's coverage and attribute view. Premium is a different matter: when a backdated change lands, the affected periods need re-rating, and that call belongs to your rating engine, not the event consumer. Emit a "re-rate from effective date X" event and let the rating service own it.

Event Hubs won't dead-letter for you

This one catches people coming from Azure Service Bus. Event Hubs has no built-in dead-letter queue, and with the Functions trigger an unhandled exception doesn't stop the checkpoint from moving forward. A bad event can simply disappear from your pipeline.

So catch failures yourself and park them, as the consumer above does. A Service Bus queue or a storage container works fine. What matters is that someone owns it. A dead-letter queue nobody reviews is just a slower way to lose data.

Run it beside the batch before you trust it

Don't switch off the nightly batch on day one. Keep it running and compare, policy by policy, the event-built view against the batch extract each morning. Write every difference to a table with the policy number, the field, both values, and the latest transaction on that policy.

Watch the trend, not the total. The first week of diffs is usually field-mapping bugs in the new pipeline. After that, what's left tends to be the interesting stuff: undocumented rules in the legacy system that fire on certain endorsements and that nobody wrote down. Better to find those in a diff table than in a regulator's data call.

Set a cutover rule in advance (for example, zero unexplained diffs for a full month-end cycle) so the parallel run has an end date.

When this pattern isn't worth it

If your downstream consumers are happy with next-morning data, the batch may be fine as it is. Streaming earns its cost when brokers need bindability in minutes, when claims needs current coverage at first notice of loss, or when you're feeding AI underwriting tools that can't wait for 2 a.m.

Getting changes out of an older core system is also its own project. On a mainframe that usually means log-based change data capture on the underlying database, which brings licensing and operations work that belongs in the estimate from the start. And events tell you what changed, not why a rule fired. If the business logic inside the old system is undocumented, streaming won't document it for you. The parallel run will just make the gaps visible sooner.

DEV Community