DEV Community

Vaibhav
Vaibhav

Posted on

Data Lineage for Insurance ML: You Capture It While Data Flows, or Not at All

#dataengineering #mlops #insurtech #compliance

Most teams treat data lineage as a governance checkbox — something the data-catalog tool draws a diagram for after the fact. Then a regulator (or, increasingly, the EU AI Act or a NAIC-style model bulletin) asks: "How did this specific premium get calculated? Which records, which transformations, which version of which logic, at what time?" — and you discover the diagram is decorative.

Here's the uncomfortable engineering truth: lineage is captured as data flows, or it doesn't exist. You cannot reconstruct, months later, exactly what produced one number in a pipeline that wasn't built to remember. This isn't a tooling gap you can buy your way out of after the fact.

What lineage actually has to record

For a single output — say, a declined application or a rated premium — you need the chain of custody:

  • Which source records fed it (row-level, not "the customer table")
  • Which transformations touched it, in order
  • Which version of which logic ran (the rating rules, the model, the feature code — pinned, not "current")
  • At what time, against what state of the world

If any link is missing, "explain this decision" becomes "trust me."

Why you can't retrofit it

Consider a feature avg_claim_amount_12m. Six months later someone asks why a policy was priced the way it was. To answer, you'd need: which claims were in the 12-month window at scoring time (not now), the exact aggregation logic as it was then (not after the refactor), and the model version that actually ran. None of that is in your current warehouse snapshot. The window moved, the code changed, the model was retrained. The information didn't degrade — it was never recorded.

What "build it in" looks like in practice

You don't need a heavyweight platform on day one. You need discipline at three points:

  1. Pin everything that runs. Model version, feature-code commit, and rule-set version travel with every prediction as metadata — not in a wiki.
  2. Emit lineage events at transform time. Each pipeline step writes {input_ids, output_id, transform_version, timestamp}. Append-only. This is cheap when the data is already in motion and impossible later.
  3. Snapshot the inputs to consequential decisions. For anything a regulator could ask about — pricing, declines, cancellations — store the feature vector as scored, immutably. 
# at scoring time, not as an afterthought
emit_lineage(
    output_id=decision_id,
    input_ids=source_record_ids,      # row-level provenance
    feature_snapshot=feature_vector,  # as-scored, immutable
    logic_version=RULESET_SHA,        # pinned
    model_version=MODEL_URI,          # pinned
    ts=scored_at,
)
Enter fullscreen mode Exit fullscreen mode

The payoff

The teams that build lineage into the pipeline from day one answer the regulator's question in an afternoon. The teams that bolted on a catalog spend three weeks reconstructing something they'll never be fully sure of — and hope nobody asks about the edge cases.

Full write-up, including the regulatory context (NAIC bulletin adoption + EU AI Act) and why an audit is a photograph of something moving:
Data Lineage, Explained: What Regulators Ask For That You Can't Fake →

This is part of a longer series on the data foundation under insurance AI from IntelliBooks.

How is your team capturing lineage — event-at-transform-time, or reconstruct-later-and-pray?

Top comments (0)