DEV Community

Vaibhav
Vaibhav

Posted on

The 2026 Insurtech Data Stack: A Reference Architecture

#insurtech #dataengineering #architecture #ai

Every insurance AI conversation eventually hits the same wall: the model is fine, but the data underneath it isn't reachable, clean, or fast enough. This post is the architecture I'd draw on a whiteboard for a carrier that wants underwriting, claims, and fraud AI to actually work — and the reasoning behind each layer.

Here's the shape:

① SOURCES            Policy Admin (mainframe/Oracle) · Claims · Billing · CRM · Docs
                     External: telematics · weather · IoT · credit · satellite
                          │
② INGESTION          Batch/CDC (dbt · Airbyte)      Streaming (Kafka)
                          │                              │
③ LAKEHOUSE          ───────────► Snowflake / Databricks ◄───────────
                     dbt transforms · dedup · stable customer key
                          │
④ GOVERNANCE         Lineage · Decision/audit log · PII tagging · Access control
                          │
⑤ AI / AGENT LAYER   Underwriting AI · Claims agent (HIL) · Fraud/graph detection
Enter fullscreen mode Exit fullscreen mode

Five layers. Each exists because something specific breaks without it.

① Sources: accept that the legacy core isn't going away tomorrow

A typical carrier's policy administration system runs on a mainframe or an Oracle/Teradata stack older than most of the team. The claims system was bought, not built, and speaks its own schema. Billing, CRM, and the document store were each integrated by a different vendor in a different decade.

The mistake is treating "rip out the legacy core" as step one. It isn't — it's a multi-year program, and every AI initiative behind it starves in the meantime. Instead, read from it (CDC where possible) while you plan a phased migration with a parallel run. Ship value while the core migration proceeds underneath.

② Ingestion: two paths, deliberately

This is where teams most often over-simplify. You need both:

  • Batch / CDC for policy, claims, billing history — the bulk of your data, where a nightly or hourly cadence is genuinely fine.
  • Streaming for the signals whose value decays in minutes: FNOL events, telematics, fraud signals, quote interactions.

If you build only batch, your "real-time" fraud detection is a day late by design. If you stream everything, you've built an expensive, brittle pipeline for data nobody needs in real time. Split by decay rate of the signal, not by fashion.

③ Lakehouse: one place the truth lives

Snowflake or Databricks — pick by workload, not logo. SQL-heavy analytics and reporting with minimal ops leans Snowflake; heavy ML, streaming, and one home for data+AI leans Databricks. Either way the job is the same: consolidate the silos, then transform.

The single highest-leverage artifact in this whole diagram is a stable customer key. Without it, the same policyholder exists as four unlinked records and every model is blind exactly where it matters. "Customer 360" should be a table you can query — not a slide.

④ Governance: not a document, a layer

This is the one people bolt on last and regret. With the NAIC Model Bulletin adopted across 23 U.S. jurisdictions and the EU AI Act treating most insurance AI as high-risk, you have to prove how a decision was made:

  • Lineage — source → transformation → feature → decision. Instrumented at build time or it doesn't exist.
  • Decision log — append-only: inputs, model version, features, output, timestamp.
  • PII tagging + access control — HIPAA/GDPR don't relax because AI showed up.

You cannot document lineage you didn't capture. Build this into the pipeline, not into a PDF.

⑤ AI / agent layer: the easy part, finally

Only now does the AI get interesting — and notice it's last. Underwriting scoring, an agentic claims adjuster with human-in-the-loop gates, graph-based fraud detection. Each is a consumer of layers ①–④.

An agent is just a very demanding data consumer. It calls tools; those tools query this stack. If the stack isn't there, the agent doesn't fail loudly — it returns confidently wrong answers, which is worse.

The sequencing that actually ships

  1. Stand up the lakehouse and get one domain in (start with policy).
  2. Build the customer key. Everything downstream compounds off it.
  3. Add streaming only for the signals that need it.
  4. Instrument lineage + the decision log before your first regulated model.
  5. Then build the AI. It'll take a fraction of the time everyone feared.

Most carriers do this in exactly the reverse order — start with a model demo, then spend two years discovering they needed layers ①–④ first. The architecture isn't complicated. The discipline to build it bottom-up is the hard part.

We do these migrations and build the data foundations insurance AI runs on — 200+ cloud migrations across Snowflake, Databricks, and the modern lakehouse. More at IntelliBooks.

Top comments (0)