The latest articles on DEV Community by Murray Evans (@murray_evans_d9de79e4044c). https://dev.to/murray_evans_d9de79e4044c https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4194225%2F5a2daf94-2d98-4438-bb9f-350b5b4471bf.png https://dev.to/murray_evans_d9de79e4044c en Murray Evans Mon, 19 Oct 2026 13:47:39 +0000 https://dev.to/murray_evans_d9de79e4044c/lakehouse-architecture-for-regulated-industries-notes-from-banking-and-energy-494 https://dev.to/murray_evans_d9de79e4044c/lakehouse-architecture-for-regulated-industries-notes-from-banking-and-energy-494 <p>I've spent years building data systems in two of the most regulated industries on earth: banking and oil &amp; gas. If a data architecture survives both a bank auditor and an energy regulator, it can survive anything. Here's what the lakehouse pattern looks like when compliance isn't optional.</p> <h2> Why the lakehouse fits regulated industries </h2> <p>The classic split — data lake for raw storage, warehouse for analytics — creates a governance seam. Data gets copied, transformed, and copied again, and every copy is a place where lineage breaks and access controls drift. In a regulated environment, that's not a technical inconvenience; it's an audit finding.</p> <p>The lakehouse (open table formats like Delta Lake, Iceberg, or Hudi over cheap object storage, with a proper governance layer) collapses that seam: <strong>one copy of the data, one set of access controls, one lineage story.</strong> When the auditor asks "who touched this field and when," there's exactly one place to look.</p> <h2> Four non-negotiables </h2> <p><strong>1. Governance before scale.</strong> Unity Catalog, Apache Ranger, or equivalent — not as phase two, as day one. Table-level and column-level access controls, audit logging of every read. In banking, "we'll add governance later" is how you end up explaining yourself to a regulator.</p> <p><strong>2. The medallion pattern, enforced.</strong> Bronze (raw, immutable) → Silver (cleaned, validated) → Gold (business-ready). The key word is <em>enforced</em>: pipelines that write directly to Gold get rejected in code review. I've seen too many "temporary" shortcuts become permanent data quality incidents.</p> <p><strong>3. Time travel isn't a luxury.</strong> Open table formats give you snapshot isolation and time travel for free. When finance asks "what did this report show last Tuesday," you check out the table version — you don't restore a backup. For reconciliation-heavy industries, this alone justifies the architecture.</p> <p><strong>4. Data contracts at ingestion.</strong> Every upstream system declares its schema, SLAs, and semantics in a contract; breaking changes fail the pipeline instead of silently corrupting downstream. In energy, where SCADA feeds and trading systems converge, this is the difference between a reliable platform and a rumor mill.</p> <h2> The mistake I see most </h2> <p>Teams buy the lakehouse <em>technology</em> (Databricks, Snowflake with Iceberg, open-source Trino stacks) and skip the lakehouse <em>discipline</em>. The tech without governance, medallion enforcement, and contracts is just an expensive data lake with a new name. Architecture is 30% tooling, 70% the rules everyone agrees to follow.</p> <p><em>Aziz is a data architect working across banking and energy. He runs <a href="https://aidatacenter.ai" rel="noopener noreferrer">AI Data Center</a> in Calgary — data engineering, analytics, and AI systems designed as one intelligent ecosystem.</em></p> dataengineering lakehouse architecture ai