Eighty-two percent of financial institutions still rely on manual Excel spreadsheets to map data lineage for BCBS 239 reporting. If that number sounds like a death sentence for your next internal audit, that’s because it is.
When the Basel Committee on Banking Supervision (BCBS) published their 239 principles, they didn't ask for a nice diagram in a Visio file. They asked for an "accurate, complete, and timely" understanding of data flow from the source system to the Risk Weighted Assets (RWA) calculation. If you can’t prove the provenance of a single column in your regulatory report, you’re looking at capital add-ons or, worse, a formal "finding" from the regulators.
We are long past the point where static documentation suffices. Your data platform is moving too fast for manual updates. You have two real choices to solve this: bet the farm on Databricks’ proprietary Unity Catalog (UC) or build an agnostic, open-source pipeline using OpenLineage.
The contenders
On one side, you have Unity Catalog. It’s the "it just works" button for the Databricks ecosystem. It captures lineage at the compute layer automatically. If your data is in Delta tables and you’re using Spark or SQL warehouses, the lineage is generated for you with zero code changes.
On the other side, you have OpenLineage. This is the CNCF standard for data lineage. It’s an open-source framework that hooks into Airflow, Spark, dbt, and Great Expectations to emit events to a backend like Marquez or DataHub. It’s the "do it yourself, but do it right" approach.
Photo by Mick Haupt on Unsplash
The operational tax
Unity Catalog is effectively a managed service. You flip a switch in the metastore settings, and suddenly, the sys.access.lineage tables are populated. The operational burden is near zero. If you are already living in the Databricks ecosystem, the marginal cost of "configuring" UC is essentially just assigning permissions.
OpenLineage is a different beast. You are managing the integration points. You need to configure the openlineage-spark jar in your cluster configurations. You need to handle the lifecycle of the OpenLineage backend (like Marquez). If the backend goes down, your jobs don't necessarily fail, but your audit trail goes dark. In a production financial environment, a gap in lineage data is often treated as a compliance breach, so you end up building high-availability infrastructure for your lineage store. That’s a FTE worth of work.
Failure modes and observability
I’ve seen Unity Catalog lineage fail when users start doing "clever" things with dynamic SQL or Python exec() calls within Spark. When the parser can't map the lineage, the graph just breaks. It’s a black box; you can’t "fix" the parser. You just stare at a gap in the DAG and hope the auditors don't ask about that specific transformation.
OpenLineage, however, is transparent. If the metadata emission fails, you see it in the logs. Because it’s an event-based system, you can implement custom retries or sidecars. The failure mode is noisy, which is exactly what you want when you’re dealing with BCBS 239. I’d rather have a noisy alert that lineage is missing than a silent, empty graph that lets a non-compliant transformation sneak into a regulatory report.
Photo by Pietro Jeng on Unsplash
The cost of vendor lock-in
This is the real elephant in the room for financial services. BCBS 239 requires resilience. If you build your entire compliance strategy on Unity Catalog, you are tethered to Databricks for the next decade. If they decide to hike prices or change their API, your compliance posture changes with them.
OpenLineage is portable. If you decide to move your processing from Databricks to an EMR cluster or a Kubernetes-native environment, your lineage strategy stays the same. You just update your Spark configs. In the eyes of an auditor, an open standard is often viewed as a more "robust" control than a proprietary feature that could be deprecated in the next product release cycle.
What I'd pick, and why
If you are a mid-sized team with a heavy Databricks footprint and you need to check the box for auditors yesterday, use Unity Catalog. It is the path of least resistance. It provides enough visibility for 90% of the standard risk reporting requirements. Just be aware that you are trading flexibility for speed. Keep a side-car documentation process for the "weird" stuff—the stored procedures or the legacy mainframe feeds that UC can't touch.
However, if you are working at a Tier-1 bank or a systemically important financial institution where the regulatory scrutiny is intense, go with OpenLineage.
My recommendation? Use OpenLineage as your primary source of truth. Use a tool like Marquez or DataHub to visualize it. Why? Because when the regulator asks "How do you know this data is correct?", you want to be able to point to an open, standard-compliant schema that you own, not a proprietary proprietary UI that lives behind a vendor login.
Here is the honest caveat: OpenLineage requires discipline. You have to ensure that every job—from the ingestion job in Python to the final aggregation in dbt—is emitting the correct events. If your engineers forget to include the openlineage-python library in a new service, your lineage is incomplete.
If you choose OpenLineage, build a "compliance gate" in your CI/CD pipeline. Use pytest to inspect the metadata emission of your jobs before they hit production. If a job doesn't emit the required lineage metadata, the build fails. It sounds draconian, but it’s the only way to ensure your BCBS 239 compliance doesn't drift when someone pushes a hotfix at 2:00 AM on a Friday.
The technology isn't the hard part here. The hard part is accepting that your lineage is only as good as the least-maintained pipeline in your stack. Whether you choose the ease of Unity Catalog or the flexibility of OpenLineage, the audit-readiness comes from your internal controls, not the tool itself. Pick your trade-off, document the failure modes, and for heaven's sake, get rid of the spreadsheets.
Top comments (0)