Most data engineers building for healthcare are dead wrong: they think that if they enable AES-256 at the storage layer, they’ve satisfied HIPAA. They haven't. They’ve just satisfied a checkbox on a low-effort audit. If you aren't governing data access at the column level, you aren't protecting PHI; you’re just waiting for a breach notification letter to hit your desk.
You are currently standing at a crossroads. You either build a lakehouse where access is controlled via coarse-grained IAM policies, or you build one where security is baked into the storage format itself.
The contenders
On one side, we have the "Infrastructure-as-Perimeter" approach. This is the classic cloud-native pattern: S3 buckets or Azure ADLS containers guarded by IAM roles. You grant a service account access to a bucket, and they get everything in it. PHI handling is done via "data zoning"—moving sensitive data into a separate, highly restricted bucket.
On the other side, we have the "Table-Format-as-Perimeter" approach. This uses modern open-table formats like Apache Iceberg or Delta Lake, paired with a metadata-driven access control layer like Unity Catalog or Starburst/Trino. Here, access is defined by SQL policies (GRANT SELECT ON TABLE...) rather than filesystem paths.
The cost of operational complexity
The "data zoning" approach—the bucket-per-sensitivity-level model—is a nightmare that scales linearly with your paranoia. I’ve managed pipelines where we had to maintain three separate copies of the same patient demographics data just to keep the "Research" team out of the "Clinical" bucket.
You end up paying for storage three times, and your egress costs go up because your Spark jobs are constantly shuffling data across bucket boundaries. More importantly, it kills your agility. Every time a data scientist needs one more column for a longitudinal study, you’re stuck updating Terraform scripts, running a terraform plan that takes six minutes, and praying you don't break a production IAM policy.
The table-format approach is more expensive in terms of licensing or specialized compute, but cheaper in terms of "human-in-the-loop." When I use Unity Catalog or similar frameworks, I can tag a column as PHI and apply a masking policy. If someone queries the column, they get ***-**-1234 instead of a real SSN. The cost is a slight overhead on every query, but I don't pay for triple-redundant storage, and I don't pay for the senior engineer’s time spent babysitting bucket policies.
Failure modes and audit footprints
When your security is tied to IAM roles, your failure mode is "over-permissioning." It’s the path of least resistance. A developer needs access to a table for a quick bug fix? You grant them the IAM role that covers the whole bucket. Six months later, that developer has left the company, but their role still exists, and they still have access to 400 gigabytes of PHI because nobody remembered to prune the permissions.
In the table-format approach, the failure mode is "metadata drift." If your catalog isn't perfectly synced with your underlying Iceberg manifests, you might experience a "ghost access" issue where a user can see a file but not the table metadata, or vice-versa.
However, the audit footprint is where the table format wins. In the IAM model, an auditor asks, "Who accessed this PHI?" and you have to parse through CloudTrail logs. Have you ever tried to map a GetBucket event to a specific patient record? It’s impossible. You get an IP address and an IAM ARN. In the table-format model, the audit log gives you the identity, the exact SQL statement executed, and the specific column mask applied. If I’m an auditor, I know exactly what you saw.
Photo by Anna Evans on Unsplash
The reality of data leakage
The biggest lie in healthcare data engineering is that you can "anonymize" data by stripping identifiers. You can’t. You have to treat everything as PHI until proven otherwise.
In the bucket-based approach, you eventually reach the "Oops, I accidentally moved raw PHI to the dev environment" stage. It happens. Someone runs a cp command from the wrong terminal, and suddenly you have production patient data in a non-compliant sandbox. Because the security is at the bucket level, that sandbox is now a massive compliance liability.
With granular column-level security, even if a dev accidentally pulls raw data into a downstream table, the access policies follow the metadata tags. If the downstream table is tagged as PHI, the masking policy still triggers. You don't have to rely on the discipline of the developer; you rely on the policy engine.
What I'd pick, and why
If you are building a lakehouse today, stop building "data zones." Move to an Iceberg-based architecture with a centralized metadata layer.
I’d pick Apache Iceberg on S3, governed by a solution like Unity Catalog (if you’re on Databricks) or Starburst/Trino (if you’re running a mix of EMR/Glue). The reason is simple: it allows you to decouple storage from policy.
The caveat—and this is a big one—is that you must treat your access control code as production-grade application code. Do not just use the UI to click "Grant Access." You need to treat your GRANT and MASK SQL statements as code, checked into Git, and reviewed by a security engineer.
If you aren't doing code reviews on your access policies, you are just replacing bucket-level IAM mess with SQL-level IAM mess. The risk profile shifts from "accidental exposure" to "logic error in a policy script." But at least with a policy script, you can write unit tests. You can’t write a unit test for an IAM bucket policy that checks if the policy prevents unauthorized access to a specific column.
Finally, keep your pii_masking_udf simple. Don't try to build a complex cryptographic hashing function inside your SQL layer unless you have the crypto-engineering expertise to manage the keys. Keep the logic predictable. A simple regex-based mask for a Social Security number is better than a complex, bespoke hashing algorithm that nobody understands and that breaks when you upgrade your Spark runtime from 3.3 to 3.5.
Your goal is not to be clever. Your goal is to pass the audit without losing your mind. Move the security to the data, not the server.
Cover photo by Lee Lawson on Unsplash.
Top comments (0)