I tagged PII only in bronze and Unity Catalog masked it all the way to gold

Vinícius de Carvalho — Thu, 12 Nov 2026 13:49:35 +0000

Most breaches don't start where the protection is. An analyst copies customer data out of the warehouse into a shared folder or a spreadsheet for a quick analysis, and the copy outlives the reason it was made. Anyone who can open that folder now has the data, whether or not they ever had warehouse access. IBM found shadow data, copies nobody tracks, in more than a third of breaches in 2024. The same gap exists inside the warehouse. Teams mask or tokenize the columns they flagged as sensitive, and the masks work, but each mask is attached to a column in one table. A table built from that column is a new table, and it starts with no protection. Protection stays where it was applied; it doesn't travel with the data.

Health data makes the gap hardest to ignore. It is a special category under both the GDPR and Brazil's LGPD, and it is also the most expensive data to lose. IBM's Cost of a Data Breach Report 2026, which covers breaches from March 2025 to February 2026, puts healthcare first at $6.64M per breach, with financial services close behind at $6.29M:

In a lakehouse, that copy is usually a derived table: someone builds a table with full_name and ssn_hash from columns you classified, and the classification stays behind. So I tested whether Unity Catalog can make the protection follow the data. I tagged 10 columns in bronze, touched nothing downstream, and checked silver and gold.

Result: 33 of 33 expected tags arrived on the derived columns within about 4 seconds, and the masks applied to them. Not a single SET TAGS outside bronze. The way there also showed one trap that destroys data.

The feature: tag propagation in Databricks

Tag propagation makes governed tags follow column lineage, from a source column to every column derived from it. Combined with ABAC policies, column masks defined once on the catalog for any column carrying a tag, a derived table is masked the moment it exists.

Before it works:

It is in Beta. A workspace admin enables it on the Previews page.
It is switched on per governed tag: Catalog → Govern → Governed Tags → Configure → Tag propagation.
You need MANAGE on the tag, and the writes must run on serverless compute.

Details in the tag propagation docs.

How I tested it

Setup

Data: Synthea synthetic health records (patients, encounters, providers), in a bronze → silver → gold medallion. None of it is real, so mistakes are cheap.
Platform: Databricks Free Edition, serverless SQL warehouse. The SQL ran both ad hoc and as Databricks Asset Bundle jobs.
Classification: two Databricks-provided governed tags that map to LGPD's two categories, sap.PersonalData.isPotentiallyPersonal (art. 5, I) and sap.PersonalData.isPotentiallySensitive (art. 5, II). Ten columns tagged in bronze, in one file. PASSPORT carries both tags on purpose. CITY, GENDER and HEALTHCARE_EXPENSES stay untagged as negative controls.
Write paths: five silver objects, one per operation: CTAS with derived columns (concat_ws, right(), sha2()), CTAS with a JOIN, INSERT ... SELECT into a pre-existing table, MERGE, and a view. Then three gold tables on top of silver, including a GROUP BY on a sensitive column.
Masks: four catalog-level ABAC policies: redact (***), pseudonymize (SHA-256), remove (NULL), and one for columns with both tags.
Personas: four account groups, combined into five personas, from no access to the pipeline identity that writes everything in clear text.

Tagging is one statement per column, in bronze only:

ALTER TABLE synthea.bronze.patients ALTER COLUMN SSN
  SET TAGS ('sap.PersonalData.isPotentiallyPersonal' = 'true');

A policy matches tags, not tables, which is what lets it reach silver and gold:

CREATE OR REPLACE POLICY pii_personal_redact
ON CATALOG synthea
COLUMN MASK synthea.governance.mask_redact
TO `account users`
EXCEPT `synthea_pipeline`, `synthea_pii_readers`, `synthea_analysts`
FOR TABLES
MATCH COLUMNS has_tag_value('sap.PersonalData.isPotentiallyPersonal', 'true')
          AND NOT has_tag_value('sap.PersonalData.isPotentiallySensitive', 'true') AS m
ON COLUMN m;

A user must never resolve to two different masks on the same column, or the query fails. The four policies cover three disjoint sets of tags. Two of them, redact and pseudonymize, share a set and split it by principal: redact skips synthea_analysts, pseudonymize applies only to synthea_analysts, and synthea_pii_readers is exempt from both.

Measurement

Propagation test. The 33 column/tag pairs I expect in silver and gold live in a table. A FULL OUTER JOIN against information_schema.column_tags labels each one OK, MISSING or EXTRA, and any MISSING calls raise_error(), which fails the job.
Integrity check. Distinct values against row count on hashed columns after every build.
Persona matrix. One query per persona reads a value from each kind of column and classifies it as clear, hash, *** or NULL.

Results

What	Result
Tags on silver and gold	33/33 OK, 0 EXTRA, no `SET TAGS` outside bronze
Write paths covered	CTAS, JOIN, `INSERT ... SELECT`, `MERGE`, view, and the second hop into gold
Time to propagate	within about 4 seconds of the last write
Bundle jobs on serverless	4 runs, 33/33 each, 30 to 50 seconds per run
Rebuild with `CREATE OR REPLACE`	all 33 tags kept
Negative controls	`city`, `gender`, `healthcare_expenses` stayed untagged

What each persona saw, set by group membership:

Groups	Personal (SSN, name)	Sensitive (race)	Passport (both tags)
none	`***`	`NULL`	`NULL`
analysts	SHA-256 hash	`NULL`	`NULL`
analysts + pii_readers	clear	`NULL`	`NULL`
analysts + pii_readers + sensitive_readers	clear	clear	clear
pipeline	clear	clear	clear

I repeated the matrix after switching to each group as a role, which makes Databricks run queries with that group's permissions only, and every result matched its policies. Analysts got hashes, pii_readers got personal columns in clear, sensitive_readers got race in clear and personal columns as ***, and the pipeline got everything. No single reader role saw PASSPORT, because that takes both groups.

Propagation is conservative. full_name, ssn_last4 and ssn_hash all inherit the Personal tag, so a hash computed in the pipeline gets masked like the raw value. That is the right call: an unsalted SHA-256 of a 9-digit SSN can be reversed by hashing every possible value, which is why the EDPB treats such hashes as personal data. In gold, race is a GROUP BY dimension in an aggregate report, and it inherits Sensitive: without access, the report still has its 18 rows, but race and ethnicity read NULL in every one, so only gender is left to tell them apart. For analysts, the pseudonymized SSN was identical in bronze and silver, and a bronze-to-silver join on it returned all 1,163 rows.

To release a derived column, change the policy (pseudonymize instead of redact). Do not remove the tag downstream, because classification is supposed to live in one place.

Problems I hit

1. A masked writer persists the mask. This one cost data. A CTAS reads the source through the masks of whoever runs it. On the first silver/gold build, ssn_hash had 1 distinct value across 1,163 rows, the hash of ***, and gold.demographics had a single race, ***. No policy was involved anymore. The data itself was gone.

Later I repeated it on purpose, after switching to the analyst group as a role. The CTAS stored SSN and names as SHA-256 and race and passport as NULL, in all 1,163 rows, and an admin reading that copy gets the same hashes and nulls. The tags still propagated, so the copy stays governed, but the clear values are gone for everyone.

So anyone without access who runs a CTAS cuts the new table off from the real values: its columns hold ***, NULL or a SHA-256 string, never the original. The hash is the one case where the value can come back, by the same rule as ssn_hash above. The pseudonymization mask does add a salt, but the salt is in the public repo, so hashing every possible SSN with it still reverses the copy. In production the salt belongs in a secret.

Fix: a synthea_pipeline group in the EXCEPT of every mask, and the pipeline running as a member of it. A job run as a pipeline-only user produced 1,163 distinct hashes, 6 races and 887 passports, with no *** written. A bundle job's run_as takes a user or service principal, never a group, so membership is what makes the pipeline write clean data.

SELECT count(DISTINCT ssn_hash) AS distinct_hashes, count(*) AS rows
FROM synthea.silver.patients;   -- expect 1,163 / 1,163, not 1 / 1,163

2. EXCEPT means OR, not AND. PASSPORT carries both tags, so only users in both reader groups should see it. The obvious policy says EXCEPT synthea_pii_readers, synthea_sensitive_readers, but EXCEPT exempts anyone in any listed group, so a user in sensitive_readers alone would skip the mask and see passport numbers. EXCEPT cannot say "in both groups", so the policy exempts only the pipeline, and the mask function checks both groups itself:

CASE WHEN is_account_group_member('synthea_pii_readers')
      AND is_account_group_member('synthea_sensitive_readers')
     THEN value ELSE NULL END

I caught this by reading the policy definitions, not through a test. The role test above confirms the fix: as sensitive_readers alone, PASSPORT reads NULL.

3. Group membership changes are not instant. After I left the reader groups in the UI, is_account_group_member() kept returning true for about 2 minutes. A persona test in that window proves nothing, so I check effective membership in the same session before each run.

4. Smaller ones.

The governance job once recreated the policies while the medallion job was mid-build. Run governance first, or use one job with a dependency.
DROP POLICY has no IF EXISTS.
Free Edition has no account API (databricks account groups list fails even for a workspace admin), so the groups were created in the UI. Its workspace also lacks Data Classification, the LLM-assisted scanner that would replace hand-tagging at scale.

What I did not test

These come from the docs, not from the lab:

Declarative pipelines (DLT) and pipeline materialized views do not propagate tags.
Until a tag arrives on a derived table, policies that reference it are not enforced there.
Two policies that resolve to different masks for the same user and column make the query fail.
A STRING mask such as '***' on a DATE or INT column breaks reads.
Writing under every role. Every role was tested for reads, but I ran a CTAS under one role only, the analysts, in Problem 1.
Whether the class.* tags from Data Classification propagate like custom governed tags.

Takeaways

Turn on propagation for each governed tag first. Nothing else works without it.
Keep every SET TAGS in one file for the source layer, and make a missing downstream tag a failing test.
Exempt the pipeline identity from every mask, and check distinct counts after each build. Masked users get CREATE only in a sandbox schema.
Make sure no user and column can resolve to two masks: disjoint tags or disjoint principals.
Expect derived columns, hashes and aggregate dimensions to stay sensitive. Loosen them with a policy, not by removing tags.

Tag propagation is still in Beta, but it is one of the most useful governance features I have tested in Unity Catalog. It moves classification out of a spreadsheet and into the lineage, so "is this derived table protected?" gets the same answer as "is its source protected?", without anyone having to remember to ask. That matters more every year. The same IBM report found that one in four malicious breaches were AI-enabled, and phishing was the most common way in. An attack that goes through people ends with a valid login, and what that login can read is decided by access policies. When every copy inherits its policy, a phished analyst account gets hashes and NULLs, not the one table somebody forgot to tag.

The full lab, with the bundle, the SQL and the tests, is on GitHub: ViniCarvalhoData/synthea_databricks.

DEV Community: Vinícius de Carvalho