Model Deletion Orders Require Training-Data Lineage

In May 2021, the FTC finalized its order against Everalbum, the company behind the Ever photo app. Everalbum had to delete the facial recognition models it had built using its users' photos. The order carried no fine. The main remedy was deleting the models.

Orders like this assume a company can find every model that touched the data in question. A team can usually point to the code and configuration that produced a model. Pointing to the exact records that went into it is harder, and that is what these orders require.

FTC orders that required model deletion

The FTC has used this remedy at least five times since 2019:

• Cambridge Analytica (December 2019). Ordered to delete the data collected through the GSRApp and any work product built from it, including algorithms.
• Everalbum (May 2021). Ordered to delete the facial recognition models and algorithms built using biometric data from Ever users.
• WW International and Kurbo (March 2022). Ordered to delete data collected from children under 13 without parental consent, destroy any algorithms built from it, and pay $1.5 million.
• Amazon Alexa (May 2023). Fined $25 million and barred from using voice and geolocation data that users had asked to delete to build or improve its data products.
• Rite Aid (December 2023). Banned from using facial recognition for surveillance for five years, and ordered to delete the images it collected and any models or algorithms built from them.

The Cambridge Analytica and Everalbum orders cover anything built from the data "in whole or in part." Read literally, a model trained on a dataset with a single covered record falls under the order.

What counts as a derived artifact

Training produces more than a final model file. Starting from one source table, a pipeline can create:

• dataset snapshots and feature tables, which copy or transform source rows
• model weights and training checkpoints, which are fitted to those rows
• fine-tuned models, which start from the weights of an earlier model
• embeddings and vector indexes, which store values the model computed from source records
• distilled student models, which are trained on the outputs of a teacher model

Each item is a node in a lineage graph, and each one connects to the source through a different kind of edge. Lawyers decide which of these an order covers. To make that call, they need engineering to produce the full list for a given source.

The minimum lineage record

That list has to come from records written at training time. Lineage rebuilt later from logs and notebooks misses runs, especially old ones. At minimum, each training run should emit:

1. Sources. The tables and partitions it read, the table version at read time, and the filter predicate, including any consent condition. With a pinned version and a deterministic predicate, the exact row set can be recomputed later without storing row IDs.
2. Dataset snapshot ID. An immutable version of the training set. Apache Iceberg, for example, assigns a snapshot ID to every commit, and VERSION AS OF <snapshot-id> in Spark SQL returns the table as it was at that commit.
3. Run metadata. Run ID, code commit, and hyperparameters, plus the base model ID for a fine-tune or the teacher model ID for distillation.
4. Outputs. A content hash, such as SHA-256, of every model, checkpoint, and embedding table the run produced. The hash ties the record to the exact bytes, not to a file path that can be overwritten.
5. Deployments. Every service and service version that loads those outputs.

Querying the lineage graph

Each record adds edges to a graph. Nodes are source tables, snapshots, training runs, artifacts, and deployments, and each edge points from an input to an output. A deletion request then becomes a traversal: start at the covered source and collect every node reachable from it. Figure 1 shows the result for one source.

Figure 1.

Model M2 and Deployment D2 stay out of scope because Snapshot S2 never read Source A. Without the snapshot-level edges, there is no way to show that, and M2 would have to be treated as affected too.

If the edges are stored in a table with columns src and dst, a recursive query returns the full set:

WITH RECURSIVE downstream(node_id) AS (
  SELECT 'source:A'
  UNION
  SELECT e.dst
  FROM lineage_edges e
  JOIN downstream d ON e.src = d.node_id
)
SELECT node_id FROM downstream;

UNION, unlike UNION ALL, drops rows that are already in the result, so the recursion stops when no new nodes are found. The output is the list that legal review starts from.

What existing tools record

OpenLineage models Jobs, Runs, and Datasets, emits a START and a COMPLETE event for each run, and accepts custom metadata as facets. MLflow, since version 2.4, records the datasets a run consumed through mlflow.log_input(), with a digest of each dataset. Either can supply the source and run fields above. Neither has a standard field for the consent condition applied at read time, so that field needs a custom facet or tag. For sharing lineage outside the company, SPDX 3.0 and CycloneDX define bill-of-materials formats that cover datasets and models.

Retraining and unlearning

Once the affected models are identified, each one is either deleted or retrained without the covered data. Retraining from scratch on the filtered snapshot removes the records exactly, but its cost scales with model size and training-set size.

SISA training (Bourtoule et al., IEEE Symposium on Security and Privacy 2021) reduces that cost. It splits the training set into disjoint shards and trains one model per shard, then aggregates their predictions at inference time. Within each shard, training runs over slices of the data, with a checkpoint after each slice. Deleting a record means retraining only its shard, starting from the checkpoint taken before the slice that contained it. The authors report unlearning 4.63× faster on the Purchase dataset and 2.45× faster on SVHN than retraining from scratch.

Guo et al. (ICML 2020) define certified removal: after the update, the model is statistically hard to distinguish from a model that never saw the removed records. Their method applies a Newton update to the parameters and comes with a proof for L2-regularized linear models with a convex loss, such as logistic regression. Approximate unlearning methods for deep networks do not have a comparable guarantee.

Every one of these approaches still depends on lineage. Without it, the only safe scope for a deletion is every model trained while the covered data was in use.

References

1. Federal Trade Commission. In the Matter of Cambridge Analytica, LLC. Final Order. December 2019.
2. Federal Trade Commission. In the Matter of Everalbum, Inc. Decision and Order. May 2021.
3. Federal Trade Commission. FTC Takes Action Against Company Formerly Known as Weight Watchers for Illegally Collecting Kids' Sensitive Health Data. March 2022.
4. Federal Trade Commission. FTC and DOJ Charge Amazon with Violating Children's Privacy Law. May 2023.
5. Federal Trade Commission. Rite Aid Banned from Using AI Facial Recognition. December 2023.
6. Bourtoule, L., et al. Machine Unlearning. IEEE Symposium on Security and Privacy, 2021.
7. Guo, C., Goldstein, T., Hannun, A., and van der Maaten, L. Certified Data Removal from Machine Learning Models. ICML, 2020.
8. OpenLineage. Object Model.
9. MLflow. Dataset Tracking.
10. Apache Iceberg. Spark Queries: Time Travel.
11. The Linux Foundation. SPDX Specification 3.0.
12. OWASP Foundation. CycloneDX Bill of Materials Standard.