Last month I watched a privacy-preserving query feature die on the load test bench. The baseline endpoint was doing 40ms. The homomorphic-encryption version needed 8 seconds. Two hundred times slower. The business lead didn't ask for a breakdown. He said “roll it back,” and we spent the afternoon pulling the encryption layer out of the call path.
The next few days were worse. We couldn't answer the obvious rollback questions. Which requests had actually gone through the encrypted path? Which code version had they used? What data access policy did that version enforce? We had latency percentiles and error rates. We didn't have a single record that said “this computation ran under this policy, on these inputs.” So the rollback was blind.
I'm a compliance auditor, so my first instinct is to look for evidence. That week I learned that a privacy-preserving system without evidence is just an expensive black box. The query was encrypted. Great. But if someone asked “what did you compute, with what, and for whom?”, the honest answer was “we can't show you.”
People ask me about the cost of privacy computing. They usually mean latency. I care about latency too; that load test made it impossible not to. The technical bill is actually the easy part. MPC pays in communication—every multiplication gate needs precomputed triples or oblivious transfer, and the round count scales with circuit depth. TEE pays in hardware and in the chore of keeping every enclave measurement trustworthy. Federated learning pays in synchronization and noise; each round spends a slice of the privacy budget, and the budget is a one-way street. You can estimate all of that before writing code. The cost that quietly kills you is the evidence bill.
For a TEE, that means attestation evidence: code version, boot measurements, the key that signed the quote, and a way to cross-reference that with workload IDs. For MPC, it means protocol parameters: number of parties, secret-sharing scheme, precomputation method, and proof that triples were validated. For federated learning, it means the aggregator's logic and the noise mechanism, plus a per-round ledger for the privacy budget. Call it a checklist if you want, but it's the support structure for any future “what happened?” question.
That's what we changed after the rollback. Not the encryption. The bookkeeping.
We now give every computation a unique compute ID. It's a GUID bound to the data access policy, the code version, the participant list, and the exact protocol instance. Before, a query was just a query. After, it's an event you can point to. The log line looks like this: compute_id=7f3a... policy_ref=... code_sha=... parties=bank-a,fintech-b. If you need to know what the system did on the day of the rollback, you don't ask a person on call. You grep the log.
The first time we tried this, an engineer said “we already have request IDs.” Request IDs are for network messages. A single MPC run generates dozens of them, so they don't tell you which computation a message belonged to. The compute ID is the tie.
Next, we moved the audit log out of the production database. I had a client who kept audit records in a table with a 30-day retention policy. A cleanup job removed the evidence before the auditor even asked. The job was doing exactly what it was told; the intent was wrong. So audit records now go to a separate append-only store, each batch is signed, and no cleanup job knows where it lives. We record data lineage, node identities, attestation evidence, and hash-based input commitments. We don't log raw data. We log enough to prove that a computation ran on the intended inputs.
The third practice came out of an MPC audit. The policy document said “three-party computation with at least one honest party.” The actual run had two parties. The system hadn't flagged the mismatch. So we started maintaining a registry of executed policies versus actual parameters. Every computation's log must include the policy ID it claims to satisfy, and the runtime checks that participants, protocol, and keys match what that policy allows. If they don't, the computation never starts.
I want to be honest about the trade-off. We tried recording every intermediate state of the MPC protocol, thinking it would give us perfect audit coverage. The log grew to forty times the size of the input, and the metadata started leaking timing information. So we cut it down. We log input commitments, output shares, and a hash chain over the protocol events. We skip the individual gates. That compromise has worked for the audits we've seen so far, but it hasn't been tested against a truly hostile regulator. It's a bet, not a proof.
Eventually we went back to the drawing board. We split the data into sensitive and non-sensitive fields, encrypted the sensitive ones, and moved the computation into a TEE. The query time came back to something the business could live with. The TEE came with its own evidence demands—attestations, code measurements, key signing state—but at least nobody was reaching for a stopwatch.
The load test taught me something about that bet. If you sell a privacy solution to a business team, they will eventually measure it with a stopwatch. Eight seconds per query was never going to survive contact with a user. The engineers who built it had good reasons, but they didn't bring the benchmark early. They brought a concept. The team that has to explain the cost has no time for concepts.
So now my standard questions for any privacy-computing design review are: where is the compute ID? Where is the immutable log? Are the data-encryption keys separate from the protocol-execution keys? And what does the slowest realistic query look like under load? If the answer is “200x slower,” the system will be bypassed before production, no matter how solid the crypto is.
We rolled back that feature in an afternoon. The recovery took longer, mostly because we had to convince people we hadn't broken anything without being able to show them. The lesson has stuck: privacy machinery lives or dies by the numbers you can put in front of a business lead, and the audit trail that backs those numbers.
Run the load test before you sell the privacy story. Show me the log. Show me a compute ID with a signed input commitment. If you don't have those, we're going to have a longer conversation than you'd like. And I already know how that conversation ends.
Top comments (0)