DEV Community: Andrew Stevens

What the Data Act Actually Changes on 12 January 2027

Andrew Stevens — Wed, 28 Oct 2026 15:03:23 +0000

If you buy cloud services in the EU, you have probably heard that cloud exit fees end on 12 January 2027. For the one-off move of a service's data to another provider, that is broadly right. The traffic that crosses between providers while you run both, and the traffic to your own users, are governed by other rules, and on a long migration they can be the larger part of the bill.

The short version for a finance lead: from 12 January, a provider cannot charge you for switching a service to another provider or on-premises, although minimum-term penalties in your contract still apply and some custom-built services are excluded. Until then it can charge only its own direct costs of the switch. Data sent between providers while you use both can still be charged at cost after January, with no end date. What you pay to serve your own users does not change.

Some of what follows depends on your own contract and circumstances, and the post points out where. It is general commentary, not legal advice.

Article references are to the Data Act, Regulation (EU) 2023/2854 (European Parliament and Council, 2023), unless stated otherwise. "Provider" means a provider of data processing services, the Regulation's term for cloud and similar services, and "you" means the customer.

What happens on 12 January 2027

Article 29 is titled "Gradual withdrawal of switching charges". Its first three paragraphs set out the dates:

From 12 January 2027, providers of data processing services shall not impose any switching charges on the customer for the switching process.

From 11 January 2024 to 12 January 2027, providers of data processing services may impose reduced switching charges on the customer for the switching process.

The reduced switching charges referred to in paragraph 2 shall not exceed the costs incurred by the provider of data processing services that are directly linked to the switching process concerned. (European Parliament and Council, 2023, Art. 29)
{.source-quote}

Until 12 January 2027, then, a provider may charge for switching, but only its own costs directly linked to the switch, and from that date it may not charge at all. The Commission's FAQ describes it the same way, saying providers "must reduce any switching charges (including egress charges) from 11 January 2024 onwards" and may no longer charge for switching, data egress included, from 12 January 2027 (European Commission, 2026, Q54).

Article 29 names 11 January 2024 as the start of the cost cap, but the Regulation has generally applied only since 12 September 2025, and Article 50 gives the switching chapter no earlier date (Art. 50). The FAQ repeats the 2024 date without addressing Article 50. In my reading the cap clearly applies to any switch since September 2025; if you were charged above cost for a switch between January 2024 and September 2025, that is a question for a lawyer.

Article 29 also requires information. Before you sign, the provider must give you clear information on its standard service fees, its early termination penalties, and the reduced switching charges it may impose, and where relevant it must tell you about services that would be highly complex or costly to switch (Art. 29(4) and (5)). Where applicable, that information has to be made public in a dedicated section of its website or in another easily accessible way (Art. 29(6)). Your contract must also contain a clause on the switching charges allowed under Article 29 (Art. 25(2)(i)).

What counts as a switching charge

The ban applies to "switching charges", and the definition is narrow:

(36) 'switching charges' means charges, other than standard service fees or early termination penalties, imposed by a provider of data processing services on a customer for the actions mandated by this Regulation for switching to the system of a different provider or to on-premises ICT infrastructure, including data egress charges; (European Parliament and Council, 2023, Art. 2(36))
{.source-quote}

Standard service fees are excluded, and the recitals say they remain payable until the contract ends (European Parliament and Council, 2023, recital 89). Early termination penalties are excluded too: the same recital confirms that fixed-term contracts may include "proportionate early termination penalties", in line with Union or national law. If your contract has a minimum term, January does not remove what you owe for leaving early. Because the definition covers only "the actions mandated by this Regulation", extra help you ask the provider for, beyond its switching obligations, can also be charged if you agree the price in advance (recital 89).

The charge also has to be for switching, which the Regulation defines as the process by which the customer "changes from using one data processing service to using another data processing service of the same service type, or other service, offered by a different provider of data processing services, or to an on-premises ICT infrastructure" (Art. 2(34)).

Data egress charges have their own definition: "data transfer fees charged to customers for extracting their data through the network from the ICT infrastructure of a provider of data processing services to the system of a different provider or to on-premises ICT infrastructure" (Art. 2(35)). Traffic from your application to your own users and customers is neither switching nor in-parallel use. It stays part of what you pay for the service, at whatever rate your contract sets.

Running two providers at once

Many migrations do not move everything in one go. You stand up the new platform, move some of the traffic, and run the two side by side for a while, with data flowing between them. Plenty of companies also run two providers permanently, for example an application on AWS with analytics in BigQuery. The Regulation deals with both under a separate article.

Article 34, in Chapter VIII on interoperability, covers "in-parallel use of data processing services". Its second paragraph deals with charges:

Where a data processing service is being used in parallel with another data processing service, the providers of data processing services may impose data egress charges, but only for the purpose of passing on egress costs incurred, without exceeding such costs. (European Parliament and Council, 2023, Art. 34(2)) {.source-quote}

There is no end date in that paragraph, and the recitals make clear there is not meant to be one. Recital 99 explains that in-parallel use "relates to situations in which customers do not terminate a contract to switch to a different provider", that egress for in-parallel use "can be an ongoing activity, in contrast with the one-off egress required as part of the switching process", and that providers "should therefore continue to be able to impose data egress charges, not exceeding the costs incurred, for the purposes of in-parallel use after three years from the date of entry into force of this Regulation" (European Parliament and Council, 2023, recital 99). The Commission's FAQ says the same: in-parallel egress can still be billed at cost "even after 12 January 2027" (European Commission, 2026, Q54). If you are in financial services, note that the same recital says multi-cloud can contribute to digital operational resilience, citing Regulation (EU) 2022/2554, the Digital Operational Resilience Act. A permanent setup like the AWS and BigQuery example above is the kind of use recital 99 describes.

Article 34(1) also carries some of the switching protections across to parallel use, including the duty to remove obstacles under Article 23 and the open interface requirements in Article 30(2) to (5) (Art. 34(1)). Article 29 is not on that list, which is why the charge rules differ.

Switching is defined service by service. If you move a whole service to another provider or on-premises, that is switching, even if your wider agreement with the provider carries on for other services. If you keep using a service and add another provider's service alongside it, recital 99 describes that as in-parallel use. The hard case is a partial move, where you take some of a service's workloads elsewhere and keep the rest. The Regulation does not settle it, and the providers' programmes that exclude partial moves are commercial terms, not a reading of the law. Long migrations raise a related question: Article 25(5) lets you extend the transitional period once, for a period you choose (Art. 25(5)), but how far that stretches before the move looks like in-parallel use has not been tested. Take partial moves and long transitions to counsel before you plan the transfer, because the answer decides whether the traffic is free from January or charged at cost indefinitely.

Some services are partly or wholly outside the switching rules. Where "the majority of main features has been custom-built to accommodate the specific needs of an individual customer", or all components have been developed for an individual customer, and the service is not offered at broad commercial scale from the provider's catalogue, Article 29 does not apply, along with Article 23(d) and Article 30(1) and (3) (Art. 31(1)). Non-production versions provided "for testing and evaluation purposes and for a limited period of time" are outside the whole switching chapter (Art. 31(2)). The provider has to tell you before you sign which obligations do not apply (Art. 31(3)), so if you have a bespoke platform, ask.

A worked example

The numbers below are made up for illustration. Suppose you are moving 40 TiB of data off a provider, running both platforms in parallel for a while with 8 TiB a month crossing between them, and serving 25 TiB a month to your own users. Suppose your contract rate for egress is 85 per TiB and the provider's actual cost is 8 per TiB, in whatever currency your contract uses. That cost figure is an assumption; the provider is the only one who knows the real one.

Transfer	Rule	Before 12 January 2027	From 12 January 2027
Moving the 40 TiB dataset off the provider	Switching, Art. 29	At most cost: 320	No charge
Cross-provider traffic while both run, 8 TiB a month	In-parallel use, Art. 34(2)	At most cost: 64 a month, against 680 at the contract rate	Same: at most 64 a month
Traffic to your own users, 25 TiB a month	Standard service fees	2,125 a month at the contract rate	Unchanged

Table 1. Illustrative transfer costs under the Data Act, using assumed volumes and rates.

On these assumptions, January saves you at most 320, once, and possibly nothing if you already qualify for your provider's free exit programme. If your provider bills parallel traffic at the contract rate today, holding it to cost would save up to 616 a month. Your user-facing egress is untouched by either. The result depends heavily on the inputs, above all the gap between the provider's cost and its rate, and that gap is the least certain number here. A very large one-off move with a short parallel run makes the switch charge matter more, and a long parallel run makes the Article 34(2) cap matter more. "Costs incurred" is not defined in the Regulation, so ask the provider to state its figure in writing.

How the providers' exit programmes work

AWS, Google Cloud, and Microsoft Azure all announced free data transfer out for customers leaving in early 2024: Google Cloud on 11 January (Zavery, 2024), AWS on 5 March (Stormacq, 2024), and Microsoft on 13 March (Microsoft, 2024). AWS said its waiver "follows the direction set by the European Data Act" (Stormacq, 2024), and Microsoft said its exemption "aligns with the European Data Act" (Microsoft, 2024).

	AWS	Google Cloud	Microsoft Azure
What has to move	All data from all eligible accounts, or all data off a particular service (Amazon Web Services, n.d.a)	All of a given service, ending the agreement for it; partial moves excluded (Google Cloud, 2025)	The account, with all subscriptions cancelled; single services for customers with a UK billing address transferring from UK datacentres (Microsoft, 2026a)
How you claim	Request to support at least two months before the switch; credits, which can be charged back if the transfer was for another purpose (Amazon Web Services, n.d.a)	Credit on the final invoice; Google may review the movement of data (Google Cloud, 2025)	Request to support; credit after review (Microsoft, 2026a)
Time allowed	180 days for customers outside the EU; EU customers follow the EU Data Act Addendum (Stormacq, 2024; Amazon Web Services, n.d.b)	14-day intake, 30 days to begin, a migration period of at least 30 days, completion notice within 180 days of it ending (Google Cloud, 2025)	60 days, or longer if you give a migration timeline in the request; 180 days for UK-billed customers transferring from UK datacentres (Microsoft, 2026a)
Paths not covered	Including CloudFront, Direct Connect, the Snow Family, and Global Accelerator (Amazon Web Services, n.d.a)	No excluded services named; transfers over the internet only, for products on Google's data portability list (Google Cloud, 2025)	Including ExpressRoute, ExpressRoute Direct, VPN, Front Door, and Azure CDN (Microsoft, 2026a)

Table 2. The three providers' free data transfer programmes for customers leaving, as published on 28 October 2026.

For EU customers, AWS now points to a separate document. Its blog post was updated on 8 October 2026 to send EU customers "exercising a switch under the EU Data Act" to its EU Data Act Addendum (Stormacq, 2024). The addendum follows the Regulation's process closely: a switching request at least two months ahead, a 30-day transitional period that can be extended up to seven months if AWS finds the switch technically unfeasible, and a right for you to extend once (Amazon Web Services, n.d.b).

The switching process itself still applies after January. Article 25 expects you to give notice of up to two months, followed by a transitional period of up to 30 days and a retrieval period of at least 30 days (Art. 25(2)(a), (d), and (g)), so with the maximum notice period, a 30-day transition, and the minimum retrieval period, it is about four months from notice to the end of retrieval, and longer if the transition is extended. What changes on 12 January is that the provider can no longer charge you for the switch. In my reading, the ban covers switching charges whichever network path the data takes, although port or connection fees for a dedicated link may be argued to be standard service fees. The programmes' path exclusions will still matter for transfers that are not a switch.

For parallel use, two of the three publish an at-cost or free route. Microsoft offers at-cost transfer over the internet (and, for UK-billed customers transferring from UK datacentres, over its own premium network) for organisations with a billing address in the EEA, EFTA, or the United Kingdom, excluding customers who store their data outside those areas, using another provider "in parallel, in an interoperable manner", refunded after validation (Microsoft, 2026b). Google's Data Transfer Essentials covers "data transfer between services of the same organization that are distributed across different cloud service providers", excludes applications serving end users on other providers, and is "initially offered at no charge" with advance notice if that changes (Google Cloud, n.d.). Google's EU contract terms also restate Article 34(2) almost word for word, limiting any charge for parallel data export to "passing on the egress costs incurred by Google, without exceeding such costs" (Google Cloud, 2026). AWS's EU Data Act Addendum deals with the switching process and has no parallel-use rates (Amazon Web Services, n.d.b), unlike its UK addendum, so for EU parallel traffic on AWS, raise the Article 34(2) cost cap with your account team.

What your contract should contain

The switching chapter has applied since 12 September 2025, so contracts for in-scope services are expected to carry a set of terms. Article 25(2) lists the minimum. In summary, and in the order the article sets them out:

a transitional period of at most 30 calendar days, starting after the notice period, during which the provider gives "reasonable assistance" with the switch, keeps providing the service with due care, maintains a high level of security, and tells you about known risks to continuity (Art. 25(2)(a));
an obligation on the provider "to support the customer's exit strategy relevant to the contracted services, including by providing all relevant information" (Art. 25(2)(b));
a clause that the contract ends when the switch completes, or at the end of the notice period if you are leaving without switching and want your data erased (Art. 25(2)(c));
a notice period for starting the switch that "shall not exceed two months" (Art. 25(2)(d));
an "exhaustive specification of all categories of data and digital assets that can be ported", covering at least all exportable data (Art. 25(2)(e)), and an exhaustive list of any categories of data specific to the provider's internal functioning that are held back because of a risk to its trade secrets, which must not "impede or delay the switching process" (Art. 25(2)(f));
at least 30 calendar days after the transitional period to retrieve your data (Art. 25(2)(g));
full erasure of the exportable data and digital assets "generated directly by the customer, or relating to the customer directly" after the retrieval period, provided the switch completed (Art. 25(2)(h));
the switching charges the provider may impose under Article 29 (Art. 25(2)(i)).

If a provider says 30 days is not technically feasible, it has to tell you within 14 working days, explain why, and propose an alternative period that "shall not exceed seven months" (Art. 25(4)). Separately, you can extend the transitional period once, for a period you consider appropriate (Art. 25(5)).

On the technical side, Article 30(1) covers infrastructure providers: those offering "scalable and elastic computing resources limited to infrastructural elements such as servers, networks and the virtual resources necessary for operating the infrastructure", without access to the software and applications running on them. They must take "all reasonable measures in their power" to help you reach functional equivalence on a destination service of the same type, and support the switch with "capabilities, adequate information, documentation, technical support and, where appropriate, the necessary tools" (Art. 30(1)). Other providers must make open interfaces available "free of charge" (Art. 30(2)). Where no relevant common specifications have been published in the central Union standards repository, providers must on request export all exportable data "in a structured, commonly used and machine-readable format" (Art. 30(5)). None of this requires a provider to build new technology, hand over protected intellectual property or trade secrets, or compromise security (Art. 30(6)).

The Commission has also published voluntary standard contractual clauses for cloud contracts, three of which turn the switching chapter into ready-to-use terms, although its site still presents them as a draft Recommendation (European Commission, 2025). If you are renegotiating, they are a useful comparison.

The Commission's Digital Omnibus package of 19 November 2025 would exempt some adapted standard services, and SME and small mid-cap providers of non-infrastructure services, from parts of the switching regime (Sidley Austin, 2025; Greenberg Traurig, 2026). As reported in July 2026, the Council's compromise text applies those exemptions to contracts concluded on or before 12 September 2025, keeps Article 29(1) to (3) applying to them, and was still under negotiation (Greenberg Traurig, 2026). If your provider is small or your service is heavily customised, keep an eye on it.

Where UK buyers stand

The Data Act applies to providers "irrespective of their place of establishment" when they provide services to "customers in the Union" (Art. 1(3)(f)). The Regulation does not spell out how a customer's location is decided, but a UK customer contracting through a UK entity is unlikely to get these rights from the Act. In the UK, the pressure on egress fees has come from the Competition and Markets Authority.

The CMA's cloud market investigation concluded in July 2025 that competition "is not working well" and named egress fees as "a key commercial barrier" to switching and multi-cloud (Competition and Markets Authority, 2025). It recommended that the CMA Board "prioritise commencing SMS investigations" into AWS and Microsoft, meaning investigations into whether they have strategic market status (Competition and Markets Authority, 2025). In March 2026 the CMA announced that the two had "set out actions on cloud egress fees and interoperability" for UK customers, "following engagement with the CMA and subject to ongoing review" (Competition and Markets Authority, 2026a). The CMA describes the arrangement as a "voluntary dialogue" (Competition and Markets Authority, 2026b).

The CMA's October 2026 update reports that AWS and Microsoft have put free switching for UK customers into their contracts, extended their free switching windows to 180 days, and introduced reduced egress for multi-cloud, while noting that "the uptake of these schemes has been limited so far" (Competition and Markets Authority, 2026b). AWS's UK addendum gives UK accounts a 180-day transitional period, free data transfer credits for the switch, and "reduced rates not exceeding AWS's costs" for traffic to another provider used alongside AWS (Amazon Web Services, n.d.c). Microsoft's at-cost parallel transfer covers UK billing addresses (Microsoft, 2026b), and Google's Data Transfer Essentials was launched for customers in the EU and the UK (Manfra, 2025). The CMA Board plans to review whether this voluntary route is working in spring 2027 (Competition and Markets Authority, 2026b).

So on AWS and Microsoft, a UK buyer ends up with terms that look much like the EU position. The difference is that they rest on contracts and the CMA's oversight, which its Board will review in spring 2027. If your company has entities in both the EU and the UK, check which entity holds each cloud contract, because that decides which set of terms you can rely on.

What to do before January

Check your current contracts against the Article 25(2) list. If a contract is silent on the transitional period, the categories of portable data, or the retrieval window, raise it at renewal.

Before you talk to the provider, estimate how much data each part of the move involves: the switch itself, parallel running, and normal operations. Each has different rules, and on a long migration the parallel traffic is often the line most worth negotiating.

Ask for the provider's Article 34(2) cost basis in writing: what its egress costs are, how it calculates them, and where that is documented. If it has an at-cost or free parallel-use programme, find out how to enrol before the traffic starts, since some apply only to future usage.

If you are switching before 12 January, use the provider's programme and check its conditions against your plan, especially the requirement to move everything off a service or account, and any excluded network paths.

Ask which switching obligations the provider says do not apply to your services, and why. Article 31(3) requires it to tell you before you contract.

If a switch will straddle 12 January, ask the provider in writing how it will bill transfers made before and after that date.

Disclosure: Sakura Sky is an UpCloud reseller, so it has a commercial interest in customers being able to move between cloud providers. UpCloud did not fund this article and did not review or approve it. Sakura Sky also provides managed cloud and GRC services.

This article is general commentary on the EU Data Act and related UK developments as at 28 October 2026. It is not legal advice. Several points are open, including how the 2024 and 2025 dates interact, how partial moves are treated, and what counts as a provider's cost for in-parallel egress. Whether a particular transfer is switching or in-parallel use, and which terms apply to your contracts, depends on your facts, so take advice from a qualified lawyer before relying on it.

References

Amazon Web Services, n.d.a. Amazon EC2 FAQs: AWS Free Data Transfer Out Program for Switching Customers. Available at: https://aws.amazon.com/ec2/faqs/ [Accessed 28 October 2026].

Amazon Web Services, n.d.b. AWS EU Data Act Addendum. Available at: https://d1.awsstatic.com/onedam/marketing-channels/website/aws/en_US/legal/approved/eu-data-act-addendum.pdf [Accessed 28 October 2026].

Amazon Web Services, n.d.c. AWS UK Customer Switching and Portability Addendum. Available at: https://d1.awsstatic.com/onedam/marketing-channels/website/aws/en_US/legal/approved/aws-uk-customer-switching-addendum.pdf [Accessed 28 October 2026].

Competition and Markets Authority, 2025. Cloud services market investigation: summary of final decision. 31 July. Available at: https://assets.publishing.service.gov.uk/media/688b20e6ff8c05468cb7b120/summary_of_final_decision.pdf [Accessed 28 October 2026].

Competition and Markets Authority, 2026a. CMA announces package of actions on business software and cloud services. Press release, 31 March. Available at: https://www.gov.uk/government/news/cma-announces-package-of-actions-on-business-software-and-cloud-services [Accessed 28 October 2026].

Competition and Markets Authority, 2026b. Update on Cloud actions: Covering March-October 2026. 16 October. Available at: https://assets.publishing.service.gov.uk/media/6ad1cb60e5db84746b4796d1/Update_on_cloud_actions_-_covering_March_to_October_2026.pdf [Accessed 28 October 2026].

European Commission, 2025. Draft Recommendation on non-binding model contractual terms on data access and use and non-binding standard contractual clauses for cloud computing contracts. 19 November. Available at: https://digital-strategy.ec.europa.eu/en/library/draft-recommendation-non-binding-model-contractual-terms-data-access-and-use-and-non-binding [Accessed 28 October 2026].

European Commission, 2026. Frequently Asked Questions: Data Act. Version 1.4, 22 January. Available at: https://digital-strategy.ec.europa.eu/en/library/commission-publishes-frequently-asked-questions-about-data-act [Accessed 28 October 2026].

European Parliament and Council, 2023. Regulation (EU) 2023/2854 of the European Parliament and of the Council of 13 December 2023 on harmonised rules on fair access to and use of data and amending Regulation (EU) 2017/2394 and Directive (EU) 2020/1828 (Data Act). Official Journal of the European Union, L series, 22 December. Available at: https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32023R2854 [Accessed 28 October 2026].

Google Cloud, n.d. Data Transfer Essentials overview. Available at: https://docs.cloud.google.com/data-transfer-essentials/docs/overview [Accessed 28 October 2026].

Google Cloud, 2025. Applying for free data transfer when exiting Google Cloud. Last updated 11 September 2025. Available at: https://cloud.google.com/exit-cloud [Accessed 28 October 2026].

Google Cloud, 2026. Service Specific Terms (General Service Terms, Section 16: EU Data Act Terms). Last modified 27 October 2026. Available at: https://cloud.google.com/terms/service-terms [Accessed 28 October 2026].

Greenberg Traurig, 2026. EU Digital Omnibus Package Proposes Amendments to Data Act. 15 July. Available at: https://www.gtlaw.com/en/insights/2026/7/eu-digital-omnibus-package-proposes-amendments-to-data-act [Accessed 28 October 2026].

Manfra, J., 2025. Introducing no-cost, multicloud Data Transfer Essentials for EU and U.K. customers. Google Cloud Blog, 10 September. Available at: https://cloud.google.com/blog/products/networking/new-for-the-uk-and-eu-no-cost-multicloud-data-transfer-essentials [Accessed 28 October 2026].

Microsoft, 2024. Now available: Free data transfer out to internet when leaving Azure. Azure Updates, 13 March. Available at: https://azure.microsoft.com/en-us/updates?id=now-available-free-data-transfer-out-to-internet-when-leaving-azure [Accessed 28 October 2026].

Microsoft, 2026a. Cancel and delete your Azure subscription. Microsoft Learn. Available at: https://learn.microsoft.com/en-us/azure/cost-management-billing/manage/cancel-azure-subscription [Accessed 28 October 2026].

Microsoft, 2026b. Azure data transfer fees. Microsoft Learn. Available at: https://learn.microsoft.com/en-us/azure/cost-management-billing/manage/data-transfer-fees [Accessed 28 October 2026].

Sidley Austin, 2025. EU Digital Omnibus: The European Commission Proposes Important Changes to the EU's Digital Rulebook. Data Matters, 9 December. Available at: https://datamatters.sidley.com/2025/12/09/eu-digital-omnibus-the-european-commission-proposes-important-changes-to-the-eus-digital-rulebook/ [Accessed 28 October 2026].

Stormacq, S., 2024. Free data transfer out to internet when moving out of AWS. AWS News Blog, 5 March, updated 8 October 2026. Available at: https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-internet-when-moving-out-of-aws/ [Accessed 28 October 2026].

Zavery, A., 2024. Cloud switching just got easier: Removing data transfer fees when moving off Google Cloud. Google Cloud Blog, 11 January. Available at: https://cloud.google.com/blog/products/networking/eliminating-data-transfer-fees-when-migrating-off-google-cloud [Accessed 28 October 2026].

Before You Scale the AI, Fix the Evaluation

Andrew Stevens — Tue, 27 Oct 2026 10:15:27 +0000

When your organisation funds AI work today, much of the budget and attention tends to go to AI engineering: assembling foundation models, retrieval, tools, and agents into products. That's usually the right place to start, because it shows quickly what a system can do. Once a system has to make a measurable decision in production, though, a different set of questions starts to matter. What counts as a good result? Which cases does the system get wrong? Do the numbers in the status report mean what everyone assumes they mean?

Those questions are the core of ML engineering. AI engineering is incomplete without ML engineering's habits, especially evaluation, and programmes that treat ML engineering as a first-class part of AI engineering are better placed to ship systems that hold up. That matters most at the handoffs, where an ML model's output feeds a generative AI system or a generative model's output feeds ML. This post sets out why, using five learning outcomes from a recent advisory engagement on a production ML system.

Two practices, one system

Chip Huyen's account of the AI engineering stack puts the difference well. AI engineering "differs from ML engineering in that it's less about model development, and more about adapting and evaluating models". Because foundation models can produce open-ended outputs that are harder to evaluate, she argues, "[t]his makes evaluation a much bigger problem in AI engineering" (Orosz and Huyen, 2025). That point supports the argument here: the evaluation discipline ML engineering built up is what AI engineering now needs more of. The same excerpt notes that foundation models let teams build the product first and invest in data and models once the product shows promise. That ordering is a strength, and it also means the evaluation work tends to arrive late, sometimes after commitments have already been made on the strength of a demo.

	AI engineering	ML engineering
Starting point	Pre-trained foundation models and APIs	A specific task, its data, and its success criteria
Core activities	Composition, prompting, retrieval, tool use, and orchestration	Data and labels, evaluation design, training, calibration, deployment, and monitoring
Typical question	"Can we make the system do this?"	"How well does it do it, on which cases, and how do we know?"
Risk when it's the only practice	Impressive demos that don't hold up in production	Rigour applied to the wrong problem, or too slowly to matter

Table 1. AI engineering and ML engineering compared.

The two practices meet at handoffs. Figure 1 shows the two that come up most: an ML model's scores, with a threshold the business has chosen, passed to a generative AI system that acts on them, and generated labels passed back into the data an ML model learns from. Both sides are measured against what ML engineering defines as good, and production feedback flows back into the data and the evaluation.

The engagement

The engagement was advisory, covering review, evaluation design, analysis, and recommendations for a production ML system, with the models built and trained by the system's owners. The system's output feeds an automated downstream process and is shown to end users. What stood out across the engagement was how many of the most important shifts in understanding came from evaluation and from tracing the system end to end, more than from changes to the model.

Lesson 1: a lenient headline metric can hide the real problem

Early reporting led with a lenient measure that gave credit for near misses: an output counted as correct if it partly matched its target. On that measure the model looked strong. Stricter measures already existed in the tooling but weren't the ones in the headline report.

Scored with one-to-one matching and a strict standard for what counts as correct, the same outputs told a different story, and the gap persisted across later models. Strict measures were added to the reporting. A strict standard can be harsh in its own way, so it helps to report both kinds of measure, choose thresholds deliberately, and know which question each number answers.

Evaluation design belongs to the engineering work, and it deserves the same scrutiny as the model. A single lenient metric can point effort at the wrong problem. In this case, tracing the outputs end to end (lesson 2) then showed that part of the gap sat in how outputs were exported, outside the model.

Lesson 2: two consumers, two outputs

Following the outputs from the model to the places they're used showed that exported results carried a fixed safety margin. Under the strict measure, that margin cost a lot of score, and a simulation on existing outputs showed that removing it would sharply improve the score with no retraining.

The margin met a real requirement: the downstream step needs it so that it still acts on the whole target when the model is slightly off. Removing it would have lifted the metric while reducing the coverage that step relies on. The proposal was two outputs for two consumers: precise outputs for scoring and display, and conservative outputs for the downstream step.

Before you change anything to improve a number, find out how each output is consumed. A change that lifts a metric can work against the product, and the people who run the downstream process often know why a design detail exists.

Lesson 3: measure what the user experiences

End users mainly see the result of the downstream step and see the model's raw outputs only indirectly, so what they experience is a better basis for acceptance. Building on that, the proposal was for acceptance measures tied to user outcomes: how completely the downstream step covers what it should, and how much it acts on beyond that.

Framed that way, the confidence threshold becomes a product decision between two kinds of error. A trade-off curve lets the business see what it gains and gives up at each setting and choose an operating point with that in view. The outcome measures would sit alongside the model-level measures that track the model. Each kind of measure answers a different question, and a release review benefits from both.

Operating points are business decisions. Part of the ML engineering job is to make the trade-off visible to the people who own that decision and leave the choice with them.

Lesson 4: an agreed gate turns opinions into experiments

An acceptance gate had been agreed, combining a recall measure, a strict accuracy measure, and a cap on false alarms, which puts the point from lesson 1 into practice.

Having the gate written down changed how decisions were made. Ideas were tested against the same measures, so debates became experiments. Experiments showed that several proposed approaches needed to be reworked, and with the gate agreed in advance, that was a normal outcome.

Lesson 5: verify the verifier, including with AI

The analysis work ran under a fixed review protocol, with key numbers re-derived independently from the raw outputs and an adversarial review of the main findings.

The protocol still let an error through. The review had run on an earlier draft, and the text that was shared differed from it: a statement about a fix held for some cases and not others. A blind review caught it, and the shared document and the accompanying message were corrected. The protocol now includes a blind review of the exact final version before anything is shared.

The same protocol had earlier surfaced existing code that, together with a data source already available, narrowed the scope of a proposed build. That helped turn the build into a proposal for a short, time-boxed test.

This is also where AI engineering and ML engineering meet in practice. AI agents are well suited to this kind of verification: re-deriving key numbers from raw outputs, measuring results against agreed outcome and acceptance measures, and giving a blind, adversarial review of a finding before it's shared. They can make verification faster and more thorough, and they can check more numbers than is practical by hand. Accountability for what is shared stays with people, and a verification protocol should assume errors will happen, including in the review itself, and check the exact artefact that goes out.

The benchmark and the training data need engineering too

Two smaller points from the engagement are worth adding, because both bear on the handoffs below.

A re-annotation substantially increased the number of labelled examples, likely in part because earlier labels were incomplete and in part because the labelling scope widened. That means some predictions scored as false alarms may have been correct. Benchmark label quality matters well beyond one project: a study of ten widely used test sets across image, text, and audio data estimated an average of at least 3.3% of test labels to be wrong, and the authors found that, on corrected labels, some lower-capacity models would outperform higher-capacity ones if the share of originally mislabelled test examples rose by about five to six percentage points (Northcutt, Athalye and Mueller, 2021). Treating a benchmark as a product, with versions and a changelog, makes it possible to say which version a result was measured on.

Runtime constraints made running several models at inference time impractical, so the chosen approach was a single model trained on the original labels plus labels fused from the outputs of several models. It performed about as well as a larger multi-model ensemble at the same false-alarm level, and a control experiment (the same model trained on the original labels alone) indicated that the gain came from the fused labels. Without that control run, the improvement could as easily have been credited to the architecture or to the training run itself. Labels produced by models are exactly what a generative AI system hands back to ML, which is where the next section picks up.

Handoffs between ML and generative AI

More systems now combine both kinds of model, and the handoffs between them are often where evaluation is thinnest. Each direction needs its own checks.

When ML outputs feed generative AI

A classifier, risk score, or extraction model often decides what a generative AI system sees or does: which requests an agent handles on its own, which documents reach a prompt, which cases go to an automated response. The ML model's threshold at that point is an operating point in the sense of lesson 3, and the agent inherits whatever errors sit above it. A few practices help at this boundary:

Pass the score and the threshold along with the output, so the agent, and the person reviewing its work, can see how confident the upstream model was.
Agree which cases go to a person, and treat that routing as part of the product.
Measure the combined result. An ML model that passes its own gate can still feed an agent cases the agent handles badly.

One way to keep that decision visible is to declare it next to the action it gates. Arkhe, an open-source ontology language I maintain, takes that approach: a version-controlled file declares the entities an agent works with and the actions it may take, by whom, and with what audit trail, and compilers turn that into tool definitions for agent frameworks (Stevens, 2026). A threshold written as a guard on the agent's action sits in a file the business can review, and changing it goes through the same review as any other change.

Listing 1 shows the basic trade-off on ten invented cases. Raising the threshold from 0.5 to 0.9 sends fewer cases to the agent (90% falling to 30%) and more to people, and the error rate among the cases the agent handles falls from about a third to none. A sample this small is only an illustration; a real trade-off curve needs far more cases and should be measured on the combined system.

Python

# Invented scores for illustration only; not data from any system.
# Each case: (model score, whether the model's answer was actually correct).
CASES = [
    (0.97, True), (0.93, True), (0.91, True), (0.86, False), (0.82, True),
    (0.76, False), (0.71, True), (0.64, False), (0.58, True), (0.41, False),
]

def handoff_at(threshold: float, cases: list[tuple[float, bool]]) -> tuple[float, float]:
    """Cases at or above the threshold go to the agent; the rest go to a person.
    Returns the share sent to the agent and the error rate among those cases."""
    to_agent = [ok for score, ok in cases if score >= threshold]
    share = len(to_agent) / len(cases)
    error_rate = to_agent.count(False) / len(to_agent) if to_agent else 0.0
    return share, error_rate

if __name__ == "__main__":
    for t in (0.5, 0.7, 0.9):
        share, err = handoff_at(t, CASES)
        print(f"threshold {t:.1f}: {share:.0%} to the agent, {err:.0%} of those wrong")

Rust

// Invented scores for illustration only; not data from any system.
// Each case: (model score, whether the model's answer was actually correct).
const CASES: [(f64, bool); 10] = [
    (0.97, true), (0.93, true), (0.91, true), (0.86, false), (0.82, true),
    (0.76, false), (0.71, true), (0.64, false), (0.58, true), (0.41, false),
];

/// Cases at or above the threshold go to the agent; the rest go to a person.
/// Returns the share sent to the agent and the error rate among those cases.
fn handoff_at(threshold: f64, cases: &[(f64, bool)]) -> (f64, f64) {
    let to_agent: Vec<bool> = cases
        .iter()
        .filter(|(score, _)| *score >= threshold)
        .map(|(_, ok)| *ok)
        .collect();
    let share = to_agent.len() as f64 / cases.len() as f64;
    let wrong = to_agent.iter().filter(|ok| !**ok).count();
    let error_rate = if to_agent.is_empty() { 0.0 } else { wrong as f64 / to_agent.len() as f64 };
    (share, error_rate)
}

fn main() {
    for t in [0.5, 0.7, 0.9] {
        let (share, err) = handoff_at(t, &CASES);
        println!("threshold {t:.1}: {:.0}% to the agent, {:.0}% of those wrong", share * 100.0, err * 100.0);
    }
}

Listing 1. The share of cases handed to an agent, and the error rate among them, at three thresholds on invented scores.

When generative AI feeds ML

The reverse handoff is increasingly common: labels generated by an LLM used to train a model, synthetic examples used to fill gaps in a dataset, or an LLM acting as a judge in an evaluation. These are model outputs, and they carry model errors into the data that everything else is measured against. The fused-label result above shows the pattern that helps. Check generated labels against a sample labelled by people before you rely on them, version them like any benchmark, and run a control experiment so you know whether they helped. An LLM judge needs the same treatment, with its agreement with people measured and tracked as models and prompts change.

What this means for agents

The disciplines above matter more once agents act on a model's outputs. A weak evaluation in a reporting pipeline produces a wrong number that someone may question. In an agentic system the same weakness can produce a wrong action, possibly before a person reviews it. Sculley et al. (2015) noted that "[a] model may directly influence the selection of its own future training data", and agents that act on the world, or generate the labels a model learns from, can widen that kind of feedback loop.

The learning outcomes above have direct counterparts in agent work. A task-success rate that counts partial completions behaves like a lenient headline metric. A tool output read by both an agent and a person has two consumers with different needs. The confidence level at which an agent acts without escalating to a person is an operating point, and the business should be the one choosing it.

Foundation models change where the effort goes. They don't remove the need for evaluation, data quality, or operating-point decisions, and EU law and US guidance point the same way. For high-risk AI systems, the EU AI Act will require "[t]he levels of accuracy and the relevant accuracy metrics" to be declared in their instructions of use once those obligations apply (European Parliament and Council, 2024, Article 15(3)), on a timetable the 2026 Digital Omnibus on AI pushed back (European Parliament and Council, 2026). In the US, the voluntary NIST AI Risk Management Framework asks that "AI system performance or assurance criteria are measured qualitatively or quantitatively and demonstrated for conditions similar to deployment setting(s)" (National Institute of Standards and Technology, 2023, MEASURE 2.3). Both assume someone has decided which metrics matter, and the NIST text adds that they are demonstrated under conditions similar to deployment. That is the ML engineering question from Table 1.

On the governance side, the Trustworthy Agentic AI Blueprint and GATE, the open agent governance framework I maintain, describe wrapping the probabilistic model in a deterministic perimeter of governance, with controls at the tool boundary, where an agent's actions take effect. The evaluation work described here helps you know how far to trust what the model inside that perimeter produces.

Where to start

If you're about to scale an AI initiative, a useful first step is to ask whoever reports the headline metric how a correct result is defined, which benchmark version it was measured on, and who chose the operating point. Then list the handoffs between ML and generative AI in your system and ask who measures each one. If the answers are unclear, that is the work to fund before scaling.

Separately from the engagement above, if you'd like a second view on evaluation and production readiness for an AI or ML system, our Data & AI team works on machine learning in production and would be glad to talk it through.

Disclosure: GATE is authored and maintained by me personally rather than by Sakura Sky, and there is no paid tier, hosted version, or commercial product built on it. I co-authored the Trustworthy Agentic AI Blueprint with Sakura Sky. Arkhe (arkhelang) is also my personal open-source project, authored and maintained by me rather than by Sakura Sky, released under Apache-2.0, with no commercial product built on it. Agent governance is also my day job at Sakura Sky, which is a commercial interest worth stating.

References

European Parliament and Council, 2024. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), Article 15: Accuracy, robustness and cybersecurity. AI Act Service Desk, European Commission. Available at: https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-15 [Accessed 26 October 2026].

European Parliament and Council, 2026. Regulation (EU) 2026/1744 of the European Parliament and of the Council of 8 July 2026 amending Regulations (EU) 2024/1689, (EU) 2018/1139 and (EU) 2023/1230 as regards the simplification of the implementation of harmonised rules on artificial intelligence (Digital Omnibus on AI). Official Journal of the European Union, L series, 24 July 2026. Available at: https://eur-lex.europa.eu/eli/reg/2026/1744/oj [Accessed 26 October 2026].

National Institute of Standards and Technology, 2023. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. Gaithersburg, MD: U.S. Department of Commerce. Available at: https://www.nist.gov/node/1731741 [Accessed 26 October 2026].

Northcutt, C.G., Athalye, A. and Mueller, J., 2021. Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks. 35th Conference on Neural Information Processing Systems (NeurIPS 2021), Track on Datasets and Benchmarks. Available at: https://arxiv.org/abs/2103.14749 [Accessed 26 October 2026].

Orosz, G. and Huyen, C., 2025. The AI Engineering Stack. The Pragmatic Engineer, 20 May. Available at: https://newsletter.pragmaticengineer.com/p/the-ai-engineering-stack [Accessed 26 October 2026].

Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F. and Dennison, D., 2015. Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems 28 (NIPS 2015), pp. 2503-2511. Available at: https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf [Accessed 26 October 2026].

Stevens, A., 2026. Arkhe: An ontology language for AI systems. GitHub repository. Available at: https://github.com/arkhelang/arkhelang [Accessed 27 October 2026].

Whose Authority Reaches the Tools?

Andrew Stevens — Mon, 26 Oct 2026 17:22:04 +0000

Every tool call your agent makes carries someone's authority. When it searches documents, the document server has to decide whose documents it may return. When it sends an email, the mail server has to decide whether this send is allowed and for whom. If the answer at every server is "the agent's service account", each server sees one caller doing everything, and the employee who asked has dropped out of the picture.

Part 2 got the employee as far as the agent: the backend checks their Entra token, binds the session to tid:oid (the tenant and object IDs from that token), and calls Agent Runtime as itself. This part follows the agent's calls out to its four MCP servers from Part 1 and decides, one server at a time, whose authority each call runs on. The answer isn't the same for all four, and getting that mix right matters more than any single choice. Five of the checks at the end should hold a release.

Four servers, four decisions

Server	Runs as	How the token gets there	Why
Documents (company)	The employee	The backend gets an on-behalf-of token for this server and passes it to the agent with the query	Results should be what the employee could read themselves
Email (company)	The employee who approved the send	The agent only proposes a send. After approval, the backend gets an on-behalf-of token from the approver's sign-in and sends the message itself	The agent never holds a mail token, and the mail server knows who approved
BigQuery (Google)	The agent	The agent's Agent Identity, Google's per-agent workload identity	The tasks read a fixed set of tables, and making rows follow an Entra-only employee needs plumbing this design avoids
Ticketing (vendor)	The employee	Google's Agent Identity auth manager holds the employee's vendor token after they consent at the vendor	Only the employee can grant access to their vendor account

Table 1. Whose authority each MCP server sees, and why.

The documents and mail servers then apply their own rules to a named employee and log that employee. An injected agent can't read documents its user couldn't, as long as the documents server enforces the store's access rules, and it has no mail token to send with. Some exposure remains (Table 2).

What remains	How long it lasts	Where it's covered
Whoever controls the backend, or can add a credential to its registration, can act as recently active employees at the company's servers	Until revoked. Each refresh token lasts about 90 days, and using one returns a fresh one (Microsoft, 2025e)	The backend
The documents token goes to the agent, against Microsoft's rule that a middle tier's token goes only to its audience, so the documents server can't send its own claims challenge mid-query (Conditional Access policies on its enterprise app still apply when the backend mints the token)	For every triage query	Documents
BigQuery's logs show only the agent	For every query	BigQuery
Code in the agent's process can use the documents and vendor tokens it holds, including posting ticket comments as the employee, in both triage and ticket reply sessions	For each token's lifetime	Documents, Ticketing
One Agent Identity principal reaches the model, sessions, BigQuery, and the vendor tokens	Until the agent is contained	BigQuery
The agent, and anyone else who can retrieve credentials or reconfigure the auth provider, can get any consented employee's vendor token	Until the employee's authorization is revoked	Ticketing
A compromised agent can finish a vendor consent without the backend, and an employee can consent with a personal vendor account	Until found	Ticketing
A documents token the agent used stays valid	60 to 90 minutes by default (Microsoft, 2026b)	Dropping a token doesn't revoke it

Table 2. What this design leaves, and for how long.

Some of this rests on things we haven't tested on a deployed agent: that Agent Runtime passes the extra tool_tokens argument through, that Agent Platform Sessions trims temp: state the way the in-memory service does, that ADK's mTLS path reaches BigQuery as the agent, and the Workforce Identity Federation route for per-employee rows. The first two are release-blocking checks.

Three ways a call can carry authority

	The agent's own identity	The employee, delegated	The backend's app identity
What the server sees	One principal per agent	The employee, and the app that obtained the token	One principal for every employee
What limits the call	Grants to the agent	The employee's own access, narrowed by scope	Grants to the backend
Who did it, in the server's logs	The agent	The employee, through the backend	The backend
What it costs	The agent can read everything the task allows, whoever asked	A token per employee that someone has to obtain, carry, and drop	The server can't tell employees apart
Used here for	BigQuery	Documents, email, and ticketing	Nothing

Table 3. Three sources of authority for a tool call.

OAuth's token exchange RFC distinguishes impersonation from delegation. With impersonation, "[f]or all intents and purposes, when A is impersonating B, A is B within the context of the rights authorized by the token" (Jones et al., 2020). With delegation, A keeps its own identity and the token carries both parties, and the RFC defines an act claim naming the acting party, nested for chains. Entra's on-behalf-of flow is a JWT bearer grant with a Microsoft parameter, so the comparison is only an analogy. Microsoft refers to it as delegation (Microsoft, 2025c), and the token does name two parties: the employee as subject, and the backend in azp, "[t]he application ID of the client using the token" (Microsoft, 2026a). Microsoft's access token claims reference lists no act claim (Microsoft, 2026a), so nothing documented in the token records a chain. The front end that first got the employee's token has dropped out, and the agent never appears. The documents and mail servers see "the employee, through the backend", and a token the backend got for some other feature would look the same. That's why the Entra setup below gives the agent feature its own registration if the backend does anything else.

The third column is a reasonable default for many integrations, since it needs no per-user plumbing. For this agent it's the wrong trade. A mail server that only ever sees the backend can't apply a rule about who may email a customer, and its log can't say who asked. OWASP's agentic Top 10 covers that kind of gap under ASI03, Identity and Privilege Abuse (OWASP, 2025).

What MCP's authorization spec asks of each side

MCP's authorization spec, version 2026-07-28, is optional, and stdio servers are told to take credentials from the environment instead (Model Context Protocol, 2026a). For HTTP servers, three of its rules matter here. A server publishes where to get a token, using protected resource metadata (RFC 9728). A client names the server it wants a token for, using resource indicators (RFC 8707), and "MCP servers MUST validate that access tokens were issued specifically for them as the intended audience" (Model Context Protocol, 2026a). And a server that calls an upstream API gets its own token for it, and "MUST NOT pass through the token it received from the MCP client" (Model Context Protocol, 2026b). The audience rule is the one to check first in any server you inherit.

The core spec doesn't say how a server, or an agent, gets a token for a user it isn't talking to directly. MCP has an optional Enterprise-Managed Authorization extension, in which the MCP client exchanges the identity assertion it got at single sign-on for an "Identity Assertion JWT Authorization Grant" from the company's identity provider, then trades that for an access token from the MCP server's authorization server (Model Context Protocol, 2026d). It assumes the MCP client signed the user in. Here the MCP client is the agent, which never sees the employee's sign-in, and the extension's launch post names Okta as "the first supported identity provider" (Carleton, 2026). This part uses Entra's on-behalf-of flow instead.

Google says its remote MCP servers "implement the MCP authorization specification 2026-07-28", but they "don't support Dynamic Client Registration or OAuth Client ID Metadata Documents" (Google Cloud, n.d.d). A client needs a pre-registered OAuth client or a token it already holds. The spec puts pre-registration first in its client priority order and now deprecates Dynamic Client Registration (Model Context Protocol, 2026c), so that isn't unusual.

The company's own servers use Entra, which doesn't take the resource indicator. Microsoft's error reference lists "AADSTS901002: The 'resource' request parameter isn't supported" (Microsoft, 2025b), and a v2.0 token request names its audience through the scope instead. The spec says clients must send resource whether or not the authorization server supports it (Model Context Protocol, 2026a). Here the token request comes from the backend's on-behalf-of call, which sits outside the MCP client's authorization flow. Listing 1 leaves resource out deliberately and lets the server's aud check bind the audience. A generic MCP client pointed at an Entra-protected server may hit the same error.

Setting up Entra for the company servers

Seven token claims do most of the work below. aud is the API a token is for, which in v2.0 tokens is that API's client ID. azp is the client that obtained the token. azpacr says how that client authenticated, with 0 for a public client, 1 for a secret, and 2 for a certificate. scp lists the delegated scopes and roles the app roles assigned to the user (Microsoft, 2026a). idtyp marks app-only tokens, and acct is 1 for guests and 0 for members of the tenant (Microsoft, 2026d).

Most of the later checks fail unless the app registrations match. For the documents and mail APIs:

Set requestedAccessTokenVersion to 2. "If the value is null, this defaults to 1" (Microsoft, 2025g), and a v1.0 token has a different issuer and audience format, so every token would fail the checks.
Expose one delegated scope each, Docs.Read and Mail.SendFromSupport. Make them admin-consent only, so employees can't consent to them for some other app, and leave preAuthorizedApplications empty, since "[u]sers aren't required to consent to any preauthorized application" (Microsoft, 2025g).
On the mail API, define an app role, Mail.SendExternal, and assign it to the people allowed to email customers. The scope says what the backend may do with the mail API, and the role says what this employee may do.
Configure acct and idtyp as optional claims on access tokens, as Part 2 did for the agent API.

For the backend, which is also the API the front end's tokens are for:

Use the same app registration for the API and the on-behalf-of client. The employee's token "must have an audience (aud) claim of the app making this OBO request" (Microsoft, 2025c).
Grant it the delegated permissions Docs.Read and Mail.SendFromSupport with admin consent. The backend can't show the employee a consent prompt, which is why Microsoft points tenants at admin consent for the middle tier (Microsoft, 2025c). It also gets offline_access whether you grant it or not: "[i]f any delegated permission is granted, offline_access is implicitly granted" (Microsoft, 2026f).
If the backend serves other features, give the agent feature a registration of its own, which the front end requests tokens for, so azp names it.
Don't make it capable of Continuous Access Evaluation, which a client opts into by declaring the cp1 client capability (Microsoft, 2024). CAE-capable clients "may receive long-lived tokens (up to 28 hours)" where the resource supports it too (Microsoft, 2026b), and Listing 1 would refuse those anyway.
Grant it Microsoft Graph's profile permission with admin consent as well. Listing 1 asks for profile, and in on-behalf-of the middle tier "has no user interaction to get the user's consent" (Microsoft, 2025c), so that consent has to be in place before the first request.
Give it exactly one credential, a certificate, with no secrets and no federated credentials. Microsoft's client assertion is a short-lived JWT signed with the certificate's key (Microsoft, 2025d), and it shows up downstream as azpacr of 2. Listing 1 signs the assertion with a key in memory. Keeping the key non-exportable means signing in something like Cloud KMS instead, which changes one function.
Give the registration no owners beyond the deployment pipeline. Owners aren't the only way in, since directory roles and apps with permission to manage applications can add credentials too, on the application or its service principal. Check who holds those, and alert on any change to keys, secrets, or federated credentials on both objects. Anyone who can add a certificate gets tokens with the same azp and azpacr, and a federated credential may produce the same azpacr, so test that before treating the two as equivalent.
Turn off public client flows and leave out SPA and native redirect URIs.
Don't give it custom signing keys, since Microsoft says such apps "can't be used as middle-tier APIs in the OBO flow" (Microsoft, 2025c).

Documents: the employee, with a token minted per query

The documents server sits in front of a store where access already follows the employee. If the agent searched it with its own identity, it would need read access to everything, and a triage session for one team could return another team's runbooks. So the documents server should see the employee.

At the start of each triage query, the backend sends the employee's token, the one Part 2 checked, to Entra's token endpoint with grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer and requested_token_use=on_behalf_of, and asks for the Docs.Read scope (Figure 1). What comes back is an access token for the documents API. Microsoft's page says a refresh token comes only if offline_access was requested (Microsoft, 2025c), but since that permission is implicit, and MSAL's source notes that Entra "currently still issues an RT" (Microsoft, 2026c), Listing 1 expects one and throws it away. The backend passes the access token to the agent with the query, and the agent sends it on every documents call in that query. In ADK, keys with the temp: prefix are visible for one invocation and are meant to be trimmed before the session is saved, which makes them the natural place for it.

Microsoft is explicit: "DO NOT send access tokens that were issued to the middle tier to anywhere except the intended audience for the token" (Microsoft, 2025c). The rule is about the middle tier, and the agent is a separate workload the backend hands the token to, even though the token only ever goes on to the documents server. That's the risk Microsoft warns about, and this design accepts it on purpose. One consequence Microsoft names is that the agent can't complete a Conditional Access step-up challenge, so the documents server can't send its own claims challenge in the middle of a query. Policies on the documents API's enterprise app still apply when the backend mints the token, and Listing 1 passes any claims challenge from that request back to the front end. The alternative is for the backend to make the documents calls itself, as a proxy for the agent's tool calls. That keeps employee tokens out of the agent and lets the documents server issue its own claims challenge again, at the cost of the backend seeing all tool traffic and holding more code. It's the better choice if your documents store is more sensitive than this one, or if Conditional Access on it matters to you.

The documents server has its own hop to make. It can't forward the employee's token to the store behind it, so it either enforces the store's access rules itself, with a service identity, or makes its own on-behalf-of request for the store, in which case the store sees the documents server as azp. Either way, the promise that the agent reads no more than the employee could holds only if that hop is right. None of this stops the agent leaking what it does read, which is what Part 1's policy check and Parts 4 and 6 are about.

Email: the employee who approved, from the support mailbox

If email is giving you trouble, start with whose token the mail server sees.

Don't give the mail server the backend's app-only token, even though a customer reply is the company speaking. The company does speak through the mailbox: replies go out from a shared support mailbox, never from the employee's own. But the mail server needs to know who asked, so that a send that leaks data can be traced to an employee at the mail server itself, without joining it against the backend's and the agent's logs.

The mail server also needs the employee's identity to decide whether the approver may email customers. Part 2 made sure the person approving a held send owns the session and signed in recently, but neither check covers that. Entra puts app roles assigned to a user into the roles claim of their delegated tokens (Microsoft, 2026a), so a mail server that sees the employee can refuse an external send for anyone without Mail.SendExternal, approved or not. As long as the server enforces that role, an approval can't push an external send past what the approver could have sent without the agent.

Don't mint a mail token at the start of every query that might send, either. Part 1's policy check runs in the agent's process, so anything that gets past it could send with that token, as the employee. Withholding the token until release narrows the window but doesn't close it if the token still goes to the agent, where any code running there could reuse it for its lifetime.

So the agent never gets a mail token. Its send_email tool is a local function that reaches nothing, and the proposal is the function call itself, in the session's history. Part 2 starts every session with the untrusted-content flag set, so under Part 1's decision table every email the agent proposes is either denied or held for approval. When an employee approves, the backend records the exact arguments the approver saw, gets a mail token from the approver's step-up sign-in, and sends that message itself, without re-reading it from session state the agent can write to. Part 4 builds that release path.

Add two checks at the mail server. It should refuse a second send for the same approval ID, and a second send carrying the same token, identified by its uti claim, which Microsoft describes as a unique per-token identifier (Microsoft, 2026a). It has to count sends, meaning calls to the send tool, rather than requests, because the MCP spec puts the token on every request, and one send takes several. Those limit what a double release or a replayed mail token can do. They don't protect against a compromised backend, which can mint a new token and write its own approval record. A Conditional Access policy on the mail API's enterprise app could go further, since Microsoft documents that the on-behalf-of request fails when the downstream API's policy isn't met (Microsoft, 2025c). Paired with a sign-in frequency policy, that could stop a backend holding only an ordinary front-end token from minting a mail token, so test it before relying on it.

BigQuery: the agent, with grants sized to the task

BigQuery is the case Part 2 left open: should the data question run as the employee, so that row-level security applies to them, or as the agent?

Running as the agent is the path Google documents best. Agent Identity has been generally available since 22 April 2026 (Google Cloud, n.d.k). It gives each agent a SPIFFE-based principal, such as principal://agents.global.org-ORG_ID.system.id.goog/resources/aiplatform/projects/PROJECT_NUMBER/locations/LOCATION/reasoningEngines/ENGINE_ID (or agents.global.proj-PROJECT_NUMBER... for a project with no organization), and Google says agent identities "can't be impersonated, and don't allow developers to generate long-lived service account keys" (Google Cloud, n.d.c). Client libraries on Agent Runtime pick the agent's token up through Application Default Credentials (Google Cloud, n.d.e). To use the BigQuery MCP server, the agent needs MCP Tool User and BigQuery Job User on the project, and BigQuery Data Viewer (Google Cloud, n.d.x). Google's page grants Data Viewer on the project, but for this design it belongs on the datasets the tasks need and nowhere else.

Agent Identity gives each agent one principal, so Part 2's question about per-tool service accounts no longer applies. The same principal calls the model, writes the agent's sessions, and can fetch any consented employee's vendor tokens. It also comes with two default roles, roles/aiplatform.agentContextEditor and roles/aiplatform.agentDefaultAccess (Google Cloud, n.d.v). A compromise of the agent's runtime compromises all of those together. Google describes it as "a secure, per-agent identity" and documents no per-tool option (Google Cloud, n.d.v), so per agent is the finest split it offers. Separate auth providers, each with its own IAM grant, narrow which vendor tokens the agent can reach, but not which of its tools uses them.

Moving to Agent Identity changes Part 2's deny policy.

Rule 3 (sessions and memory) let only the agent's service account and a break-glass group read and write sessions. Its exception has to name the agent's new principal, which deny policies accept (Google Cloud, n.d.q), or the agent is denied its own sessions.
Rule 4 (changing the agent or its access) stopped anyone but the pipeline impersonating service accounts. An agent identity can't be impersonated, so outside the runtime itself, the way to act as the agent is to update it, with new code or new environment variables, including the one that turns off token binding. Keeping reasoningEngines.update and reasoningEngines.create with the pipeline becomes rule 4's main job. Grant to the agent's own principal:// rather than to Google's principal sets for every agent in a project (Google Cloud, n.d.q), or a newly deployed agent inherits the grants.
If you delete an agent and deploy a replacement, Google says it "receives a new resource ID and therefore a new principal identifier" (Google Cloud, n.d.c). Dataset grants, the auth manager grant, and the deny exceptions all have to follow, and the old bindings stay behind "as inactive grants" until someone removes them (Google Cloud, n.d.c).

The cost of running as the agent is that the agent can read every row its grants allow, whichever employee asked. That makes its grants the access of every employee who may run the task. My rule for this agent is that its BigQuery grants never exceed what the least-entitled employee allowed to run data_question or ticket_reply may see, with Part 2's backend restricting who may run each task by app role. In practice that can mean splitting a task by entitlement tier, or accepting a less useful data question. With Data Viewer only on the datasets those tasks need, and Part 1's deny on read-write MCP tools through mcp.googleapis.com/tool.isReadOnly (Google Cloud, n.d.i), the agent's identity is acceptable here. It stops being acceptable when different employees should see different rows. And because ticket_reply reads BigQuery and can propose a customer email, the decision also leans on Part 1's policy check and the approval release holding that email after a sensitive read.

It costs attribution too. BigQuery's records will show the agent's principal. Have deterministic agent code, not the model, log the session's tid:oid and session ID with every BigQuery tool call. That log is still self-reported by the component most likely to be compromised, so the backend should also record which employee's session was active and when, as a cross-check against the job times. Google's sample prompts identify jobs run through the MCP server by a goog-mcp-server:true tag (Google Cloud, n.d.x). Any other labels come from the model's tool arguments, so don't rely on them for attribution.

If rows must follow the employee

None of the routes is simple.

An on-behalf-of token, exchanged with Workforce Identity Federation. Part 2 promised to say how a server-side component gets an Entra ID token for the federation app. Microsoft's page documents no ID token in the on-behalf-of response (Microsoft, 2025c). MSAL's source asks for one with openid (Microsoft, 2026c), but an ID token's audience "is your app's Application ID" (Microsoft, 2025a), which here would be the backend's, so it doesn't fit. But Google's token exchange doesn't strictly need an ID token. For an OIDC JWT, it accepts a subjectTokenType of urn:ietf:params:oauth:token-type:jwt (Google Cloud, n.d.p), and the workforce provider's client ID "must match the aud claim of the JWT" (Google Cloud, n.d.h). An on-behalf-of access token for a dedicated federation API, issued as v2.0 so its aud is that API's client ID, looks like it would fit, with google.subject mapped to assertion.oid as Google's Entra guide does. The provider would also need an attribute condition requiring azp to be the backend, or any client that can get a token for that API could swap it for Google credentials. We haven't tested this with a workforce pool, and it's the route to try first.
The front end's own ID token. If the federation provider trusts the front end's client ID, the front end already holds a fresh ID token and can send it with each data question. Part 2 argued against this, because every sign-in to the front end could then be swapped for Google credentials.
The auth manager's three-legged OAuth, pointed at Google. Google's worked example uses the BigQuery MCP server with the https://www.googleapis.com/auth/bigquery scope (Google Cloud, n.d.f). That signs in a Google account. Neither that guide nor the auth manager overview mentions Workforce Identity Federation (Google Cloud, n.d.b; n.d.f), so for employees who exist only in Entra, don't plan on it without testing it.

Row-level security has problems of its own on any route. The usage guide shows workforce principals in a policy's GRANT TO list as principal://iam.googleapis.com/locations/global/workforcePools/POOL_ID/subject/... (Google Cloud, n.d.w), while the DDL reference lists only user:, serviceAccount:, group:, domain:, and the all-users forms (Google Cloud, n.d.j), and the guide's own example for workforce groups uses principal://.../subject/ with a group's email rather than principalSet://. For third-party users, SESSION_USER() "returns the principal identifier of the user that's running the query" (Google Cloud, n.d.t), so a filter written as email = SESSION_USER() won't match a federated employee unless the column holds that identifier.

The agent's own identity has a related problem. Principals outside a policy's grantees "cannot see any rows" (Google Cloud, n.d.l), so if your tables already have row access policies, the agent sees nothing until one names it. Google's row-level security guide covers Google Account and Workforce Identity Federation principals as grantees and doesn't mention agent identities (Google Cloud, n.d.w). Google's BigQuery MCP guide doesn't say whether row-level security applies to queries made through MCP (Google Cloud, n.d.x). Since execute_sql_readonly looks like an ordinary query job run as the caller, it probably does, but that's an inference, so test it.

Ticketing: the employee, through the auth manager

The ticketing vendor's server is the one where the company has no say over the authorization server. Only the employee can grant access to their account at the vendor, so this call has to run as the employee, and someone has to look after the token the vendor issues.

That's what Google's Agent Identity auth manager is for. It went GA on 22 August 2026 (Google Cloud, n.d.k), and it "[s]tores API keys, OAuth client secrets, and user tokens in a Google-managed vault" and "[h]andles multi-step OAuth 2.0 flows, such as user consent, authorization code exchange, and token refreshes" (Google Cloud, n.d.b). Figure 2 is the consent flow from Google's three-legged OAuth guide (Google Cloud, n.d.f), applied to the ticketing vendor.

Google says "[a]ccess to Agent Identity auth manager is governed by IAM" (Google Cloud, n.d.c), and the agent is inside that boundary. A credential request names the auth provider and a user_id. ADK 2.12.0 passes the session's own user_id (Google, 2026a), which Part 2's backend sets to tid:oid, but the API takes "[t]he identity of the end user" as an input (Google Cloud, n.d.n), so a compromised agent calling it directly can name anyone. Google's API reference describes the stored user ID as "[n]ot verified by Google" (Google Cloud, n.d.s). Google's guide grants roles/agentidentity.user to the agent's principal, and also to a developer's own account for local testing (Google Cloud, n.d.f). That role carries four permissions that release vendor tokens, retrieveCredentials and exchangeCredentials on both auth providers and authorizations, and the legacy Connector User role carries them too (Google Cloud, n.d.a). Whoever holds any of them, directly or through an inherited or custom role, can fetch any consented employee's vendor token by naming them, and anyone who can set IAM policy on the provider or its authorizations can grant themselves the same.

Reconfiguring the provider is another way in. Its token and authorize URLs, client ID, and client secret can all be changed with authProviders.update (Google Cloud, n.d.r), which the basic Editor role carries, among others, as does the Discovery Engine Service Agent role, roles/discoveryengine.serviceAgent (Google Cloud, n.d.a). Pointing the token URL elsewhere would likely send the stored refresh tokens there on the next refresh, though we haven't tested it.

On a production provider, allow the agent's principal to retrieve credentials and nobody else, keep updating or recreating the provider with the pipeline, alert on any change to it, and use a provider for this agent alone, since revoking a user's authorization works "across all agents" (Google Cloud, n.d.o). The provider can also limit what any agent may ask the vendor for, through its allowedScopes, and turn on PKCE (Google Cloud, n.d.r). Alert on retrievals for a user_id with no active session, after confirming which audit log records credential retrievals and that it's turned on. Google supports adding the Agent Identity APIs to a VPC Service Controls perimeter (Google Cloud, n.d.k), but clients then have to come through the restricted VIP (Google Cloud, n.d.c), and the vendor's redirect reaches the auth manager from the employee's browser. A perimeter is likely to break consent unless its ingress rules allow for that, so test it end to end.

Inside the agent, how the tools are grouped matters here. ADK resolves credentials for every toolset on the running agent before each model call (Google, 2026a), so an agent that carries the ticketing toolset fetches the employee's vendor token on every step, whatever it's doing. Giving each task its own agent, with only that task's tools, should keep the vendor token out of data questions. Confirm it on a deployed agent.

Step 7 is the check Google's guide leaves to the app. The guide has the continue_uri endpoint take the user_id and consent nonce "from your session storage" and call finalize with them (Google Cloud, n.d.f). Listing 5 keys the pending consent to the employee and the session, so for consents that come back through the company app, only the employee who started one can finish it.

That holds only while the agent behaves. The agent supplies the continue_uri when it asks for a credential, it already has the nonce, and Google's reference for finalize lists no IAM permission (Google Cloud, n.d.m), so a compromised agent could send the browser somewhere else and finish the consent itself. Google's own sample calls finalize without credentials, so check whether the audit log records who called it before you plan an alert on calls that don't come from the backend. The same listing checks the consent URL before the front end redirects to it, since that URL comes out of the agent, and Part 1 treats the agent's output as untrusted.

Nothing checks which vendor account the employee chose, so an employee who consents with a personal account at the vendor attaches that one. In this design only the agent's principal retrieves the vendor token, so a check against the vendor's "who am I" endpoint would have to run in the agent's own code, outside the model's tool choices.

Ticket comments need the same caution as email. The triage agent holds the employee's vendor token and the add_ticket_comment tool, and since both tasks share one provider and its scopes, a ticket reply session's token can write comments too, even though its agent lacks the tool. The triage agent's comments are held for confirmation by Part 1's policy check, inside the agent's process. Code that gets past it can post what the agent read, documents included, into a vendor ticket as the employee, with no backend release step. If the vendor offers internal-only notes, limit the scope to those. A second, read-only provider for ticket reply should keep its token from writing, if the vendor enforces its scopes (Listings 3 and 5 show a single shared provider). Otherwise treat comments as one of the ways data leaves, which Part 6 covers, or route them through the backend's release path, which means the backend retrieving the vendor token too.

And the agent does see the vendor token. The auth manager returns the credential to the agent, and ADK attaches it to the request (Google Cloud, n.d.b). Only with Agent Gateway and Gemini Enterprise connectors does Google say credentials are decrypted at the gateway so "the agent can never access the raw credential" (Google Cloud, n.d.c). On the path in Figure 2, code in the agent's process holds a working vendor token for the length of the call, the same as Part 1 assumed for every credential.

Getting a token to the agent for one query

On the backend (Listing 1), each query starts with an on-behalf-of request for each company server the task may reach, which means the documents server on triage queries. The mail token comes only from tokens_for_release, for the backend's own use. The backend authenticates with its certificate, drops any refresh token, refuses an unexpected lifetime, and passes a Conditional Access claims challenge back to the front end rather than failing. It also asks for the profile scope, since Microsoft documents tid and oid as needing it (Microsoft, 2026a). Microsoft gives the default access token lifetime as "a random value ranging between 60-90 minutes" (Microsoft, 2026b). Listing 1 sets the ceiling at two hours, which allows any default token. A token lifetime policy in your tenant can set something longer, and then every request fails the check, so find that out before you deploy.

Python

"""Per-invocation tool credentials, minted by the company backend.

The agent holds no long-lived credentials for the company's MCP servers.
For each query, the backend gets short-lived on-behalf-of tokens whose
audience is one MCP server, and passes them to the agent for that
invocation only. The mail token is minted only to release an approved send.
"""

from __future__ import annotations

import base64
import hashlib
import json
import time
import uuid
import urllib.error
import urllib.parse
import urllib.request
from dataclasses import dataclass
from enum import Enum

import jwt

class Authority(Enum):
    EMPLOYEE = "employee"          # Entra on-behalf-of token for the employee
    AGENT_IDENTITY = "agent"       # the agent's Agent Identity, no token passed
    AUTH_MANAGER = "auth_manager"  # vendor OAuth held in Agent Identity auth manager

@dataclass(frozen=True)
class Server:
    authority: Authority
    scope: str | None = None       # one explicit Entra scope, never .default

DOCS_API = "dddddddd-dddd-dddd-dddd-dddddddddddd"   # client IDs of the two
MAIL_API = "eeeeeeee-eeee-eeee-eeee-eeeeeeeeeeee"   # company APIs (placeholders)

SERVERS = {
    "documents": Server(Authority.EMPLOYEE, f"api://{DOCS_API}/Docs.Read"),
    "email": Server(Authority.EMPLOYEE, f"api://{MAIL_API}/Mail.SendFromSupport"),
    "bigquery": Server(Authority.AGENT_IDENTITY),
    "ticketing": Server(Authority.AUTH_MANAGER),
}

# Company servers each of Part 1's tasks may reach during an ordinary query.
# Email is absent: the agent only proposes a send, and the backend's
# release path gets a mail token (tokens_for_release).
TASK_SERVERS = {
    "triage_ticket": {"ticketing", "documents"},
    "data_question": {"bigquery"},
    "ticket_reply": {"ticketing", "bigquery"},
}

# Entra's default access token lifetime is 60 to 90 minutes. A tenant token
# lifetime policy can make it much longer; refuse those rather than hand
# them to the agent.
MAX_LIFETIME = 2 * 3600

class TokenError(Exception):
    pass

class ClaimsChallenge(TokenError):
    """Conditional Access needs the employee back at the front end."""

    def __init__(self, claims: str):
        super().__init__("interaction_required")
        self.claims = claims

class EntraTokens:
    """On-behalf-of calls to the Entra v2.0 token endpoint, authenticating
    with a certificate, which company servers see as azpacr 2."""

    def __init__(self, tenant_id: str, client_id: str, private_key,
                 cert_der: bytes,
                 authority: str = "https://login.microsoftonline.com"):
        self.url = f"{authority}/{tenant_id}/oauth2/v2.0/token"
        self.aud = self.url   # the client assertion's audience is the endpoint
        self.client_id, self.key = client_id, private_key
        self.x5t = base64.urlsafe_b64encode(
            hashlib.sha256(cert_der).digest()).rstrip(b"=").decode()

    def _client_assertion(self) -> str:
        now = int(time.time())
        return jwt.encode(
            {"aud": self.aud, "iss": self.client_id, "sub": self.client_id,
             "jti": str(uuid.uuid4()),
             "nbf": now, "iat": now, "exp": now + 300},
            self.key, algorithm="PS256", headers={"x5t#S256": self.x5t})

    def on_behalf_of(self, employee_token: str, scope: str) -> dict:
        # The employee's token must have been issued for this backend's own
        # API (Part 2 checked that); Entra won't redeem anything else.
        form = {
            "grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
            "client_id": self.client_id,
            "client_assertion_type":
                "urn:ietf:params:oauth:client-assertion-type:jwt-bearer",
            "client_assertion": self._client_assertion(),
            "assertion": employee_token,
            "scope": f"{scope} profile",   # profile: Microsoft documents tid
            "requested_token_use": "on_behalf_of",   # and oid as needing it
        }
        req = urllib.request.Request(
            self.url, data=urllib.parse.urlencode(form).encode(), method="POST")
        try:
            with urllib.request.urlopen(req, timeout=10) as r:
                return json.load(r)
        except urllib.error.HTTPError as e:
            try:
                body = json.load(e)
            except ValueError:
                body = {}
            if body.get("error") == "interaction_required" and body.get("claims"):
                raise ClaimsChallenge(body["claims"]) from e
            raise TokenError(f"token endpoint returned {e.code} "
                             f"{body.get('error', '')}") from e

def _mint(names: set[str], employee_token: str, entra: EntraTokens) -> dict[str, str]:
    out = {}
    for name in sorted(names):
        server = SERVERS[name]
        if server.authority is not Authority.EMPLOYEE:
            continue   # Google and the vendor are handled inside the agent
        resp = entra.on_behalf_of(employee_token, server.scope)
        if resp.get("token_type", "").lower() != "bearer" or "access_token" not in resp:
            raise TokenError(f"no bearer token for {name}")
        resp.pop("refresh_token", None)   # Entra may send one anyway; keep nothing
        if not 0 < int(resp.get("expires_in", 0)) <= MAX_LIFETIME:
            raise TokenError(f"unexpected lifetime for {name}")
        out[name] = resp["access_token"]
    return out

def tokens_for_invocation(task: str, employee_token: str,
                          entra: EntraTokens) -> dict[str, str]:
    """Tokens for the company servers this task may reach, and no others."""
    return _mint(TASK_SERVERS[task], employee_token, entra)

def tokens_for_release(approver_token: str, entra: EntraTokens) -> dict[str, str]:
    """The mail token, minted from the approver's step-up token (Part 2)
    only after an approval is recorded. The backend sends the approved
    message itself; this token never goes to the agent. Part 4 builds the
    release path."""
    return _mint({"email"}, approver_token, entra)

Listing 1. tool_credentials.py: the backend mints one token per company server for each query, and the mail token only for its own release step. It uses plain HTTP so each field is visible. MSAL's acquire_token_on_behalf_of does the same, but its token cache would keep any refresh token Entra returns, so use a cache per request or clear it.

TASK_SERVERS does for tokens what Part 1's task allowlist does for tools, but only for the company's own servers. BigQuery and the vendor aren't scoped by it, which is why Listing 3 builds an agent per task. We ran the backend against a fake Entra endpoint. Each request carried the form fields and certificate assertion we expected, each task got tokens only for its own servers, and ordinary queries made no mail request. A day-long token and a missing lifetime were refused, a refresh token was dropped, and a claims challenge reached the caller.

Inside the agent, the documents token arrives as temp: state. ADK applies temp: keys to the session for the current invocation and trims them before the session is saved, and in ADK 2.12.0 VertexAiSessionService does the trimming before its remote write (Google, 2026a). The Agent Platform SDK's streaming_agent_run_with_events, the entry point meant for AgentSpace, passes per-request OAuth tokens to the agent the same way (Google, 2026b), and Listing 2 follows that pattern. We tested it with the in-memory session service: the token was in state during the invocation, reached the documents toolset's request headers through ADK's own tool listing, was absent from the stored state and every event afterwards, and was missing from the next invocation. We haven't run the same test against Agent Platform Sessions, or confirmed that a deployed agent receives the extra tool_tokens argument, so do both before you rely on this.

Listing 2 extends Part 2's narrowed app template in two ways. It takes one new argument, tool_tokens, which may only carry a documents token. And it accepts one structured message, the adk_request_credential reply that resumes a query after vendor consent, in the shape Google's guide sends (Google Cloud, n.d.f). Part 2's app took only text, so without this the ticketing flow could never finish. The auth manager's provider is registered in set_up(), because Google's guide warns that registering at import time doesn't carry over to a container deployed with the SDK, and the agent then fails with "No auth provider registered" (Google Cloud, n.d.f). AdkApp comes from the agentplatform package, which google-cloud-aiplatform provides.

Python

"""Inside the agent: per-invocation tokens arrive as temp: state.

ADK applies temp: keys to the session for the current invocation and trims
them before the session is stored, so the tokens never reach Agent Platform
Sessions or the event history. We checked that with the in-memory session
service; test it against your deployment.
"""

import re

from agentplatform.frameworks import AdkApp
from google.adk.auth.credential_manager import CredentialManager
from google.adk.integrations.agent_identity import GcpAuthProvider

# Part 2, Listing 3
from backend_only_app import MAX_MESSAGE, USER_ID, BackendOnlyAdkApp

TOKEN_KEY = "temp:tool_token:{}"
COMPANY_SERVERS = {"documents"}   # the mail token never comes to the agent
TOKEN = re.compile(r"[!-~]{1,16384}")   # opaque: printable, no spaces
RESUME = "adk_request_credential"

def _is_consent_resume(message) -> bool:
    """The one structured message allowed: the reply to a consent request."""
    try:
        (part,) = message["parts"]
        fr = part["function_response"]
        return (set(message) == {"role", "parts"} and message["role"] == "user"
                and set(part) == {"function_response"}
                and set(fr) == {"id", "name", "response"}
                and fr["name"] == RESUME and isinstance(fr["id"], str)
                and isinstance(fr["response"], dict))
    except (KeyError, TypeError, ValueError):
        return False

class ToolTokenAdkApp(BackendOnlyAdkApp):
    def set_up(self):
        # Runs in the deployed container; module-level registration doesn't
        # carry over when the SDK serializes the app (Google's 3LO guide).
        CredentialManager.register_auth_provider(GcpAuthProvider())
        super().set_up()

    async def async_stream_query(self, *, message, user_id, session_id,
                                 tool_tokens=None, **extra):
        # Part 2's checks, plus a consent reply and a documents token.
        if extra:
            raise ValueError(f"unexpected arguments: {sorted(extra)}")
        if not session_id:
            raise ValueError("session_id is required")
        if not isinstance(user_id, str) or not USER_ID.fullmatch(user_id):
            raise ValueError("unexpected user_id")
        if not ((isinstance(message, str) and len(message) <= MAX_MESSAGE)
                or _is_consent_resume(message)):
            raise ValueError("message must be text or a consent reply")
        tokens = tool_tokens or {}
        if (not isinstance(tokens, dict) or not set(tokens) <= COMPANY_SERVERS
                or not all(isinstance(v, str) and TOKEN.fullmatch(v)
                           for v in tokens.values())):
            raise ValueError("unexpected tool_tokens")
        # The only state a query can carry: temp: token keys.
        delta = {TOKEN_KEY.format(k): v for k, v in tokens.items()}
        async for event in AdkApp.async_stream_query(
                self, message=message, user_id=user_id,
                session_id=session_id, state_delta=delta):
            yield event

def bearer_for(server: str):
    """McpToolset header_provider: this server's token, or no call at all."""
    key = TOKEN_KEY.format(server)

    def provider(context) -> dict[str, str]:
        token = context.state.get(key)
        if not token:
            raise PermissionError(f"no {server} token for this invocation")
        return {"Authorization": f"Bearer {token}"}
    return provider

Listing 2. agent_tool_tokens.py: the agent accepts a documents token as temp: state, and a consent reply as its only structured message. BackendOnlyAdkApp is Part 2's Listing 3.

Listing 2 calls AdkApp's method directly, because Part 2's method would refuse the token argument. That leaves two copies of Part 2's checks to keep in step. We tested that a caller's state_delta is still refused, a consent reply passes through unchanged, and any other structured message is refused, including one with a second part or another function name.

Listing 3 builds the agent. Each task gets its own LlmAgent with only that task's tools, and a small router runs the one named in the session's policy_task, which Part 2's backend sets when it creates the session. The task agents can't hand the conversation to each other, so a data question has no path to the triage agent's tools short of a bug in the router or in ADK.

Each toolset gets its authority differently. Documents reads its token through ADK's header_provider, which builds headers on each call (Google, n.d.a). Ticketing uses the auth manager's scheme. BigQuery carries nothing and relies on ADK 2.12.0, which by default opens an mTLS channel from Application Default Credentials for every HTTP MCP server, when its setup probe succeeds, and attaches the token only for googleapis.com hosts (Google, 2026a). On Agent Runtime that should be the agent's Agent Identity. The same channel presents the agent's certificate to the vendor's server, if it asks for one, which shows the vendor the agent's identity. BigQuery is the weak point. If the mTLS setup fails, ADK sends BigQuery requests with no token, tools/list still answers because it needs no authentication, and only the queries fail. We haven't tested this on Agent Runtime.

Python

"""The agent's tools, one sub-agent per task, one authority per server.

documents: the employee's on-behalf-of token for this invocation only.
ticketing: the employee's vendor token, held by the auth manager.
bigquery:  the agent's own Agent Identity, through ADC (untested on Runtime).
send_email: a local tool that reaches nothing. The function call itself is
            the proposal; the backend sends the approved message.
"""

from google.adk.agents import BaseAgent, LlmAgent
from google.adk.integrations.agent_identity import GcpAuthProviderScheme
from google.adk.tools.mcp_tool import McpToolset, StreamableHTTPConnectionParams
from google.adk.utils.context_utils import Aclosing

from agent_tool_tokens import bearer_for

PROJECT, LOCATION = "ops-agent-prod", "europe-west1"
CONTINUE_URI = "https://ops.example.com/agent/consent-done"   # the company app

# Part 1's tool allowlist per task, grouped by server.
TASK_TOOLS = {
    "triage_ticket": {"ticketing": ["get_ticket", "add_ticket_comment"],
                      "documents": ["search_documents"]},
    "data_question": {"bigquery": ["execute_sql_readonly"],
                      "email": ["send_email"]},
    "ticket_reply": {"ticketing": ["get_ticket"],
                     "bigquery": ["execute_sql_readonly"],
                     "email": ["send_email"]},
}

def send_email(to: list[str], cc: list[str], bcc: list[str],
               subject: str, body: str) -> dict:
    """Proposes an email. It is sent only if an employee approves it."""
    return {"status": "held for approval"}   # the policy check holds it first

def _toolset(server: str, tools: list[str]):
    if server == "documents":
        return McpToolset(
            connection_params=StreamableHTTPConnectionParams(
                url="https://docs-mcp.example.com/mcp"),
            header_provider=bearer_for("documents"),   # no token, no call
            tool_filter=tools)
    if server == "ticketing":
        # ToolTokenAdkApp.set_up() registers the auth provider (Listing 2).
        return McpToolset(
            connection_params=StreamableHTTPConnectionParams(
                url="https://mcp.ticketing.example.net/mcp"),
            auth_scheme=GcpAuthProviderScheme(
                name=(f"projects/{PROJECT}/locations/{LOCATION}"
                      "/authProviders/ticketing-3lo"),
                scopes=["tickets.read", "comments.write"],   # the vendor's
                continue_uri=CONTINUE_URI),
            tool_filter=tools)
    if server == "bigquery":
        # ADK sends the agent's ADC token to *.googleapis.com over mTLS.
        return McpToolset(
            connection_params=StreamableHTTPConnectionParams(
                url="https://bigquery.googleapis.com/mcp"),
            tool_filter=tools)
    if server == "email":
        return send_email
    raise ValueError(server)

class TaskRouter(BaseAgent):
    """Runs the session's task agent, and no other.

    ADK resolves auth for every toolset on the running agent before each
    model call, so an agent per task keeps the vendor token out of tasks
    that never use ticketing.
    """

    async def _run_async_impl(self, ctx):
        task = ctx.session.state.get("policy_task")
        agent = next((a for a in self.sub_agents if a.name == task), None)
        if agent is None:
            raise PermissionError(f"no agent for task {task!r}")
        async with Aclosing(agent.run_async(ctx)) as events:
            async for event in events:
                yield event

def build_agent(model, **llm_kwargs) -> TaskRouter:
    return TaskRouter(name="ops", sub_agents=[
        LlmAgent(name=task, model=model,
                 tools=[_toolset(s, t) for s, t in servers.items()],
                 # No handing the conversation to another task's agent.
                 disallow_transfer_to_parent=True,
                 disallow_transfer_to_peers=True,
                 **llm_kwargs)
        for task, servers in TASK_TOOLS.items()])

Listing 3. agent_tools.py: one agent per task, each with only its own tools. The scopes and URLs are placeholders, and the vendor's real scope names will differ.

With no token in state, the documents toolset raises before it sends anything, including tools/list. We tested that against McpToolset itself with the network blocked. Reading ADK's source, a full run logs the error and carries on without that toolset, so the model doesn't see a documents tool in a query with no documents token. That fails closed, but gives no sign. A triage answer written without documents looks like any other, so turn that log line into an alert. We also checked that the router runs only the session's task agent, that the data question agent has no toolset with vendor credentials, and that no task agent can transfer to another.

What a company server checks

The documents and mail servers are where the spec's audience rule gets enforced. Listing 4 accepts a token only if it was issued for this server's API by the right tenant, obtained by the backend with a certificate, for a member of the tenant rather than a guest, and with the scope this server needs. Microsoft defines acct only as whether the user is a member or a guest of the tenant (Microsoft, 2026d), so an external user added with the Member user type is likely to pass it. If "employee" has to mean someone from your own directory, also check the idp claim or require user assignment on the API. Its checks are Part 2's, applied to a different audience. It returns the employee and their app roles, so the mail server's own send rule can look for Mail.SendExternal, and it can refuse tokens issued before a cutoff, for use in an incident. Wiring it into an MCP server means answering a bad token with HTTP 401 and a valid token that may not do this with 403, each with a WWW-Authenticate header naming the resource_metadata URL, plus error="insufficient_scope" when the scope is missing, as the spec recommends (Model Context Protocol, 2026a). Each refusal carries its status. The cutoff compares against iat, which Microsoft describes as when authentication occurred (Microsoft, 2026a), so set it to the time containment finished and test that a newer token passes.

Microsoft says "[a] client application that's acting on behalf of a user (referred to as the actor), must also be authorized" and points to scp for that (Microsoft, 2025f), which Listing 4 checks. The azp and azpacr checks add a restriction on which client that is, and azp "may be used in authorization decisions" (Microsoft, 2026a), though Microsoft's claims validation page has a note against using it to authorize the subject of a delegated token, because such tokens "can potentially be obtained by entities other than the application" (Microsoft, 2025f), and Listing 4 keeps the employee as the subject. The remaining risk is another client getting a token for these APIs, and the setup narrows it. The scopes are admin-consent only, so another app can't get one unless an admin consents for it. The backend's registration has no public client path. And the azpacr check refuses a token obtained without a certificate. That shows some certificate registered on the app was used. It can't show the certificate was yours, so the registration's owners and credentials carry the risk.

Python

"""What a company MCP server checks on each request.

The server accepts only an employee's delegated token that was issued for
its own API and obtained by the company backend, with the scope it needs.
"""

from __future__ import annotations

from dataclasses import dataclass
from typing import Callable

import jwt

class Refused(Exception):
    """status: 401 for a bad token, 403 for a valid one that may not do this."""

    def __init__(self, reason: str, status: int = 401):
        super().__init__(reason)
        self.status = status

@dataclass(frozen=True)
class Caller:
    employee: str            # tid:oid
    roles: frozenset[str]    # the employee's app roles on this server's API

@dataclass(frozen=True)
class ServerConfig:
    tenant_id: str
    api_client_id: str       # this server's client ID (aud in v2 tokens)
    backend_client_id: str   # the only client allowed to get tokens for it
    scope: str               # e.g. Docs.Read or Mail.SendFromSupport
    client_auth: frozenset[str] = frozenset({"2"})   # azpacr: 2 = certificate
    issued_after: int = 0    # after an incident: when containment finished

def check_server_token(token: str, cfg: ServerConfig,
                       key_for: Callable[[str], object]) -> Caller:
    try:
        if jwt.get_unverified_header(token).get("alg") != "RS256":
            raise Refused("unexpected alg")
        c = jwt.decode(
            token, key_for(token), algorithms=["RS256"],
            audience=cfg.api_client_id,   # a token for any other API fails
            issuer=f"https://login.microsoftonline.com/{cfg.tenant_id}/v2.0",
            leeway=60,
            options={"require": ["exp", "nbf", "iat", "aud", "iss", "tid",
                                 "ver", "oid"]})
    except jwt.PyJWTError as e:
        raise Refused(f"token rejected: {e}") from e

    if c["iat"] < cfg.issued_after:
        raise Refused("issued before the cutoff")
    if c["ver"] != "2.0" or c["tid"] != cfg.tenant_id:
        raise Refused("wrong token version or tenant")
    if c.get("azp") != cfg.backend_client_id:
        raise Refused("token not obtained by the backend", 403)
    if str(c.get("azpacr")) not in cfg.client_auth:
        raise Refused("backend didn't authenticate as expected", 403)  # 0 = public
    # A missing scp catches app-only tokens even if idtyp isn't configured.
    if c.get("idtyp") == "app" or "scp" not in c:
        raise Refused("expected an employee's delegated token", 403)
    if str(c.get("acct")) != "0":
        raise Refused("guest account, or acct claim not configured", 403)
    if cfg.scope not in str(c["scp"]).split():
        raise Refused("missing scope", 403)   # with error="insufficient_scope"
    return Caller(f"{c['tid']}:{c['oid']}", frozenset(c.get("roles") or []))

Rust

//! What a company MCP server checks on each request: an employee's delegated
//! token, issued for this server's API, obtained by the company backend.

use jsonwebtoken::{Algorithm, DecodingKey, Validation, decode, decode_header};
use serde::Deserialize;
use serde_json::Value;

pub struct ServerConfig {
    pub tenant_id: String,
    pub api_client_id: String,     // this server's client ID (aud in v2 tokens)
    pub backend_client_id: String, // the only client allowed (azp)
    pub scope: String,             // e.g. Docs.Read or Mail.SendFromSupport
    pub client_auth: Vec<String>,  // allowed azpacr values: "2" = certificate
    pub issued_after: u64,         // after an incident: when containment finished
}

#[derive(Deserialize)]
struct Claims {
    tid: String,
    ver: String,
    oid: String,
    azp: Option<String>,
    azpacr: Option<Value>, // documented as a string; accept a number too
    acct: Option<Value>, // 0 or "0"; Microsoft doesn't give the type
    scp: Option<String>,
    idtyp: Option<String>,
    #[serde(default)]
    roles: Vec<String>,
    iat: u64, // required: a token without it fails to deserialize
}

#[derive(Debug, PartialEq)]
pub struct Caller {
    pub employee: String,   // tid:oid
    pub roles: Vec<String>, // the employee's app roles on this server's API
}

#[derive(Debug, PartialEq)]
pub enum Refused {
    Token(String),
    BeforeCutoff,
    WrongVersionOrTenant,
    NotFromBackend,
    WeakClientAuth,
    NotDelegated,
    Guest,
    MissingScope,
}

impl Refused {
    /// 401 for a bad token; 403 (insufficient_scope for MissingScope) for a
    /// valid token that may not do this.
    pub fn status(&self) -> u16 {
        match self {
            Refused::Token(_) | Refused::BeforeCutoff | Refused::WrongVersionOrTenant => 401,
            _ => 403,
        }
    }
}

pub fn check(token: &str, key: &DecodingKey, cfg: &ServerConfig) -> Result<Caller, Refused> {
    let bad = |e: jsonwebtoken::errors::Error| Refused::Token(e.to_string());
    if decode_header(token).map_err(bad)?.alg != Algorithm::RS256 {
        return Err(Refused::Token("unexpected alg".into()));
    }
    let mut v = Validation::new(Algorithm::RS256);
    v.set_audience(&[&cfg.api_client_id]); // a token for any other API fails
    v.set_issuer(&[format!("https://login.microsoftonline.com/{}/v2.0", cfg.tenant_id)]);
    v.set_required_spec_claims(&["exp", "nbf", "aud", "iss"]);
    v.validate_nbf = true;
    let c = decode::<Claims>(token, key, &v).map_err(bad)?.claims;

    if c.iat < cfg.issued_after {
        return Err(Refused::BeforeCutoff);
    }
    if c.ver != "2.0" || c.tid != cfg.tenant_id {
        return Err(Refused::WrongVersionOrTenant);
    }
    if c.azp.as_deref() != Some(cfg.backend_client_id.as_str()) {
        return Err(Refused::NotFromBackend);
    }
    let azpacr = match &c.azpacr {
        Some(Value::String(s)) => Some(s.clone()),
        Some(Value::Number(n)) => Some(n.to_string()),
        _ => None,
    };
    if !azpacr.is_some_and(|a| cfg.client_auth.contains(&a)) {
        return Err(Refused::WeakClientAuth); // "0" is a public client
    }
    // A missing scp catches app-only tokens even if idtyp isn't configured.
    let Some(scp) = c.scp.filter(|_| c.idtyp.as_deref() != Some("app")) else {
        return Err(Refused::NotDelegated);
    };
    let member = match &c.acct {
        Some(Value::Number(n)) => n.as_u64() == Some(0),
        Some(Value::String(s)) => s == "0",
        _ => false,
    };
    if !member {
        return Err(Refused::Guest); // a guest, or acct not configured
    }
    if !scp.split_whitespace().any(|s| s == cfg.scope) {
        return Err(Refused::MissingScope);
    }
    Ok(Caller { employee: format!("{}:{}", c.tid, c.oid), roles: c.roles })
}

Listing 4. mcp_server_auth: the documents and mail servers' token check, in Python with PyJWT and Rust with jsonwebtoken. Key fetching is left to the caller, as in Part 2.

We signed tokens with local keys and tried the ones a server is most likely to be handed, by mistake or on purpose: the employee's original token passed straight through, a token for the other company server, one obtained by the front end, one obtained by a public client or with a secret, the backend's app-only token, a guest's token, one without acct, one with the wrong scope or algorithm, expired, wrong-tenant, and wrong-key tokens, and a token issued before the cutoff. Both languages refused all of them, each for the reason and with the status expected. As with Part 2, that shows the code does what it says. Only a real on-behalf-of token from your tenant shows that Entra issues the claims the code expects.

Finishing vendor consent

Listing 5 is the backend's side of Figure 2. When the agent asks for consent, the backend checks that the consent URL goes to the vendor's own authorize endpoint and sends the code back to the auth manager's callback, then records the pending consent under the employee and the session. When the browser comes back to continue_uri, the front end calls the backend with the employee's current Entra token and the session, and the backend finishes only the pending consent for that pair. It works once, expires after ten minutes, and refuses a second consent while one is pending in the same session. Finishing returns the finalize request and the consent reply that Listing 2 accepts.

The backend then calls finalize. Google's reference lists only the cloud-platform OAuth scope for it, and no IAM permission (Google Cloud, n.d.m). The consent nonce is stored in the session's events, so treat the validation state as the only secret, keep the continue_uri query string out of access logs, and strip the auth config from events the backend relays to the front end. The reference describes the validation state only as "[t]he encrypted state passed back from the consent flow" and doesn't say how finalize checks it against the userId it's given (Google Cloud, n.d.m), so test that a finalize naming another employee fails. A resumed triage query also needs a fresh documents token, or the agent runs without its documents tool. If the backend has more than one instance, keep the store somewhere shared.

Python

"""The backend's side of vendor consent (Figure 2, steps 4 to 9).

When the agent asks for consent, the backend checks the consent URL and
records who the consent is for, keyed by employee and session. When the
browser comes back to continue_uri, the front end calls the backend with the
employee's current Entra token, and the backend finishes only the pending
consent for that employee and that session. Use a shared store with expiry
when the backend runs on more than one instance.
"""

from __future__ import annotations

import time
import urllib.parse
from dataclasses import dataclass

from google.adk.auth.auth_tool import AuthToolArguments

PROVIDER = "projects/ops-agent-prod/locations/europe-west1/authProviders/ticketing-3lo"
CALLBACK = f"https://agentidentitycredentials.googleapis.com/v1/{PROVIDER}/oauthcallback"
VENDOR_AUTHORIZE = "https://auth.ticketing.example.net/oauth/authorize"
VENDOR_CLIENT_ID = "ops-agent"                      # the auth provider's client
VENDOR_SCOPES = {"tickets.read", "comments.write"}  # also the provider's allowedScopes
# The provider has PKCE on, so a real consent URL carries a code challenge.
TTL = 600   # seconds a consent may take

@dataclass(frozen=True)
class Pending:
    call_id: str           # the adk_request_credential function call id
    auth_config: dict      # echoed back to the agent on resume
    consent_nonce: str
    expires: float

class ConsentRefused(Exception):
    pass

class ConsentStore:
    def __init__(self, clock=time.time):
        self._pending: dict[tuple[str, str], Pending] = {}
        self._clock = clock

    def start(self, employee: str, session_id: str, call_id: str,
              args: dict) -> str:
        """employee is the session owner (Part 2). args are the agent's
        adk_request_credential arguments. Returns the URL to send the
        browser to, once it's been checked."""
        auth = AuthToolArguments.model_validate(args).auth_config
        oauth2 = auth.exchanged_auth_credential.oauth2
        url = urllib.parse.urlsplit(oauth2.auth_uri)
        query = urllib.parse.parse_qs(url.query)
        # The agent's output is untrusted (Part 1): only the vendor's own
        # authorize endpoint, returning to the auth manager's callback.
        if (f"{url.scheme}://{url.netloc}{url.path}" != VENDOR_AUTHORIZE
                or query.get("redirect_uri") != [CALLBACK]
                or query.get("client_id") != [VENDOR_CLIENT_ID]
                or not query.get("code_challenge")
                or query.get("code_challenge_method") != ["S256"]
                or not query.get("state")
                or query.get("response_type") != ["code"]
                or not 0 < len(scopes := set(" ".join(query.get("scope", [])).split()))
                or not scopes <= VENDOR_SCOPES
                or not oauth2.nonce):
            raise ConsentRefused("unexpected consent URL")
        held = self._pending.get((employee, session_id))
        if held and self._clock() <= held.expires:
            raise ConsentRefused("a consent is already pending for this session")
        # Echo ADK's own camelCase arguments back unchanged: re-serializing
        # through AuthConfig drops the custom scheme's fields.
        self._pending[(employee, session_id)] = Pending(
            call_id, args["authConfig"], oauth2.nonce, self._clock() + TTL)
        return oauth2.auth_uri

    def finish(self, signed_in: str, session_id: str, provider: str,
               validation_state: str):
        """signed_in is tid:oid from Part 2's checks on this request."""
        if provider != PROVIDER:
            raise ConsentRefused("unexpected auth provider")
        p = self._pending.pop((signed_in, session_id), None)   # single use
        if p is None or self._clock() > p.expires:
            raise ConsentRefused("no pending consent for this employee")
        finalize = {"userId": signed_in,
                    "userIdValidationState": validation_state,
                    "consentNonce": p.consent_nonce}
        resume = {"role": "user", "parts": [{"function_response": {
            "id": p.call_id, "name": "adk_request_credential",
            "response": p.auth_config}}]}
        return finalize, resume

Listing 5. consent.py: the backend checks the consent URL, including its client ID, scopes, state, and S256 PKCE challenge, and binds the consent to the employee and session that started it.

In our tests, a second employee couldn't finish the first one's consent, and neither could the same employee from another session. A consent couldn't be finished twice or after it expired, an abandoned one stopped blocking the session once it expired, and an unexpected provider name was refused. So was a consent URL that pointed anywhere but the vendor, returned anywhere but the callback, named another client, asked for broader or unspecified scopes, lacked a state, lacked an S256 PKCE challenge, or asked for a response type other than code. We fed it arguments serialized the way ADK puts them in the function call, but the URLs were ones we built, so check the real provider's consent URL against these rules before release.

Where tokens sit between queries

Credentials sit in four places between queries.

ADK's MCP session pool. McpToolset keeps MCP sessions open and pools them by a hash of the request headers, which include the bearer token. In ADK 2.12.0 an idle session is evicted after 900 seconds, checked the next time the toolset opens or reuses a session (Google, 2026a). The pool also holds the vendor token, since the auth manager's credential goes into the same headers. A token therefore leaves session state when the invocation ends but can stay in the agent's memory for at least fifteen minutes after, and on an idle instance until it expires. The pool belongs to the toolset, so it holds recent tokens for every employee that agent instance has served.
The invocation's credential map. The vendor token from the auth manager is held in the invocation's context and fetched again before each model call in the tasks that use ticketing. The auth manager's scheme doesn't use ADK's credential service (Google, 2026a), so the default InMemoryCredentialService that AdkApp creates (Google, 2026b) doesn't hold it.
The session-state credential service. SessionStateCredentialService is marked experimental, and its docstring in ADK 2.12.0 says the end user's access and refresh tokens "are kept in clear text" in session state, which "is returned verbatim by the session-read endpoints" (Google, 2026a). ADK's authentication guide also warns that keeping tokens in session state carries risk, depending on the session backend (Google, n.d.b). It isn't suited to real user tokens.
The auth manager's vault. This is where the vendor's tokens live between queries, behind the IAM grant described above. Google describes it as "a Google-managed vault" (Google Cloud, n.d.b), and the auth manager pages don't cover customer-managed keys or who at Google can read it.

The design never stores the documents token. The backend gets a new one for each query that needs it, which costs one token request per query. That's cheap compared with keeping delegated tokens around.

Cutting off one employee

Disabling an account in Entra stops new tokens, but Microsoft says that "[f]or applications using access tokens, the user loses access when the access token expires" (Microsoft, 2026e). Part 2's checks validate the token itself, so an employee's current front-end token keeps passing them until it expires. The first offboarding step should therefore be a deny list in the backend, keyed by tid:oid and checked on every request.

Server	What its logs show	How to cut the employee off	When it takes effect
All, through the backend	The backend's own records	Add `tid:oid` to the backend's deny list, end their agent sessions, and cancel pending approvals and consents	On the next request
Documents	The employee, via the backend	Disable the account in Entra, then revoke sessions. Whether revoking sessions alone stops on-behalf-of requests made with an unexpired token is something to test	When on-behalf-of requests start failing. Tokens already issued work until they expire, 60 to 90 minutes by default and, for tokens minted by Listing 1, never more than its two-hour ceiling
Email	The approver, via the backend	The same, and remove `Mail.SendExternal`	The same
BigQuery	The agent only	Nothing per employee beyond the deny list	On the next request, through the backend's deny list. Past queries are attributable only through the agent's and backend's logs
Ticketing	The employee, at the vendor	Call the auth provider's `revokeAuthorization` with their `user_id`, and remove their access at the vendor	Depends on the vendor. A token already fetched works until it expires there

Table 4. What it takes to cut off one employee at each server.

Google's revokeAuthorization method "[r]evokes all authorizations for a specific user on an auth provider" (Google Cloud, n.d.o), and disabling the provider is the blunt fallback. Write the steps into the offboarding runbook and test them, the vendor step included, before release.

Containing the agent or the backend is a different job. For a suspected agent compromise, disable the ticketing provider, remove the agent's BigQuery grants, stop the backend minting documents tokens, and treat every consented employee's vendor token as exposed, since a compromised agent can name any user_id. Revoking at the vendor, or rotating the provider's OAuth client there, depends on the vendor.

If the backend is the suspect, disable its service principal first, then remove all of its registration's credentials and have an admin revoke the sessions of the employees whose tokens passed through it. Microsoft's revocation table shows an admin revoke ending confidential clients' refresh tokens, where a user's own password change doesn't (Microsoft, 2025e). Then set the cutoff in Listing 4 to the time containment finished, so each company server with the cutoff set refuses every token minted before it, including the attacker's. The same check could be scoped to one employee's oid to cut off a single person at once. Entra's sign-in logs for the backend's application ID show whose tokens those were. Part 8 builds detection on top of these records.

What this design doesn't settle

Dropping a token doesn't revoke it

Once the documents token has left state and the session pool, it's still valid until it expires, usually 60 to 90 minutes after issue. Anything that copied it in the meantime has it for that long: a log line, a trace span with content capture on, a debug dump, or code in the agent's process. Whoever holds a copy can use it. The backend's ceiling stops it passing on an unusually long-lived token. A token lifetime policy can make tokens shorter, down to ten minutes, and can be set for specific service principals, though "an organization-level policy takes precedence over an application-level policy" (Microsoft, 2026b). A short policy on the documents and mail APIs is worth testing.

The backend can act as recently active employees

Part 2 noted that whoever controls the backend can be any employee as far as the agent is concerned. With on-behalf-of, they can also be that employee at the documents and mail servers, using any employee token that passes through. The window isn't limited to token lifetimes either. Because offline_access comes with any delegated permission (Microsoft, 2026f), whoever holds the backend's certificate can get refresh tokens. Each "is typically valid for 90 days" (Microsoft, 2026f), and since "[r]efresh tokens replace themselves with a fresh token upon every use" (Microsoft, 2025e), a holder who keeps redeeming them keeps access until they're revoked. Listing 1 drops them, but an attacker holding the certificate wouldn't be running Listing 1. That puts the weight on the registration controls from the Entra section, plus one more alert: the backend redeeming a refresh token, which this code never does.

BigQuery's wiring is untested on Agent Runtime

Agent Identity's tokens for Google APIs are bound to the agent's certificate, and replaying one elsewhere fails with "Context-Aware Access requirements are not met" (Google Cloud, n.d.v). ADK 2.12.0's mTLS path for MCP looks designed for exactly this, but we haven't run it on Agent Runtime. Google's pages name two different environment variables for turning the binding off, GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN (Google Cloud, n.d.e) and GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES (Google Cloud, n.d.v), and both discourage it. Turning it off wouldn't help in any case if the mTLS setup fails, since ADK then sends no token at all.

The same identity offers a way to let the company's servers see the agent as well as the employee. ADK already presents the agent's certificate to any MCP server that asks for one over mTLS (Google, 2026a). An agent can get OIDC ID tokens of its own, and Google's client libraries bind them to its certificate "by including the certificate fingerprint in the token's cnf (x5t#S256) claim" (Google Cloud, n.d.e). A company server that required one of those alongside the on-behalf-of token, over mTLS, should refuse a copied employee token used on its own, which is worth testing for the most sensitive servers.

The auth manager's ADK support is still marked experimental

The auth manager service is GA, but in ADK 2.12.0 the pluggable auth feature that GcpAuthProvider builds on is marked experimental and switched on by default (Google, 2026a). Pin the ADK version.

The query carries a token through Agent Runtime's API

The documents token travels to the agent as an argument of a query request to Agent Runtime. Google's Agent Runtime logging guide treats logging a query's input as something the agent does itself, through stdout or its own logger (Google Cloud, n.d.u), and doesn't say whether the platform records request arguments. Content capture in traces should be off, as Part 2 said, and sending a marked test token and searching the logs for it is a cheap check to repeat.

Anyone can list BigQuery's MCP tools

The BigQuery MCP server's tools/list "doesn't require authentication" (Google Cloud, n.d.x), and deny policies don't hide tools from the list (Google Cloud, n.d.i). Tool names and descriptions are untrusted input either way, as Part 1 said, and Part 5 covers servers you didn't write.

Where GATE fits

GATE, the open agent governance framework I maintain, lists "[s]hared service accounts across agents" as a failure mode of C01, Workload Identity and Attestation, because they collapse attribution (Stevens, 2026). Agent Identity removes the shared account for BigQuery. C01 doesn't cover the employee behind a call. Part 2 proposed adding that in GATE v1.5, and Table 1 is a candidate for the same proposal, since it records whose identity each server sees.

Versions and platform facts

Item	Version or stage	Checked
ADK for Python	Tested with 2.12.0, with the `agent-identity` and `mcp` extras	22 October 2026
Agent Platform SDK (`google-cloud-aiplatform`)	Tested with 2.5.0	22 October 2026
PyJWT	Tested with 2.15.1	22 October 2026
`cryptography`	Tested with 50.0.2	22 October 2026
`jsonwebtoken` (Rust)	Tested with 11.1.0, `rust_crypto` feature	22 October 2026
Agent Identity	GA since 22 April 2026	22 October 2026 (Google Cloud, n.d.k)
Agent Identity auth manager	GA since 22 August 2026	22 October 2026 (Google Cloud, n.d.k)
ADK pluggable auth (`GcpAuthProvider`)	Experimental, on by default, in ADK 2.12.0	22 October 2026 (Google, 2026a)
BigQuery MCP server	GA since 20 April 2026	22 October 2026 (Google Cloud, n.d.g)
MCP specification	2026-07-28	22 October 2026 (Model Context Protocol, 2026a)

Table 5. Versions tested and launch stages relied on in this part.

What I'd check in your own deployment

As with Part 2, these depend on your tenant, projects, and servers, so treat each as something to confirm. Five should stop a release, and they're marked.

Entra and the company servers

Release-blocking. With Graph profile admin-consented for the backend, a real on-behalf-of token from your tenant, for each company server, has ver 2.0, aud set to that server's client ID, azp set to the backend's client ID, azpacr 2, acct 0, the employee's tid and oid, and the expected scope, and a real on-behalf-of response passes Listing 1's checks.
Release-blocking. Each company server refuses the employee's original token, a token for the other server, a token from any other client, a token obtained by a public client, an app-only token, and a guest's token, tested with real tokens.
The backend's registration has one certificate and no other credentials, no owners outside the pipeline, no public client flows, and no SPA or native redirect URIs. You know which directory roles and apps can add credentials to it, and a change to keys, secrets, or federated credentials on the application or its service principal raises an alert.
Neither company API lists anything in preAuthorizedApplications, and a new consent grant to either raises an alert.
A claims challenge from Conditional Access reaches the front end and can be completed.
The mail server refuses an external send for an employee without Mail.SendExternal, approved or not, and refuses a second send for the same approval or carrying the same token's uti, while a single send's several requests succeed.
A token minted after the Listing 4 cutoff passes, and one minted before it is refused.

Tokens inside the agent

Release-blocking. No mail token ever reaches the agent, and the only path to the mail server is the backend's release of an approved message.
Release-blocking. Against the deployed agent and Agent Platform Sessions, a temp: token arrives through tool_tokens, is in state during the invocation, and is absent from the stored session and its events afterwards.
A query that runs without its documents toolset raises an alert.
A data question session makes no call to retrieve vendor credentials, and a BigQuery tool call succeeds on the deployed agent with no "Failed to configure mTLS" line in its logs.
AdkApp isn't given SessionStateCredentialService, trace content capture is off, and a marked test token doesn't appear in Agent Runtime's request logs or the agent's telemetry.

The vendor and the auth manager

Release-blocking. On the production ticketing auth provider, no principal other than the agent holds retrieveCredentials or exchangeCredentials on the provider or its authorizations, including through inherited, legacy, or custom roles. If your project has a Discovery Engine service agent, know that its role can update auth providers, and confirm it has no reason to touch this one. Apart from that service agent, only the pipeline can set IAM policy on either, update, create, or delete the provider and its authorizations, or revoke authorizations. A test with two employees shows one can't finish the other's consent, and a finalize naming a different employee fails.
Retrievals of vendor credentials are logged, and one for a user_id with no active session raises an alert, as does any change to the provider. If the audit log records who calls finalize, a call that doesn't come from the backend raises one too.
The provider's allowedScopes lists only the vendor scopes the agent needs, PKCE is on, and the backend accepts the real provider's consent URL but refuses one that points elsewhere.
Ticket comments are limited to internal notes, or go through the release path, and ticket reply sessions get a read-only vendor token.
Consent completes end to end with any VPC Service Controls perimeter enforced.
Revoking one employee's vendor authorization is tested and in the offboarding runbook.

BigQuery and Part 2's deny policy

The agent's BigQuery grants don't exceed what the least-entitled employee allowed to run data_question or ticket_reply may see, and the backend restricts who may run each task.
If any of those tables have row access policies, a query through MCP as the agent returns what you expect.
The agent logs tid:oid and the session ID with every BigQuery tool call, and the backend's own records can be joined to BigQuery's job times.
Part 2's session and memory deny rule names the agent's Agent Identity principal, grants name that principal rather than a project-wide principal set, only the pipeline can create or update agents, and recreating the agent is a pipeline step that also updates its grants and removes the old ones.

The rest of the series

Part 4 takes on indirect injection through tickets, documents, and query results, and builds the approval release path, which now ends with the backend sending the approved message. Part 5 covers MCP servers you didn't write, including the vendor's. Part 6 covers the ways data leaves, Part 7 memory, Part 8 detection and containment, and Part 9 red-teaming before release.

If you asked each of your agent's tool servers who made the last call it served, how many could name the person who asked?

Disclosure: The author is a lifetime member of the OWASP Foundation. This article reflects an independent reading of public OWASP material and does not represent the views of the Foundation.

Disclosure: GATE is authored and maintained by me personally rather than by Sakura Sky, and there is no paid tier, hosted version, or commercial product built on it. Agent governance is also my day job at Sakura Sky, which is a commercial interest worth stating.

References

Carleton, P., 2026. Enterprise-Managed Authorization: Zero-touch OAuth for MCP. Model Context Protocol Blog, 18 June. Available at: https://blog.modelcontextprotocol.io/posts/enterprise-managed-auth/ [Accessed 26 October 2026].

Google, 2026a. google-adk 2.12.0: Agent Development Kit [source code]. PyPI. Available at: https://pypi.org/project/google-adk/2.12.0/ [Accessed 22 October 2026].

Google, 2026b. google-cloud-aiplatform 2.5.0: Vertex AI API client library [source code]. PyPI. Available at: https://pypi.org/project/google-cloud-aiplatform/2.5.0/ [Accessed 22 October 2026].

Google, n.d.a. Advanced MCP configuration. Agent Development Kit documentation. Available at: https://adk.dev/tools-custom/mcp-tools/advanced/ [Accessed 22 October 2026].

Google, n.d.b. Authenticating with tools. Agent Development Kit documentation. Available at: https://adk.dev/tools-custom/authentication/ [Accessed 22 October 2026].

Google Cloud, n.d.a. Agent Identity API roles and permissions. IAM documentation. Available at: https://docs.cloud.google.com/iam/docs/roles-permissions/agentidentity [Accessed 22 October 2026].

Google Cloud, n.d.b. Agent Identity auth manager overview. IAM documentation. Available at: https://docs.cloud.google.com/iam/docs/auth-manager-overview [Accessed 22 October 2026].

Google Cloud, n.d.c. Agent Identity overview. IAM documentation. Available at: https://docs.cloud.google.com/iam/docs/agent-identity-overview [Accessed 22 October 2026].

Google Cloud, n.d.d. Authenticate to Google and Google Cloud MCP servers. Google Cloud MCP servers documentation. Available at: https://docs.cloud.google.com/mcp/authenticate-mcp [Accessed 22 October 2026].

Google Cloud, n.d.e. Authenticate to Google Cloud using an agent's own identity. IAM documentation. Available at: https://docs.cloud.google.com/iam/docs/auth-agent-own-identity [Accessed 22 October 2026].

Google Cloud, n.d.f. Authenticate using 3-legged OAuth with auth manager. IAM documentation. Available at: https://docs.cloud.google.com/iam/docs/auth-with-3lo-v2 [Accessed 22 October 2026].

Google Cloud, n.d.g. BigQuery release notes. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/release-notes [Accessed 22 October 2026].

Google Cloud, n.d.h. Configure Workforce Identity Federation with Microsoft Entra ID and sign in users. IAM documentation. Available at: https://docs.cloud.google.com/iam/docs/workforce-sign-in-microsoft-entra-id [Accessed 22 October 2026].

Google Cloud, n.d.i. Control MCP use with Identity and Access Management. Google Cloud MCP servers documentation. Available at: https://docs.cloud.google.com/mcp/control-mcp-use-iam [Accessed 22 October 2026].

Google Cloud, n.d.j. Data definition language (DDL) statements in GoogleSQL. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language [Accessed 22 October 2026].

Google Cloud, n.d.k. IAM release notes. IAM documentation. Available at: https://docs.cloud.google.com/iam/docs/release-notes [Accessed 22 October 2026].

Google Cloud, n.d.l. Introduction to BigQuery row-level security. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/row-level-security-intro [Accessed 22 October 2026].

Google Cloud, n.d.m. Method: projects.locations.authProviders.credentials.finalize. Agent Identity Credentials API reference. Available at: https://docs.cloud.google.com/iam/docs/reference/agentidentitycredentials/rest/v1/projects.locations.authProviders.credentials/finalize [Accessed 23 October 2026].

Google Cloud, n.d.n. Method: projects.locations.authProviders.credentials.retrieve. Agent Identity Credentials API reference. Available at: https://docs.cloud.google.com/iam/docs/reference/agentidentitycredentials/rest/v1/projects.locations.authProviders.credentials/retrieve [Accessed 23 October 2026].

Google Cloud, n.d.o. Method: projects.locations.authProviders.revokeAuthorization. Agent Identity API reference. Available at: https://docs.cloud.google.com/iam/docs/reference/agentidentity/rest/v1/projects.locations.authProviders/revokeAuthorization [Accessed 22 October 2026].

Google Cloud, n.d.p. Method: token. Security Token Service API reference. Available at: https://docs.cloud.google.com/iam/docs/reference/sts/rest/v1/TopLevel/token [Accessed 22 October 2026].

Google Cloud, n.d.q. Principal identifiers. IAM documentation. Available at: https://docs.cloud.google.com/iam/docs/principal-identifiers [Accessed 22 October 2026].

Google Cloud, n.d.r. REST Resource: projects.locations.authProviders. Agent Identity API reference. Available at: https://docs.cloud.google.com/iam/docs/reference/agentidentity/rest/v1/projects.locations.authProviders [Accessed 22 October 2026].

Google Cloud, n.d.s. REST Resource: projects.locations.authProviders.authorizations. Agent Identity API reference. Available at: https://docs.cloud.google.com/iam/docs/reference/agentidentity/rest/v1/projects.locations.authProviders.authorizations [Accessed 22 October 2026].

Google Cloud, n.d.t. Security functions. BigQuery GoogleSQL reference. Available at: https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/security_functions [Accessed 22 October 2026].

Google Cloud, n.d.u. Set up logging. Gemini Enterprise Agent Platform documentation. Available at: https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/runtime/logging [Accessed 26 October 2026].

Google Cloud, n.d.v. Use Agent Identity with Agent Runtime. Gemini Enterprise Agent Platform documentation. Available at: https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/runtime/agent-identity [Accessed 22 October 2026].

Google Cloud, n.d.w. Use row-level security. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/managing-row-level-security [Accessed 22 October 2026].

Google Cloud, n.d.x. Use the BigQuery MCP server. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/use-bigquery-mcp [Accessed 22 October 2026].

Jones, M., Nadalin, A., Campbell, B., Bradley, J. and Mortimore, C., 2020. OAuth 2.0 Token Exchange. RFC 8693. IETF. Available at: https://datatracker.ietf.org/doc/html/rfc8693 [Accessed 22 October 2026].

Microsoft, 2024. Claims challenges, claims requests and client capabilities. Microsoft identity platform documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity-platform/claims-challenge [Accessed 23 October 2026].

Microsoft, 2025a. ID token claims reference. Microsoft identity platform documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity-platform/id-token-claims-reference [Accessed 22 October 2026].

Microsoft, 2025b. Microsoft Entra authentication and authorization error codes. Microsoft identity platform documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity-platform/reference-error-codes [Accessed 22 October 2026].

Microsoft, 2025c. Microsoft identity platform and OAuth 2.0 On-Behalf-Of flow. Microsoft identity platform documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-on-behalf-of-flow [Accessed 22 October 2026].

Microsoft, 2025d. Microsoft identity platform application authentication certificate credentials. Microsoft identity platform documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity-platform/certificate-credentials [Accessed 22 October 2026].

Microsoft, 2025e. Refresh tokens in the Microsoft identity platform. Microsoft identity platform documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity-platform/refresh-tokens [Accessed 23 October 2026].

Microsoft, 2025f. Secure applications and APIs by validating claims. Microsoft identity platform documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity-platform/claims-validation [Accessed 22 October 2026].

Microsoft, 2025g. Understand the app manifest (Microsoft Graph format). Microsoft identity platform documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity-platform/reference-microsoft-graph-app-manifest [Accessed 22 October 2026].

Microsoft, 2026a. Access token claims reference. Microsoft identity platform documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity-platform/access-token-claims-reference [Accessed 22 October 2026].

Microsoft, 2026b. Configurable token lifetimes in the Microsoft identity platform. Microsoft identity platform documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity-platform/configurable-token-lifetimes [Accessed 22 October 2026].

Microsoft, 2026c. msal 1.39.0: The Microsoft Authentication Library (MSAL) for Python [source code]. PyPI. Available at: https://pypi.org/project/msal/1.39.0/ [Accessed 22 October 2026].

Microsoft, 2026d. Optional claims reference. Microsoft identity platform documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity-platform/optional-claims-reference [Accessed 22 October 2026].

Microsoft, 2026e. Revoke user access in Microsoft Entra ID. Microsoft Entra ID documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity/users/users-revoke-access [Accessed 22 October 2026].

Microsoft, 2026f. Scopes and permissions in the Microsoft identity platform. Microsoft identity platform documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity-platform/scopes-oidc [Accessed 22 October 2026].

Model Context Protocol, 2026a. Authorization. Specification, version 2026-07-28. Available at: https://modelcontextprotocol.io/specification/2026-07-28/basic/authorization [Accessed 22 October 2026].

Model Context Protocol, 2026b. Authorization Security Considerations. Specification, version 2026-07-28. Available at: https://modelcontextprotocol.io/specification/2026-07-28/basic/authorization/security-considerations [Accessed 22 October 2026].

Model Context Protocol, 2026c. Client Registration. Specification, version 2026-07-28. Available at: https://modelcontextprotocol.io/specification/2026-07-28/basic/authorization/client-registration [Accessed 22 October 2026].

Model Context Protocol, 2026d. Enterprise-Managed Authorization. MCP extensions documentation. Available at: https://modelcontextprotocol.io/extensions/auth/enterprise-managed-authorization [Accessed 22 October 2026].

OWASP, 2025. OWASP Top 10 for Agentic Applications for 2026 [PDF]. OWASP GenAI Security Project. Available at: https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/ [Accessed 22 October 2026].

Stevens, A., 2026. C01 - Workload Identity and Attestation. GATE v1.4 knowledge bundle. GitHub. Available at: https://github.com/deterministic-agents/gate-knowledge/blob/main/controls/C01.md [Accessed 22 October 2026].

From a Leaked Pipeline Credential to Bulk Data

Andrew Stevens — Thu, 22 Oct 2026 18:21:24 +0000

Your dbt job's service account has a key file sitting on a shared drive the early data team used, and another copy in a former contractor's laptop backup. The key has no expiry date. Key creation has been blocked in your organisation for a while, but this key predates the block, and it still gets an access token today. Whoever has the file can use the job's access to your BigQuery datasets, unless a VPC Service Controls perimeter limits where the account's requests can come from.

If your platform is older than your key-creation policy, it's worth assuming at least one key like this exists until an inventory says otherwise.

In brief. Blocking key creation doesn't retire keys made before the block. A legacy pipeline key can read, and possibly export, curated BigQuery data. BigQuery's Data Access logs, the record of which tables were read, with which key, and from where, stay in the _Default bucket for 30 days unless you change it, so after that you may not be able to say what a key read. This week: export your service account key inventory with Cloud Asset Inventory, find the user-managed keys with no expiry, and check when each one last authenticated.

This series treats the data platform's own machinery as the target: the service accounts that run jobs, the orchestrator, CI for SQL and dbt, connectors, keys, and exports. It uses the same example platform as The Lakehouse in Production: Microsoft Entra ID for workforce identity, Google Cloud for the data platform (BigQuery, Datastream, Cloud Composer, and dbt), and Azure SQL Database at source. It's a composite, and it already has the hardening from that series' first part, including an organisation policy that blocks key creation and upload.

Part 1 follows one credential to the data it can reach, then goes through the controls that narrow that path, roughly in the order they'd pay off on the example platform. The first one is an inventory, and the rest of the series builds on it.

Why blocking key creation didn't fix this

Google Cloud has two pairs of organisation policy constraints that stop service account keys at the source. iam.managed.disableServiceAccountKeyCreation and its legacy form iam.disableServiceAccountKeyCreation block new keys. iam.managed.disableServiceAccountKeyUpload and iam.disableServiceAccountKeyUpload block uploaded public keys. Organisations created on or after 3 May 2024 get the legacy pair enforced by default (Google Cloud, n.d.aa).

They only apply going forward. Google says they "are not retroactive; they do not affect previously created and configured service accounts" (Google Cloud, n.d.aa). iam.serviceAccountKeyExpiryHours caps key lifetime, but only for keys created after it applies (Google Cloud, n.d.aa). So a key made in the platform's early days keeps working until someone disables or deletes it.

We tested this in a sandbox project in October 2026. With creation blocked, a request for a new key failed with FAILED_PRECONDITION. A key we'd made during a short, approved exception still got a fresh token after the block went back on. We used a client that always asks for a new token, because gcloud can return a cached one, which says nothing about whether the key still works.

So the hardening stops new keys, subject to any exceptions, and finding the old ones is separate work.

From the file to the data

Each hop below says what the grant allows and what record it leaves. Figure 1 shows the same path and records.

Hop 1: the file. The key is a JSON credentials file made for dbt's BigQuery service-account method, which points at it with a keyfile field (dbt Labs, n.d.b). Files like this tend to outlive the projects they were made for. They turn up on a shared drive from a migration, in a laptop backup, or on a branch nobody deleted, and storing or copying them usually leaves no record in Google Cloud. The file store may keep its own access logs, which are worth pulling during an investigation. ATT&CK calls this Credentials In Files, searching "local file systems and remote file shares for files containing insecurely stored credentials" (MITRE, 2026b).

Hop 2: the token. Whoever has the file can authenticate as the service account. Each access token typically expires after an hour (Google Cloud, n.d.ab), but the key doesn't, so it can be used again for as long as it stays valid. Each authentication with the key is counted by the Key Authentication Events metric (Google Cloud, n.d.e). Google can automatically disable keys it finds exposed publicly, and that's on by default (Google Cloud, n.d.aa; Google Cloud, n.d.w), but a private share or a backup isn't public, so it's unlikely to help here. From this point the activity looks like the account's own, which ATT&CK files under valid cloud accounts (MITRE, 2026a). Audit log entries for its calls record the key in authenticationInfo.serviceAccountKeyName, next to the account's email in principalEmail, for services that log it. Google notes that some, including App Engine and Compute Engine, don't (Google Cloud, n.d.o).

Hop 3: the grants. dbt's documentation lists BigQuery Data Editor, BigQuery User, and BigQuery Job User as "the typical starting point" for dbt access (dbt Labs, n.d.b). Data Editor has bigquery.tables.getData, bigquery.tables.updateData, and bigquery.tables.export, and can be granted on a dataset or a single table. Job User has bigquery.jobs.create and can only be granted on a project, folder, or organisation (Google Cloud, n.d.g). On the example platform the job has Data Editor on the platform's datasets and Job User on the project. BigQuery User would add more, and Part 2 measures how much. A dbt job that builds the silver and gold layers (the cleaned and business-ready tables) usually needs to read the layers below and write its own outputs, so the key can reach most of the curated data.

Job User decides where the account can run jobs, not what it can read. bigquery.tables.getData alone lets it read a table's rows through the API's tabledata.list method (Google Cloud, n.d.u), and a query job can be billed to a different project from the one that holds the data (Google Cloud, n.d.f). So removing Job User from your project doesn't contain the key; the dataset grants do. Data Editor can also change and delete those tables, so the key is an integrity risk as well as a confidentiality one. Changes to table data appear as TableDataChange entries in Data Access (Google Cloud, n.d.f).

Hop 4: the reads. A query job reads the tables the account can see. ATT&CK's Databases sub-technique covers this kind of collection (MITRE, 2026c). BigQuery records table reads as TableDataRead entries in the Data Access log, including reads through tabledata.list that run no job, and the job itself as JobInsertion and JobChange entries (Google Cloud, n.d.f). When a job is billed to another project, the read is logged in the project that holds the data while the job's own entries go to the billing project, and the billing project's ID is redacted there unless the two projects share a domain or an organisation (Google Cloud, n.d.f). For BigQuery those logs are on by default and land in the _Default bucket unless you've routed them elsewhere (Google Cloud, n.d.i), and _Default keeps them for 30 days unless someone has changed it (Google Cloud, n.d.j).

Hop 5: the data leaves. The same grants allow more than queries. Exporting a table to Cloud Storage needs bigquery.tables.export, bigquery.jobs.create, and write access to the destination bucket (Google Cloud, n.d.q). The roles above cover the first two. The third is granted on the bucket, and that bucket can sit in a project or organisation you don't control, so your own allow policies don't limit it. A VPC Service Controls perimeter around BigQuery is the usual control for that path. A Principal Access Boundary policy on the account may also keep it to resources in your organisation, for services that enforce one. Any log of the write to the bucket belongs to the bucket's owner.

In ATT&CK, one of the Databases sub-technique's detection analytics names BigQuery among the platforms to watch for unusual table exports (MITRE, 2026c). Transfer Data to Cloud Account covers moving data to another account on the same provider (MITRE, 2025), and applying it to a BigQuery export is our reading. Part 5 covers the export, copy, transfer, and connection paths, and the perimeter controls around them.

None of these hops needs a vulnerability; each uses a permission the account already holds.

Other credentials like this

A legacy key is the clearest example, but the pattern shows up elsewhere. Application-default credentials on an engineer's laptop let tools act as that engineer, with everything the engineer can reach, and the key constraints generally don't apply to them. Later parts cover a short-lived token printed in a CI log (Part 3), an Airflow connection holding a key (Part 4), and a connector vendor's credential (Part 6).

Controls that narrow the path

These are ordered roughly by how much each one narrows this path on the example platform. The right order for you depends on what you already have, your team, and your tooling, and none of them closes the path on its own.

1. Find the keys you already have

The constraint stops new keys. To find the old ones, Cloud Asset Inventory can export service accounts (iam.googleapis.com/ServiceAccount) and their keys (iam.googleapis.com/ServiceAccountKey) as they stood at a point in time you choose (Google Cloud, n.d.p; Google Cloud, n.d.b). Ours came out as newline-delimited JSON, one asset per line. Each key record had its type, origin, and validity window, and disabled keys had "disabled": true.

Listing 1 reads that file. For each user-managed key it reports the key's age, whether it ever expires, whether it's disabled or was uploaded, and whether its account is in the same export. It also writes out the graph the series builds on: accounts and keys as nodes, a disabled flag on each key, and an edge from each key to the account it authenticates as. It skips Google-managed keys. Google holds their private halves in escrow and nobody can access them directly (Google Cloud, n.d.ab), and using them to act as an account is impersonation, which Part 2 covers.

Python

"""Credential inventory over a Cloud Asset Inventory export (Part 1).

Reads newline-delimited JSON exported with --content-type=resource and emits
the series graph format: principals, credentials, and the edges between them,
plus one finding per user-managed key.
"""
import json
import sys
from datetime import datetime, timezone

SERVICE_ACCOUNT = "iam.googleapis.com/ServiceAccount"
KEY = "iam.googleapis.com/ServiceAccountKey"

def parse_time(value):
    # validAfterTime looks like 2023-03-02T09:20:11Z; a bare date means UTC midnight
    parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
    return parsed if parsed.tzinfo else parsed.replace(tzinfo=timezone.utc)

def inventory(lines, as_of):
    accounts, keys = {}, []
    for line in lines:
        if not line.strip():
            continue
        asset = json.loads(line)
        data = asset["resource"]["data"]
        if asset["asset_type"] == SERVICE_ACCOUNT:
            accounts[data["uniqueId"]] = data["email"]
        elif asset["asset_type"] == KEY and data.get("keyType") == "USER_MANAGED":
            parent_id = asset["resource"]["parent"].rsplit("/", 1)[1]
            keys.append((parent_id, data, asset.get("ancestors") or []))

    nodes = {}
    for email in accounts.values():
        principal = f"serviceAccount:{email}"
        nodes[principal] = {"id": principal, "kind": "principal",
                            "attrs": {"in_export": True}}

    edges, findings = [], []
    for parent_id, data, ancestors in keys:
        email = data["name"].split("/serviceAccounts/")[1].split("/keys/")[0]
        principal = f"serviceAccount:{email}"
        credential = f"key:{data['name']}"
        in_export = parent_id in accounts
        # A key with no end time is treated as never expiring, so it's reported
        ends = data.get("validBeforeTime", "9999-")
        no_expiry = ends.startswith("9999-")
        nodes.setdefault(principal, {"id": principal, "kind": "principal",
                                     "attrs": {"in_export": in_export}})
        nodes[credential] = {"id": credential, "kind": "credential",
                             "attrs": {"key_type": "USER_MANAGED",
                                       "disabled": bool(data.get("disabled"))}}
        edges.append({"from": credential, "to": principal,
                      "type": "authenticates_as", "evidence": "asset-export"})
        findings.append({
            "key": data["name"],
            "account": email,
            "age_days": (as_of - parse_time(data["validAfterTime"])).days,
            "no_expiry": no_expiry,
            "expired": not no_expiry and parse_time(ends) <= as_of,
            "disabled": bool(data.get("disabled")),
            "uploaded": data.get("keyOrigin") == "USER_PROVIDED",
            "account_in_export": in_export,
            # Audit logs name the key with this prefix (serviceAccountKeyName)
            "log_key_name": f"//iam.googleapis.com/{data['name']}",
            "project": data["name"].split("/")[1],
            "ancestors": ancestors,
        })

    return {
        "format": "pas-graph/1",
        "nodes": sorted(nodes.values(), key=lambda n: n["id"]),
        "edges": sorted(edges, key=lambda e: (e["from"], e["to"], e["type"])),
        "findings": sorted(findings, key=lambda f: f["key"]),
    }

if __name__ == "__main__":
    path, as_of = sys.argv[1], parse_time(sys.argv[2])
    with open(path, encoding="utf-8") as export:
        print(json.dumps(inventory(export, as_of), indent=2, sort_keys=True))

Rust

//! Credential inventory over a Cloud Asset Inventory export (Part 1).
//!
//! Reads newline-delimited JSON exported with --content-type=resource and
//! emits the series graph format: principals, credentials, and the edges
//! between them, plus one finding per user-managed key.
use chrono::{DateTime, NaiveDate, Utc};
use serde_json::{json, Value};
use std::collections::BTreeMap;

const SERVICE_ACCOUNT: &str = "iam.googleapis.com/ServiceAccount";
const KEY: &str = "iam.googleapis.com/ServiceAccountKey";

pub fn parse_time(value: &str) -> DateTime<Utc> {
    // validAfterTime looks like 2023-03-02T09:20:11Z; a bare date means UTC midnight
    match DateTime::parse_from_rfc3339(value) {
        Ok(t) => t.with_timezone(&Utc),
        Err(_) => NaiveDate::parse_from_str(value, "%Y-%m-%d").expect("RFC 3339 timestamp or date")
            .and_hms_opt(0, 0, 0).unwrap().and_utc(),
    }
}

fn text<'a>(v: &'a Value, field: &str) -> &'a str {
    v[field].as_str().unwrap_or("")
}

pub fn inventory<'a>(lines: impl Iterator<Item = &'a str>, as_of: DateTime<Utc>) -> Value {
    let mut accounts = BTreeMap::new();
    let mut keys = Vec::new();
    for line in lines.filter(|l| !l.trim().is_empty()) {
        let asset: Value = serde_json::from_str(line).expect("one JSON asset per line");
        let data = asset["resource"]["data"].clone();
        match text(&asset, "asset_type") {
            SERVICE_ACCOUNT => {
                accounts.insert(text(&data, "uniqueId").to_string(), text(&data, "email").to_string());
            }
            KEY if text(&data, "keyType") == "USER_MANAGED" => {
                let parent = text(&asset["resource"], "parent");
                let (_, parent_id) = parent.rsplit_once('/').expect("parent is a resource name");
                let ancestors = asset["ancestors"].clone();
                keys.push((parent_id.to_string(), data, ancestors));
            }
            _ => {}
        }
    }

    let mut nodes = BTreeMap::new();
    for email in accounts.values() {
        let principal = format!("serviceAccount:{email}");
        nodes.insert(principal.clone(),
            json!({"id": principal, "kind": "principal", "attrs": {"in_export": true}}));
    }

    let (mut edges, mut findings) = (Vec::new(), Vec::new());
    for (parent_id, data, ancestors) in &keys {
        let name = text(data, "name");
        let email = name.split("/serviceAccounts/").nth(1).expect("key name holds its account")
            .split("/keys/").next().unwrap_or("");
        let principal = format!("serviceAccount:{email}");
        let credential = format!("key:{name}");
        let in_export = accounts.contains_key(parent_id);
        let created = parse_time(text(data, "validAfterTime"));
        // A key with no end time is treated as never expiring, so it's reported
        let ends = data["validBeforeTime"].as_str().unwrap_or("9999-");
        let no_expiry = ends.starts_with("9999-");
        nodes.entry(principal.clone()).or_insert_with(||
            json!({"id": principal, "kind": "principal", "attrs": {"in_export": in_export}}));
        nodes.insert(credential.clone(),
            json!({"id": credential, "kind": "credential", "attrs": {"key_type": "USER_MANAGED",
                "disabled": data["disabled"].as_bool().unwrap_or(false)}}));
        edges.push(json!({"from": credential, "to": principal,
                          "type": "authenticates_as", "evidence": "asset-export"}));
        findings.push(json!({
            "key": name,
            "account": email,
            // Whole days, rounded down, to match Python's timedelta.days
            "age_days": (as_of - created).num_microseconds().expect("age in range")
                .div_euclid(86_400_000_000),
            "no_expiry": no_expiry,
            "expired": !no_expiry && parse_time(ends) <= as_of,
            "disabled": data["disabled"].as_bool().unwrap_or(false),
            "uploaded": text(data, "keyOrigin") == "USER_PROVIDED",
            "account_in_export": in_export,
            // Audit logs name the key with this prefix (serviceAccountKeyName)
            "log_key_name": format!("//iam.googleapis.com/{name}"),
            "project": name.split('/').nth(1).unwrap_or(""),
            "ancestors": if ancestors.is_array() { ancestors.clone() } else { json!([]) },
        }));
    }

    let order = |v: &Value, f: &str| v[f].as_str().unwrap_or("").to_string();
    edges.sort_by_key(|e| (order(e, "from"), order(e, "to"), order(e, "type")));
    findings.sort_by_key(|f| order(f, "key"));
    json!({
        "format": "pas-graph/1",
        "nodes": nodes.into_values().collect::<Vec<_>>(),
        "edges": edges,
        "findings": findings,
    })
}

Listing 1. A service account key inventory over a Cloud Asset Inventory export, emitting the series graph format.

If you run Security Command Center Premium, some of this already reaches you as findings. Security Health Analytics flags user-managed keys (USER_MANAGED_SERVICE_ACCOUNT_KEY) and keys that haven't been rotated for more than 90 days (SERVICE_ACCOUNT_KEY_NOT_ROTATED) (Google Cloud, n.d.af). Event Threat Detection flags service accounts used from a new geography or with a new user agent (Persistence: New Geography, Persistence: New User Agent) with an organisation-level activation, though both rules run on Admin Activity logs, so a key that only reads BigQuery may not trigger them (Google Cloud, n.d.y). Use them if you have them.

Listing 1 does a different job. It runs on an export without a Premium activation, the export is a file you can keep and compare over time, and its output is the graph the rest of the series uses to show what a key can reach. Part 2 compares that graph with Policy Analyzer and with SCC's attack path simulation (Google Cloud, n.d.x).

Run against the example platform, the first finding is the key this post is about: user-managed, on the dbt job's account, created years ago, with a validBeforeTime of 9999-12-31T23:59:59Z. In our export, that end date marked a key with no expiry. One way to retire it is to move the job to another way of authenticating (control 2), run it on the new credential through at least one full schedule cycle, confirm the old key has stopped authenticating, then disable it. Until you delete the key, re-enabling it is your rollback.

To check when a key last authenticated, Google points to the Key Authentication Events metric (Google Cloud, n.d.e). Activity Analyzer's serviceAccountKeyLastAuthentication gives a last-authentication date per key, but it's in Preview, it's queried per project with up to ten keys per filter, and it may miss very recent events (Google Cloud, n.d.ae). If a key was used at times that don't fit the dbt job's schedule, or the audit logs show it used from somewhere the job doesn't run, look into that before anything else.

Listing 2 exports the organisation's service accounts and keys. Listing 1 expects --content-type=resource, as shown.

Shell

gcloud asset export --organization=ORG_ID --content-type=resource \
  --asset-types=iam.googleapis.com/ServiceAccount,iam.googleapis.com/ServiceAccountKey \
  --output-path=gs://BUCKET/keys.ndjson

Listing 2. Exporting service accounts and their keys for Listing 1. Placeholder values.

A few things trip this up in larger organisations. The caller needs cloudasset.assets.exportResource at the organisation, which the Cloud Asset Viewer role includes (Google Cloud, n.d.h). The destination bucket can't have a retention policy and can't be encrypted with customer-managed Cloud KMS keys, so an organisation that enforces either needs a dedicated, locked-down bucket for this. The export takes time to finish: wait for it before copying the file, and expect a second export to the same path to be rejected until the first one finishes or 15 minutes pass (Google Cloud, n.d.p).

Copy the file down from the bucket, then give Listing 1 the file and a date or RFC 3339 time to measure against, for example python3 inventory.py keys.ndjson 2026-10-15. The Python version needs 3.11 or later. Neither version validates its input. Both expect well-formed records, which is what the export writes, and a partial download can cause a parse error. Treat the export file as sensitive, since it's a map of your accounts and keys.

Each finding also carries expired, the key's project and ancestors so you can route it to an owner, and log_key_name, the key's name in the form audit logs use. To list live, enabled keys oldest first: jq '.findings | map(select((.expired or .disabled) | not)) | sort_by(-.age_days)'.

The field names come from a real export. Top-level fields are snake_case (asset_type, update_time), but fields inside resource.data use the IAM API's camelCase (keyType, validAfterTime). A loader that assumes one style will miss fields.

Our sandbox test turned up three limits worth designing around.

Exports lag. An export we took within ten minutes of creating a key and its account had neither. One taken about an hour later had both. That's why Listing 1 reports a key whose account is missing instead of dropping it, and why containment is better confirmed in the IAM API than in an export.
Google-managed keys didn't show up. The IAM API listed one for the test account, and Asset Inventory never did. Listing 1 only reports user-managed keys, so it doesn't matter here, but don't treat the export as a complete list of keys.
The export doesn't say why a key was disabled. After we disabled ours, the IAM API showed disableReason: SERVICE_ACCOUNT_KEY_DISABLE_REASON_USER_INITIATED, while the inventory record only had disabled: true. If the reason matters, for example after an exposure, read the key from the IAM API, which also has an extendedStatus for exposed or compromised keys.

So Listing 1 is only as complete as the export behind it. It can miss keys created shortly before the export, keys in projects outside the export's scope, and credentials that live outside Google Cloud. Treat a finding as a lead to confirm in the IAM API. A clean run tells you about that export, and doesn't prove there are no keys.

2. Get the job off the key file

Which approach fits depends on where the job runs, and one platform often needs more than one:

Scheduled runs inside Google Cloud, such as on Cloud Composer: a service account attached to the runtime. In the dbt-bigquery 1.x adapter, method: oauth picks up application default credentials, so there's no key file to store (dbt Labs, n.d.c). Part 4 covers the Composer side.
CI on GitHub Actions: Google documents Workload Identity Federation for deployment pipelines, where a workflow uses its GitHub OIDC token to get short-lived Google Cloud credentials. Google requires an attribute condition that restricts tokens to your GitHub organisation, and recommends the numeric *_id claims over names (Google Cloud, n.d.m). For a production data role, it's worth narrowing the condition to the repository and branch that run dbt. Part 3 covers CI.
Engineers' laptops: each engineer's own login, with impersonate_service_account pointing at a development account if you want local runs to use one identity (dbt Labs, n.d.c). Impersonation needs a permission held by roles such as Service Account Token Creator (Google Cloud, n.d.ad), so grant it on the development account only, never on the production job's account.
A runner outside Google Cloud whose identity lives in Entra: method: external-oauth-wif, shown in Listing 3. Where that runner is an Azure VM with a managed identity, Google documents federation from the managed identity's token, which avoids a stored secret (Google Cloud, n.d.l). dbt doesn't document that combination, so test it before relying on it.

dbt v2 documents external-oauth-wif with Microsoft Entra ID (dbt Labs, n.d.b), dbt-bigquery's changelog lists a Workload Identity Federation method from the 1.10.0 beta (dbt Labs, n.d.a), and the adapter's source uses the same external-oauth-wif method name (dbt Labs, n.d.c). So dropping the key file may not need a dbt v2 upgrade; check your installed version. Either way, dbt swaps an Entra-issued token for short-lived Google credentials, which needs a workload identity pool and provider configured to trust your Entra application (Google Cloud, n.d.l).

YAML

# Before: dbt 1.x, service account key file
platform:
  target: prod
  outputs:
    prod:
      type: bigquery
      method: service-account
      keyfile: /path/to/dbt-transform-key.json
      project: example-data-platform
      dataset: silver
      location: EU

# After: Workload Identity Federation with Microsoft Entra (dbt v2 syntax; check dbt-bigquery 1.10+ for 1.x)
platform:
  target: external_runner
  outputs:
    external_runner:
      type: bigquery
      method: external-oauth-wif
      database: example-data-platform
      schema: silver
      location: EU
      workload_pool_provider_path: //iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/providers/PROVIDER_ID
      token_endpoint:
        type: entra
        request_url: https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/token
        request_data: "grant_type=client_credentials&client_id={{ env_var('ENTRA_CLIENT_ID') }}&client_secret={{ env_var('ENTRA_CLIENT_SECRET') }}&scope=SCOPE/.default"

Listing 3. A dbt profile before (key file) and after (Workload Identity Federation with Entra, in the syntax dbt v2 documents) for a runner outside Google Cloud. Placeholder values throughout.

The old profile uses the older field names project and dataset. The 1.x adapter maps them to database and schema (dbt Labs, n.d.c); on v2, use the new names.

dbt's documented example authenticates to Entra with a client secret, so a stored credential remains. It now lives on the Entra side, and that secret is what you protect, rotate, and inventory. Entra also accepts a certificate or a federated credential instead of a shared secret for this kind of request (Microsoft, n.d.), but dbt doesn't document either for external-oauth-wif, so check your version before relying on them. Keep the secret in the runtime's secret store and out of development targets, and give it an expiry on the Entra side. Listing 1 won't find it, so it needs its own line in your inventory.

What you get in return is Google credentials that are short-lived and issued per run, with the trust sitting in a pool provider whose conditions you control. Restrict who the provider accepts, with an attribute condition that accepts only the dbt job's own Entra identity, or with app role assignments on the Entra application. Google notes that with Azure's default settings, a new application lets every user in the tenant get access tokens for it (Google Cloud, n.d.l).

The federated identity can hold BigQuery roles directly, or the profile can impersonate a service account through the optional service_account_impersonation_url (dbt Labs, n.d.b). If the profile impersonates, use a new service account, not the one with the legacy key. Federation doesn't retire that key, and while the old account stays in the path you may not be able to disable it without stopping the job. Parts 2 and 3 come back to pool conditions and CI identities.

Plan this as a joint change. Entra administrators register the application, own its secret or certificate, and limit its app role assignments. The Google Cloud IAM team creates the pool and provider, writes the attribute condition, and grants roles to the federated identity. The data team updates the profile for each target and runs the cutover.

3. Keep Google's leaked-key response on

iam.serviceAccountKeyExposureResponse decides what happens when Google detects that a service account key has been "exposed publicly". If it isn't set, it defaults to DISABLE_KEY, which "automatically disables any publicly exposed service account key, and creates an entry in the audit log" (Google Cloud, n.d.w). That has been the default for new and existing customers since 16 June 2024 (Google Cloud, n.d.aa; Liu and Martucci, 2024), and Google says it disables keys found in public repositories in partnership with secret-scanning programmes at sites including GitHub and GitLab (Liu and Martucci, 2024).

It only covers public exposure, and Google says it "doesn't guarantee that it will detect leaked keys" (Google Cloud, n.d.e). The key in this post was never public, so the response wouldn't see it. As with a manual disable, tokens already issued from a key it disables may stay valid until they expire (control 6). Check nobody has set the policy to WAIT_FOR_ABUSE, and consider setting DISABLE_KEY explicitly so the behaviour doesn't depend on a default.

4. Make the keys you keep expire

Some integrations still need a key. For those, iam.serviceAccountKeyExpiryHours caps the lifetime of new keys, picked from a fixed list running from one hour to 2,160 hours (Google Cloud, n.d.aa; Google Cloud, n.d.w). It does nothing for keys that already exist, so it sits alongside the inventory and doesn't replace it.

5. Keep BigQuery's access logs past 30 days

BigQuery's Data Access entries are what tell you which tables a credential read. By default they sit in _Default for 30 days (Google Cloud, n.d.i; Google Cloud, n.d.j). Admin Activity entries, such as a key being created or disabled, go to _Required and stay for 400 days, which you can't change (Google Cloud, n.d.i; Google Cloud, n.d.j).

BigQuery's INFORMATION_SCHEMA.JOBS view keeps job history for the past 180 days, including the account that ran each job, its query text, and the tables it referenced (Google Cloud, n.d.t). It only covers jobs run in that project and region, it doesn't record the key or the caller's IP address, and reads through tabledata.list don't create jobs at all. If a key was misused a few months before anyone noticed, you may be able to see that the key authenticated and which queries the account ran, but not where each request came from, which is usually what separates the key holder's activity from the job's own.

If you want to answer what a credential read beyond 30 days, route BigQuery's audit logs, Data Access included, somewhere with retention set for investigations. That can be a user-defined log bucket, with retention from 1 to 3,650 days and an irreversible lock if you want one (Google Cloud, n.d.k), or a BigQuery dataset you can query. It costs money, and the routing becomes one more thing to protect. Part 8 comes back to that, since the detection path is a pipeline too.

For regulated firms, this gap has a compliance side too. In the EU, DORA asks financial entities to identify, classify, and document their ICT assets (Article 8(1)) and to limit logical access to what's required for approved functions (Article 9(4)(c)) (European Union, 2022). The supporting technical standard asks them to set a retention period for their logs and to log access control and identity management events (Article 12) (European Commission, 2024). In the US, New York's cybersecurity regulation asks covered entities to review all user access privileges at least annually and to promptly terminate access following departures (23 NYCRR 500.7(a)(4) and (a)(6)) (Legal Information Institute, n.d.). A key nobody has inventoried, and logs that expire before an investigation starts, may be hard to square with expectations like these. How they apply depends on your entity and your regulator.

BigQuery is unusual in having Data Access logs on by default (Google Cloud, n.d.i). Token requests for other service accounts, such as GenerateAccessToken, are Data Access entries in the Service Account Credentials API's logs, and you turn those on by enabling Data Access logs for the IAM API or for all services (Google Cloud, n.d.ac). Without them, Part 2's question of which accounts a key could act as has no records behind it.

6. If you find a key that's still live

Disabling a key stops it authenticating. In our test, a disabled key and a deleted key failed the same way, with invalid_grant: Invalid JWT Signature., so the error alone won't tell you which happened. Google also notes that disabling a key "does not revoke short-lived credentials that were issued based on the key" (Google Cloud, n.d.n). Google says the way to revoke those early is to disable or delete the service account itself (Google Cloud, n.d.n), which stops every workload using it.

The sequence below is one to adapt to your own runbooks and roles. Keep the order, except where step 4 says otherwise:

Open an incident and an emergency change. Tell the data platform on-call and the owners of downstream data, since disabling credentials will fail scheduled jobs. Confirm the responder holds the roles to disable keys and service accounts, and record each key's current state before changing anything.
Start preserving evidence. Copy the account's existing Data Access entries out of _Default, for example with gcloud logging copy to a Cloud Storage bucket, and add a sink for new entries, since a sink doesn't pick up entries stored before it existed (Google Cloud, n.d.d). Admin Activity entries stay in _Required for 400 days, so they can wait. A copy takes at least an hour, so let it run while you contain. Also export the account's rows from INFORMATION_SCHEMA.JOBS in each project and region it runs jobs in, which needs bigquery.jobs.listAll, since those age out after 180 days.
Confirm in the IAM API that the key is live (gcloud iam service-accounts keys list --iam-account=SA_EMAIL --managed-by=user), and check when it last authenticated (Google Cloud, n.d.e).
Decide whether you can wait, and name in your runbook who makes that call, because it trades pipeline downtime against continued exposure. If step 3 shows use that doesn't fit the job's schedule or runtime, or you can't rule that out, go to step 5 now and accept that the job stays down until step 6 is done. Containment doesn't delete logs, so don't wait for step 2 to finish. Otherwise, do step 6 first, then step 5, with a deadline in days, and alert on any use of the key until it's disabled.
Disable every user-managed key on the account, not just the one you found. On the can't-wait branch, disable the service account too, since disabled keys leave tokens already issued valid until they expire (Google Cloud, n.d.n). On the other branch, once nothing else uses the account, disabling it as well closes the remaining token window.
Move the job to a different principal (control 2). If control 2 isn't already in place, this is planned work through change control, not an incident step. Don't create a new key as a stopgap.
Review what the account did while the key was live: Data Access and Admin Activity entries for that serviceAccountKeyName, tokens it requested for other service accounts, and anything it created or changed that runs as it. Include TableDataRead entries in each project that holds data, since reads by jobs billed elsewhere are logged only there. If it did get tokens for other service accounts, contain those accounts too, because disabling this one doesn't revoke tokens issued to them.
Check who could recreate the exposure: who can create keys for the account, who can impersonate it, and who can grant an exception to the key-creation constraint.
Delete the disabled keys once nothing needs them, the investigation included, and treat deletion as final. For a key exposed in a repository, Google says to delete it in IAM as quickly as possible (Google Cloud, n.d.e).

For a planned cutover, the rollback until step 9 is to re-enable what you disabled. On the can't-wait branch, re-enabling the key or the account hands the holder their access back too, so roll forward to step 6 instead. Disabled keys can only be re-enabled with gcloud or the API, not the console (Google Cloud, n.d.n).

Where Part 2 takes the graph

Listing 1's output has three parts: nodes (principals, credentials, and later resources), edges with a type and an evidence source, and findings. Part 1 adds one edge type, authenticates_as, from a key to its account. Part 2 adds edges for what an account can reach and whom it can impersonate, plus a traversal over them. Later parts add CI identities, DAG buckets, export paths, connector credentials, decrypt rights, and the log sinks themselves.

The graph shows possible reach. Where a deny policy, a Principal Access Boundary policy, or an IAM Condition might block a step, Part 2's traversal will mark the step as possibly blocked and leave the question open. On the edges it has, that means it over-reports, which suits an audit, but it can't report edges its inputs don't contain. Google's Policy Analyzer answers who has what access under allow policies, though Google notes its results don't account for deny policies (Google Cloud, n.d.a). Part 2 compares the two, case by case.

What we tested, and what to check yourself

We ran the key checks in a sandbox project with no client data, in October 2026, under a short, approved exception to the key-creation policy. Besides the token, export-lag, and disable results described above, we confirmed that:

the export uses the field names and mixed casing described above, a key with no expiry carries an end time of 9999-12-31T23:59:59Z, and the disabled field only appears once a key is disabled;
the two constraint forms report a denied key creation differently. The legacy one returns a PreconditionFailure naming constraints/iam.disableServiceAccountKeyCreation, and the managed one returns an ErrorInfo with reason CUSTOM_ORG_POLICY_VIOLATION.

In your own environment, check the dbt job's actual grants, which may be narrower or wider than dbt's starting point. Check whether anything requested tokens for other service accounts during an exposure, since disabling the key doesn't revoke tokens issued to other accounts. Then check the Entra side of the federation, including where the client secret lives and which identities the provider's condition accepts.

What to alert on

Event	Where it's seen
A service account key created or uploaded (`google.iam.admin.v1.CreateServiceAccountKey`, `UploadServiceAccountKey`)	Admin Activity (Google Cloud, n.d.s)
A key disabled, enabled, or deleted (`DisableServiceAccountKey`, `EnableServiceAccountKey`, `DeleteServiceAccountKey`)	Admin Activity (Google Cloud, n.d.s)
A key disabled by the exposure response: `DisableServiceAccountKey` by `gcp-compromised-key-response@system.gserviceaccount.com`	Admin Activity (Google Cloud, n.d.aa; Google Cloud, n.d.s)
A change to an organisation policy on the key constraints, including a project-level exception or the exposure response set to `WAIT_FOR_ABUSE` (`google.cloud.orgpolicy.v2.OrgPolicy.CreatePolicy`, `UpdatePolicy`, `DeletePolicy`, and the older `cloudresourcemanager.v1` `setOrgPolicy` and `clearOrgPolicy` on organisations, folders, and projects)	Admin Activity (Google Cloud, n.d.v; Google Cloud, n.d.z)
Before a job moves off its key: a call whose `protoPayload.authenticationInfo.serviceAccountKeyName` matches a finding's `log_key_name`, from a caller IP or user agent the dbt runtime doesn't use. Build the baseline from the runtime's actual `callerIp` values, and don't treat `gce-internal-ip` as safe, since a VM outside your organisation can show it too. The user agent is supplied by the client, so treat it as a hint	Data Access and Admin Activity, where the service logs the key name (Google Cloud, n.d.o; Google Cloud, n.d.c)
After a job moves off its key: any authentication with that key	The Key Authentication Events metric (`iam.googleapis.com/service_account/key/authn_events_count`, by `key_id`), alerting on any count; data can take up to three hours to appear (Google Cloud, n.d.e; Google Cloud, n.d.r)
A key's account requesting tokens for another service account (`GenerateAccessToken` and related methods)	Service Account Credentials Data Access logs, once enabled (Google Cloud, n.d.ac)
A BigQuery extract job to a bucket that isn't on your list	BigQuery Data Access: the job's own entries when it runs in your project, and `TableDataRead` entries on your datasets for reads by jobs run elsewhere (Google Cloud, n.d.f); for buckets outside your organisation, Event Threat Detection's BigQuery data extraction rule where you run it (Google Cloud, n.d.y); Part 5 covers exports in detail
A user-managed key with no expiry, or older than your rotation limit	Listing 1, run on a schedule

The ATT&CK mapping

The mapping uses MITRE ATT&CK v20.0, released on 13 October 2026, and only covers the techniques in this post, plus the key creation the alerts above watch for.

Step	Technique	Tactic
The key file on a share or in a backup	T1552.001 Unsecured Credentials: Credentials In Files (MITRE, 2026b)	Credential Access
A new key on a pipeline account	T1098.001 Account Manipulation: Additional Cloud Credentials (MITRE, 2026d)	Persistence and Privilege Escalation
Acting as the service account	T1078.004 Valid Accounts: Cloud Accounts (MITRE, 2026a)	Initial Access, Persistence, Privilege Escalation, and Stealth
Query jobs reading silver and gold	T1213.006 Data from Information Repositories: Databases (MITRE, 2026c)	Collection
An export to a bucket the holder controls	T1537 Transfer Data to Cloud Account (MITRE, 2025), our reading for BigQuery	Exfiltration

This week

Run the inventory, then take these questions to your platform team:

How many user-managed service account keys exist across the organisation, and how many never expire?
Which of them sit on accounts that run data pipelines, and when did each last authenticate?
Do we keep BigQuery Data Access logs for longer than 30 days, and where?
Is BigQuery inside a VPC Service Controls perimeter, and would it stop an export to a bucket outside our organisation?
If we move to Workload Identity Federation with Entra, where does the client secret live, who rotates it, and which identities does the provider accept?
Is the key exposure response explicitly set to DISABLE_KEY?

Part 2 looks at the account itself: what the dbt job's account can reach beyond its own datasets, which other accounts it can act as, and how to work that out from the same export.

References

dbt Labs, n.d.a. CHANGELOG, dbt-bigquery adapter. GitHub. Available at: https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-bigquery/CHANGELOG.md [Accessed 22 October 2026].

dbt Labs, n.d.b. Connect BigQuery to dbt v2. dbt Developer Hub. Available at: https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup [Accessed 15 October 2026].

dbt Labs, n.d.c. credentials.py, dbt-bigquery adapter. GitHub. Available at: https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-bigquery/src/dbt/adapters/bigquery/credentials.py [Accessed 15 October 2026].

European Commission, 2024. Commission Delegated Regulation (EU) 2024/1774 of 13 March 2024 supplementing Regulation (EU) 2022/2554 of the European Parliament and of the Council with regard to regulatory technical standards specifying ICT risk management tools, methods, processes, and policies and the simplified ICT risk management framework. Official Journal of the European Union. Available at: https://eur-lex.europa.eu/eli/reg_del/2024/1774/oj [Accessed 22 October 2026].

European Union, 2022. Regulation (EU) 2022/2554 of the European Parliament and of the Council of 14 December 2022 on digital operational resilience for the financial sector and amending Regulations (EC) No 1060/2009, (EU) No 648/2012, (EU) No 600/2014, (EU) No 909/2014 and (EU) 2016/1011. Official Journal of the European Union, L 333. Available at: https://eur-lex.europa.eu/eli/reg/2022/2554/oj [Accessed 22 October 2026].

Google Cloud, n.d.a. Analyze allow policies. Google Cloud Documentation. Available at: https://docs.cloud.google.com/policy-intelligence/docs/analyze-iam-policies [Accessed 15 October 2026].

Google Cloud, n.d.b. Asset types. Google Cloud Documentation. Available at: https://docs.cloud.google.com/asset-inventory/docs/asset-types [Accessed 15 October 2026].

Google Cloud, n.d.c. AuditLog. Google Cloud Documentation. Available at: https://docs.cloud.google.com/logging/docs/reference/audit/auditlog/rest/Shared.Types/AuditLog [Accessed 22 October 2026].

Google Cloud, n.d.d. Batch and route log entries retroactively. Google Cloud Documentation. Available at: https://docs.cloud.google.com/logging/docs/routing/copy-logs [Accessed 22 October 2026].

Google Cloud, n.d.e. Best practices for managing service account keys. Google Cloud Documentation. Available at: https://docs.cloud.google.com/iam/docs/best-practices-for-managing-service-account-keys [Accessed 15 October 2026].

Google Cloud, n.d.f. BigQuery audit logs overview. Google Cloud Documentation. Available at: https://docs.cloud.google.com/bigquery/docs/reference/auditlogs [Accessed 15 October 2026].

Google Cloud, n.d.g. BigQuery IAM roles and permissions. Google Cloud Documentation. Available at: https://docs.cloud.google.com/bigquery/docs/access-control [Accessed 15 October 2026].

Google Cloud, n.d.h. Cloud Asset Inventory roles and permissions. Google Cloud Documentation. Available at: https://docs.cloud.google.com/iam/docs/roles-permissions/cloudasset [Accessed 22 October 2026].

Google Cloud, n.d.i. Cloud Audit Logs overview. Google Cloud Documentation. Available at: https://docs.cloud.google.com/logging/docs/audit [Accessed 11 October 2026].

Google Cloud, n.d.j. Cloud Logging quotas and limits. Google Cloud Documentation. Available at: https://docs.cloud.google.com/logging/quotas [Accessed 11 October 2026].

Google Cloud, n.d.k. Configure log buckets. Google Cloud Documentation. Available at: https://docs.cloud.google.com/logging/docs/buckets [Accessed 11 October 2026].

Google Cloud, n.d.l. Configure Workload Identity Federation with AWS or Azure VMs. Google Cloud Documentation. Available at: https://docs.cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds [Accessed 15 October 2026].

Google Cloud, n.d.m. Configure Workload Identity Federation with deployment pipelines. Google Cloud Documentation. Available at: https://docs.cloud.google.com/iam/docs/workload-identity-federation-with-deployment-pipelines [Accessed 22 October 2026].

Google Cloud, n.d.n. Disable and enable service account keys. Google Cloud Documentation. Available at: https://docs.cloud.google.com/iam/docs/keys-disable-enable [Accessed 15 October 2026].

Google Cloud, n.d.o. Example logs for service accounts. Google Cloud Documentation. Available at: https://docs.cloud.google.com/iam/docs/audit-logging/examples-service-accounts [Accessed 15 October 2026].

Google Cloud, n.d.p. Export asset metadata to Cloud Storage. Google Cloud Documentation. Available at: https://docs.cloud.google.com/asset-inventory/docs/export-cloud-storage [Accessed 11 October 2026].

Google Cloud, n.d.q. Export table data to Cloud Storage. Google Cloud Documentation. Available at: https://docs.cloud.google.com/bigquery/docs/exporting-data [Accessed 15 October 2026].

Google Cloud, n.d.r. Google Cloud metrics: I through O. Google Cloud Documentation. Available at: https://docs.cloud.google.com/monitoring/api/metrics_gcp_i_o [Accessed 22 October 2026].

Google Cloud, n.d.s. IAM audit logging. Google Cloud Documentation. Available at: https://docs.cloud.google.com/iam/docs/audit-logging [Accessed 15 October 2026].

Google Cloud, n.d.t. JOBS view. Google Cloud Documentation. Available at: https://docs.cloud.google.com/bigquery/docs/information-schema-jobs [Accessed 22 October 2026].

Google Cloud, n.d.u. Method: tabledata.list. Google Cloud Documentation. Available at: https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/list [Accessed 22 October 2026].

Google Cloud, n.d.v. Organization Policy audit logging. Google Cloud Documentation. Available at: https://docs.cloud.google.com/organization-policy/audit-logging [Accessed 22 October 2026].

Google Cloud, n.d.w. Organization policy constraints. Google Cloud Documentation. Available at: https://docs.cloud.google.com/organization-policy/reference/org-policy-constraints [Accessed 15 October 2026].

Google Cloud, n.d.x. Overview of attack exposure scores and attack paths. Google Cloud Documentation. Available at: https://docs.cloud.google.com/security-command-center/docs/attack-exposure-learn [Accessed 22 October 2026].

Google Cloud, n.d.y. Overview of Event Threat Detection. Google Cloud Documentation. Available at: https://docs.cloud.google.com/security-command-center/docs/concepts-event-threat-detection-overview [Accessed 22 October 2026].

Google Cloud, n.d.z. Resource Manager audit logging. Google Cloud Documentation. Available at: https://docs.cloud.google.com/resource-manager/docs/audit-logging [Accessed 22 October 2026].

Google Cloud, n.d.aa. Restricting service account usage. Google Cloud Documentation. Available at: https://docs.cloud.google.com/organization-policy/restrict-service-accounts [Accessed 15 October 2026].

Google Cloud, n.d.ab. Service account credentials. Google Cloud Documentation. Available at: https://docs.cloud.google.com/iam/docs/service-account-creds [Accessed 15 October 2026].

Google Cloud, n.d.ac. Service Account Credentials audit logging. Google Cloud Documentation. Available at: https://docs.cloud.google.com/iam/docs/audit-logging/audit-logging-iamcreds [Accessed 15 October 2026].

Google Cloud, n.d.ad. Service account impersonation. Google Cloud Documentation. Available at: https://docs.cloud.google.com/iam/docs/service-account-impersonation [Accessed 22 October 2026].

Google Cloud, n.d.ae. View recent usage for service accounts and keys. Google Cloud Documentation. Available at: https://docs.cloud.google.com/policy-intelligence/docs/activity-analyzer-service-account-authentication [Accessed 22 October 2026].

Google Cloud, n.d.af. Vulnerability findings. Google Cloud Documentation. Available at: https://docs.cloud.google.com/security-command-center/docs/concepts-vulnerabilities-findings [Accessed 22 October 2026].

Legal Information Institute, n.d. N.Y. Comp. Codes R. & Regs. Tit. 23 § 500.7, Access privileges and management. Cornell Law School. Available at: https://www.law.cornell.edu/regulations/new-york/23-NYCRR-500.7 [Accessed 22 October 2026].

Liu, S. and Martucci, A., 2024. Automatically disabling leaked service account keys: What you need to know. Google Cloud Blog, 15 May. Available at: https://cloud.google.com/blog/products/identity-security/automatically-disabling-leaked-service-account-keys-what-you-need-to-know [Accessed 11 October 2026].

Microsoft, n.d. Microsoft identity platform and the OAuth 2.0 client credentials flow. Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow [Accessed 15 October 2026].

MITRE, 2025. Transfer Data to Cloud Account (T1537). MITRE ATT&CK. Available at: https://attack.mitre.org/techniques/T1537/ [Accessed 15 October 2026].

MITRE, 2026a. Valid Accounts: Cloud Accounts (T1078.004). MITRE ATT&CK. Available at: https://attack.mitre.org/techniques/T1078/004/ [Accessed 15 October 2026].

MITRE, 2026b. Unsecured Credentials: Credentials In Files (T1552.001). MITRE ATT&CK. Available at: https://attack.mitre.org/techniques/T1552/001/ [Accessed 15 October 2026].

MITRE, 2026c. Data from Information Repositories: Databases (T1213.006). MITRE ATT&CK. Available at: https://attack.mitre.org/techniques/T1213/006/ [Accessed 15 October 2026].

MITRE, 2026d. Account Manipulation: Additional Cloud Credentials (T1098.001). MITRE ATT&CK. Available at: https://attack.mitre.org/techniques/T1098/001/ [Accessed 22 October 2026].

What Your Agents Inherit From Your Data Engineering

Andrew Stevens — Wed, 21 Oct 2026 11:58:39 +0000

When one of your customers moves to a higher plan partway through a billing period, several decisions come with the change. Is the first period prorated or charged in full? Does the contract term restart? Does the churn report count a retained customer, or one cancellation and one new sale? If the customer downgrades a few weeks later, does the original discount come back? Each answer is a commercial decision, and each one changes the revenue finance reports and the bill the customer receives.

Cases like the downgrade after an upgrade usually turn up once real customers start doing them, and surfacing the sequences a specification didn't cover is part of the engineering work. How the data platform handles them decides what your analysts can trust. As more organisations put agents to work on their data, it also decides what an agent can explain and what it can safely do.

In short: much of the risk in putting agents on your data sits in the data platform, alongside the model. Agents need the history of your business rules, corrections that are attributed when they're captured, and limits on what they can change. A staged readiness bar (read, supervised write, then write within limits) gives a way to report progress dataset by dataset, and a handful of measures, such as time to explain a variance and restatements after close, show whether the work is paying off.

An agent gets it wrong

Take an illustrative case. A customer contacts support asking why their bill went up, and an AI agent working for the support team looks it up.

It reads last month's invoice total and the current price table, which holds this month's prices and no history of when they changed, and concludes the customer was overcharged. They weren't: a price change took effect between the two invoices. The agent apologises and issues a credit. The credit is larger than the agent's approval threshold, so it issues two smaller credits instead. Neither credit says which product line it belongs to, so both land in a catch-all bucket in revenue reporting. The customer's message also included a pasted line of text that read like an instruction, and the agent followed it, adding a note to the account that the customer was owed a further refund.

Most of what made that sequence costly comes from the platform: no rule history, no attribution at capture, an approval rule that looks at one credit at a time, and free text passed to the agent unscreened. The rest comes from what the agent was allowed to do at all. Each is a design decision, and the first is who decides what the rules are.

Who owns the rules

Business definitions belong to the business. Finance, the commercial teams, and product each have a view on what counts as revenue or an active customer, and settling who has the final say is often the hardest part of the work. Around that decision, analysts work with the business to write definitions down and find where they conflict, analytics engineers turn agreed definitions into models and tests, and data engineers own ingestion, history, and the guarantees the platform makes: whether a definition can be represented in the data, enforced when events arrive, and rebuilt later. The lines vary, and in smaller teams one person may cover all three. Gaps tend to appear where each role assumes another has a case covered, and making sure none goes unowned is part of the engineering job.

One practice that helps is a set of worked examples, agreed before a rule change goes live. The business describes the change. The analyst, usually the person closest to how the rule will be used, works with the engineers to write out a handful of customers' event sequences with the figures the business expects at each step: a customer who upgrades and then downgrades, a customer whose activity straddles the change date, a record that arrives late. The engineers turn those into tests the pipeline has to pass. Everyone works from the same examples, which is usually where disagreements about a rule come out, well before a customer or an auditor finds them.

History is part of the rule

Business rules rarely stay fixed. Prices change, a tier is added, the definition of an active customer is tightened. When a rule changes, someone has to decide whether it applies only from now on or to the past as well. Google Cloud's description of the medallion pattern gives the case for reprocessing: if finance changes how it calculates profit, old data can be re-run to match the new rule (Google Cloud, n.d.e). That fits when the goal is comparability across years. Where the old rule is what customers were actually billed under, a rebuild would describe a past that never happened. Both cases come up in most businesses, so the choice belongs in the rule's metadata, recorded when the rule changes.

In practice that usually means rules live in versioned reference tables with effective dates, joined to events on the date each event happened, with the version of the transformation code recorded alongside the output. A rebuild then applies the rules that were in force at the time, unless the rule is marked to apply retrospectively. That's the history the support agent in the example was missing.

There's a second kind of history as well. Fowler distinguishes actual time, when something was true, from record time, when the business learned it (Fowler, 2021). A usage record that arrives a week late was true on the day it happened and recorded a week later. A price change backdated to the start of the month changes what was true for weeks that have already been invoiced. Keeping both times lets you answer what was actually true on a given date and what you believed when you acted on it, which is the question an auditor or a customer dispute usually turns on.

The familiar warehouse tools cover part of this. Kimball's type 2 approach adds a new row for each change to an attribute, with effective and expiration dates and a current-row indicator (Kimball Group, n.d.), so a report can ask what plan a customer was on when an event happened. That tracks one timeline. If a correction later changes when a plan actually started and the existing type 2 rows are edited to match, what the business believed before the correction is lost, so the record-time view needs its own columns, such as when each row was recorded and when it was superseded. Platform time travel doesn't fill the gap either. BigQuery's time travel window runs from two to seven days, with seven by default (Google Cloud, n.d.a). It's a recovery feature, and record history needs to be modelled explicitly.

Where the lakehouse decisions are made

Google Cloud describes the bronze layer of a medallion lakehouse as the landing zone that keeps incoming data unchanged, silver as the cleaned and standardised source of truth, and gold as refined data tailored to specific business needs (Google Cloud, n.d.e). Within that pattern there are real choices about where each kind of rule goes, and teams make them differently.

Bronze keeps raw data so it can be reprocessed when rules change. That's a reason to keep source records complete, including fields that aren't modelled yet, within limits. Personal data in bronze needs a stated purpose, a retention period, and an erasure process that reaches every layer, and data minimisation applies to it as much as anywhere else. Erasure and history pull against each other. Common approaches, to be confirmed with your privacy counsel, are to keep direct identifiers out of history tables and join on a pseudonymous key instead (pseudonymised data is still personal data, so it still needs a retention rule), or to encrypt personal fields with a key per person and delete the key when the person is erased. Deciding what bronze keeps, for how long, and why, is an early decision that depends on understanding the business.

For rules, a split we often see working is to put anything that changes what an event is into silver, and metric logic into gold. Linking a refund to the sale it reverses, merging duplicate accounts, and assigning the rule version that applied are silver concerns, because every downstream table needs the same answer. How revenue is aggregated for a board report is gold, published with its definition. Some teams keep more business logic in gold, which works well when each definition has one owner. Where several teams build on the same events, pushing event-level rules down into silver tends to reduce the number of slightly different answers.

Reprocessing also has a cost. Rebuilding years of history every time a rule changes can be expensive, so most pipelines process incrementally, with a lookback window that picks up late-arriving records, and keep full rebuilds for rule changes marked as retrospective. Backfills need to be idempotent, so running one twice doesn't double-count. Much of the medallion work we deliver is written in SQL, with whichever orchestration your team already runs, and most of the effort goes into the decisions above.

Corrections and adjustments

Manual corrections show how a decision made at capture affects everything downstream. Every industry has its own version: fee reversals in banking, seller and refund adjustments on a marketplace, virtual-currency grants in a game, goodwill credits and billing corrections almost everywhere. Each one is usually recorded correctly against the account, so the balance is right. What can be missing is the attribute downstream reporting needs, such as the product line, channel, or cost centre the correction belongs to. If an adjustment doesn't reference the record it corrects, reporting has to infer where it belongs, or the amount lands in a catch-all bucket without a clear owner.

The capture screen belongs to the team that owns the source application, usually billing or CRM. The data team's route is a data contract agreed with that team: the fields every adjustment carries, the allowed reason codes, which attribution options each reason code permits, and the approvals required. Attribution needs explicit options because there often isn't one obviously correct answer. A correction might belong to the unit of the original record, the unit that owns the account, the unit that raised it, or a central bucket the business has chosen deliberately. Which option applies to which reason code is a decision for finance and the analyst, written into the contract. In our experience these contracts hold best when a business owner, often in finance, sponsors them, because finance owns the figures the adjustments move.

The source system enforces the parts of the contract that have to hold before money moves: approvals, limits, and idempotency. An approval rule that looks at each credit on its own can be sidestepped by splitting one credit into two, which is what the agent in the example did. A cumulative rule is designed to close that gap: a credit needs approval if it, added to the credits issued or pending on the same account over the previous 24 hours by anyone, would reach the threshold, with the check and the write handled one at a time per account so parallel requests can't both slip under it. Daily limits per creator, set per currency or converted to one reporting currency, cap the total. An idempotency key on every write stops a retried request from issuing the same credit twice; for agents, the key works best when it's derived from the case and the type of action, because an agent that retries by re-planning will often generate a new request, sometimes with a different amount.

The data engineer's part is to check every adjustment again as it lands. An adjustment the billing system has already issued is a fact, so the lakehouse loads it whatever the check finds, and silver keeps reconciling to billing. What the check produces is a set of flags. Rows with attribution flags are reported as unattributed until they're fixed, rather than being assigned to a unit by inference, and every flag is counted.

In plain terms, Listing 1 flags an adjustment if its amount is zero; its reason code is unknown or doesn't allow the attribution chosen; its sign doesn't match the reason; it's held centrally but names a unit, or, if not central, has a missing or unknown unit; it's attributed to an original record that doesn't exist or belongs to another account or unit; its currency isn't covered; a large amount lacks an approval from an authorised approver other than its creator; or its effective date fell in a period finance had already closed when it was recorded. The approver comes from the approval system, joined on the adjustment's ID and version, so an approval for one version doesn't carry over to a larger correction, and the comparison with the creator flags an adjustment approved by whoever raised it.

Python

from dataclasses import dataclass
from datetime import date
from enum import Enum

class Basis(Enum):
    ORIGINATING_RECORD = "originating_record"  # unit of the record being corrected
    ACCOUNT_OWNER = "account_owner"            # unit that owns the account
    RAISED_BY = "raised_by"                    # unit that raised the adjustment
    CENTRAL = "central"                        # held centrally, by decision

@dataclass(frozen=True)
class Adjustment:
    adjustment_id: str
    version: int                   # a correction arrives as a new version
    account_key: str
    amount_minor: int              # signed, in minor units of `currency`
    currency: str                  # ISO 4217 code, e.g. "EUR"
    reason_code: str
    basis: Basis
    unit_key: str | None           # product line, channel, or cost centre; None if central
    original_record_id: str | None
    created_by: str                # person, service, or per-agent identity
    approved_by: str | None        # joined from the approval system on id and version
    effective_date: date           # actual time; may be backdated

@dataclass(frozen=True)
class OriginalRecord:
    account_key: str
    unit_key: str

@dataclass(frozen=True)
class Contract:
    """Agreed with the source system's owners; versioned like any other rule."""
    allowed_bases: dict[str, frozenset[Basis]]  # reason code -> bases it permits
    allowed_sign: dict[str, int]                # reason code -> -1 credit, 1 debit, 0 either
    approval_at_or_above: dict[str, int]        # currency -> minor units
    authorised_approvers: frozenset[str]
    units: frozenset[str]                       # valid unit keys; whether the unit is the
                                                # account's or raiser's own is checked elsewhere

def contract_flags(
    adj: Adjustment,
    contract: Contract,
    originals: dict[str, OriginalRecord],       # record id -> record, already loaded
    closed_through_at_recording: date,          # from the close calendar, as of the date the
                                                # source recorded the adjustment, so reruns agree
) -> list[str]:
    """Reasons an adjustment breaks the contract. The row is loaded either way."""
    flags: list[str] = []
    if adj.amount_minor == 0:
        flags.append("amount must be non-zero")
    threshold = contract.approval_at_or_above.get(adj.currency)
    if threshold is None:
        flags.append(f"currency {adj.currency} not in contract")

    bases = contract.allowed_bases.get(adj.reason_code)
    if bases is None:
        flags.append(f"unknown reason code {adj.reason_code}")
    else:
        if adj.basis not in bases:
            flags.append(f"reason does not permit basis {adj.basis.value}")
        sign = contract.allowed_sign.get(adj.reason_code, 0)
        if sign and adj.amount_minor * sign < 0:
            flags.append("sign does not match reason code")

    unit = (adj.unit_key or "").strip()
    if adj.basis is Basis.CENTRAL:
        if unit:
            flags.append("central adjustments carry no unit")
    elif unit not in contract.units:
        flags.append("missing or unknown unit")

    if adj.basis is Basis.ORIGINATING_RECORD:
        original = originals.get((adj.original_record_id or "").strip())
        if original is None:
            flags.append("original record not found")
        else:
            if original.account_key != adj.account_key:
                flags.append("original record belongs to another account")
            if original.unit_key != unit:
                flags.append("unit does not match the original record")

    if threshold is not None and abs(adj.amount_minor) >= threshold:
        if adj.approved_by is None:
            flags.append("needs an approval")
        elif adj.approved_by == adj.created_by:
            flags.append("approver must differ from creator")
        elif adj.approved_by not in contract.authorised_approvers:
            flags.append("approver not authorised")

    if adj.effective_date <= closed_through_at_recording:
        flags.append("effective date falls in a closed period")
    return flags

Rust

use chrono::NaiveDate;
use std::collections::{HashMap, HashSet};

#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum Basis {
    OriginatingRecord, // unit of the record being corrected
    AccountOwner,      // unit that owns the account
    RaisedBy,          // unit that raised the adjustment
    Central,           // held centrally, by decision
}

impl Basis {
    pub fn as_str(self) -> &'static str {
        match self {
            Basis::OriginatingRecord => "originating_record",
            Basis::AccountOwner => "account_owner",
            Basis::RaisedBy => "raised_by",
            Basis::Central => "central",
        }
    }
}

#[derive(Debug, Clone)]
pub struct Adjustment {
    pub adjustment_id: String,
    pub version: u32,             // a correction arrives as a new version
    pub account_key: String,
    pub amount_minor: i64,        // signed, in minor units of `currency`
    pub currency: String,         // ISO 4217 code, e.g. "EUR"
    pub reason_code: String,
    pub basis: Basis,
    pub unit_key: Option<String>, // product line, channel, or cost centre; None if central
    pub original_record_id: Option<String>,
    pub created_by: String,          // person, service, or per-agent identity
    pub approved_by: Option<String>, // joined from the approval system on id and version
    pub effective_date: NaiveDate,   // actual time; may be backdated
}

#[derive(Debug, Clone)]
pub struct OriginalRecord {
    pub account_key: String,
    pub unit_key: String,
}

/// Agreed with the source system's owners; versioned like any other rule.
#[derive(Debug)]
pub struct Contract {
    pub allowed_bases: HashMap<String, HashSet<Basis>>, // reason code -> bases it permits
    pub allowed_sign: HashMap<String, i8>,              // reason code -> -1 credit, 1 debit, 0 either
    pub approval_at_or_above: HashMap<String, u64>,     // currency -> minor units
    pub authorised_approvers: HashSet<String>,
    pub units: HashSet<String>,                         // valid unit keys; whether the unit is the
                                                        // account's or raiser's own is checked elsewhere
}

/// Reasons an adjustment breaks the contract. The row is loaded either way.
pub fn contract_flags(
    adj: &Adjustment,
    contract: &Contract,
    originals: &HashMap<String, OriginalRecord>, // record id -> record, already loaded
    closed_through_at_recording: NaiveDate,      // from the close calendar, as of the date the
                                                 // source recorded the adjustment, so reruns agree
) -> Vec<String> {
    let mut flags = Vec::new();
    if adj.amount_minor == 0 {
        flags.push("amount must be non-zero".to_string());
    }
    let threshold = contract.approval_at_or_above.get(&adj.currency).copied();
    if threshold.is_none() {
        flags.push(format!("currency {} not in contract", adj.currency));
    }

    match contract.allowed_bases.get(&adj.reason_code) {
        None => flags.push(format!("unknown reason code {}", adj.reason_code)),
        Some(bases) => {
            if !bases.contains(&adj.basis) {
                flags.push(format!("reason does not permit basis {}", adj.basis.as_str()));
            }
            let sign = contract.allowed_sign.get(&adj.reason_code).copied().unwrap_or(0);
            if sign != 0 && adj.amount_minor.signum() == -i64::from(sign) {
                flags.push("sign does not match reason code".to_string());
            }
        }
    }

    let unit = adj.unit_key.as_deref().map(str::trim).unwrap_or("");
    if adj.basis == Basis::Central {
        if !unit.is_empty() {
            flags.push("central adjustments carry no unit".to_string());
        }
    } else if !contract.units.contains(unit) {
        flags.push("missing or unknown unit".to_string());
    }

    if adj.basis == Basis::OriginatingRecord {
        let id = adj.original_record_id.as_deref().map(str::trim).unwrap_or("");
        match originals.get(id) {
            None => flags.push("original record not found".to_string()),
            Some(original) => {
                if original.account_key != adj.account_key {
                    flags.push("original record belongs to another account".to_string());
                }
                if original.unit_key != unit {
                    flags.push("unit does not match the original record".to_string());
                }
            }
        }
    }

    if threshold.is_some_and(|t| adj.amount_minor.unsigned_abs() >= t) {
        match &adj.approved_by {
            None => flags.push("needs an approval".to_string()),
            Some(a) if *a == adj.created_by => {
                flags.push("approver must differ from creator".to_string())
            }
            Some(a) if !contract.authorised_approvers.contains(a) => {
                flags.push("approver not authorised".to_string())
            }
            Some(_) => {}
        }
    }

    if adj.effective_date <= closed_through_at_recording {
        flags.push("effective date falls in a closed period".to_string());
    }
    flags
}

Listing 1. Checking an adjustment against its data contract as it lands in silver. The adjustment is loaded either way; the check returns the reasons it breaks the contract.

Both implementations apply the same rules and return the same messages. Listing 1 looks at one adjustment at a time, so it can't see a credit split into two. Catching splits after the fact needs a separate aggregate test, for example summing credits per account over a rolling 24 hours across every creator.

On a SQL-first platform, the rules that compare an adjustment with reference tables are usually written as queries that return the failing rows. A Dataform assertion is a data quality query that fails if it returns any rows (Google Cloud, n.d.d), and dbt's tests work the same way, selecting the failing rows (dbt Labs, n.d.). Failing the run suits rules that should stop a pipeline. Contract flags are expected in normal operation, so Listing 2 is better kept as a view of open flags whose count is monitored. It covers the reason, basis, and unit rules for the latest version of each adjustment. The reference tables are shown current-only for brevity; in practice they're versioned and joined on the adjustment's effective date, like any other rule.

-- Open attribution flags on the latest version of each adjustment. Used as a
-- monitored view; as a Dataform assertion, any row returned would fail the run.
-- Reference tables are shown current-only and keyed uniquely, for brevity.
WITH latest AS (
  SELECT *,
    ROW_NUMBER() OVER (PARTITION BY adjustment_id ORDER BY version DESC) AS rn
  FROM silver.adjustments
)
SELECT a.adjustment_id, a.version, a.reason_code, a.basis, a.unit_key
FROM latest AS a
LEFT JOIN ref.reason_bases AS rb
  ON rb.reason_code = a.reason_code AND rb.basis = a.basis
LEFT JOIN ref.units AS u
  ON u.unit_key = TRIM(a.unit_key)
WHERE a.rn = 1
  AND (
    rb.reason_code IS NULL                                         -- reason doesn't permit this basis
    OR (a.basis = 'central' AND NULLIF(TRIM(a.unit_key), '') IS NOT NULL)
    OR (a.basis != 'central' AND u.unit_key IS NULL)               -- missing or unknown unit
  )

A correction to an adjustment arrives as a new version. Reporting uses the latest version, and the earlier ones are kept, so the platform can still show what it believed before, in line with the history section above. An adjustment whose effective date falls in a closed period is flagged for finance to decide whether it's carried into the open period or the closed one is restated.

Flags only help if someone works them. The contract should say who clears them (usually the source team for content problems and the data team for pipeline problems), how quickly, and how a fixed adjustment comes back, normally as a new version through the same check. The count, value, and age of open flags then becomes one of the measures below.

Adjustments captured before the contract existed need a backfill rule. Where an adjustment references an original record, the attribution can usually be derived from it. Where it doesn't, the safer choice is the central bucket under a reason code such as BACKFILL_UNATTRIBUTED, so it stays visible in reporting and unit figures don't absorb adjustments by inference.

When agents read the data

An analyst who finds an ambiguous column can ask a colleague. An agent answering a question has only what the platform exposes to it, so the work above becomes the agent's context. Agents tend to work best when they read through governed tools and the same semantic layer people use, which holds the agreed metric definitions, rather than writing raw SQL against tables. A catalog adds descriptions, a business glossary, and lineage. On Google Cloud, Knowledge Catalog (formerly Dataplex Universal Catalog) provides a business glossary, data quality checks, and lineage across the platform (Google Cloud, n.d.c). Lineage matters to regulators as well as to agents. In its January 2026 newsletter on the BCBS 239 principles for risk data aggregation, the Basel Committee describes data lineage, the traceability of data from its origin to its final use, as important for confirming data quality (Basel Committee on Banking Supervision, 2026).

Identity needs two parts. The agent needs its own identity, so its actions can be audited, limited, and scoped to the tools it's allowed to use. When it reads data for a member of staff, it also needs to act with that person's access, through pass-through or delegated credentials, which limits what it shows them to what they could see themselves. What the agent can reach is then the overlap of the two: only what its own scope allows and the person can see. Audit records should show both, which agent acted and for whom. With delegated credentials, the platform's own logs usually show the person as the principal, so recording the agent as well typically means labelling its queries or logging at the tool layer. Controls such as BigQuery's row-level security, which filters the rows a principal can read based on conditions you define (Google Cloud, n.d.b), then apply as they would to the person. An agent running only under its own broad service account sees what the service account sees, which is usually more than any one user should.

Customer-facing agents, like the one in the example, act for someone with no identity on your data platform. There, the agent's tools do the scoping: the customer's account comes from their authenticated session, and every tool the agent can call is bound to that account, which helps keep lookups to that customer's records even when a message tries to point the agent elsewhere.

Free text needs care too. Customer messages, reason notes on adjustments, and comments in source records can all carry instructions an agent might follow, as the pasted line in the example did. Screening inputs and outputs for prompt injection and personal data is one layer. It can be bypassed, so it works alongside limits on what the agent can do at all: which tools it has, which records they reach, and how much it can change. A safer assumption is that some free text will be read as instructions, and to make sure anything that follows stays within those limits.

One example from our own work shows these pieces together. A bank wanted to roll out generative AI to its staff without weakening its privacy and security controls. Its data platform brought data from many sources together, and its agents were set up to inherit the same role-based access and data access policies as employees. Agents used pass-through authentication, so retrieval was limited to documents the user could already open in the source system. Inputs were screened for prompt injection and outputs for personal data. With access controls inherited instead of rebuilt, the project could spend more of its time on adoption.

When agents write

Agents also take actions, and those actions usually land in operational systems, the billing platform or the CRM, before they reach the lakehouse. An agent that issues a goodwill credit, corrects a charge, or changes a customer's plan is making a manual adjustment by another name, and the same contract applies. In Listing 1, created_by holds the agent's own identity. Approvals for agent-raised adjustments come from a person, or from a policy service owned outside the agent's team, which mirrors the segregation of duties finance and audit teams already expect for manual entries. The cumulative approval rule and daily limits in the source system are designed to catch the split credit from the example, and an idempotency key derived from the case means a retried action doesn't issue the credit twice. The pipeline's checks are the second line, because they see an adjustment only after it has happened.

The worked examples are useful again here. The event sequences the analyst and engineers wrote for a rule change, with the expected figures at each step, make good evaluation cases for an agent that applies or explains the same rule. Passing them shows the pipeline and the agent agree on the cases the business cared enough to write down. An agent's output can vary between runs, so each case is worth running several times, and again whenever the rule changes.

A readiness bar for agents

A staged bar turns agent readiness into something that can be reported per dataset or use case, for example as the share of gold tables that meet the read stage. One version looks like this.

Read. Before an agent reads from a dataset:

Published definitions for the metrics it uses, in a semantic layer or catalog, with a named owner.
Rule versions and effective dates recorded, so answers about the past use the rules that applied.
Lineage from the figures it reads back to their sources.
Its own identity, plus delegated access or tool-level scoping, limited to what the person it serves can see.
Screening for free-text inputs and outputs.

Supervised write. Before an agent proposes changes:

A data contract for the records it creates, enforced at capture and checked again on landing.
Every proposed write approved by a person before it's applied.
An owner for contract flags, with an agreed time to clear them.
Worked examples used as evaluation cases, rerun whenever the rule changes.

Write within limits. Before an agent acts without per-action approval:

Approval from a separate identity above a cumulative threshold, and daily limits per agent.
Idempotency keys on every write.
A period of supervised writes with a low, agreed flag rate.
A way to pause the agent and revoke its access quickly.

How to tell whether it's working

A few measures show whether the work is paying off. Where one isn't tracked yet, the first step is to start recording it.

The count, value, and age of flagged or unattributed adjustments, since unattributed amounts are a misstatement risk in unit reporting.
Time to explain a variance when finance asks, which feeds into how long the close takes.
How much analyst time goes to the business's questions instead of rebuilding data.
Restatements after a period has closed. In regulated industries these matter to supervisors as well as auditors.
Lead time from an agreed rule change to correct figures in production. A long lead time leaves a gap between what customers are billed and what reporting shows, and gaps like that tend to turn into disputes.
The share of gold tables that meet the read stage of the bar, the headline measure of agent readiness.

The value of the role

The data engineer who asks what happens at a plan change, and keeps the history that answers it, is looking after things customers and analysts notice directly. For customers, that shows up as a bill or a balance they can trust and get explained when they ask. For analysts, it shows up as history they can query without rebuilding it first. An agent built on that platform can answer the plan-change question with the rules that applied at the time.

A practical first step is to take one dataset an agent will read and work through the read stage of the bar for it. Sakura Sky's Data & AI practice builds and runs data platforms from the lakehouse to the agents that use them, and on the platforms we run, we review the readiness bar and these measures with your teams as part of that work.

References

Basel Committee on Banking Supervision, 2026. Implementation of the Principles for effective risk data aggregation and risk reporting (BCBS 239 Principles). Basel Committee newsletter, Bank for International Settlements, 6 January. Available at: https://www.bis.org/publ/bcbs_nl36.htm [Accessed 21 October 2026].

dbt Labs, n.d. Add data tests to your DAG. dbt Developer Hub. Available at: https://docs.getdbt.com/docs/build/data-tests [Accessed 21 October 2026].

Fowler, M., 2021. Bitemporal History. martinfowler.com, 7 April. Available at: https://www.martinfowler.com/articles/bitemporal-history.html [Accessed 21 October 2026].

Google Cloud, n.d.a. Data retention with time travel and fail-safe. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/time-travel [Accessed 21 October 2026].

Google Cloud, n.d.b. Introduction to BigQuery row-level security. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/row-level-security-intro [Accessed 21 October 2026].

Google Cloud, n.d.c. Knowledge Catalog overview. Google Cloud documentation. Available at: https://docs.cloud.google.com/dataplex/docs/introduction [Accessed 21 October 2026].

Google Cloud, n.d.d. Test data quality. Dataform documentation. Available at: https://docs.cloud.google.com/dataform/docs/assertions [Accessed 21 October 2026].

Google Cloud, n.d.e. What is medallion architecture? Google Cloud. Available at: https://cloud.google.com/discover/what-is-medallion-architecture [Accessed 21 October 2026].

Kimball Group, n.d. Type 2: Add New Row. Kimball Dimensional Modeling Techniques. Available at: https://www.kimballgroup.com/data-warehouse-business-intelligence-resources/kimball-techniques/dimensional-modeling-techniques/type-2/ [Accessed 21 October 2026].

From an Entra Sign-In to an Agent Session

Andrew Stevens — Mon, 19 Oct 2026 12:44:36 +0000

When an employee asks your operations agent to reply to a ticket, the agent has no idea who they are. Entra gives your web front end a token, your backend checks it, and then the backend calls Agent Runtime as itself and passes along a user ID it picked. Everything that depends on the employee, including the policy check from Part 1, rests on that hand-off. Get it wrong, or let something other than the backend call the agent, and the policy check runs with whatever task and flags the caller hands it.

This part walks through that hand-off, from the sign-in to the session. It covers what the backend checks on the Entra token and how it remembers which employee owns which session. It covers who else can reach the agent through IAM, which turned out to be a lot more roles than I expected, and a narrower version of ADK's app template that limits what a call can change. Then it covers a step-up sign-in for approvals, and where Workforce Identity Federation fits, which for this agent is at most one task of the three. The checks at the end mark the four I'd hold a release for.

The hand-off

Figure 1, below, is the main path. The front end gets a token from Entra and sends it with every request. The backend checks it, works out which task the request is for, makes sure the employee owns the session, and calls the agent with its own Google Cloud identity. The agent keeps its session state, including the policy check's keys, in Agent Platform Sessions.

Two things in that picture matter more than the rest. The agent never sees the Entra token, so it knows who the employee is only because the backend says so. And nothing but IAM stops something other than the backend from making the same call.

The token Entra issues

The agent API gets its own app registration in Entra with one delegated scope, which I'll call Agent.Use. The front end asks for a token with that scope and passes it on without looking inside. That is what Microsoft asks for: client applications "should treat them as opaque strings" and "shouldn't attempt to validate access tokens", because the token belongs to the API (Microsoft, 2025a).

One setting on the API's registration decides which token format turns up, and the default may not be the one you expect. Web APIs that only accept Entra accounts get v1.0 tokens unless you change requestedAccessTokenVersion, where "[t]he values of null and 1 result in v1.0 tokens, and the value of 2 results in v2.0 tokens" (Microsoft, 2025a). The two formats use different issuers, allow different things in aud, and name the client app differently (azp in v2.0, appid in v1.0) (Microsoft, 2026a). I'd set it to 2 and have the backend refuse anything else, so there's only one format to get right. The code below assumes that.

What the backend checks

Table 1 is the list, in the order the backend runs it. A JWT library handles the first five if you configure it properly. The rest are specific to Entra and to this agent.

Check	What it rules out
The algorithm is RS256, from a list the backend sets rather than the token's header	Unsigned tokens, and HMAC tokens keyed with the public key (Sheffer, Hardt and Jones, 2020, sections 2.1 and 3.1)
The signature verifies with a key from the tenant's published key set, chosen by `kid`	Forged tokens. Entra rotates its keys and Microsoft asks applications to handle that automatically (Microsoft, 2025a)
`iss` is `https://login.microsoftonline.com/{tenant}/v2.0` for the company's tenant	Tokens from another tenant, and v1.0 tokens
`aud` is the agent API's client ID	Tokens issued for another API, or for the front end itself. Microsoft says to verify `aud` "[b]efore validating claims" (Microsoft, 2025c)
`exp` and `nbf`, with a minute of leeway	Expired tokens and tokens not yet valid
`tid`, `oid`, `iat`, and `ver` are present, and `ver` is `2.0`	Tokens the rest of the checks can't evaluate
`scp` is present and `idtyp` isn't `app`	App-only tokens. `scp` is "[o]nly included for user tokens", and the optional `idtyp` claim is `app` for an app-only token (Microsoft, 2026a; Microsoft, 2026c)
`azp` is the front end's client ID	Tokens obtained through other registered client apps. It doesn't prove the request came from your front end: a browser app is a public client, and Microsoft cautions that delegated user tokens "can potentially be obtained by entities other than the application" (Microsoft, 2025c). With a confidential backend-for-frontend, `azpacr` of `1` or `2` shows the client authenticated with a secret or certificate (Microsoft, 2026a)
`scp` contains `Agent.Use`	Tokens for the API that didn't ask for this scope
`acct` is `0`, unless the task allows guests	Guest accounts, for which the optional `acct` claim is `1` (Microsoft, 2026c). Requiring `0` also fails closed if the claim isn't configured

Table 1. The backend's checks on each request, in order.

You only get idtyp and acct if you configure them as optional claims on the API's registration. Without the include_user_token property, idtyp shows up only on app-only tokens, which is all this check needs (Microsoft, 2026c), and a missing scp catches the same case if nobody configured idtyp.

The ADK user_id is built from tid and oid. Microsoft's validation guidance says to "use the immutable claim values tid and oid as a combined key", and never to decide access on email, preferred_username, or unique_name (Microsoft, 2025c). sub looks tempting but it is "a pairwise identifier that's unique to a particular application ID", so a second app reading the same user sees a different value, while oid stays put across applications (Microsoft, 2026a). The client has to request the profile scope to get either tid or oid (Microsoft, 2026a).

Listing 1 is Table 1 in code, Python with PyJWT and Rust with the jsonwebtoken crate. None of the Microsoft pages I read name a Python library for validating access tokens in an API, so the choice is mine. The two libraries don't agree on defaults. PyJWT checks nbf, but only "if present in payload" (PyJWT, n.d.), so the listing makes it required. jsonwebtoken 11.1.0 doesn't check nbf at all unless you set validate_nbf (jsonwebtoken, n.d.), and one of our tests failed until the Rust code did.

The entra_keys function reads the key set's location from the tenant's metadata document instead of hard-coding it. When a token names a key it hasn't seen, PyJWKClient fetches the set again, but no more than once per cooldown period (30 seconds by default), so a token signed with a freshly rotated key can be turned away for up to that long (PyJWT, n.d.). If your API uses claims mapping, Microsoft says to add an appid parameter to the metadata URL, because the keys are different (Microsoft, 2025a).

Python

from __future__ import annotations

import base64
import json
import secrets
import threading
import time
import urllib.request
from dataclasses import dataclass, field
from typing import Callable, Protocol

import jwt

class Rejected(Exception):
    """The token or the request is not acceptable. The reason is for logs."""

@dataclass(frozen=True)
class EntraConfig:
    tenant_id: str
    api_client_id: str          # the agent API's app registration (aud in v2.0)
    allowed_clients: frozenset  # client apps allowed to call the API (azp)
    required_scope: str         # delegated scope the client must hold
    allow_guests: bool = False
    leeway_seconds: int = 60

    @property
    def issuer(self) -> str:
        return f"https://login.microsoftonline.com/{self.tenant_id}/v2.0"

@dataclass(frozen=True)
class Employee:
    tenant_id: str
    object_id: str
    client_id: str
    claims: dict = field(repr=False, compare=False)

    @property
    def user_id(self) -> str:
        # oid is the same for this user across apps in the tenant; tid
        # keeps it unique if more than one tenant is ever accepted.
        return f"{self.tenant_id}:{self.object_id}"

KeyResolver = Callable[[str], object]   # token -> verification key

def entra_keys(tenant_id: str,
               authority: str = "https://login.microsoftonline.com"
               ) -> KeyResolver:
    """Signing keys from the tenant's OpenID Connect metadata.

    PyJWKClient caches the key set and fetches it again when a token names
    a key ID it hasn't seen, which covers Entra's key rotation.
    """
    meta = f"{authority}/{tenant_id}/v2.0/.well-known/openid-configuration"
    with urllib.request.urlopen(meta, timeout=10) as r:
        jwks_uri = json.load(r)["jwks_uri"]
    client = jwt.PyJWKClient(jwks_uri, timeout=5, cooldown_duration=30)
    return lambda token: client.get_signing_key_from_jwt(token).key

def validate_token(token: str, cfg: EntraConfig, key_for: KeyResolver) -> Employee:
    try:
        alg = jwt.get_unverified_header(token).get("alg")
        if alg != "RS256":
            raise Rejected(f"unexpected alg {alg!r}")
        claims = jwt.decode(
            token,
            key_for(token),
            algorithms=["RS256"],
            audience=cfg.api_client_id,
            issuer=cfg.issuer,
            leeway=cfg.leeway_seconds,
            options={"require": ["exp", "nbf", "iat", "aud", "iss",
                                 "tid", "oid", "ver"]},
        )
    except jwt.PyJWTError as e:
        raise Rejected(f"token rejected: {e}") from e

    if claims["ver"] != "2.0":
        raise Rejected("not a v2.0 token")
    if claims["tid"] != cfg.tenant_id:
        raise Rejected("wrong tenant")
    if claims.get("idtyp") == "app" or "scp" not in claims:
        raise Rejected("app-only token")
    if claims.get("azp") not in cfg.allowed_clients:
        raise Rejected("client app not allowed")
    if cfg.required_scope not in str(claims["scp"]).split():
        raise Rejected("missing scope")
    if not cfg.allow_guests and str(claims.get("acct")) != "0":
        raise Rejected("guest account, or acct claim not configured")
    return Employee(claims["tid"], claims["oid"], claims["azp"], claims)

Rust

//! Entra access-token checks for the company backend.
//!
//! `jsonwebtoken` verifies the signature, algorithm, audience, issuer, and
//! lifetime. The Entra-specific checks that follow are what turn a valid
//! token into an employee the backend will bind a session to.

use jsonwebtoken::{Algorithm, DecodingKey, Validation, decode, decode_header};
use serde::Deserialize;

pub struct EntraConfig {
    pub tenant_id: String,
    pub api_client_id: String,        // aud in v2.0 tokens
    pub allowed_clients: Vec<String>, // azp
    pub required_scope: String,
    pub allow_guests: bool,
}

#[derive(Debug, Deserialize)]
struct Claims {
    tid: String,
    oid: String,
    ver: String,
    azp: Option<String>,
    scp: Option<String>,
    idtyp: Option<String>,
    acct: Option<u8>,
    #[allow(dead_code)]
    iat: u64, // required: a token without it fails to deserialize
}

#[derive(Debug, PartialEq)]
pub struct Employee {
    pub tenant_id: String,
    pub object_id: String,
    pub client_id: String,
}

impl Employee {
    pub fn user_id(&self) -> String {
        format!("{}:{}", self.tenant_id, self.object_id)
    }
}

#[derive(Debug, PartialEq)]
pub enum Rejected {
    Token(String),
    NotV2,
    WrongTenant,
    AppOnly,
    ClientNotAllowed,
    MissingScope,
    Guest,
}

pub fn validate(token: &str, key: &DecodingKey, cfg: &EntraConfig) -> Result<Employee, Rejected> {
    let bad = |e: jsonwebtoken::errors::Error| Rejected::Token(e.to_string());
    if decode_header(token).map_err(bad)?.alg != Algorithm::RS256 {
        return Err(Rejected::Token("unexpected alg".into()));
    }
    let mut v = Validation::new(Algorithm::RS256);
    v.set_audience(&[&cfg.api_client_id]);
    v.set_issuer(&[format!("https://login.microsoftonline.com/{}/v2.0", cfg.tenant_id)]);
    v.set_required_spec_claims(&["exp", "nbf", "aud", "iss"]);
    v.validate_nbf = true; // off by default in this crate
    let c = decode::<Claims>(token, key, &v).map_err(bad)?.claims;

    if c.ver != "2.0" {
        return Err(Rejected::NotV2);
    }
    if c.tid != cfg.tenant_id {
        return Err(Rejected::WrongTenant);
    }
    if c.idtyp.as_deref() == Some("app") || c.scp.is_none() {
        return Err(Rejected::AppOnly);
    }
    let client = c.azp.unwrap_or_default();
    if !cfg.allowed_clients.contains(&client) {
        return Err(Rejected::ClientNotAllowed);
    }
    if !c.scp.unwrap().split(' ').any(|s| s == cfg.required_scope) {
        return Err(Rejected::MissingScope);
    }
    if !cfg.allow_guests && c.acct != Some(0) {
        return Err(Rejected::Guest); // a guest, or acct not configured
    }
    Ok(Employee { tenant_id: c.tid, object_id: c.oid, client_id: client })
}

Listing 1. Validating the Entra access token and deriving the ADK user ID. The Rust version takes the decoding key as an argument and leaves key fetching to the caller.

The tests sign tokens with local RSA keys. They cover every row of Table 1, plus an unsigned token, an HMAC token keyed with the public key's PEM, an unknown kid, a key set served from a local metadata endpoint, and a token where sub changes but oid doesn't. That shows the code does what Table 1 says. Whether your tenant actually issues the claims Table 1 expects is something only a real token from your tenant can tell you.

Why the agent takes the backend's word

The backend calls Agent Runtime with a Google Cloud OAuth access token for its own service account, the same kind of bearer token Google's examples send (Google Cloud, n.d.r). The Entra token goes no further than the backend. All the agent learns about the employee is the user_id argument, and Google's documentation for it says "[c]hoose your own user ID with a character limit of 128" (Google Cloud, n.d.r). I found nothing in the documentation suggesting Agent Runtime checks it against anything.

So in practice the backend is the agent's identity provider, whatever it says on the architecture diagram. Session lists, session ownership, and per-user state (the user: prefix) are all keyed on a string the backend supplies, and the agent has no Entra token to compare it with. If someone else can make the same call with a different user_id, then as far as the agent can tell, they are that employee. That leaves two questions: who else can make the call, and what else can the call carry.

Who else can call the agent

Google's sharing page says that to share an agent "you grant the aiplatform.reasoningEngines.query permission on the specific agent resource", and suggests a custom role with only that permission in it. It also warns that most default ADK agents trust their front end, "which gives the frontend full control of sessions and users" (Google Cloud, n.d.q). Part 1 recommended that custom role for the backend, and said only the backend and the agent's own identity should hold the query and session permissions. When I went through the roles reference for this part, I found far more roles carrying them than I'd assumed, and the Part 1 rule needs two exceptions in practice: a break-glass group and Google's own service agents. Both come up below.

As of 19 October 2026, the permissions reference lists aiplatform.reasoningEngines.query in the basic Owner, Editor, and Viewer roles (and in Admin, Writer, and Reader), in Agent Platform Viewer (roles/aiplatform.viewer) and Agent Platform User, and in Data Scientist, ML Engineer, and Support User, among others (Google Cloud, n.d.f). Most of the same roles also carry aiplatform.reasoningEngineRuntimeRevisions.query, which talks to one specific revision of the agent (Google Cloud, n.d.f). Revisions are a Preview feature, and querying a revision's path "explicitly bypasses traffic rules", so an older revision stays reachable for as long as it's active (Google Cloud, n.d.i). Session and memory permissions are spread just as widely. Table 2 is what that means for this agent.

Permission	Example roles that carry it	What a holder can do to this agent
`aiplatform.reasoningEngines.query`, `aiplatform.reasoningEngineRuntimeRevisions.query`	Owner, Editor, Viewer, Agent Platform User, Data Scientist, ML Engineer, Support User; Agent Platform Viewer for the first	Call the agent, or one of its revisions, with any `user_id`: create a session with chosen state, resume another employee's session, and, with the default template, search or add to any employee's memory
`aiplatform.sessions.get`, `aiplatform.sessions.list`, `aiplatform.sessionEvents.list`	Viewer, Agent Platform Viewer, Agent Platform User; Security Reviewer and Security Admin for some	Read sessions and their event history through the Sessions API, including tool results such as query rows
`aiplatform.sessions.create`, `aiplatform.sessions.update`, `aiplatform.sessionEvents.append`	Editor, Agent Platform User	Write sessions directly, including events that change state
`aiplatform.memories.get`, `aiplatform.memories.list`, and the rest of `aiplatform.memories.*`	Viewer, Agent Platform Viewer, Security Reviewer, and Security Admin (read); Editor and Agent Platform User (write)	Read or change Memory Bank memories, which are generated from sessions and can hold what the agent read
`aiplatform.reasoningEngines.update`, `iam.serviceAccounts.actAs`, `iam.serviceAccounts.getAccessToken`	Editor, Agent Platform User, Data Scientist (update); Editor and Service Account User (`actAs`); Service Account Token Creator (`getAccessToken`)	Redeploy the agent with other code, or act as the identities the deny policy exempts

Table 2. Permissions that reach the agent, its sessions, or its memory, from Google's IAM references as of 19 October 2026 (Google Cloud, n.d.f; Google Cloud, n.d.g). Grants on the project, folder, or organisation are inherited by the agent.

Don't skip the read rows. A session's events hold whatever the agent read on the employee's behalf, so Viewer on the agent's project may be enough to read query results that BigQuery would have refused that same person. The last row is there because anyone who can redeploy the agent, or act as the identities it trusts, can undo most of this part.

What I'd do is give the agent its own project with no basic roles on it, run the backend's service account from a different project, and add a deny policy so a broad grant made later, or one inherited from a folder, doesn't count for these permissions. Deny policies attach to a project, folder, or organisation, and every permission in Table 2 is on Google's list of ones a deny policy supports (Google Cloud, n.d.e; Google Cloud, n.d.o). Listing 2 has four rules:

Calling the agent. Only the backend's service account can query the agent.
Calling a revision. Nobody can query a specific revision, since the backend never needs to.
Reading and writing sessions and memory. Only the agent's own identity and a break-glass group can.
Changing the agent or its access. Only the deployment pipeline's identity can redeploy the agent, change the project's, the agent's, or a service account's IAM policy, or impersonate or create keys for service accounts in the project.

Exceptions in a deny rule apply to whoever is calling. That's why the backend's and the pipeline's service accounts live in their own projects, where the fourth rule doesn't get in the way of their own workloads using them. The fourth rule also does nothing about keys that already exist, so the agent's project shouldn't have any.

Without Agent Identity, a deployed agent runs as either the AI Platform Reasoning Engine Service Agent or a custom service account (Google Cloud, n.d.l), and the listing assumes a custom one. Every rule also exempts the project's service agents. Google is clear on this one, "[d]on't deny permissions for service agents", and gives a principal set for exempting them (Google Cloud, n.d.p). That exemption covers every Google service agent in the project, though, so I'd keep products that call agents on their users' behalf, Gemini Enterprise for example, out of the agent's project. If one has to be there, treat it as another caller with its own idea of who the user is.

JSON

{
  "displayName": "Ops agent: who can call it, read its sessions, and change it",
  "rules": [
    {
      "denyRule": {
        "deniedPrincipals": ["principalSet://goog/public:all"],
        "exceptionPrincipals": [
          "principal://iam.googleapis.com/projects/-/serviceAccounts/agent-backend@backend-prod.iam.gserviceaccount.com",
          "principalSet://cloudresourcemanager.googleapis.com/projects/PROJECT_NUMBER/type/ServiceAgent"
        ],
        "deniedPermissions": [
          "aiplatform.googleapis.com/reasoningEngines.query"
        ]
      }
    },
    {
      "denyRule": {
        "deniedPrincipals": ["principalSet://goog/public:all"],
        "exceptionPrincipals": [
          "principalSet://cloudresourcemanager.googleapis.com/projects/PROJECT_NUMBER/type/ServiceAgent"
        ],
        "deniedPermissions": [
          "aiplatform.googleapis.com/reasoningEngineRuntimeRevisions.query"
        ]
      }
    },
    {
      "denyRule": {
        "deniedPrincipals": ["principalSet://goog/public:all"],
        "exceptionPrincipals": [
          "principal://iam.googleapis.com/projects/-/serviceAccounts/ops-agent@ops-agent-prod.iam.gserviceaccount.com",
          "principalSet://goog/group/ops-agent-break-glass@example.com",
          "principalSet://cloudresourcemanager.googleapis.com/projects/PROJECT_NUMBER/type/ServiceAgent"
        ],
        "deniedPermissions": [
          "aiplatform.googleapis.com/sessions.*",
          "aiplatform.googleapis.com/sessionEvents.*",
          "aiplatform.googleapis.com/memories.*",
          "aiplatform.googleapis.com/agentMemories.*"
        ]
      }
    },
    {
      "denyRule": {
        "deniedPrincipals": ["principalSet://goog/public:all"],
        "exceptionPrincipals": [
          "principal://iam.googleapis.com/projects/-/serviceAccounts/agent-deployer@deploy-prod.iam.gserviceaccount.com",
          "principalSet://cloudresourcemanager.googleapis.com/projects/PROJECT_NUMBER/type/ServiceAgent"
        ],
        "deniedPermissions": [
          "aiplatform.googleapis.com/reasoningEngines.create",
          "aiplatform.googleapis.com/reasoningEngines.update",
          "aiplatform.googleapis.com/reasoningEngines.delete",
          "aiplatform.googleapis.com/reasoningEngines.setIamPolicy",
          "cloudresourcemanager.googleapis.com/projects.setIamPolicy",
          "iam.googleapis.com/serviceAccounts.setIamPolicy",
          "iam.googleapis.com/serviceAccounts.actAs",
          "iam.googleapis.com/serviceAccounts.getAccessToken",
          "iam.googleapis.com/serviceAccounts.getOpenIdToken",
          "iam.googleapis.com/serviceAccounts.implicitDelegation",
          "iam.googleapis.com/serviceAccounts.signBlob",
          "iam.googleapis.com/serviceAccounts.signJwt",
          "iam.googleapis.com/serviceAccountKeys.create"
        ]
      }
    }
  ]
}

Listing 2. A deny policy for the agent's project. Replace the service account emails, the group, and the project number. If the agent runs with Agent Identity, use its principal from Google's principal identifiers page instead of the ops-agent service account.

The deny policy gets in the way of some legitimate work, and it leaves some paths open. Whoever handles an erasure request or an incident needs to read or delete sessions, and so does a security team whose organisation-wide reviewer roles would otherwise have reached them. A just-in-time role grant won't help, because a deny rule beats an allow grant, so the way in has to be a short, logged spell in the break-glass group. Deployments have to go through the pipeline identity.

Deny Admin is the only predefined role that can delete a deny policy, and Google only lets you grant it at the organisation level, so anyone holding it there, or able to change IAM at the organisation, can still remove this one (Google Cloud, n.d.g). The pipeline identity can't, because its rights stop at the project. A deny policy can't protect itself either, since Google's list of deny-supported permissions has nothing for managing deny policies. Changes to the policy, and to organisation-level IAM, need an alert, and Deny Admin should sit with a small group that someone reviews. Google says policy changes generally take effect within two minutes, though "in some cases, it can take 7 minutes or more" (Google Cloud, n.d.d), so give it time before you test that a pipeline deployment still works and that the agent can still read and write its own sessions.

If tracing is turned on with the template's enable_tracing flag, the SDK we read switches on capture of message content in ADK's spans, so prompts and tool results can end up in Cloud Trace and Logging, where the deny policy doesn't reach. I'd leave content capture off for this agent, or send its telemetry somewhere with the same access as its sessions.

What a query can carry

Locking down who can call the agent still leaves the question of what the backend's own calls can carry. A REST query names a class_method and an input. Google's runtime contract, which is about agents deployed as custom containers, includes an example server that looks the method up by name and calls it with the input as keyword arguments (Google Cloud, n.d.a). The SDK's client sends keyword arguments the same way. Google doesn't say how the managed deployment of AdkApp dispatches, which is why the tests below matter.

AdkApp, ADK's app template for Agent Runtime, accepts a lot more than the backend needs. We read version 2.5.0 of the Agent Platform SDK and ran the template locally, in memory, with ADK 2.12.0 and a stub model in place of Gemini:

async_create_session takes an initial state, which Google's ADK session page also documents (Google Cloud, n.d.j), and a session ID of the caller's choosing. Initial state is how the backend sets policy_task. It's also how anyone else would.
async_stream_query, and the synchronous stream_query, hand any keyword arguments they don't recognise to ADK's runner, and the runner accepts a state_delta. In our local test, a query on a session with both policy flags set, carrying a state_delta that set them false, left them false. The next tool call would be checked as though the session had read nothing.
async_stream_query also takes session_events, which builds a temporary session from events the caller supplies, and it takes the message as a content dictionary, which can carry files. In our test a file part reached the model without setting the untrusted flag that Part 1 says attachments should set.
streaming_agent_run_with_events, also registered by default, creates a session when given an unknown session ID and replays the caller's events into it, which can set the policy check's keys directly.
The template registers memory and artifact operations alongside the session ones, including async_add_session_to_memory and async_search_memory, each of which takes a user ID or session from the caller.

We haven't tried this against a deployed agent. Whether the managed endpoint passes every input field through, and whether it will dispatch a method that isn't registered, are things to test in your own project. I'd assume it does both until a test says otherwise.

The fix lives in two places. The backend builds every query from its own fields and never forwards a client's JSON. And the agent refuses anything the backend wouldn't send. Google's custom-agent guide uses register_operations to pick which methods a deployed agent registers (Google Cloud, n.d.c), and AdkApp defines that method, so a subclass can narrow it. Listing 3 registers two operations and checks what each one is given:

Creating a session takes a tid:oid user ID and the policy check's three keys, and nothing else: a known task, the untrusted flag set, and the sensitive flag clear. The caller can't choose a session ID or add user: or app: keys.
Sending a message takes a text message, a tid:oid user ID, and the ID of a session that exists, and nothing else.

As a second line of defence, every other public method on the template raises an error, in case the runtime dispatches it anyway. That's a deny list, so if dispatch really does work by name, set_up, clone, and the private methods are still callable. Only a test against the deployed endpoint settles that. The narrower app also breaks integrations that rely on those methods: the SDK's docstring says streaming_agent_run_with_events is meant for calls from AgentSpace, which is now part of Gemini Enterprise.

Python

"""An AdkApp that exposes only what the company backend uses.

The default template registers memory, artifact, and session operations,
and its query methods pass keyword arguments they don't name on to the
ADK runner, which accepts a state_delta. This subclass registers two
operations, refuses the template's other public methods as a second line
in case the runtime dispatches methods that aren't registered, and checks
the arguments of the two it keeps.
"""

import re

from agentplatform.frameworks import AdkApp

POLICY_KEYS = {"policy_task", "policy_untrusted_seen", "policy_sensitive_read"}
TASKS = {"triage_ticket", "data_question", "ticket_reply"}
USER_ID = re.compile(r"[0-9a-f-]{36}:[0-9a-f-]{36}")   # tid:oid
MAX_MESSAGE = 20_000
KEEP = {"async_create_session", "async_stream_query",
        "register_operations", "set_up", "clone"}

class BackendOnlyAdkApp(AdkApp):
    def register_operations(self):
        return {"async": ["async_create_session"],
                "async_stream": ["async_stream_query"]}

    async def async_create_session(self, *, user_id, state, **extra):
        # Only the policy check's keys, no caller-chosen session ID,
        # no user: or app: state.
        if (extra or not isinstance(state, dict) or set(state) != POLICY_KEYS
                or state["policy_task"] not in TASKS
                or state["policy_untrusted_seen"] is not True
                or state["policy_sensitive_read"] is not False
                or not isinstance(user_id, str)
                or not USER_ID.fullmatch(user_id)):
            raise ValueError("unexpected session arguments")
        return await super().async_create_session(user_id=user_id, state=state)

    async def async_stream_query(self, *, message, user_id, session_id,
                                 **extra):
        # A text message for an existing session, and nothing else: no
        # implicit sessions, no session_events, no state_delta, and no
        # Content dict that could carry files.
        if extra:
            raise ValueError(f"unexpected arguments: {sorted(extra)}")
        if not session_id:
            raise ValueError("session_id is required")
        if not isinstance(user_id, str) or not USER_ID.fullmatch(user_id):
            raise ValueError("unexpected user_id")
        if not isinstance(message, str) or len(message) > MAX_MESSAGE:
            raise ValueError("message must be text")
        async for event in super().async_stream_query(
                message=message, user_id=user_id, session_id=session_id):
            yield event

def _refuse(name):
    def refused(self, *args, **kwargs):
        raise PermissionError(f"{name} is not exposed by this agent")
    return refused

for _name in dir(AdkApp):
    if (not _name.startswith("_") and _name not in KEEP
            and callable(getattr(AdkApp, _name))):
        setattr(BackendOnlyAdkApp, _name, _refuse(_name))

Listing 3. An AdkApp that exposes only what the backend uses. Tested locally with google-cloud-aiplatform 2.5.0 and google-adk 2.11.0 and 2.12.0. The module has to be shipped with the deployment, like any other code the agent imports.

Our tests run the default template and the subclass side by side. With the default, a query through either the async or the sync method resets the flags. With the subclass, every route we tried raises an error and the flags stay as they were. Listing 3 doesn't stop someone with the query permission from creating a session with policy keys of their choosing, because the backend needs that operation. Keeping other callers out is Listing 2's job.

There are two other ways to change session state outside a query. The ADK API server, if anyone puts it in front of the agent, accepts initial state when it creates a session and has a PATCH endpoint for sessions, except in the Go server, whose users are told to pass a stateDelta on runs instead (Google, n.d.c). Rewind is a method on ADK's runner that AdkApp doesn't register. It restores session-level state but not "app-level or user-level state" (Google, n.d.a), so a rewind can bring back old flag values. ADK's own advice is to "[a]void directly modifying the session.state collection" outside an invocation (Google, n.d.b). For this agent, the only code that should ever create a session or touch the policy check's keys is the backend's start call and the policy check itself.

Binding the session to one employee

Once the callers are limited, the backend decides which employee a session belongs to. Listing 4 is the part of the backend that does it, and it does three things.

When a session starts, the employee has picked an entry point, replying to a ticket for example. The backend maps that to one of the policy check's tasks and creates the session with policy_task set. Part 1 said a session that takes pasted text should start with the untrusted flag set. Every entry point here takes free text, and the backend can't tell typed text from text pasted out of a customer's email, so all three start with the flag set. That has a cost under Part 1's decision table. Every internal side effect needs a confirmation, and in the data question task an external email after any query is denied, so that task can only send results inside the company. The deny that Part 1 expected to fire rarely will now fire every time someone tries, and the alert on it needs to tell that case apart. Finally the backend records the session ID, the owner's user_id, and the task in its own store.

When a session is resumed, the backend looks up the record and compares the owner with the user_id from the current token. An unknown session and someone else's session get the same answer, so a caller learns very little about which session IDs exist.

When the backend calls the agent, the user_id comes from the record it just matched against the token. The only part of the request body that reaches the agent is the message.

Python

# ---------------------------------------------------------------------------
# Sessions

class AgentRuntime(Protocol):
    def create_session(self, user_id: str, state: dict) -> str: ...
    def query(self, user_id: str, session_id: str, message: str) -> str: ...

@dataclass(frozen=True)
class SessionRecord:
    session_id: str
    user_id: str
    task: str
    created_at: float

class SessionStore(Protocol):
    def put(self, rec: SessionRecord) -> None: ...
    def get(self, session_id: str) -> SessionRecord | None: ...

class MemoryStore:
    def __init__(self):
        self._d: dict[str, SessionRecord] = {}

    def put(self, rec):
        self._d[rec.session_id] = rec

    def get(self, session_id):
        return self._d.get(session_id)

# The entry points the backend offers, matching the policy check's tasks in
# Part 1, and whether each starts as having read untrusted content. Every
# entry point here takes free text from the employee, and the backend can't
# tell typed text from text pasted out of a customer's email, so all three
# start with the flag set. The client never supplies session state.
TASKS = {"triage_ticket": True, "data_question": True, "ticket_reply": True}

class NotFound(Exception):
    """Unknown session, or one owned by someone else (same answer)."""

class Backend:
    def __init__(self, cfg: EntraConfig, key_for: KeyResolver,
                 runtime: AgentRuntime, store: SessionStore,
                 clock: Callable[[], float] = time.time):
        self.cfg, self.key_for = cfg, key_for
        self.runtime, self.store, self.clock = runtime, store, clock
        self.decided = DecidedApprovals()

    def _employee(self, bearer: str) -> Employee:
        return validate_token(bearer, self.cfg, self.key_for)

    def start(self, bearer: str, task: str) -> str:
        emp = self._employee(bearer)
        if task not in TASKS:
            raise Rejected(f"unknown task {task!r}")
        state = {"policy_task": task,
                 "policy_untrusted_seen": TASKS[task],
                 "policy_sensitive_read": False}
        sid = self.runtime.create_session(emp.user_id, state)
        self.store.put(SessionRecord(sid, emp.user_id, task, self.clock()))
        return sid

    def send(self, bearer: str, session_id: str, message: str) -> str:
        emp = self._employee(bearer)
        rec = self.store.get(session_id)
        if rec is None or not secrets.compare_digest(rec.user_id, emp.user_id):
            raise NotFound(session_id)
        # user_id comes from the record and the token, never the request
        return self.runtime.query(rec.user_id, rec.session_id, message)

    def approve(self, bearer: str, approval: "PendingApproval",
                rule: "StepUp") -> dict:
        """Record an employee's approval of a call the policy check held.

        The backend loads `approval` from its queue by ID; the client only
        names it.
        """
        emp = self._employee(bearer)
        rec = self.store.get(approval.session_id)
        if rec is None or not secrets.compare_digest(rec.user_id, emp.user_id):
            raise NotFound(approval.session_id)
        now = self.clock()
        if now - approval.requested_at > rule.approval_ttl:
            raise Rejected("approval request expired")
        try:
            check_step_up(emp, rule, approval.requested_at, now)
        except StepUpNotConfigured:
            raise
        except Rejected:
            if "cp1" in [str(v).lower() for v in _as_list(emp.claims.get("xms_cc"))]:
                raise StepUpRequired(*step_up_challenge(rule, self.cfg.tenant_id))
            raise
        if not self.decided.claim(approval.approval_id):
            raise Rejected("approval already decided")
        return {"approval_id": approval.approval_id,
                "session_id": rec.session_id,
                "request_hash": approval.request_hash,
                "approver": emp.user_id,
                "approved_at": now}

class DecidedApprovals:
    """Each approval is decided once. In production, a unique key in the
    backend's database does this; the lock stands in for it here."""

    def __init__(self):
        self._ids: set[str] = set()
        self._lock = threading.Lock()

    def claim(self, approval_id: str) -> bool:
        with self._lock:
            if approval_id in self._ids:
                return False
            self._ids.add(approval_id)
            return True

@dataclass(frozen=True)
class PendingApproval:
    """Written by the policy check's approval queue when it holds a call."""
    approval_id: str
    session_id: str
    request_hash: str   # hash of the tool name and arguments it held
    requested_at: float

Listing 4. The backend's session binding. AgentRuntime is a small interface over the deployed agent; the tests use an in-memory fake.

The record lives outside the agent on purpose. Agent Platform Sessions stores a user ID with each session too, but the backend put it there, so it can't act as an independent check on the backend. The backend's store is also where Part 4's request-level record will go, next to the approvals.

As written, Listing 4 lets any employee who passes Table 1 start any of the three tasks. If only some people should be answering data questions, Entra app roles assigned to users appear in the roles claim of their delegated tokens (Microsoft, 2026a), and the backend can tie each entry point to a role.

When the employee approves a call

Part 1's policy check asks for confirmation before some calls, an internal email after the session has read untrusted content, for instance. The person approving should be the employee who owns the session, and ideally someone who is at the keyboard right now. An access token doesn't tell you that. Microsoft gives the default lifetime as "a random value ranging between 60-90 minutes" (Microsoft, 2025a). Figure 2 shows one way to ask Entra for a recent sign-in instead.

All the pieces come from Microsoft's Conditional Access documentation.

You start with an authentication context. A tenant can define contexts with IDs from c1 to c99, and a Conditional Access policy targets one of them (Microsoft, 2025b). Here the policy sets sign-in frequency to "Every time", which "requires full reauthentication when the session is evaluated" (Microsoft, 2026b). Microsoft warns that "every time, without multifactor authentication might result in sign-in looping" (Microsoft, 2026b), and a password on its own only proves someone knows the password, so I'd have the policy require a phishing-resistant authentication strength.

When a token is missing the context, the API sends back a claims challenge: a 401 whose WWW-Authenticate header carries error="insufficient_claims" and a base64-encoded request for acrs (Microsoft, 2024). The client then asks Entra for a new token with those claims. A client says it can handle these challenges by declaring the cp1 capability, which reaches the API as the optional xms_cc claim. Microsoft's example code only sends the formatted challenge when that claim shows cp1, and sends a plain error otherwise (Microsoft, 2024). Listing 5 does the same.

The API also has to opt in to the optional acrs, auth_time, and xms_cc claims on access tokens. For acrs, Microsoft notes that "[e]ach token type needs to be individually opted-in" (Microsoft, 2025b), and auth_time is the "[t]ime when the user last authenticated" (Microsoft, 2026c). Forget auth_time and Listing 5 rejects every approval.

One detail changes how the check has to work. Microsoft says: "The policy factors in five minutes of clock skew when every time is selected, so that it doesn't prompt users more often than once every five minutes" (Microsoft, 2026b). If the backend insists on a sign-in after the moment the call was held, an employee who signed in two minutes earlier gets rejected, sent back to Entra, and isn't prompted, round and round. So Listing 5 accepts a sign-in up to five and a half minutes before the hold, which allows for some clock drift, and no more than ten minutes before the approval arrives. A held call waits at most fifteen minutes for a decision, and each approval can be decided once. The backend's database should enforce that with a unique key, not a read followed by a write.

acrs on its own proves less than it seems to. Once the API opts in, Entra can add acrs to an ordinary token when the policy is already satisfied (Microsoft, 2025b), so auth_time is the claim that actually shows a recent sign-in. The five-minute window also means one sign-in can approve several calls held a few minutes apart, and the backend can't demand a fresh prompt for each one because Entra won't give it. Putting every pending call on one approval screen at least lets the employee see what they're approving. The numbers in Listing 5 are mine, so tune them, and test the whole sequence in your tenant. Microsoft documents each piece separately, not this combination end to end.

Python

# ---------------------------------------------------------------------------
# Step-up for approvals

@dataclass(frozen=True)
class StepUp:
    auth_context: str = "c1"   # the tenant's authentication context ID
    reuse_window: int = 330    # Entra's five minutes, plus clock skew
    max_age: int = 600         # time allowed from sign-in to approval
    approval_ttl: int = 900    # how long a held call waits for a decision

class StepUpNotConfigured(Rejected):
    """The tenant isn't issuing a claim the step-up needs. A new sign-in
    won't fix that, so no challenge is sent."""

class StepUpRequired(Exception):
    """Carries the 401 claims challenge for a client that can handle it."""

    def __init__(self, status: int, headers: dict):
        super().__init__(status)
        self.status, self.headers = status, headers

def _as_list(v) -> list:
    return [v] if isinstance(v, str) else list(v or [])

def step_up_challenge(rule: StepUp, tenant_id: str) -> tuple[int, dict]:
    """401 response asking the client for a token with the context.

    Sent only to clients whose token shows xms_cc = cp1, as in Microsoft's
    sample; other clients get a plain rejection.
    """
    claims = {"access_token": {"acrs": {"essential": True,
                                        "value": rule.auth_context}}}
    b64 = base64.b64encode(json.dumps(claims, separators=(",", ":"))
                           .encode()).decode()
    uri = f"https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/authorize"
    return 401, {"WWW-Authenticate":
                 f'Bearer realm="{tenant_id}", authorization_uri="{uri}", '
                 f'error="insufficient_claims", claims="{b64}"'}

def check_step_up(emp: Employee, rule: StepUp, requested_at: float,
                  now: float) -> None:
    """The approval token must show a recent sign-in under the context.

    `requested_at` is when the policy check held the call. A sign-in from
    well before that moment was not made for this approval. Calls held
    within the same few minutes can share one sign-in.
    """
    auth_time = emp.claims.get("auth_time")
    if not isinstance(auth_time, (int, float)):
        raise StepUpNotConfigured("no auth_time claim: configure it as optional")
    if rule.auth_context not in _as_list(emp.claims.get("acrs")):
        raise Rejected("authentication context not satisfied")
    if auth_time < requested_at - rule.reuse_window:
        raise Rejected("sign-in predates the approval request")
    if now - auth_time > rule.max_age:
        raise Rejected("sign-in too old")

Listing 5. The step-up check behind Backend.approve in Listing 4, and the claims challenge it returns to clients that can handle one.

The approval record holds the employee, the session, and a hash of the held call. Part 4 builds the release path that checks the call against that hash before running it, and makes the hash compatible with the canonical-JSON hashes used elsewhere in the series. Rate limits on approval requests belong there too.

When the employee's identity has to reach Google Cloud

So far the employee hasn't needed a Google Cloud identity. The backend calls the agent as itself, and the agent's tools run with whatever authority Part 3 gives them. Part 1 promised this part would cover federating the employee's Entra identity into Google Cloud "for the calls that need it", so this section is about which calls those actually are.

Workforce Identity Federation is Google's mechanism for people. In Google's words, it "federates user identities whereas Workload Identity Federation federates workload identities" (Google Cloud, n.d.t). A workforce pool sits at the organisation level and maps an identity provider's claims to Google attributes, and for Entra over OIDC Google recommends mapping google.subject to assertion.oid (Google Cloud, n.d.t; Google Cloud, n.d.b). An application swaps an Entra ID token for a short-lived Google token at https://sts.googleapis.com/v1/token, with a subject token type of urn:ietf:params:oauth:token-type:id_token and the workforce provider's resource name as the request's audience (Google Cloud, n.d.n). The ID token's own aud has to "match the client ID specified in the provider configuration" (Google Cloud, n.d.m), which is the Entra app you register for federation. What comes back acts in IAM as principal://iam.googleapis.com/locations/global/workforcePools/POOL_ID/subject/OID.

This agent only needs that where Google Cloud enforces access per employee, and BigQuery row-level security is the clearest example. A row access policy can name workforce principals as grantees (Google Cloud, n.d.s), and BigQuery is listed as GA for Workforce Identity Federation, with "[n]o known limitations" for API access and some console, connector, and data-loading features excepted (Google Cloud, n.d.h). If the data question task should only return rows that employee is allowed to see, then either the query runs as the employee through the federated token, or the agent's own identity reads everything and something else does the filtering. Part 3 compares those options, along with on-behalf-of flows and the agent's own identity.

The federated path costs something, and Part 3 weighs that against the alternatives:

The token is broad. Whatever does the exchange ends up holding a token that acts as the employee in Google Cloud, with all of their IAM grants. The STS reference notes that "workforce pools do not support Credential Access Boundaries" (Google Cloud, n.d.m), so you can't narrow the token to one dataset.
It lasts as long as the pool's session duration. That "defaults to one hour" and can be set anywhere from 15 minutes to 12 hours (Google Cloud, n.d.k). I'd set it short for this path. I haven't found anything saying that disabling the user in Entra ends a Google token that's already been issued, so I'd assume it doesn't.
It needs its own Entra app and a billing project. The ID token for the exchange should come from an app registered only for federation, ideally exchanged server-side. If the provider used the front end's client ID, every sign-in to the front end could be swapped for Google credentials. Google's exchange example passes a userProject for quota and billing, which the employee needs serviceusage.services.use on (Google Cloud, n.d.n), and BigQuery needs a project where the employee can run jobs.

For triage and ticket replies, I can't see what the federated path would add, so I wouldn't build it for them.

What this binding doesn't establish

With all of the above in place, the backend is a fairly reliable witness to who sent each request. There are still gaps, and these are the ones I'd want a reviewer to know about.

A stolen token passes the checks

A stolen access token passes every check in Table 1 until it expires, and the backend has no way to tell. Token protection, device compliance, and Continuous Access Evaluation live in Entra and the front end, not in this code, so I've left them out. Microsoft also gives a default "long lived token lifetime" of 20 to 28 hours, in a passage about tenants with Continuous Access Evaluation turned on (Microsoft, 2025a), so it's worth finding out what lifetime your tenant gives tokens for the agent API. The step-up only narrows the window for approvals.

The backend can act as any employee

Anyone who can run code as the backend's service account, or change the backend's code, can be any employee as far as the agent is concerned. Deploy rights on the backend belong in the same review as deploy rights on the agent, and its service account shouldn't have keys anyone can download. If Part 3 ends up with the backend holding a delegated Entra token for each employee, that concentration gets worse.

Sessions outlive the sign-in

A session carries on after the employee signs out, after their account is disabled, and after they change roles. Listing 4 doesn't end sessions when Entra disables someone. Checking account status on resume, or keeping sessions short, closes most of that gap.

Session history is sensitive data

The session records, the approval records, and the agent's own session history show who asked the agent what, and the history holds what the agent read on their behalf. Each of those stores needs its own access review, which is part of the reason Table 2's read permissions matter.

GATE covers the agent's identity, not the employee's

GATE is the open agent governance framework I maintain, and its C01 control, Workload Identity and Attestation, is about the agent's own identity: it binds each agent instance to runtime claims you can verify (Stevens, 2026). C01 doesn't cover the employee behind a request, which is what this part has been about. That gap has been put forward as a candidate for GATE v1.5, and Part 9 will say what came of it.

Versions and platform facts

Item	Version or stage	Checked
ADK for Python	Tested with 2.11.0 and 2.12.0 (the current release)	19 October 2026
Agent Platform SDK (`google-cloud-aiplatform`, providing `agentplatform`)	Tested with 2.5.0	19 October 2026
PyJWT	Tested with 2.15.1	19 October 2026
`jsonwebtoken` (Rust)	Tested with 11.1.0, `rust_crypto` feature	19 October 2026
Workforce Identity Federation with BigQuery	GA	19 October 2026 (Google Cloud, n.d.h)
Workforce Identity Federation with Agent Platform	GA	19 October 2026 (Google Cloud, n.d.h)
Agent Runtime revisions and traffic	Preview	19 October 2026 (Google Cloud, n.d.i)

Table 3. Versions tested and launch stages relied on in this part.

What I'd check in your own deployment

All of this depends on how your tenant, projects, and backend are set up, so treat each item as something to confirm in your own environment. Four of them would stop a release for me, and they're marked. The backend runs Table 1's checks on v2.0 tokens. Nobody outside the backend, the agent, the break-glass group, and Google's service agents holds the query, session, or memory permissions. A query can't change the policy check's keys. And a session only resumes for its owner.

Entra registration

Owner: the identity team.

The agent API's registration sets requestedAccessTokenVersion to 2, exposes one delegated scope, and has idtyp, acct, acrs, auth_time, and xms_cc configured as optional claims on access tokens. A real token from the tenant shows them.
The front end requests the profile scope along with the API's scope, so tokens carry tid and oid.
User assignment is required on the API's enterprise application, and only the intended users or groups are assigned. If tasks differ by role, app roles map to entry points.
The authentication context used for approvals has a Conditional Access policy with sign-in frequency set to every time and a phishing-resistant authentication strength, and the front end declares the cp1 client capability.

The backend's token checks

Owner: the backend's developers.

Release-blocking. The backend runs every check in Table 1, on every request, and a test with real tokens from your tenant shows that an app-only token, a guest token, a token for another API, and a token from another client app are each rejected.
The user_id is built from tid and oid and nothing else, and no request field can override it.
Signing keys come from the tenant's metadata and refresh on an unknown kid, and a failure to fetch them denies requests instead of skipping validation.

Who can reach the agent

Owner: the platform team, with security.

Release-blocking. In the agent's project, only the backend's service account can query the agent, no one can query a revision directly, and only the agent's identity and the break-glass group hold the session and memory permissions, apart from the project's Google service agents, which Listing 2 exempts. A review of effective permissions, including inherited, basic, and organisation-wide security roles, shows no one else, and a deny policy like Listing 2 is attached. After it has propagated, a deployment through the pipeline and a query from the backend both still work. That test also settles whether a query needs any session permission from the caller, such as aiplatform.sessions.run, which the third rule denies.
Only the deployment pipeline can redeploy the agent, change IAM on the project, the agent, or its service accounts, or act as those service accounts. Deny Admin and organisation-level IAM admin are held by a small, reviewed group, and changes to either raise an alert. The project has no user-managed service account keys, revisions that aren't serving traffic are removed, and the agent isn't registered with other products that would call it on their own users' behalf.
Membership of the break-glass group is granted for a short time, approved by someone other than the requester, and logged.
The backend's and the pipeline's service accounts live in their own projects. In each, only the workload that runs as the service account can use it, no one can create keys for it, and who can impersonate it is reviewed alongside deploy rights on the agent, since Listing 2 exempts both identities.

Session state and approvals

Owner: the agent's developers, with the backend's developers.

Release-blocking. A query to the deployed agent that carries state_delta, session_events, a content dictionary, or any field beyond the message, user ID, and session ID is rejected, and so is a call to any method other than the two the agent registers, including stream_query and streaming_agent_run_with_events. A test against the deployed endpoint shows both.
Release-blocking. A session resumes only for the employee recorded as its owner, and a test with two employees' tokens shows the second one gets the same answer as for a session that doesn't exist.
An approval needs a token from the session's owner with the approval context in acrs and a recent auth_time, expires if no one decides it, and can be decided once. A test in the tenant shows the whole sequence works, including for an employee who signed in shortly before the call was held.
Sessions are refused on resume when the employee's account is disabled.

Logging and response

Owner: security operations, with the backend's developers.

The backend logs every rejected token, session start, refused resume, and approval with the employee's user_id, the session ID, and a request ID, so an Entra sign-in, a backend request, a session, and an approval can be joined.
Changes to the deny policy and to IAM on the agent's project raise an alert.
Message content capture in traces is off, or the agent's telemetry goes somewhere with the same access as its sessions.
There is a tested way to end all of one employee's sessions and pending approvals, and to disable the backend's service account. Part 8 builds detection on top of these records.

The rest of the series

Part 3 follows the agent's calls out to its four MCP servers and works out which should run as the employee and which as the agent, including the BigQuery row-level security case above and where delegated tokens should live. Part 4 takes on indirect injection and builds the approval release path that Listing 5 feeds into. Parts 5 to 9 cover MCP servers you didn't write, the ways data leaves, memory, detection and containment, and red-teaming before release.

If you listed every principal that holds aiplatform.reasoningEngines.query on your agent's project today, inherited roles included, would the list match the people you expect to use the agent?

Disclosure: The author is a lifetime member of the OWASP Foundation. This article reflects an independent reading of public OWASP material and does not represent the views of the Foundation.

References

Google, n.d.a. Rewind sessions for agents. Agent Development Kit documentation. Available at: https://adk.dev/sessions/session/rewind/ [Accessed 19 October 2026].

Google, n.d.b. State: The Session's Scratchpad. Agent Development Kit documentation. Available at: https://adk.dev/sessions/state/ [Accessed 19 October 2026].

Google, n.d.c. Use the API Server. Agent Development Kit documentation. Available at: https://adk.dev/runtime/api-server/ [Accessed 19 October 2026].

Google Cloud, n.d.a. Agent Platform runtime contract. Gemini Enterprise Agent Platform documentation. Available at: https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/runtime/runtime-contract [Accessed 19 October 2026].

Google Cloud, n.d.b. Configure Workforce Identity Federation with Microsoft Entra ID and sign in users. IAM documentation. Available at: https://docs.cloud.google.com/iam/docs/workforce-sign-in-microsoft-entra-id [Accessed 19 October 2026].

Google Cloud, n.d.c. Create a custom agent. Gemini Enterprise Agent Platform documentation. Available at: https://docs.cloud.google.com/gemini-enterprise-agent-platform/build/runtime/create-a-custom-agent [Accessed 19 October 2026].

Google Cloud, n.d.d. Deny access to resources. IAM documentation. Available at: https://docs.cloud.google.com/iam/docs/deny-access [Accessed 19 October 2026].

Google Cloud, n.d.e. Deny policies. IAM documentation. Available at: https://docs.cloud.google.com/iam/docs/deny-overview [Accessed 19 October 2026].

Google Cloud, n.d.f. Gemini Enterprise Agent Platform roles and permissions. IAM documentation. Available at: https://docs.cloud.google.com/iam/docs/roles-permissions/aiplatform [Accessed 19 October 2026].

Google Cloud, n.d.g. Identity and Access Management roles and permissions. IAM documentation. Available at: https://docs.cloud.google.com/iam/docs/roles-permissions/iam [Accessed 19 October 2026].

Google Cloud, n.d.h. Identity federation: products and limitations. IAM documentation. Available at: https://docs.cloud.google.com/iam/docs/federated-identity-supported-services [Accessed 19 October 2026].

Google Cloud, n.d.i. Manage revisions and traffic. Gemini Enterprise Agent Platform documentation. Available at: https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/runtime/manage-revisions-and-traffic [Accessed 19 October 2026].

Google Cloud, n.d.j. Manage sessions with Agent Development Kit. Gemini Enterprise Agent Platform documentation. Available at: https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/sessions/manage-with-adk [Accessed 19 October 2026].

Google Cloud, n.d.k. Manage workforce identity pool providers. IAM documentation. Available at: https://docs.cloud.google.com/iam/docs/manage-workforce-identity-pools-providers [Accessed 19 October 2026].

Google Cloud, n.d.l. Managing access for deployed agents. Gemini Enterprise Agent Platform documentation. Available at: https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/runtime/manage-agent-access [Accessed 19 October 2026].

Google Cloud, n.d.m. Method: token. Security Token Service API reference. Available at: https://docs.cloud.google.com/iam/docs/reference/sts/rest/v1/TopLevel/token [Accessed 19 October 2026].

Google Cloud, n.d.n. Obtain short-lived tokens for Workforce Identity Federation. IAM documentation. Available at: https://docs.cloud.google.com/iam/docs/workforce-obtaining-short-lived-credentials [Accessed 19 October 2026].

Google Cloud, n.d.o. Permissions supported in deny policies. IAM documentation. Available at: https://docs.cloud.google.com/iam/docs/deny-permissions-support [Accessed 19 October 2026].

Google Cloud, n.d.p. Principal identifiers. IAM documentation. Available at: https://docs.cloud.google.com/iam/docs/principal-identifiers [Accessed 19 October 2026].

Google Cloud, n.d.q. Share an agent. Gemini Enterprise Agent Platform documentation. Available at: https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/share-agent [Accessed 19 October 2026].

Google Cloud, n.d.r. Use an Agent Development Kit agent. Gemini Enterprise Agent Platform documentation. Available at: https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/runtime/use-an-adk-agent [Accessed 19 October 2026].

Google Cloud, n.d.s. Use row-level security. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/managing-row-level-security [Accessed 19 October 2026].

Google Cloud, n.d.t. Workforce Identity Federation. IAM documentation. Available at: https://docs.cloud.google.com/iam/docs/workforce-identity-federation [Accessed 19 October 2026].

jsonwebtoken, n.d. Validation (jsonwebtoken 11.1.0). docs.rs. Available at: https://docs.rs/jsonwebtoken/11.1.0/jsonwebtoken/struct.Validation.html [Accessed 19 October 2026].

Microsoft, 2025a. Access tokens in the Microsoft identity platform. Microsoft identity platform documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity-platform/access-tokens [Accessed 19 October 2026].

Microsoft, 2025b. Developer guide to Conditional Access authentication context. Microsoft identity platform documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity-platform/developer-guide-conditional-access-authentication-context [Accessed 19 October 2026].

Microsoft, 2025c. Secure applications and APIs by validating claims. Microsoft identity platform documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity-platform/claims-validation [Accessed 19 October 2026].

Microsoft, 2026b. Conditional Access adaptive session lifetime policies. Microsoft Entra ID documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity/conditional-access/concept-session-lifetime [Accessed 19 October 2026].

Microsoft, 2026c. Optional claims reference. Microsoft identity platform documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity-platform/optional-claims-reference [Accessed 19 October 2026].

PyJWT, n.d. API Reference (PyJWT 2.15.1). Available at: https://pyjwt.readthedocs.io/en/stable/api.html [Accessed 19 October 2026].

Sheffer, Y., Hardt, D. and Jones, M., 2020. JSON Web Token Best Current Practices. RFC 8725, BCP 225. IETF. Available at: https://www.rfc-editor.org/rfc/rfc8725 [Accessed 19 October 2026].

Stevens, A., 2026. C01 - Workload Identity and Attestation. GATE v1.4 knowledge bundle. Available at: https://github.com/deterministic-agents/gate-knowledge/blob/main/controls/C01.md [Accessed 19 October 2026].

Native Tables, Managed Iceberg, or Iceberg in an Open Catalog

Andrew Stevens — Fri, 16 Oct 2026 14:53:22 +0000

On Google Cloud, a lakehouse table can live in BigQuery's own storage, or as Iceberg files in your bucket, either managed by BigQuery or registered in an open catalog that other engines write to. Native tables give you every BigQuery control, and anything outside BigQuery has to read through it. Managed Iceberg puts Parquet in your bucket, where anyone with the right Storage role can open it, and Google documents it as not supporting authorised views, row-level security, table snapshots, fail-safe, or managed disaster recovery. An open catalog lets Spark and other engines write the files themselves, and hands you the maintenance and the catalog's uptime.

Each fits some tables. The trouble comes from picking one for the whole platform, when each layer has its own readers, writers, and controls.

This part of the series is written as an architecture decision record: context, drivers, the three options and what each costs, and then a decision for an example platform, layer by layer. Part 1 introduced the options as A, B, and C and left two things for this post, the full comparison and how erasure works when the files belong to an open catalog.

The short version: start from native tables. Move a layer to managed Iceberg only when a named team or pipeline has to read its files, and to an open catalog only when another engine has to write it. BigQuery's policies apply only to reads through BigQuery, so every role on a bucket is another way to read the data. Each step away from native removes BigQuery controls and adds an erasure path and identities to review.

Two things are out of scope here. Consent withdrawal, which for a publisher is often more common than erasure, and erasure requests from readers known only by a cookie both need their own handling before anything reaches storage. This post assumes Part 1's intake has already turned a request into tokens.

Status and context

Most of the calls below depend on who writes each table and who reads it.

Status: accepted for the example platform below. Revisit it whenever Google changes what managed Iceberg or the Lakehouse runtime catalog supports. Several of the features that matter here were still in Preview when we read the pages in October 2026.

Part 1 used a composite retailer. This post uses a second composite, a publisher, because its data arrives in more shapes, from change data capture to anonymous event streams, ad logs, and consent records, and goes to more kinds of reader. The publisher runs a national news site, a few magazine brands, and an app, on BigQuery Enterprise edition. Subscriptions and billing live in an operational database and reach the lakehouse through change data capture. Article views, scroll depth, and paywall events arrive as a high-volume event stream, and most of those readers never log in, so the only thing identifying them is a cookie or device ID. Ad servers deliver impression and click logs in batches, and those logs carry IP addresses and device identifiers. The content management system supplies article metadata: sections, authors, tags, and publication times. Consent records decide which of those events can be used for personalisation and advertising at all.

The readers want different things from the same data. Editors and the audience team watch Looker dashboards through the day and notice when the numbers are stale. Ad operations reconcile delivery against bookings. A data science team trains recommendation and churn models in Spark, partly because its feature pipelines also feed a partner platform outside Google Cloud, and it wants long histories of reading behaviour, often more history than anyone else needs, while the recommendation models themselves care most about what someone read recently. Finance wants subscription revenue it can tie back to the billing system.

Part 1's design maps across without much change. There's a restricted raw landing area, a split job that swaps identifiers for tokens, an identity table that maps tokens back to people, and an erasure job that deletes in the restricted stores and hands tokens to downstream jobs. For the publisher, the identifiers going into the identity table are subscriber IDs and emails, plus the cookie and device IDs of readers who never sign in. Google lists fail-safe windows among the features managed Iceberg tables don't support (Google Cloud, n.d.c), which settles the question Part 1 left open about option B and fail-safe. The split lands differently, though. Part 1's option B variant put silver and gold on managed Iceberg; here gold stays native, for reasons the decision explains.

Two things carry over directly. Bronze, the identity table, and the restricted stores stay on native tables, because the design reads them only through authorised views and managed Iceberg doesn't support those (Google Cloud, n.d.c). And whatever storage a layer ends up on, the erasure process has to reach it, so each option's deletion path is part of this decision.

Decision drivers

These are the questions we ask for each layer. Their weight shifts by layer: gold usually turns on BigQuery features, and feature tables on who writes them.

Who writes the files. Is BigQuery the only writer, or do Spark jobs, streaming pipelines, or other engines need to write the layer too?
Who reads it, and how. Looker through BigQuery, Spark through a connector, and Spark straight from the files each make different demands.
Which BigQuery features the layer leans on. Authorised views, row-level security, column-level policies, masking, snapshots, clones, materialised views, and fail-safe aren't available on every option.
Where access is enforced. BigQuery's policies only protect data that's read through BigQuery. Once an engine reads files from the bucket, the catalog and the bucket are doing the enforcing.
Which edition you run. Fine-grained security, VPC Service Controls, customer-managed keys, BI Engine, and creating materialised views aren't on Standard edition, and managed disaster recovery is Enterprise Plus only (Google Cloud, n.d.y).
Who runs the maintenance, and who pays. Compaction, snapshot expiry, and orphan cleanup either happen for you or become jobs on someone's list.
Catalog availability and recovery. An open catalog is one more service that has to be up, backed up, and recoverable.
Exit cost. How hard it would be to move the layer later, in either direction.

Option A: BigQuery native tables

This is the BigQuery most teams already know. BigQuery stores and manages the data, and every BigQuery feature applies.

What you get

All the security features are available, subject to edition. Authorised views share a subset of a dataset with specific users and groups (Google Cloud, n.d.d). Row access policies filter rows by who's asking (Google Cloud, n.d.k). Policy tags restrict sensitive columns (Google Cloud, n.d.l), and dynamic masking, which builds on column-level access control, masks values at query time based on the role of whoever runs the query (Google Cloud, n.d.m). Standard edition has none of the fine-grained controls (Google Cloud, n.d.y), so on Standard most of option A's security lead over option B goes away, and the edition needs settling before the storage.

Recovery is the other strength. Time travel keeps changed and deleted data for a window you set between two and seven days, and fail-safe then keeps deleted data for a further seven days that you can't configure or extend (Google Cloud, n.d.i). Google's page describes fail-safe in terms of deleted or replaced tables and deleted or expired partitions, and recovery from it works at the table level (Google Cloud, n.d.i). It doesn't spell out rows removed by DML, so for erasure planning it's safer to assume those can sit in fail-safe too, which is how Part 1 counted it. Table snapshots preserve a table's contents at a point in time, and you pay only for data a snapshot doesn't share with another table (Google Cloud, n.d.o). Clones give you a cheap writable copy, charged only for what differs from the base table (Google Cloud, n.d.n). On Enterprise Plus, managed disaster recovery keeps a secondary replica, and with turbo replication Google aims to keep it within 15 minutes of the primary, though a hard failover can lose data (Google Cloud, n.d.r).

What you give up

You can't open the files yourself. Other engines read native tables through BigQuery, usually through the Storage Read API, which gives fast RPC-based access to BigQuery-managed storage (Google Cloud, n.d.aa). The Spark BigQuery connector uses that API (Google Cloud, n.d.e). Google's row-level security page notes that policies using subqueries aren't compatible with the Storage Read API, which supports only simple filter predicates (Google Cloud, n.d.k). Reading through the API also needs the Read Session User role on the project (Google Cloud, n.d.aa), and reads from Managed Service for Apache Spark clusters, or Spark on GKE, are charged at standard Storage Read API rates (Google Cloud, n.d.e). Figure 1 follows one Spark read through that path. The client creates a read session naming the table, columns, filter, and snapshot time, and then reads each stream the session returns (Google Cloud, n.d.aa).

Billing changes with this choice too: under logical billing, the default, time travel and fail-safe are included. Under physical billing they're charged separately at active storage rates, and once you switch you have to wait 14 days before switching again (Google Cloud, n.d.t). Tables that churn, such as bronze tables with frequent deletes, can carry more time travel bytes than you'd guess.

Replication has limits too. Cross-region dataset replication doesn't copy policy tags or data policies, and Google says it isn't meant as a disaster recovery plan (Google Cloud, n.d.h). If gold depends on column policies, they have to be set up separately in the secondary region.

Consequences

Native tables suit layers that depend on BigQuery's controls or are read mostly through it, and they're the safest home for personal data an erasure job has to reach. They fit less well when another engine has to write the layer in place, or when a team wants to read it as files without going through BigQuery.

Option B: Apache Iceberg managed tables

Managed Iceberg tables, formerly called BigLake tables for Apache Iceberg in BigQuery, keep their data as Parquet files in a Cloud Storage bucket you own. BigQuery commits every change to them.

What you get

Writes are ordinary GoogleSQL DML, load jobs, and streaming. Spark and Dataflow write through the Storage Write API, BigQuery's gRPC interface for streaming data in (Google Cloud, n.d.c; n.d.p). BigQuery does the housekeeping: it compacts and clusters the data files, and garbage-collects old data files once the time travel window has passed (Google Cloud, n.d.c).

Other engines can get at the data two ways. Through BigQuery, Spark can use the BigQuery connector, which reads with the Storage Read API (Google Cloud, n.d.e). Google's fine-grained access page shows Spark reading this way, with spark.read.format("bigquery"), for managed Iceberg tables that use a BigQuery connection override (Google Cloud, n.d.b). Or Spark can read the files. You export Iceberg V2 metadata with EXPORT TABLE METADATA, and Google's example has Spark reading the exported table through a path-based catalog on the bucket (Google Cloud, n.d.c). Google also describes automatic refresh of that metadata, turned on by request to Google, with export costs on each refresh (Google Cloud, n.d.c).

Column-level access control and masking are supported (Google Cloud, n.d.c), but only for reads that go through BigQuery, and Standard edition has no fine-grained controls at all (Google Cloud, n.d.y). A Spark job reading Parquet straight from the bucket never asks BigQuery, so it sees every column its Storage role lets it open.

Storage is billed by Cloud Storage, including historical table data, with no BigQuery storage fees on top. The automatic maintenance isn't free, though. Google bills it in Data Compute Units, per second, and load and export operations run on Enterprise edition pay-as-you-go slots, which standard tables aren't charged for (Google Cloud, n.d.c). For a table that gets many small streaming writes, maintenance and metadata export are the costs to watch.

What you give up

Google's full list is longer than this; check the page. As of this writing, managed Iceberg tables don't support any of the following (Google Cloud, n.d.c):

renaming, table copies, table clones, or table snapshots
materialised views
authorised views (column-level access control is supported)
managed disaster recovery
row-level security
fail-safe windows
extract jobs
partition evolution, or partitioning on columns other than DATE, DATETIME, or TIMESTAMP
some data types, including JSON, BIGNUMERIC, INTERVAL, RANGE, and GEOGRAPHY

Concurrency is tighter. Only one mutating DML statement, an UPDATE, DELETE, or MERGE, runs at a time on each managed Iceberg table, and others queue behind it (Google Cloud, n.d.c). A table that takes a merge every few minutes and also has erasure deletes running against it will queue. On the platforms we work on, batching helps: one DELETE per erasure run that removes every token in the run's hand-off batch, scheduled into a gap between merges, rather than one statement per person.

Readers on exported metadata see the table as of the last export. Iceberg metadata might not include data streamed through the Storage Write API in the last 90 minutes (Google Cloud, n.d.c), so a notebook reading through exported metadata can lag behind BigQuery's view. The same applies to deletes. Until the metadata is exported again, a reader using it still sees rows the erasure job has already deleted in BigQuery, so the erasure job for a managed Iceberg table ends with an export, unless automatic refresh is turned on and you've confirmed it runs after deletes. Even then, the old Parquet files stay in the bucket until garbage collection, anything with read access to the bucket can open them, and a reader pinned to an older metadata file can keep seeing the deleted rows. Figure 2 shows that sequence for a pipeline that reads the files directly.

On row-level security, Google's pages disagree. The managed tables page says row-level security isn't supported (Google Cloud, n.d.c). A separate Lakehouse runtime catalog page describes fine-grained access control, row-level security included, on a managed Iceberg table in BigQuery that uses a BigQuery connection override (Google Cloud, n.d.b). The two pages use overlapping terms for what may be different table setups, so we'd treat row-level security on managed Iceberg as unavailable unless a test in your own project shows otherwise.

The bucket

Your data now lives in objects that Storage roles can read without any BigQuery permission, so the bucket deserves the same care as the dataset. Google's setup has the connection's service account hold Storage Object User and Storage Legacy Bucket Reader on the bucket, and its guidance is to use a single-region bucket in the same region as the dataset, with uniform bucket-level access and public access prevention turned on (Google Cloud, n.d.c). It also recommends restricting write access for tools outside BigQuery, because files added or changed outside BigQuery can make a table unreadable or get garbage-collected (Google Cloud, n.d.c). We make the connection's service account the only identity outside break-glass access that can write or delete objects there.

A few more of Google's rules matter for erasure and rebuilds. Avoid object versioning, Object Lock, and Bucket Lock. Keep the default seven-day soft delete, and contact support if data needs restoring rather than restoring objects yourself (Google Cloud, n.d.c). Only create a table in an empty prefix, because files already there aren't tracked and background garbage collection deletes them (Google Cloud, n.d.c). Creating a table takes connection rights as well as dataset rights, BigQuery Data Owner and BigQuery Connection Admin in Google's list (Google Cloud, n.d.c). It's also an access problem. The connection's service account can read the whole bucket, so anyone who can use the connection and create tables could, as we read it, define an external table over the same Parquet files and read every column, or create a managed table on that prefix and have the existing files garbage-collected. As in Part 1, BigQuery Connection User and Connection Admin on the connection stay with the pipeline's deployment identity, since either one lets someone use it, and changes to them are audited.

Deleted data leaves by a slightly different route from native tables. Data files are garbage-collected after the time travel window (Google Cloud, n.d.c). Google doesn't say so in as many words, but we'd expect the collected files then to sit in the bucket's soft delete, which defaults to seven days and can be set as long as 90 (Google Cloud, n.d.x). There's no fail-safe stage. So the chain is time travel, then a garbage-collection lag you have to measure, as Part 1's worked example does, then soft delete. At defaults that's at least as long as a native table's chain, with soft delete standing where fail-safe would be, and longer by however long collection takes. Dropping a table is the case to watch: its data files aren't garbage-collected at all (Google Cloud, n.d.c), so a rebuild that drops and recreates a table leaves the old files behind until someone cleans up the path, and the recreated table needs a fresh, empty prefix anyway.

Consequences

Managed Iceberg suits a layer that BigQuery can own as the writer, that another engine wants to read as files, and that doesn't need anything on that list. Silver tables with long histories that data scientists read from Spark are the usual candidates. It suits personal data less well than native tables, because the controls you'd lean on for that data are the ones it drops, and because any file reader sits outside BigQuery's policies.

Option C: Iceberg in an open catalog

Here the catalog lives outside BigQuery's own metadata, and other engines write the files directly. On Google Cloud the obvious choice is the Lakehouse runtime catalog, formerly BigLake metastore, which offers an Apache Iceberg REST catalog endpoint (Google Cloud, n.d.q; n.d.w). Outside Google, Apache Polaris graduated to an Apache top-level project in February 2026 (Apache Polaris, 2026), Lakekeeper is an Iceberg REST catalog written in Rust and built on iceberg-rust (Lakekeeper, n.d.), and Project Nessie labels its Iceberg REST support experimental (Project Nessie, n.d.).

What you get

Spark writes through the endpoint in Google's setup guide (Google Cloud, n.d.w), and Google lists read and write support for Flink, Trino, and third-party engines such as Snowflake as a Preview capability (Google Cloud, n.d.s). If the catalog is set up for credential vending, a client sends the X-Iceberg-Access-Delegation: vended-credentials header and the catalog hands it credentials for the table's storage (Google Cloud, n.d.w). In that mode, people and jobs hold BigLake roles, and only the catalog's auto-provisioned service account needs Storage Object User on the bucket (Google Cloud, n.d.w). Google's setup page grants those roles on the project (Google Cloud, n.d.w), and its page on how the Lakehouse works says roles can also be granted on a namespace to cover the tables in it (Google Cloud, n.d.j). BigQuery reads these tables with a four-part name, project, catalog, namespace, then table, and sends the query to the catalog's primary region (Google Cloud, n.d.v; n.d.a). As we understand them, vended credentials work like bearer tokens until they expire, so a credential that leaves the engine still works, and write credentials can change files without a catalog commit. A VPC Service Controls perimeter around the bucket's project narrows where they can be used.

Google describes the read path in a few steps: the engine asks the catalog for the table's metadata, the catalog checks the request against IAM, and against fine-grained policies where the endpoint supports them, which the REST endpoint doesn't (Google Cloud, n.d.w), and it returns the metadata, plus a short-lived token if credential vending is on. The engine then reads the data files straight from Cloud Storage (Google Cloud, n.d.j). Figure 3 shows that path for a Spark job, followed by a write.

Iceberg's format and the REST protocol are open, so moving engines or catalogs later should cost less than leaving A or B. That's C's main case on lock-in. Exit costs for A and B follow the comparison table.

What you give up

By default, BigQuery is a reader. The REST endpoint setup page says you can't create or modify tables in the endpoint with BigQuery DDL or DML, and that changes made through the BigQuery API might be ones the external engine can't handle (Google Cloud, n.d.w).

Newer Lakehouse pages, marked Preview, say otherwise. They describe opting a REST catalog table into BigQuery DML with the table property gcp.biglake.bigquery-dml.enabled, and into automatic table management with gcp.biglake.table-management.enabled (Google Cloud, n.d.f). Tables created from BigQuery get both by default, and the overview page lists BigQuery DDL among the Preview capabilities (Google Cloud, n.d.s; n.d.f). The opt-in comes with conditions (Google Cloud, n.d.f):

the table has to be Iceberg format V2 and use copy-on-write for deletes, updates, and merges, so merge-on-read tables can't opt in
automatic table management needs credential vending turned on for the catalog
all the existing limitations of BigQuery-managed Iceberg tables apply
BigQuery's project and dataset time travel settings don't apply, and only the table's own Iceberg properties count
snapshot expiry is skipped for the whole table if it has any tags or branches
orphan files aren't cleaned up

We read that as the setup page describing the default and the newer pages describing a Preview opt-in. Until it's generally available, we'd plan option C tables as written by other engines and test the opt-in in a project of your own before relying on it.

BigQuery's fine-grained controls don't apply either. Tables managed through the REST catalog endpoint don't support fine-grained access control, row-level or column-level (Google Cloud, n.d.w). Access sits at the catalog and the bucket. Credential vending narrows who needs bucket roles, but anyone who does hold read access on the bucket can read every row in it.

The endpoint supports only Parquet files, doesn't support Iceberg V1, supports V2 tables as generally available and V3 tables in Preview, and caps each metadata.json at 1 MB (Google Cloud, n.d.w). Google's setup page also says to avoid paths that overlap with other catalogs, to prevent credentials being exposed to the wrong data, and to nest table paths within their namespace's path (Google Cloud, n.d.w), and its overview says nested namespaces aren't supported (Google Cloud, n.d.j). Administering the catalog needs Storage Admin on its buckets (Google Cloud, n.d.w), so the people who run it can read the data too.

The catalog now has to be available and recoverable. The Lakehouse runtime catalog offers cross-region replication and disaster recovery in Preview, for catalogs backed by dual-region or multi-region buckets (Google Cloud, n.d.z). With a self-run catalog such as Polaris or Lakekeeper, the catalog's database is one more thing to back up and restore.

And unless you opt in to table management, the maintenance is yours. Iceberg keeps every old snapshot, with the files it references, until something expires it. Iceberg's own guidance is to expire snapshots regularly so data files that are no longer needed get deleted and to remove orphan files left by failed writes, with compaction listed as optional (Apache Iceberg, n.d.d). Even with table management on, orphan cleanup stays with you (Google Cloud, n.d.f). The erasure section below goes through what that means.

Consequences

Option C suits tables another engine has to write: feature tables a data science team builds in Spark, or data shared with engines outside Google Cloud. It also needs the most operational work: someone has to own the maintenance schedule and the catalog, and anyone with bucket roles can read the data without the catalog.

The three side by side

The table below puts the three side by side, as Google's documentation described them when we read it in October 2026.

	A. Native	B. Managed Iceberg	C. Open catalog
Who writes	BigQuery	BigQuery, including Spark and Dataflow through the Storage Write API	Any engine through the catalog; BigQuery DML only by opt-in, in Preview
Where the files sit	BigQuery storage	Your bucket	Your bucket
Who maintains the files	Google	Google, billed in compute units	You, or Google's table management in Preview, with orphan cleanup still yours
Where access is enforced	BigQuery	BigQuery for reads through BigQuery; bucket roles for file reads	The catalog and the bucket
Authorised views	Yes	No	Not documented; treat as no
Row-level security	Yes, not on Standard	No on the managed tables page; unclear, see above	No
Column-level access control	Yes, not on Standard	Yes, for reads through BigQuery, not on Standard	No
Table snapshots and clones	Yes	No	Not documented; Iceberg tags and branches instead
Fail-safe	Yes, seven days	No; bucket soft delete (keep the seven-day default)	No; bucket soft delete, if enabled
Managed disaster recovery	Enterprise Plus	No	Catalog replication in Preview
Materialised views	Yes (Standard can query, not create)	No	Not documented; treat as no

The table below lists the line items each option adds, without prices, because most of them scale with data volume or team size.

What it takes to run	A. Native	B. Managed Iceberg	C. Open catalog
Storage	BigQuery storage, with time travel and fail-safe charged separately under physical billing (Google Cloud, n.d.t)	Cloud Storage, including historical table data (Google Cloud, n.d.c)	Cloud Storage
Maintenance compute	None of yours	Data Compute Units for automatic maintenance, and Enterprise pay-as-you-go slots for loads and metadata export (Google Cloud, n.d.c)	Your Spark jobs for expiry, orphan cleanup, and compaction
Reads from Spark	Storage Read API rates (Google Cloud, n.d.e)	Storage Read API rates through the connector; Cloud Storage reads for files	Cloud Storage reads
Skills	BigQuery and SQL	BigQuery and SQL, plus bucket IAM	Iceberg operations, Spark, catalog IAM, and bucket IAM
Operational duties	BigQuery's own	Metadata export after erasure, and bucket reviews	Maintenance schedule, catalog availability and recovery, and tag rules

Listing 1 shows the three in SQL: the same silver table created natively and as managed Iceberg, the copy from one to the other, the metadata export that ends an erasure batch on option B, and a query against a table Spark writes in an open catalog. The connection, bucket, and catalog names are illustrative.

SQL

-- Option A: a native table, partitioned by day and clustered by article.
CREATE TABLE silver.article_engagement (
  token STRING,
  article_id STRING,
  event_date DATE,
  seconds_engaged INT64
)
PARTITION BY event_date
CLUSTER BY article_id;

-- Option B: a managed Iceberg table with the same columns. BigQuery writes
-- its Parquet files under the storage URI, through a Cloud resource
-- connection whose service account can read and write that bucket, a
-- single-region bucket in the dataset's region (europe-west2 here). The
-- prefix must be empty: files already there would be garbage-collected.
CREATE TABLE silver.article_engagement_iceberg (
  token STRING,
  article_id STRING,
  event_date DATE,
  seconds_engaged INT64
)
PARTITION BY event_date
CLUSTER BY article_id
WITH CONNECTION `publisher-lake.europe-west2.lake-connection`
OPTIONS (
  file_format = 'PARQUET',
  table_format = 'ICEBERG',
  storage_uri = 'gs://publisher-lake-silver/article_engagement');

-- Moving data from A to B is ordinary DML. There's no in-place conversion,
-- and CREATE OR REPLACE can't swap one kind of table for the other.
INSERT INTO silver.article_engagement_iceberg
SELECT token, article_id, event_date, seconds_engaged
FROM silver.article_engagement;

-- After each erasure batch on option B, export the metadata again so readers
-- of the exported Iceberg metadata stop seeing the deleted rows.
EXPORT TABLE METADATA FROM silver.article_engagement_iceberg;

-- Option C: read a table that Spark writes through the Lakehouse runtime
-- catalog's Iceberg REST endpoint, using a four-part name. The catalog
-- (feature_store) lives in its own project, publisher-features. By default
-- BigQuery only reads it; BigQuery DML on these tables is a Preview opt-in.
SELECT article_id, COUNT(*) AS readers
FROM `publisher-features.feature_store.features.reader_affinity`
GROUP BY article_id;

Listing 1. The same silver table on native storage and managed Iceberg, and a query against an open-catalog table, in BigQuery SQL.

Moving a table between options later means creating a new table and copying the data, as the listing does. Google doesn't document an in-place conversion, and CREATE OR REPLACE can't swap a native table for a managed Iceberg table or the other way round (Google Cloud, n.d.c). Readers that go through views can be repointed once the copy is checked, so putting views in front of tables early keeps that exit cheaper.

Delta Lake and Apache Hudi come up too, but BigQuery's documentation describes them as read paths, Delta Lake as external tables and Hudi through exported manifests (Google Cloud, n.d.g; n.d.u), so they mostly turn up where another platform already owns the data.

Erasure when the files belong to an open catalog

Part 1 worked erasure through on options A and B, where BigQuery runs the deletes and Google's retention windows decide when deleted data is gone. On option C, a DELETE, or a MERGE that deletes as Listing 3 does, starts the process, and the rows stay in files until every later step finishes.

How many steps depends first on the table's write mode. Iceberg defaults to copy-on-write for deletes, updates, and merges (Apache Iceberg, n.d.b), though that's a table property each writing engine is expected to honour, so check what any Flink or Trino writers actually do, or keep them read-only on tables with personal data. A delete writes new data files without the deleted rows, the old files drop out of the current snapshot, and earlier snapshots still point to them. Tables set to merge-on-read write delete files instead: position deletes or equality deletes in format V2, or deletion vectors in V3 (Apache Iceberg, n.d.c). That's faster to write, and it means the deleted rows are still physically present in data files the current snapshot uses. Queries skip them, but anyone reading the Parquet directly can see them, and they stay until compaction rewrites those files.

Compaction at default settings may not do that. Spark's rewrite_data_files only picks a file because of its deletes once at least 30% of its rows are deleted, and the other trigger, delete-file-threshold, defaults to 2,147,483,647 deletes (Apache Iceberg, n.d.e). One reader's handful of rows in a large file reaches neither. So on a merge-on-read table, the erasure job runs compaction itself on the affected partitions, with delete-file-threshold set to 1, and, for V2 position deletes, rewrite_position_delete_files after it (Apache Iceberg, n.d.e). Equality delete files hold the deleted values themselves, tokens included. Compaction's remove-dangling-deletes option, off by default, removes delete files that no longer apply to any live data file. It goes by data sequence numbers and skips global or invalid equality deletes that match no data file (Apache Iceberg, n.d.e). Keeping equality deletes off tables with personal data avoids the question. For a table holding personal data, copy-on-write avoids that step altogether, and it's the only mode Google's table management accepts anyway (Google Cloud, n.d.f).

After that, the old files go only when every snapshot that references them is expired. Spark's expire_snapshots procedure, by default, removes snapshots older than five days and keeps at least the last one (Apache Iceberg, n.d.e). Tags and branches change that. A tagged snapshot is kept until the tag's maximum reference age passes, and the default retention for branches and tags is forever (Apache Iceberg, n.d.a; n.d.b). A tag a data scientist makes to pin an experiment's training data keeps every row it covers until someone removes the tag. Under Google's automatic table management it's stricter still: a table with tags or branches skips automatic expiry entirely (whether that counts the main branch is one of the tests below), though open-source engines can still run expiry themselves (Google Cloud, n.d.f).

Failed or abandoned writes leave a third kind of file, files no metadata references. remove_orphan_files deletes them, by default only files older than three days (Apache Iceberg, n.d.e), and Google warns that a threshold shorter than your longest-running query can delete files that query still needs (Google Cloud, n.d.f). Whatever gets deleted then moves into the bucket's soft delete, seven days by default (Google Cloud, n.d.x).

Working out the date

Part 1's calculator takes each store's stages as a chain and adds them up. For an option C table, one detail shortens it. Snapshot expiry goes by when a snapshot was created, and the last snapshot that still references a deleted row was created before the delete. So with the five-day setting, that snapshot is old enough to expire no later than five days after the delete, and it actually goes at the first expiry run after that.

Take a copy-on-write feature table with expiry run daily at five days, the bucket's soft delete left at seven days, and no tags. Day 0 here is the delete itself. A delete on day 0 leaves the old files referenced until the first expiry run after day 5, so day 5 or 6, depending on when the job runs. The files then sit in soft delete for seven days, which puts the end of this chain at about day 13. In Part 1's calculator the chain starts after the hand-off stage, so the lag before the feature-table job runs its delete comes first. A merge-on-read version of the same table starts the clock when the erasure job's own compaction finishes. Any tag over the old snapshot stops the clock entirely until the tag goes, so the tag rule belongs in the platform standards: a table with personal data has no long-lived tags, or the erasure job removes or re-points the ones covering an erased person's rows.

Counted from the request, add Part 1's hand-off lag and up to a day for the daily feature job in front of this chain. The latest date across all the stores is the figure to compare with GDPR's one month for responding under Article 12(3), before any extension, and Article 17's "without undue delay", and with the 45 days the CCPA allows, also before any extension (European Parliament and Council of the European Union, 2016; State of California Department of Justice, n.d.). A single long-lived tag can push past both. Neither law is written in terms of storage windows, so treat the dates as engineering input for your privacy team. The date also covers the Iceberg files only. The scheduled job's native output tables are rebuilt the day after, and the rows they replace go through time travel and fail-safe, so with time travel set to the two-day minimum on that dataset, that store ends around day 10 from the delete, or day 15 at the seven-day default. Silver's own chain, with its measured garbage-collection lag, may run later still. Feature tables keep no column bounds for the token, so their manifests don't add to it, but any table that does keep bounds on an identifying column holds them until a later commit replaces the deleting snapshot, as the next section explains.

Under Google's table management the arithmetic changes. Expiry runs in the background after data changes, on Google's timing (Google Cloud, n.d.f). A table that's written every day will expire snapshots much as a daily job would. A table nobody writes to after the erasure may never run expiry at all, so if you count on table management, check the jobs in INFORMATION_SCHEMA.JOBS, where Google logs them (Google Cloud, n.d.f).

Listing 2 shows snapshot expiry from Python with PyIceberg, for teams that run maintenance outside Spark. PyIceberg's maintenance API expires snapshots older than a given time (Apache Iceberg, n.d.f), and its reference documentation says it expires only unprotected snapshots, treating the heads of branches and all tagged snapshots as protected (Apache Iceberg, n.d.h). It doesn't say whether expiry also deletes the files those snapshots referenced. If it doesn't, those files become orphans, and their chain runs from expiry to the next orphan cleanup and then soft delete. So we'd still run orphan cleanup in Spark, with the prefix_listing option Google suggests for these catalogs (Google Cloud, n.d.f), and look in the bucket after the first run. The Rust client, iceberg-rust, has an expire_snapshots action on its transactions, but its options aren't documented yet (Apache Iceberg, n.d.i), which is why this listing is Python only.

Python

from datetime import datetime, timedelta, timezone

from pyiceberg.catalog import load_catalog

# Catalog settings (REST endpoint, warehouse, auth) come from .pyiceberg.yaml.
catalog = load_catalog("lake")
table = catalog.load_table("features.reader_affinity")

# Expire snapshots older than five days, matching Spark's default age. If
# expiry here doesn't delete the files it stops referencing, they become
# orphans, so add the next orphan-cleanup run to this table's chain.
table.maintenance.expire_snapshots().older_than(
    datetime.now(timezone.utc) - timedelta(days=5)
).commit()

Listing 2. Expiring snapshots older than five days with PyIceberg, to match Spark's default.

What the manifests hold

Iceberg's manifests can also hold personal data, because they store column statistics. By default it stores lower and upper bounds for each column, truncated to 16 characters for strings, for up to the first 100 columns (Apache Iceberg, n.d.b). A truncated bound on a name or email column can be enough to identify someone, and it sits in manifests that no query against the data ever touches. Deletes don't clear it straight away either. A delete records the removed file in a manifest as a deleted entry, with its statistics still attached, and that entry can only go once the snapshot that deleted it is expired (Apache Iceberg, n.d.c). Older manifests that added the file keep its bounds too. So once a later commit has replaced the deleting snapshot, the statistics leave on the same chain as the data files, snapshot expiry and then soft delete. On a table nobody writes to after the erasure, the deleting snapshot stays current and expiry always keeps it, so the bounds stay until the next commit. The check after expiry is the all_entries metadata table, which covers every snapshot the table still has, rather than entries, which covers only the current one (Apache Iceberg, n.d.g).

The fix is a table property. Setting write.metadata.metrics.column.<name> to none or counts for each identifying column stops Iceberg storing its bounds (Apache Iceberg, n.d.b). In Part 1's design the identifying columns live in native tables anyway, but any option C table that carries a free-text field or an address should have this set from the start. Google doesn't say what statistics BigQuery writes into the manifests of managed Iceberg tables, so for option B that's something to check in your own bucket.

The decision for the example platform

The publisher decides layer by layer. A platform whose data scientists need bucket access would split differently.

Bronze, the identity table, and the restricted stores: native. Part 1's design reads them only through authorised views, keeps deleted rows away from anyone without table access, and counts time travel and fail-safe as a known recovery chain. Managed Iceberg would take away the authorised views, and with them the design's main access control.

Silver reading history: managed Iceberg, with conditions. Training jobs scan years of reading behaviour, the team wants that history as Parquet that open tooling can read, and BigQuery is the only writer, so managed Iceberg fits the shape of the work. Silver here carries tokens and no names, and the data scientists already have table access to it, so losing authorised views and row-level security costs little.

Reading history is the part that needs care. What someone reads can point to their health, their religion, or their politics, and data revealing those is in the special categories that Article 9 of the GDPR restricts (European Parliament and Council of the European Union, 2016). Recital 26 treats pseudonymised data that can be attributed to a person with additional information as information about an identifiable person (European Parliament and Council of the European Union, 2016), and the identity table is that additional information. California's CCPA treats information about a consumer's health and religious or philosophical beliefs as sensitive personal information (State of California Department of Justice, n.d.), so the same care applies to US readers. That's the strongest argument for keeping this table native, and on a platform with less control over who reads files, it would be the right call. We weighed the obvious alternative: keep the history native, with its authorised views and row-level security, and export Parquet for training. It meets the training pipeline's need too. For years of history, a full re-export on every refresh is a large recurring job, so in practice the export would be incremental, by partition, and erasure would need a second path that rewrites every export partition holding an erased token. Set side by side, the export's exposure is the lag before that rewrite plus the export bucket's soft delete; managed Iceberg's is time travel, the garbage-collection lag, and the seven-day soft delete Google asks you to keep, during which bucket roles can read the old files. Managed Iceberg also needs a metadata export after each erasure and a change to when silver confirms. We chose managed Iceberg to avoid running an incremental export and its partition-rewrite path, and we accept a longer window in which the training pipeline's identity can read old files. If that window is too long for your privacy team, take the export route. The read paths here:

The reading history sits in a dataset of its own, so dataset roles on it don't reach other silver tables.
Notebooks and ad hoc work read the table through the Spark BigQuery connector's direct reads, which use the Storage Read API, so access is governed by BigQuery roles and every read goes through BigQuery. Each data scientist works on a personal runtime with its own service account, and only that person holds Service Account User on it, granted on the service account itself, so access stays per person; those service accounts get BigQuery Data Viewer on the dataset and Read Session User on the project the connector bills (Google Cloud, n.d.aa; GoogleCloudDataproc, n.d.), and no bucket roles. On this dataset they don't use the connector's query mode, which saves results into temporary tables (GoogleCloudDataproc, n.d.). IAM can't stop someone who can read the table copying it somewhere they can write, so every scratch dataset and bucket data scientists can write to inside the perimeter counts as an erasure store, with a short table expiration or lifecycle rule and a sweep that removes anything older. Time travel still lets them read erased rows until its window closes, which this table's erasure chain counts, as Part 1 does for its own tables.
The table carries no column that would need a column policy, because the next path would go around it.
The scheduled training pipeline reads the Parquet files directly, as a deliberate exception. It runs as one service account with Storage Object Viewer on a bucket that holds only this table, and its runtime sits in a separate project from the bucket. Every way to run as that service account is held as tightly as the bucket role: Service Account User and Token Creator on it, including grants inherited from the project, folder, or organisation; job-submit rights on its runtime project; login access to its runtime's VMs; Workload Identity User if it runs on GKE; the right to trigger its workflow; and write access to its code, image, and workflow definition. No basic roles are held on any of these projects. Creating keys for it is blocked by organisation policy.
The connection's service account is the only identity that can write or delete in the bucket. The bucket sits in a project of its own, and an IAM deny policy on that project denies object reads to everyone except the pipeline's service account, the connection's service account, and break-glass administrators, and denies writes, deletes, and restores from soft delete to everyone except the last two. Whether Cloud Storage's object permissions can be denied this way is one of the tests below, and if they can't, the fallback is a bucket project under a folder where no organisation- or folder-level role grants object read, with those inherited grants reviewed, and the rights to change the deny policy are held as tightly as the bucket roles. One VPC Service Controls perimeter holds every project involved, the silver bucket's, the feature catalog's and bucket's, the Spark runtimes', the jobs', and BigQuery's, with BigQuery, Cloud Storage, and the Lakehouse runtime catalog API restricted, so those APIs can't move the data to a project outside it. Managed Service for Apache Spark may need extra steps to sit inside a perimeter, so check Google's current page. Looker, people using the console, and the partner feed each need an ingress or egress rule, and those rules are reviewed like any other way out. BigQuery Standard edition can't run inside a perimeter, which is one more reason this platform runs Enterprise. A perimeter covers Google API traffic only, so the runtimes have no external IP addresses, reach Google APIs through the restricted endpoint with Private Google Access, and allow egress only to that endpoint and its DNS, plus the traffic Spark's workers need inside the subnet, and the partner feed leaves through one named, reviewed path. The partner feed is a downstream recipient under Part 1's process, with its own erasure arrangement.
The silver erasure job, which holds DML rights on the history table and the permission EXPORT TABLE METADATA needs, found by testing, removes each run's tokens in one batched DELETE, between merges, because of the single-writer limit on mutating DML, then exports the metadata again so reads through it stop returning the deleted rows, and only then writes its confirmations, one row per request and pass, as in Part 1. That's a change from Part 1's silver job, which confirmed in the same transaction as its delete; here the confirmation waits for the export, so the mapping can't go while the training pipeline still sees the rows through old metadata. The old files stay readable to the training pipeline's service account until garbage collection after the time travel window, so that window is part of this table's erasure chain. The pipeline's code reads only the current snapshot, which is a code-review rule; IAM can't enforce it, and anyone holding the service account could read older metadata files.

Other silver tables, and gold: native. Subscription revenue in silver is read through BigQuery and has no use for open files. Gold feeds Looker dashboards, uses column policies on revenue fields, and benefits from materialised views. Gold is built by scheduled SQL that reads silver, including the managed Iceberg history table, and writes native gold tables. Gold stays a registered downstream erasure job, as in Part 1. Its builds leave out tokens still in the hand-off table, and like the feature job it confirms a pass only after silver has. Gold's materialised views and authorised views sit over those native tables. Google lists materialised views and authorised views among the features managed Iceberg tables don't support (Google Cloud, n.d.c), so we'd keep both off the Iceberg table itself and not spend time testing where that line falls. That's the same answer Part 1 reached for a gold built over managed Iceberg silver: a table built by its own job.

Feature tables written in Spark: open catalog, in the Lakehouse runtime catalog. Spark jobs write them, and Spark and, now and then, BigQuery read them.

What they hold. Tokens and derived features, no direct identifiers. Copy-on-write, with column statistics set to counts for the token and any free-text field, so manifests hold no bounds on them.
Where the catalog sits. In its own project, in the same region as the datasets it's joined with, because BigQuery sends queries against it to the catalog's primary region (Google Cloud, n.d.a). Paths don't overlap with any other catalog's, and there are no nested namespaces.
Who has access. BigLake roles on the feature namespace, which Google's page on how the Lakehouse works allows (Google Cloud, n.d.j), so a grant doesn't reach every table in the catalog. The catalog runs in credential vending mode, so its service account is the only identity with write rights through a bucket-wide role, apart from break-glass access (Google Cloud, n.d.w). Catalog administrators hold Storage Admin because Google requires it, treated as break-glass access with alerting. Analysts get no bucket roles. The same deny-policy pattern as silver covers the feature bucket, with exceptions only for the identities named here, and no basic roles are held on the catalog or bucket projects, since those would get around the namespace grants.
How BigQuery reads them. BigQuery reading with vended credentials is in Preview (Google Cloud, n.d.s), so for now BigQuery reads run as one scheduled job's service account, with BigLake Viewer on the namespace, BigQuery Job User, Data Editor on its output dataset only, and, pending a test, Storage Object Viewer on the bucket. That bucket grant reaches every table in the bucket, so the bucket holds feature tables only, and the job rebuilds its native output tables in full on every run, so an erased token drops out on the next run after the feature tables lose it. Those tables aren't a registered erasure job, so for up to a day after the mapping goes they can still hold an erased token. The output dataset's time travel is set to the two-day minimum, because analysts with dataset roles could otherwise read earlier versions, and those tables count in the erasure chain.
How erasure runs. BigQuery can't run DML on these tables without the Preview opt-in, so erasure is a Spark job. It reads Part 1's hand-off table, collects the passes it hasn't confirmed once, deletes those tokens, and writes its own confirmation for exactly that set, request ID and pass number only. Feature builds read silver through the BigQuery connector, never through older exported metadata. Each day the workflow runs the feature builds and then this job, never at the same time, so every row a build writes is in the tables before the job deletes. The job confirms a pass only after silver has confirmed it, so no build that starts later can read the token from silver, and it deletes every token still in the hand-off table on every run, so a token written back by a build that read silver earlier goes again on the next run. Builds also leave out tokens still in the hand-off table. The job handles every registered feature table and confirms once every MERGE has committed and its checks on refs and table properties pass. The chain in "Working out the date" starts when the job runs.
What the job's identity holds. BigLake Editor on the feature namespace; BigQuery Data Viewer on the hand-off table and silver's confirmation table only, at table level; a custom role on its own confirmation table with bigquery.tables.get, getData, and updateData, and no bigquery.jobs.create anywhere, so it can append confirmations but can't run DML against them or delete the table; Read Session User on the project the connector bills; and Service Usage Consumer on the catalog's project. Google's quickstart grants that role to the Compute Engine default service account (Google Cloud, n.d.v); giving it to the job is our reading, because the catalog's project is the one named in the billing header (Google Cloud, n.d.w). The job runs in a project of its own, separate from the buckets, and the same run-as rights as the training pipeline's are held tightly. Because it waits for silver, it can confirm up to a day after silver does, so Part 1's downstream confirmation timeout for the feature job needs at least that margin, and the identity table's date moves out by the same amount. Feature-build jobs hold BigLake Editor on the namespace too, which also lets them set table properties, so a build could opt a table into Google's Preview capabilities. The job checks for that before it confirms.
Who maintains them. Snapshot expiry and orphan cleanup run in the same daily job as erasure, under the same identity, and the platform team owns the schedule. Orphan cleanup lists and deletes files under the table paths, which vended credentials may not allow. If it needs a bucket role, that role is granted on managed folders for the table paths, never bucket-wide, because an IAM condition on the object name doesn't narrow listing. Even path-scoped delete rights let the job remove live files without a catalog commit, so the role is watched like the catalog's own. Tags and extra branches aren't allowed, and since anyone with BigLake Editor could create one, the job checks the tables' refs before it confirms and fails if it finds anything other than main (Apache Iceberg, n.d.g). If namespace-only grants turn out not to work, the fallback is a separate catalog for the feature tables.

Listing 3 is that job, run daily. It uses the Spark catalog settings from Google's setup guide (Google Cloud, n.d.w), reads and writes the BigQuery tables through the connector (GoogleCloudDataproc, n.d.), fixes the set of pending passes once, removes every token still in the hand-off table with one MERGE per feature table, checks refs and table properties, confirms exactly the passes silver has already confirmed, expires snapshots older than five days, and removes orphan files with prefix listing (Apache Iceberg, n.d.e; Google Cloud, n.d.f). The snapshots holding today's deleted rows go on a later run, which is why the date arithmetic above counts the expiry run after day 5. It's Python only, because the job is Spark.

Python

"""Daily erasure and maintenance job for the feature tables (option C).

Runs on Managed Service for Apache Spark against the Lakehouse runtime
catalog's Iceberg REST endpoint, with Iceberg 1.10 or later (Google's
setup guide gives 1.10 for GoogleAuthManager) and the Iceberg GCP bundle. The
catalog settings follow that guide. The catalog, feature_store, lives in the
publisher-features project; other names are illustrative.

It runs in the same daily workflow as the feature builds, after them and
never at the same time, so every row a build writes is in the tables before
this job deletes. It confirms a pass only after silver has confirmed it, so
no later build can read the token from silver, and it deletes every token
still in the hand-off table on every run, so one written back by a build that
read silver earlier goes again. It handles every registered feature table and
confirms once every MERGE has committed and the refs and properties checks
pass.

The job's service account holds:
- BigLake Editor on the feature namespace
- no basic roles on any of the projects involved
- BigQuery Data Viewer on erasure_handoff.tokens and
  erasure_confirm_silver.confirmations only, granted at table level
- a custom role on erasure_confirm_features.confirmations with
  bigquery.tables.get, bigquery.tables.getData, and bigquery.tables.updateData,
  and no bigquery.jobs.create anywhere, so it can append rows but can't run
  DML against the table or delete it
- Read Session User on publisher-lake, the connector's parentProject
- Service Usage Consumer on publisher-features, the project named in
  x-goog-user-project
- if orphan cleanup needs it, an object role on managed folders for the table
  paths only, never bucket-wide
The job runs in a project of its own, separate from the buckets.
"""

from datetime import datetime, timedelta, timezone

from pyspark.sql import SparkSession

CATALOG = "lake"  # Spark's name for the catalog
FEATURE_TABLES = ["features.reader_affinity"]  # every registered feature table
BQ_PROJECT = "publisher-lake"
HANDOFF = f"{BQ_PROJECT}.erasure_handoff.tokens"
SILVER_DONE = f"{BQ_PROJECT}.erasure_confirm_silver.confirmations"
FEATURES_DONE = f"{BQ_PROJECT}.erasure_confirm_features.confirmations"

spark = (
    SparkSession.builder.appName("feature-erasure")
    .config("spark.sql.session.timeZone", "UTC")
    .config("spark.sql.defaultCatalog", CATALOG)
    .config(f"spark.sql.catalog.{CATALOG}", "org.apache.iceberg.spark.SparkCatalog")
    .config(f"spark.sql.catalog.{CATALOG}.type", "rest")
    .config(
        f"spark.sql.catalog.{CATALOG}.uri",
        "https://biglake.googleapis.com/iceberg/v1/restcatalog",
    )
    .config(
        f"spark.sql.catalog.{CATALOG}.warehouse",
        "bl://projects/publisher-features/catalogs/feature_store",
    )
    .config(f"spark.sql.catalog.{CATALOG}.header.x-goog-user-project", "publisher-features")
    .config(
        f"spark.sql.catalog.{CATALOG}.rest.auth.type",
        "org.apache.iceberg.gcp.auth.GoogleAuthManager",
    )
    .config(f"spark.sql.catalog.{CATALOG}.io-impl", "org.apache.iceberg.gcp.gcs.GCSFileIO")
    .config(
        f"spark.sql.catalog.{CATALOG}.header.X-Iceberg-Access-Delegation",
        "vended-credentials",
    )
    .config(
        "spark.sql.extensions",
        "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",
    )
    .getOrCreate()
)

def read_bq(table: str):
    return spark.read.format("bigquery").option("parentProject", BQ_PROJECT).load(table)

def utc_cutoff(days: int) -> str:
    return (datetime.now(timezone.utc) - timedelta(days=days)).strftime("%Y-%m-%d %H:%M:%S")

def erase() -> None:
    # 1. Read, once, the passes silver has confirmed and this job hasn't.
    #    Collecting them fixes the set, so the job confirms exactly the
    #    passes it deleted, even if new rows arrive while it runs. Erasure
    #    batches are small, so they fit on the driver.
    read_bq(SILVER_DONE).createOrReplaceTempView("silver_done")
    read_bq(FEATURES_DONE).createOrReplaceTempView("features_done")
    read_bq(HANDOFF).createOrReplaceTempView("handoff")
    pending_df = spark.sql("""
        SELECT s.request_id, s.pass_number
        FROM silver_done s
        LEFT ANTI JOIN features_done f
          ON f.request_id = s.request_id AND f.pass_number = s.pass_number
    """)
    pending = pending_df.distinct().collect()

    # Delete every token still in the hand-off table, not just the new
    # passes, so a token written back before silver confirmed goes again.
    spark.sql("SELECT DISTINCT token FROM handoff").createOrReplaceTempView("erasure_batch")

    # 2. One MERGE per feature table removes those tokens. The tables
    #    are copy-on-write, so this writes new files without the rows, as a
    #    DELETE would. A failed commit raises, and nothing is confirmed.
    for table in FEATURE_TABLES:
        spark.sql(f"""
            MERGE INTO {CATALOG}.{table} t
            USING erasure_batch b
            ON t.token = b.token
            WHEN MATCHED THEN DELETE
        """)

    # 3. Refuse to confirm if any table has a tag or a non-main branch, which
    #    would hold erased rows indefinitely, or a gcp.biglake.* property
    #    opting it into Google's Preview capabilities.
    check_tables()

    # 4. Confirm each pass in the fixed set, request ID and pass number only,
    #    after every MERGE has committed.
    if not pending:
        return
    passes = spark.createDataFrame(pending, schema=pending_df.schema)
    (
        passes.write.format("bigquery")
        .option("parentProject", BQ_PROJECT)
        .option("writeMethod", "direct")
        .mode("append")
        .save(FEATURES_DONE)
    )

def check_tables() -> None:
    problems = []
    for table in FEATURE_TABLES:
        refs = spark.sql(
            f"SELECT name, type FROM {CATALOG}.{table}.refs WHERE name <> 'main'"
        ).collect()
        problems += [f"{table}: {r.type} {r.name}" for r in refs]
        props = spark.sql(f"SHOW TBLPROPERTIES {CATALOG}.{table}").collect()
        problems += [
            f"{table}: {r.key}={r.value}"
            for r in props
            if r.key.startswith("gcp.biglake.") and r.value.lower() == "true"
        ]
    if problems:
        raise RuntimeError(f"not safe to confirm: {problems}")

def maintain() -> None:
    for table in FEATURE_TABLES:
        # 5. Expire snapshots older than five days, keeping the last one. The
        #    snapshots that still hold today's deleted rows go on a later run.
        spark.sql(f"""
            CALL {CATALOG}.system.expire_snapshots(
                table => '{table}',
                older_than => TIMESTAMP '{utc_cutoff(5)}',
                retain_last => 1)
        """)

        # 6. Remove files no snapshot references, left by failed writes (or
        #    by a non-Spark expiry that doesn't delete them), using the prefix
        #    listing Google's table-options page suggests. Three days is
        #    Iceberg's default; raise it if any query runs longer than that.
        spark.sql(f"""
            CALL {CATALOG}.system.remove_orphan_files(
                table => '{table}',
                older_than => TIMESTAMP '{utc_cutoff(3)}',
                prefix_listing => true)
        """)

if __name__ == "__main__":
    # Maintenance runs only after a clean erasure pass, so a refused
    # confirmation (a tag, a branch, or an opt-in) surfaces as the job's error.
    erase()
    maintain()

Listing 3. The daily erasure and maintenance job for the open-catalog feature tables, in PySpark.

Listing 4 turns the whole decision into code. Each option lists only the capabilities Google documents for it as generally available, and each layer lists what it can't do without. It's deliberately small, and the capability lists are the part to update whenever Google changes what managed Iceberg or the Lakehouse runtime catalog supports.

Python

"""Check what each layer needs against what each storage option documents.

The capability lists come from Google's documentation for each option, as
read in October 2026. They change, so re-check them before relying on this.
Edition matters for options A and B: row-level security, column-level
access control, and data masking aren't on Standard, and managed disaster
recovery is Enterprise Plus only.
"""

from __future__ import annotations

KNOWN = frozenset({
    "authorised_views",
    "row_level_security",
    "column_level_security",
    "data_masking",
    "table_snapshots",
    "fail_safe",
    "managed_disaster_recovery",
    "materialised_views",
    "bigquery_dml",
    "files_in_your_bucket",
    "other_engines_write_files",
})

CAPABILITIES: dict[str, frozenset[str]] = {
    # A. BigQuery native tables
    "A": frozenset({
        "authorised_views",
        "row_level_security",
        "column_level_security",
        "data_masking",
        "table_snapshots",
        "fail_safe",
        "managed_disaster_recovery",
        "materialised_views",
        "bigquery_dml",
    }),
    # B. Apache Iceberg managed tables: Parquet in your bucket, written
    # only through BigQuery.
    "B": frozenset({
        "column_level_security",
        "data_masking",
        "bigquery_dml",
        "files_in_your_bucket",
    }),
    # C. Iceberg in an open catalog: other engines write the files, and
    # BigQuery reads them. BigQuery DML on these tables is an opt-in in
    # Preview, so it's left out here until it's generally available.
    "C": frozenset({
        "files_in_your_bucket",
        "other_engines_write_files",
    }),
}

def _check(needs: set[str]) -> None:
    unknown = needs - KNOWN
    if unknown:
        raise ValueError(f"unknown needs: {sorted(unknown)}")

def missing(option: str, needs: set[str]) -> list[str]:
    """What a layer needs that the option doesn't document."""
    _check(needs)
    if option not in CAPABILITIES:
        raise ValueError(f"unknown option: {option}")
    return sorted(needs - CAPABILITIES[option])

def options_for(needs: set[str]) -> list[str]:
    """Options that document everything the layer needs."""
    _check(needs)
    return [o for o in sorted(CAPABILITIES) if not missing(o, needs)]

Rust

//! Check what each layer needs against what each storage option documents.
//!
//! The capability lists come from Google's documentation for each option,
//! as read in October 2026. They change, so re-check them before relying on
//! this. Edition matters for options A and B: row-level security,
//! column-level access control, and data masking aren't on Standard, and
//! managed disaster recovery is Enterprise Plus only.

use std::collections::BTreeSet;

pub const KNOWN: [&str; 11] = [
    "authorised_views",
    "row_level_security",
    "column_level_security",
    "data_masking",
    "table_snapshots",
    "fail_safe",
    "managed_disaster_recovery",
    "materialised_views",
    "bigquery_dml",
    "files_in_your_bucket",
    "other_engines_write_files",
];

/// Options in order, each with the capabilities its documentation lists.
pub fn capabilities() -> Vec<(&'static str, BTreeSet<&'static str>)> {
    vec![
        // A. BigQuery native tables
        (
            "A",
            BTreeSet::from([
                "authorised_views",
                "row_level_security",
                "column_level_security",
                "data_masking",
                "table_snapshots",
                "fail_safe",
                "managed_disaster_recovery",
                "materialised_views",
                "bigquery_dml",
            ]),
        ),
        // B. Apache Iceberg managed tables: Parquet in your bucket,
        // written only through BigQuery.
        (
            "B",
            BTreeSet::from([
                "column_level_security",
                "data_masking",
                "bigquery_dml",
                "files_in_your_bucket",
            ]),
        ),
        // C. Iceberg in an open catalog: other engines write the files,
        // and BigQuery reads them. BigQuery DML on these tables is an
        // opt-in in Preview, so it's left out here until it's generally
        // available.
        (
            "C",
            BTreeSet::from(["files_in_your_bucket", "other_engines_write_files"]),
        ),
    ]
}

fn check(needs: &BTreeSet<&str>) -> Result<(), String> {
    let unknown: Vec<&str> = needs.iter().copied().filter(|n| !KNOWN.contains(n)).collect();
    if unknown.is_empty() {
        Ok(())
    } else {
        Err(format!("unknown needs: {unknown:?}"))
    }
}

/// What a layer needs that the option doesn't document.
pub fn missing(option: &str, needs: &BTreeSet<&str>) -> Result<Vec<String>, String> {
    check(needs)?;
    let caps = capabilities()
        .into_iter()
        .find(|(name, _)| *name == option)
        .map(|(_, caps)| caps)
        .ok_or_else(|| format!("unknown option: {option}"))?;
    Ok(needs
        .iter()
        .filter(|n| !caps.contains(*n))
        .map(|n| n.to_string())
        .collect())
}

/// Options that document everything the layer needs.
pub fn options_for(needs: &BTreeSet<&str>) -> Result<Vec<&'static str>, String> {
    check(needs)?;
    Ok(capabilities()
        .into_iter()
        .filter(|(_, caps)| needs.iter().all(|n| caps.contains(n)))
        .map(|(name, _)| name)
        .collect())
}

Listing 4. Checking each layer's needs against what each storage option documents.

The Python needs version 3.9 or later, and the Rust uses only the standard library.

Consequences of the decision

The platform team now runs three kinds of storage, each with its own identities to review. Native tables need dataset roles. Managed Iceberg adds the bucket, with the connection's service account, the training pipeline's service account, break-glass administrators, the deny policy, and the perimeter, plus the silver erasure job, which deletes and exports, and the gold build, which reads the history through BigQuery. The open catalog adds BigLake roles on a namespace, the catalog's service account, catalog administrators, the feature-build jobs, the read job and its output tables, and the erasure job with its BigQuery, confirmation, and possibly managed-folder object roles.

The erasure job from Part 1 gains two paths. The managed Iceberg silver table has no fail-safe, so its chain is time travel, a measured garbage-collection lag, then soft delete, at least as long as a native table's, and the job has to batch its deletes and re-export metadata at the end. The feature tables get a daily Spark job for erasure and maintenance, and a tag rule that someone has to own and check.

In return, the training pipeline reads the full history as Parquet without a separate copy of the data, and Spark writes feature tables directly, while identifying data stays on native tables.

What a data engineer still has to test

Test these in your own project before anything moves.

Test	How	Pass
Row-level security on managed Iceberg	Create a row access policy on a test table with a connection override	It's accepted and filters connector reads
Connector reads with the narrowed roles	Read the history table from Spark as a data scientist with direct reads, then try to write the rows to a project outside the perimeter and to the internet	Reads work with Data Viewer and Read Session User, and both writes fail
Metadata export after erasure	Delete a token, export, then read the files from Spark	The row is gone after the export, and you know whether older snapshots are still reachable
Stale reads and refresh	Stream rows in, read through exported metadata	You know the lag, and whether automatic refresh runs after deletes
Manifest statistics	Inspect the manifests of a managed Iceberg table	You know which column bounds BigQuery writes
Who can read the buckets	List IAM on the silver and feature buckets, including inherited roles, then try reads and deletes as an identity the deny policy should stop	Only the identities named above can read, and the deny policy blocks the rest
The training pipeline's identity	Try to write and delete as it; list who can impersonate it or submit to its runtime	Writes fail, and the lists match what you intended
The BigQuery connection	Try to create an external table over the silver files as a non-deployment identity	It fails
Vended credentials	Load the catalog config and tables as BigLake Viewer and Editor granted only on the feature namespace	Namespace-only grants are enough, credentials cover only those paths, and Viewer's are read-only
BigQuery reads of feature tables	Query a feature table from BigQuery	You know which identity reads the files, and analysts don't need bucket roles
Listing 3 end to end	Run it against test copies of the catalog, hand-off, and confirmation tables; add hand-off rows while it runs, write an erased token back through a build that read silver before silver confirmed, and add a tag	It confirms only the passes it deleted, confirms only after silver, deletes the written-back token on the next run, refuses to confirm with the tag, appends confirmations without `jobs.create` and with the table's schema mode, expiry deletes files through vended credentials, and you know whether an empty `MERGE` commits a snapshot and which managed-folder role orphan cleanup needs
Merge-on-read compaction	Compact a test table with position and equality deletes, with `delete-file-threshold` 1 and `remove-dangling-deletes` on	The erased token is gone from data files and delete files, and you know which equality deletes stay
Personal runtimes	Read the history as data scientist A	The audit log names A's runtime service account (confirm what the logs record), and no shared identity can read it
`EXPORT TABLE METADATA` permissions	Run the silver erasure job with only the roles listed	You know the exact permission the export needs
Table properties through the catalog	Set a `gcp.biglake.*` property on a test table and run Listing 3's check	`SHOW TBLPROPERTIES` returns it and the job refuses to confirm
Transactions on managed Iceberg	Run the silver merge as a multi-statement transaction	It commits or rolls back as one
Silver confirmation order	Run the silver erasure job and stop it after the delete	No confirmation is written until the metadata export has finished
PyIceberg expiry	Run Listing 2 on a test table, then list the data prefix	You know whether it deletes files or leaves orphans
The Preview opt-in, if you try it	Enable table management on a test table, with and without a tag	Expiry runs as often as your erasure dates assume, and you know whether the main branch alone stops it
The perimeter	Call each restricted API from outside the perimeter and from a runtime inside it	Calls from outside fail, and the runtimes reach Google APIs only through the restricted endpoint
Listing 1	Run it as a dry run against real dataset, connection, and bucket names	It validates

Turn on Data Access audit logs for Cloud Storage in both bucket projects, and for the BigLake API (biglake.googleapis.com) in the catalog's project, before anything moves, so file reads are recorded and break-glass alerts have something to fire on. We'd expect Spark reads made with vended credentials to appear in the Cloud Storage logs under the catalog's service account, so confirm that in testing and attribute those reads through the catalog's own logs.

Where this leaves you

For the example platform, the decision is native tables for bronze, the identity and restricted stores, gold, and most of silver; managed Iceberg for the silver reading history, read through the BigQuery connector apart from one locked-down training pipeline; and an open catalog for the feature tables Spark writes. Several of the calls depend on features that were in Preview when we read the pages, and these changes would reopen them:

Row-level security on managed Iceberg confirmed or generally available: other silver tables become candidates for managed Iceberg.
BigQuery DML on catalog tables generally available: feature-table erasure can move into BigQuery, and the Spark job shrinks to maintenance.
Automatic table management generally available: it can take over snapshot expiry on the feature tables, as long as they stay free of tags. Orphan cleanup stays yours.
BigQuery credential vending generally available: the scheduled-job workaround for BigQuery reads of feature tables can go.
Any move between options: it stays a copy plus repointing views.

The next part moves from storage to matching: what happens when match rules improve and the IDs they assign change underneath the tables that stored the old ones.

This post describes engineering approaches, based on Google's documentation as read in October 2026. It isn't legal advice. Product limits change, and several of the features discussed are in Preview, so check the current pages before relying on any of them.

References

Apache Iceberg, n.d.a. Branching and tagging. Apache Iceberg documentation. Available at: https://iceberg.apache.org/docs/latest/branching/ [Accessed 16 October 2026].

Apache Iceberg, n.d.b. Configuration. Apache Iceberg documentation. Available at: https://iceberg.apache.org/docs/latest/configuration/ [Accessed 16 October 2026].

Apache Iceberg, n.d.c. Iceberg table spec. Apache Iceberg. Available at: https://iceberg.apache.org/spec/ [Accessed 16 October 2026].

Apache Iceberg, n.d.d. Maintenance. Apache Iceberg documentation. Available at: https://iceberg.apache.org/docs/latest/maintenance/ [Accessed 16 October 2026].

Apache Iceberg, n.d.e. Procedures. Apache Iceberg documentation. Available at: https://iceberg.apache.org/docs/latest/spark-procedures/ [Accessed 16 October 2026].

Apache Iceberg, n.d.f. PyIceberg: API. PyIceberg documentation. Available at: https://py.iceberg.apache.org/api/ [Accessed 16 October 2026].

Apache Iceberg, n.d.g. Queries. Apache Iceberg documentation. Available at: https://iceberg.apache.org/docs/latest/spark-queries/ [Accessed 16 October 2026].

Apache Iceberg, n.d.h. snapshot - PyIceberg. PyIceberg API reference. Available at: https://py.iceberg.apache.org/reference/pyiceberg/table/update/snapshot/ [Accessed 16 October 2026].

Apache Iceberg, n.d.i. Transaction in iceberg::transaction - Rust (iceberg 0.10.1). docs.rs. Available at: https://docs.rs/iceberg/0.10.1/iceberg/transaction/struct.Transaction.html [Accessed 16 October 2026].

Apache Polaris, 2026. Apache Polaris graduates to Top Level Project! Apache Polaris blog, 19 February. Available at: https://polaris.apache.org/blog/2026/02/19/apache-polaris-graduates-to-top-level-project/ [Accessed 16 October 2026].

European Parliament and Council of the European Union, 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 (General Data Protection Regulation). Official Journal of the European Union, L 119. Available at: https://eur-lex.europa.eu/eli/reg/2016/679/oj [Accessed 16 October 2026].

Google Cloud, n.d.a. About the Apache Iceberg REST catalog endpoint. Lakehouse documentation. Available at: https://docs.cloud.google.com/lakehouse/docs/about-iceberg-rest-catalog-endpoint [Accessed 16 October 2026].

Google Cloud, n.d.b. Additional Lakehouse runtime catalog features. Lakehouse documentation. Available at: https://docs.cloud.google.com/lakehouse/docs/lakehouse-catalog-features [Accessed 16 October 2026].

Google Cloud, n.d.c. Apache Iceberg managed tables. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/biglake-iceberg-tables-in-bigquery [Accessed 16 October 2026].

Google Cloud, n.d.d. Authorized views. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/authorized-views [Accessed 16 October 2026].

Google Cloud, n.d.e. BigQuery connector. Managed Service for Apache Spark documentation. Available at: https://docs.cloud.google.com/managed-spark/docs/concepts/connectors/bigquery [Accessed 16 October 2026].

Google Cloud, n.d.f. Configure table options. Lakehouse documentation. Available at: https://docs.cloud.google.com/lakehouse/docs/table-options [Accessed 16 October 2026].

Google Cloud, n.d.g. Create BigLake external tables for Delta Lake. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/create-delta-lake-table [Accessed 16 October 2026].

Google Cloud, n.d.h. Cross-region dataset replication. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/data-replication [Accessed 16 October 2026].

Google Cloud, n.d.i. Data retention with time travel and fail-safe. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/time-travel [Accessed 16 October 2026].

Google Cloud, n.d.j. How borderless Lakehouse works. Lakehouse documentation. Available at: https://docs.cloud.google.com/lakehouse/docs/lakehouse-basics [Accessed 16 October 2026].

Google Cloud, n.d.k. Introduction to BigQuery row-level security. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/row-level-security-intro [Accessed 16 October 2026].

Google Cloud, n.d.l. Introduction to column-level access control. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/column-level-security-intro [Accessed 16 October 2026].

Google Cloud, n.d.m. Introduction to data masking. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/column-data-masking-intro [Accessed 16 October 2026].

Google Cloud, n.d.n. Introduction to table clones. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/table-clones-intro [Accessed 16 October 2026].

Google Cloud, n.d.o. Introduction to table snapshots. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/table-snapshots-intro [Accessed 16 October 2026].

Google Cloud, n.d.p. Introduction to the Storage Write API (gRPC). BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/write-api [Accessed 16 October 2026].

Google Cloud, n.d.q. Key concepts. Lakehouse documentation. Available at: https://docs.cloud.google.com/lakehouse/docs/key-concepts [Accessed 16 October 2026].

Google Cloud, n.d.r. Managed disaster recovery. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/managed-disaster-recovery [Accessed 16 October 2026].

Google Cloud, n.d.s. Overview of Apache Iceberg tables. Lakehouse documentation. Available at: https://docs.cloud.google.com/lakehouse/docs/manage-tables [Accessed 16 October 2026].

Google Cloud, n.d.t. Overview of BigQuery storage. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/storage_overview [Accessed 16 October 2026].

Google Cloud, n.d.u. Query open table formats with manifests. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/query-open-table-format-using-manifest-files [Accessed 16 October 2026].

Google Cloud, n.d.v. Quickstart: Use the Iceberg REST catalog with Cloud Storage. Lakehouse documentation. Available at: https://docs.cloud.google.com/lakehouse/docs/use-lakehouse-metastore-iceberg-rest-catalog [Accessed 16 October 2026].

Google Cloud, n.d.w. Set up the Apache Iceberg REST catalog endpoint. Lakehouse documentation. Available at: https://docs.cloud.google.com/lakehouse/docs/set-up-lakehouse-iceberg-rest-catalog [Accessed 16 October 2026].

Google Cloud, n.d.x. Soft delete. Cloud Storage documentation. Available at: https://docs.cloud.google.com/storage/docs/soft-delete [Accessed 16 October 2026].

Google Cloud, n.d.y. Understand BigQuery editions. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/editions-intro [Accessed 16 October 2026].

Google Cloud, n.d.z. Use cross-region replication and disaster recovery. Lakehouse documentation. Available at: https://docs.cloud.google.com/lakehouse/docs/use-managed-disaster-recovery [Accessed 16 October 2026].

Google Cloud, n.d.aa. Use the BigQuery Storage Read API to read table data. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/reference/storage [Accessed 16 October 2026].

GoogleCloudDataproc, n.d. spark-bigquery-connector: README. GitHub. Available at: https://github.com/GoogleCloudDataproc/spark-bigquery-connector/blob/master/README.md [Accessed 16 October 2026].

Lakekeeper, n.d. Lakekeeper: Apache Iceberg REST catalog written in Rust. Lakekeeper documentation. Available at: https://docs.lakekeeper.io/ [Accessed 16 October 2026].

Project Nessie, n.d. Configure Nessie with Iceberg REST. Project Nessie documentation. Available at: https://projectnessie.org/guides/iceberg-rest/ [Accessed 16 October 2026].

State of California Department of Justice, n.d. California Consumer Privacy Act (CCPA). Office of the Attorney General. Available at: https://oag.ca.gov/privacy/ccpa [Accessed 16 October 2026].

Mapping the Agent Before Anyone Attacks It

Andrew Stevens — Wed, 14 Oct 2026 07:20:44 +0000

An agent that reads tickets, queries BigQuery, and sends email usually gets reviewed one tool at a time, and each tool has an owner and an access review. What often has no owner is the chain between them: text a customer writes in a ticket can steer what the agent queries and where it sends the result. Much of the agent-specific risk sits in that chain.

This series takes one working agent from threat model to release. Each part picks one way to attack it, shows the attack in a generic form, and then adds controls, along with what each one costs and where it stops helping. Our Missing Primitives series covered the building blocks of trustworthy agents in general. Here I apply them, plus the platform features that have shipped since, to one system you could run.

This first part describes the agent, maps where data and instructions get in, lines the threats up against current OWASP and MITRE ATLAS material, and adds a first control: a policy check on every tool call the agent makes through ADK.

The agent this series attacks

The reference agent is an internal operations agent, a composite of a setup we often see in client work: Microsoft Entra ID for workforce identity, Google Cloud for data and agents. An employee signs in through Entra and uses the agent for three kinds of task:

Triage a ticket: read it, search internal documents for the relevant runbook, and add an internal comment.
Answer a data question: query BigQuery and email the answer to colleagues.
Reply to a ticket: read a customer's ticket, look up the facts in BigQuery, and send the customer a reply.

The agent is built with Google's Agent Development Kit (ADK) and runs on Agent Runtime (formerly Agent Engine), part of Gemini Enterprise Agent Platform (Google Cloud, n.d.c). Teams also run ADK agents on Cloud Run, and where that changes the answer, a short Cloud Run note says so. The agent reaches its tools over the Model Context Protocol (MCP):

BigQuery, through Google's remote MCP server at bigquery.googleapis.com/mcp. As of October 2026 its documentation names six tools, and "[t]he only MCP tool that isn't read-only is execute_sql". The agent is meant to use execute_sql_readonly, which doesn't support DML, DDL, or Python UDFs (Google Cloud, n.d.o).
Tickets, through an MCP server that the ticketing vendor operates.
Documents, through an MCP server the company runs in front of its document store.
Email, through an MCP server the company runs in front of its mail relay.

Keeping the agent off execute_sql is an IAM job, since the in-process controls later in this post only govern a misled model; the checks at the end of the post cover it.

Figure 1 numbers the flows. The tool results in flow 4 are where the design gets harder, because text written outside the company comes back to the agent inside them, while flow 5 sends mail to the same people who can write that text. The company backend matters as much as the runtime: it is the only caller the agent should accept, and it decides which employee and which task each session belongs to.

Two identity questions sit under every arrow in Figure 1: how an employee's Entra sign-in reaches the agent and binds to the session, which Part 2 covers, and whose authority the agent uses at each MCP server, which Part 3 covers. Google's Agent Identity gives each agent its own SPIFFE-based identity and can hold tokens that users delegate to it (Google Cloud, n.d.b). Until Part 3 shows how those tokens are handled, I treat code in the agent's process as able to use any credential the agent holds.

What is worth protecting

An agent's assets go beyond the data behind its tools. For this one:

The data behind the tools: BigQuery tables, internal documents, and ticket history, including customer details.
Credentials: tokens for each MCP server, refresh tokens held on the employee's behalf (by the agent or a credential broker), and wherever delegated tokens are stored for each user.
The ability to act: sending mail from a company domain and writing to tickets both carry the company's name, even when the content is wrong.
Session state and memory: what the agent carries from one turn or one session to the next.
The record of what happened: tool-call logs and the decisions around them, which incident response depends on.
Spend: model tokens and BigQuery bytes scanned, which a looping agent can burn through.

Where instructions get in

Web applications can keep code and data on separate paths, as parameterised queries do, and many injection defences rely on that separation. A language model gets its instructions and its data in the same context window, as text. Anything that reaches the context can be read by the model as a request, whatever channel it arrived through.

Entry point	Who controls the text	How it reaches the model
System instructions	The team that deploys the agent	Loaded at startup
The employee's request	The employee	The chat turn
Tool names and descriptions	Whoever operates each MCP server	Listed to the model so it can choose tools
Ticket bodies and comments	Anyone who can file or comment on a ticket, often outside the company	`get_ticket` results
Documents	Anyone who can write or share into the store	Search results
BigQuery rows	Whatever wrote the rows, including free-text fields that customers filled in	Query results
Memory and session state	Earlier turns and earlier sessions	Loaded with the session

Table 1. Where text enters the reference agent's context.

Only the first two rows are written by people the agent works for. The third is written by whoever runs each server: Google, the ticketing vendor, and the company's own teams. The next three are written by anyone who can get text into a ticket, a document, or a table, and the last by earlier sessions, which may themselves have read any of those. The BigQuery row is easy to miss: a table that stores ticket text or customer notes carries untrusted content even though the tool reading it is internal.

The employee's own request isn't always benign either. The confirmations later in this post are approved by the employee who asked, so they help against a misled model and do little against an employee who wants data out. Where that matters, external sends of query results need a second approver or the company's existing data loss prevention controls.

Three properties in one agent

Simon Willison names the combination that turns injected text into data loss: access to private data, exposure to untrusted content, and the ability to communicate externally, which he calls the lethal trifecta (Willison, 2025). An agent with all three can be steered by text it reads into sending what it can read to wherever the text says.

The reference agent has all three, and the ticket reply task uses all three in one session. It reads a ticket (untrusted content), queries BigQuery (private data), and sends email to a customer (external communication). It is often the task teams most want to automate, which is why this series attacks it.

One path, end to end

Figure 2 shows the path in its generic form. It needs no software vulnerability, and each step is a call the agent is allowed to make.

The ticket text asks, in ordinary language, for some extra context to be included in the reply. The model has been told to be helpful with tickets, the request reads like part of the job, and the query and the email are both tools it was given. Whether a given model follows the embedded text can't be assumed either way, and that is part of the problem. A few test runs without a leak don't show the path is closed.

Each component in Figure 2 behaves as designed. The ticketing system stores what customers write, the BigQuery MCP server runs a read-only query the credentials allow, and the email server sends mail. The leak comes from the order of the calls, so this post starts with a control that can see that order. It also comes to a simple recommendation for the ticket reply task: let the agent draft the reply and have the employee send it.

If the rows in Figure 2 hold customers' personal data, the outcome is a data breach, whatever the AI Act says about the agent. In the EU, GDPR Article 33(1) requires the controller to notify the supervisory authority "without undue delay and, where feasible, not later than 72 hours after having become aware of it", unless the breach is unlikely to result in a risk to people's rights and freedoms (European Union, 2016). In the US, every state has a breach notification law (National Conference of State Legislatures, 2022), and a financial firm regulated in New York has to notify the Department of Financial Services "no later than 72 hours after determining that a cybersecurity incident has occurred" (New York State Department of Financial Services, n.d.). An EU financial entity also reports major ICT-related incidents to its competent authority under DORA (European Union, 2022). Which of these applies depends on the data and the firm, so check with counsel, but I'd put the agent on the same breach runbook as any other system that holds customer data before it ships.

A first control: a policy check on each tool call

The first control sits where the sequence is visible: between the model proposing a tool call and the tool running. ADK provides the hook. A before_tool_callback is "[c]alled just before a specific tool's run_async method is invoked, after the LLM has generated a function call for it"; returning None lets the tool run, and returning a dictionary skips it (Google, n.d.b). ATLAS describes the same idea as mitigation AML.M0030, Restrict AI Agent Tool Invocation on Untrusted Data, with human approval as AML.M0029 (MITRE, 2026).

The policy check is deny by default and works from three inputs:

The task, fixed by the application when it creates the session, from the entry point the employee used. The model doesn't choose it, so text in a ticket shouldn't be able to promote a triage session into a reply session, provided nothing else can write that state (see Session state and credentials below).
The tool's effect class: reads untrusted content, reads sensitive data, writes internally, or sends.
What the session has already been exposed to: whether it has read untrusted content and whether it has read sensitive data.

From those it decides allow, confirm, or deny, checking the rows of Table 2 in order.

Situation	Decision
The tool is unknown, or isn't in the task's allowlist	Deny
An email with no recipients, recipient fields that aren't plain lists of addresses, an argument the policy check doesn't list, a subject or body that isn't text, or control or formatting characters in the subject	Deny
An email to any external address after the session has read untrusted content and sensitive data	Deny (the send in Figure 2)
Any other email to an external address	Confirm
An internal email or a ticket comment after the session has read untrusted content	Confirm
Anything else in the task	Allow, and record whether the call reads untrusted content or sensitive data

Table 2. How the policy check decides.

An address with anything unusual in it, including a plus tag or non-ASCII characters, counts as external. Recipient fields are denied rather than parsed when they aren't plain lists, an unexpected argument could carry an address or data, and control or bidirectional formatting characters in a subject could add mail headers or disguise what the approver sees. The policy check doesn't inspect what the body says.

Listing 1 keeps send_email in the ticket reply task so the deny is visible. For most teams a drafting tool is the better fit, as the section on costs explains.

Python

"""Deny-by-default policy check for the reference agent's tool calls."""
import re
import unicodedata
from dataclasses import dataclass
from enum import Enum

class Effect(Enum):
    READ_UNTRUSTED = "read_untrusted"  # content anyone can write: tickets, documents
    READ_SENSITIVE = "read_sensitive"  # internal data: BigQuery
    WRITE_INTERNAL = "write_internal"  # changes inside the company: ticket comments
    SEND = "send"                      # leaves the agent: email

TOOLS = {
    "get_ticket": Effect.READ_UNTRUSTED,
    "search_documents": Effect.READ_UNTRUSTED,
    "execute_sql_readonly": Effect.READ_SENSITIVE,
    "add_ticket_comment": Effect.WRITE_INTERNAL,
    "send_email": Effect.SEND,
}

# The application sets the task from the entry point the employee used.
# The model never chooses it.
TASKS = {
    "triage_ticket": {"get_ticket", "search_documents", "add_ticket_comment"},
    "data_question": {"execute_sql_readonly", "send_email"},
    "ticket_reply": {"get_ticket", "execute_sql_readonly", "send_email"},
}

# A strict local part: anything unusual counts as external, including a
# plus tag, which can route mail into a ticket or a list.
INTERNAL = re.compile(r"[a-z0-9._-]+@example\.com")

# The email tool's arguments. Any other key could carry an address or data.
SEND_FIELDS = {"to", "cc", "bcc", "subject", "body"}

@dataclass
class Session:
    untrusted_seen: bool = False
    sensitive_read: bool = False

@dataclass(frozen=True)
class Decision:
    outcome: str  # "allow", "confirm", or "deny"
    reason: str

def _recipients(args: dict):
    """Every address field, or None if any field isn't a list of strings."""
    found = []
    for field in ("to", "cc", "bcc"):
        value = args.get(field, [])
        if not isinstance(value, list) or not all(isinstance(r, str) for r in value):
            return None
        found += value
    return found

def _unsafe(c: str) -> bool:
    """Control characters, line separators, and bidirectional formatting."""
    return (unicodedata.category(c) == "Cc" or c in "\u2028\u2029\u200e\u200f"
            or "\u202a" <= c <= "\u202e" or "\u2066" <= c <= "\u2069")

def _external(recipients) -> bool:
    return any(not (r.isascii() and INTERNAL.fullmatch(r.lower())) for r in recipients)

def decide(task: str, tool: str, args: dict, s: Session) -> Decision:
    effect = TOOLS.get(tool)
    if effect is None:
        return Decision("deny", f"unknown tool {tool}")
    if tool not in TASKS.get(task, set()):
        return Decision("deny", f"{tool} is not allowed for task {task}")

    if effect is Effect.SEND:
        if set(args) - SEND_FIELDS:
            return Decision("deny", "unexpected argument")
        subject, body = args.get("subject", ""), args.get("body", "")
        if not (isinstance(subject, str) and isinstance(body, str)):
            return Decision("deny", "subject and body must be text")
        if any(_unsafe(c) for c in subject):
            return Decision("deny", "control or formatting character in subject")
        recipients = _recipients(args)
        if not recipients:
            return Decision("deny", "recipients missing or malformed")
        if _external(recipients):
            if s.untrusted_seen and s.sensitive_read:
                return Decision("deny", "external send after untrusted content and sensitive data")
            return Decision("confirm", "external recipient")
    if effect in (Effect.SEND, Effect.WRITE_INTERNAL) and s.untrusted_seen:
        return Decision("confirm", "side effect after untrusted content")

    # Record what the session has now been exposed to.
    if effect is Effect.READ_UNTRUSTED:
        s.untrusted_seen = True
    if effect is Effect.READ_SENSITIVE:
        s.sensitive_read = True
    return Decision("allow", "within task policy")

Rust

//! Deny-by-default policy check for the reference agent's tool calls.
use serde_json::Value;

#[derive(Clone, Copy, PartialEq, Debug)]
pub enum Effect {
    ReadUntrusted, // content anyone can write: tickets, documents
    ReadSensitive, // internal data: BigQuery
    WriteInternal, // changes inside the company: ticket comments
    Send,          // leaves the agent: email
}

pub fn effect_of(tool: &str) -> Option<Effect> {
    match tool {
        "get_ticket" | "search_documents" => Some(Effect::ReadUntrusted),
        "execute_sql_readonly" => Some(Effect::ReadSensitive),
        "add_ticket_comment" => Some(Effect::WriteInternal),
        "send_email" => Some(Effect::Send),
        _ => None,
    }
}

// The application sets the task from the entry point the employee used.
// The model never chooses it.
pub fn task_allows(task: &str, tool: &str) -> bool {
    let allowed: &[&str] = match task {
        "triage_ticket" => &["get_ticket", "search_documents", "add_ticket_comment"],
        "data_question" => &["execute_sql_readonly", "send_email"],
        "ticket_reply" => &["get_ticket", "execute_sql_readonly", "send_email"],
        _ => &[],
    };
    allowed.contains(&tool)
}

#[derive(Default, Debug)]
pub struct Session {
    pub untrusted_seen: bool,
    pub sensitive_read: bool,
}

#[derive(PartialEq, Debug)]
pub enum Decision {
    Allow,
    Confirm(&'static str),
    Deny(String),
}

/// Every address field, or None if any field isn't a list of strings.
fn recipients(args: &Value) -> Option<Vec<String>> {
    let mut found = Vec::new();
    for field in ["to", "cc", "bcc"] {
        match args.get(field) {
            None => {}
            Some(Value::Array(items)) => {
                for item in items {
                    found.push(item.as_str()?.to_string());
                }
            }
            Some(_) => return None,
        }
    }
    Some(found)
}

/// The email tool's arguments. Any other key could carry an address or data.
const SEND_FIELDS: [&str; 5] = ["to", "cc", "bcc", "subject", "body"];

/// Control characters, line separators, and bidirectional formatting.
fn unsafe_char(c: char) -> bool {
    c.is_control()
        || matches!(c, '\u{2028}' | '\u{2029}' | '\u{200e}' | '\u{200f}')
        || ('\u{202a}'..='\u{202e}').contains(&c)
        || ('\u{2066}'..='\u{2069}').contains(&c)
}

/// A strict local part: anything unusual counts as external, including a
/// plus tag, which can route mail into a ticket or a list.
fn is_internal(address: &str) -> bool {
    if !address.is_ascii() {
        return false;
    }
    let a = address.to_ascii_lowercase();
    match a.strip_suffix("@example.com") {
        Some(local) => {
            !local.is_empty()
                && local.chars().all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || "._-".contains(c))
        }
        None => false,
    }
}

pub fn decide(task: &str, tool: &str, args: &Value, s: &mut Session) -> Decision {
    let Some(effect) = effect_of(tool) else {
        return Decision::Deny(format!("unknown tool {tool}"));
    };
    if !task_allows(task, tool) {
        return Decision::Deny(format!("{tool} is not allowed for task {task}"));
    }

    if effect == Effect::Send {
        let known = args
            .as_object()
            .is_some_and(|o| o.keys().all(|k| SEND_FIELDS.contains(&k.as_str())));
        if !known {
            return Decision::Deny("unexpected argument".into());
        }
        if ["subject", "body"].iter().any(|f| args.get(*f).is_some_and(|v| !v.is_string())) {
            return Decision::Deny("subject and body must be text".into());
        }
        let subject = args.get("subject").and_then(Value::as_str).unwrap_or("");
        if subject.chars().any(unsafe_char) {
            return Decision::Deny("control or formatting character in subject".into());
        }
        let to = match recipients(args) {
            Some(r) if !r.is_empty() => r,
            _ => return Decision::Deny("recipients missing or malformed".into()),
        };
        if to.iter().any(|r| !is_internal(r)) {
            if s.untrusted_seen && s.sensitive_read {
                return Decision::Deny("external send after untrusted content and sensitive data".into());
            }
            return Decision::Confirm("external recipient");
        }
    }
    if matches!(effect, Effect::Send | Effect::WriteInternal) && s.untrusted_seen {
        return Decision::Confirm("side effect after untrusted content");
    }

    // Record what the session has now been exposed to.
    match effect {
        Effect::ReadUntrusted => s.untrusted_seen = true,
        Effect::ReadSensitive => s.sensitive_read = true,
        _ => {}
    }
    Decision::Allow
}

Listing 1. A deny-by-default policy check for the reference agent's tool calls. Tool names follow the MCP servers in Figure 1; example.com stands in for the company's domain.

Listing 2 wires the policy check into the agent as an ADK plugin. We read the ADK for Python 2.11.0 source for this, and our tests exercise the ordering against agent callbacks. Plugin callbacks registered on the app run before any agent's own before_tool_callback, and they cover the agents the app drives, unless a sub-agent is wrapped in an AgentTool with plugins excluded. They stop at the first plugin that returns a result, so the policy check goes first in the plugin list. Listing 2 is Python only, since it is wiring for ADK's Python SDK.

Python

"""The policy check as an ADK plugin, registered first on the App."""
from google.adk.plugins.base_plugin import BasePlugin

from policy_check import Session, decide

class PolicyCheck(BasePlugin):
    def __init__(self, raise_approval, log):
        super().__init__(name="policy_check")
        self.raise_approval = raise_approval  # the application's approval queue
        self.log = log

    async def before_tool_callback(self, *, tool, tool_args, tool_context):
        state = tool_context.state
        s = Session(
            untrusted_seen=state.get("policy_untrusted_seen", False),
            sensitive_read=state.get("policy_sensitive_read", False),
        )
        decision = decide(state.get("policy_task", ""), tool.name, tool_args, s)
        # The flags only ever go from False to True.
        if s.untrusted_seen:
            state["policy_untrusted_seen"] = True
        if s.sensitive_read:
            state["policy_sensitive_read"] = True
        self.log(tool_context, tool.name, tool_args, decision)

        if decision.outcome == "allow":
            return None  # ADK runs the tool
        if decision.outcome == "confirm":
            # The application records the employee's decision, and the runtime
            # releases the call after the policy check runs again. Nothing the
            # model writes counts as approval.
            self.raise_approval(tool_context, tool, tool_args)
            tool_context.actions.skip_summarization = True  # end the turn here
            return {"status": "awaiting_approval"}
        return {"status": "not_permitted"}  # the reason stays in the log

# app = App(name="ops", root_agent=agent, plugins=[PolicyCheck(...), *other_plugins])

Listing 2. The policy check as an ADK plugin. Returning None runs the tool; returning a dictionary skips it.

The backend sets the task in session state, and the policy check keeps its exposure flags there under keys of its own. The flags are set when a read is allowed, before it runs, so later calls in the same response see them.

When a call needs confirmation

"Confirm" means the call doesn't run. The application raises an approval request instead and shows the signed-in employee the real server, tool name, and arguments. The approval is bound to that employee, the session, the task, the server and tool, and a hash of the canonicalised arguments, and it expires and can be used once.

For external sends, asking the employee to sign in again helps confirm that the approver is the employee. In Entra, one option is a Conditional Access authentication context whose policy sets sign-in frequency to every time, requested with a claims challenge for each approval. Microsoft notes that "every time" allows five minutes of clock skew (Microsoft, n.d.b). The API opts in to the optional acrs claim, and the backend checks it along with when the user last authenticated, from the optional auth_time claim (Microsoft, n.d.a; Microsoft, 2026b). A token carrying acrs only shows that the policy was met when the token was issued, and Microsoft's pages don't describe this combination end to end, so test it before relying on it. Rate-limiting approval requests and denied calls per session keeps a looping model from flooding the employee.

The plugin also ends the model's turn when it raises a request, so a looping or misled model gets no further model steps in that turn. Other tool calls it proposed in the same response are each still checked, and each can raise its own request. In ADK 2.11.0 they are checked in the order the model listed them, as long as the policy check updates its flags before awaiting anything, as Listing 2 does. Order matters: a send listed before a query in the same response gets a confirm, while the reverse gets a deny. Running the policy check again at release is meant to close that gap, and Part 4 tests it. Our tests cover the read-then-send order, so rerun them on every ADK upgrade.

Once the employee approves, the backend resumes the session with a reference to the approval, which lives in a record outside session state. Runtime code outside the model loop then releases the call under the credential the agent would have used. That code runs decide again against the session's current flags and goes ahead only if the result is still confirm. It then calls the tool directly; sending the call back through the ADK pipeline would only raise a fresh request. Because the direct call skips ADK's after-tool callbacks, the release code also records the result in the session as a new event that references the approval, logs it, and runs any after-tool checks itself, such as the Model Armor scan described later in this post.

Where the call runs under the agent's own identity, the release should also check that the approving employee is entitled to the data and the action; Part 3 compares that with running under the employee's delegated authority.

ADK has a built-in tool confirmation feature. At the time of writing, its documentation marks it experimental, lists VertexAiSessionService among the session services it doesn't support (alongside DatabaseSessionService), and doesn't describe binding the confirmation to the signed-in user (Google, n.d.a). Check this against the ADK version you run. VertexAiSessionService is the ADK client for Agent Platform Sessions, which are accessed through an Agent Runtime instance (Google Cloud, n.d.h). The reference agent keeps its sessions in Agent Platform Sessions, so the built-in feature is out for now. Part 4 builds approvals into the policy check and the company backend instead.

We tested both listings. The release path is only described here; Part 4 builds and tests it. The Python tests run the policy check on its own and as a plugin inside an ADK 2.11.0 Runner with a scripted model, where the blocked send never reaches the tool, a confirm ends the model's turn, and the plugin denies a call before the agent's own callback runs. The Rust tests cover the same decision cases: a lookalike domain, addresses hidden in bcc, a recipient field passed as one comma-separated string, an unexpected reply_to argument, a subject or body passed as a list or object, a subject with an injected line break or a bidirectional override, a plus-tagged address, and local parts with percent signs, bang paths, encoded words, invisible characters, or non-ASCII letters.

What the policy check costs

The first cost lands on the ticket reply task. With the policy check in place, the agent can read a customer's ticket, look up the facts, and draft a reply, but it can't send that reply to the customer once both flags are set. The simplest design moves the send out of the agent: the agent drafts, and the employee sends from the ticketing system after reading the draft. For most teams, that is where I'd start. The draft can still carry the data the injected text asked for, so the review screen should show which queries fed the draft and highlight values that came from them; Part 6 covers checking outbound content. Teams that want the agent to send directly need finer-grained tracking than two session flags, which Part 4 covers.

The second cost is confirmation load. Session-level flags are coarse: one ticket read early in a conversation marks everything after it, and in the triage task every comment follows a get_ticket, so the task's main action always needs confirmation. If employees see a confirmation on most actions, they tend to start approving without reading. In our experience, narrower tasks and shorter sessions do more to reduce that load than prompt changes.

The third is upkeep. Every new tool needs an effect class and a place in each task. A tool that is added to an MCP server but not to the policy check is denied, so the table needs an owner who updates it whenever a server adds or changes tools.

Where the policy check stops helping

The policy check sees tool names, arguments, and session history, and nothing else. It closes the exact path in Figure 2, for that task, with those flags. The rest of this section is what it leaves open, and, where a later part picks a gap up, which one.

Routes data can still take

A deny stops one call. The model sees not_permitted and can try another route, such as an internal address that reaches the customer, which then arrives as a routine confirmation. Ending the session after a deny like the one in Figure 2, or denying every later side effect in it, closes that route within the session, and Part 4 adds it. That particular deny should rarely fire in normal use, so I'd alert on it from day one; Part 8 builds the detection around it.

An address on the company's domain can still reach outsiders: a ticketing system's intake address that posts mail to a customer-visible ticket, a group with external guests, or a mailbox that forwards. A colleague may also not be entitled to the rows the agent read. Resolving recipients against the directory and checking their own access belongs in Parts 3 and 6.

Another customer's ticket is a route too. In the ticket tasks, injected text can ask the agent to read a different customer's ticket and include it in the reply, which needs only a confirmation that looks routine. Binding the ticket ID when the session is created, and denying get_ticket or add_ticket_comment for any other ID, helps close that route.

Some channels have an outside audience without looking like sends. A ticket comment is classed as an internal write, but on a ticket filed by a customer, the customer may be able to read it. A shared document link works the same way. Effect classes have to follow who can read the result, which Part 6 works through.

Read tools can carry data out through their arguments. Reads are allowed even after sensitive data has been read, and a ticket ID or search query stuffed with query results reaches the vendor's server and its logs. Strict argument formats help, and so does treating calls to servers run by other organisations as sends once a session holds sensitive data.

The agent's own output never reaches the policy check. Text the model returns to the employee, including links and images the chat interface renders, never passes through a tool call, and the policy check doesn't read email bodies, so HTML and remote images in them pass. Remote images and links in mail the agent sends internally can do the same when a colleague's mail client loads them. Part 6 covers output handling.

What tool names and two flags can't capture

Some sources are sensitive and untrusted at once. If a BigQuery table holds customer-written notes, a query result is both sensitive and untrusted, and the policy check as written only sets the sensitive flag. Internal documents are the mirror case: the policy check classes them as untrusted only, though they hold internal material. Tickets are mixed too, since get_ticket returns the internal comments the triage task writes, so in the reply task it should set both flags, or the ticket server should return only customer-visible fields. Classifying by source rather than by tool helps with all three, though it needs a data catalogue to work from.

Memory loaded into a new session and text an employee pastes from a customer email set no flag. The application should set both flags when it creates a session that loads memory, since memory can carry earlier query results as well as earlier untrusted text, and the untrusted flag when a session takes pasted text or attachments.

The policy check allows execute_sql_readonly in the right tasks; it doesn't judge whether a particular query reads more than the task needs. That depends on which identity the query runs as, and on row-level controls in BigQuery, which narrow results per employee when the query runs as that employee. Part 3 compares the options.

What the policy check has to trust

The policy check trusts the approval it receives. The approver can only judge what the application shows them, so the application should render the arguments itself (ASI09 in Table 3).

The policy check also trusts that nothing changes a call after it allows it. Later plugins and the agent's own callbacks receive the same arguments and can change them after the policy check has allowed the call. In the ADK 2.11.0 source, built-in tools such as Google Search grounding are added to the model request rather than run by the agent's tool pipeline, so the policy check never sees them.

The policy check trusts tool names. With several MCP servers, two can expose tools with the same name, and a server can change what a tool does after it was approved. Tools denied by the policy check also still have their descriptions listed to the model. ADK's McpToolset accepts a tool_filter and a tool_name_prefix, which help in the meantime (the policy check's table then uses the prefixed names), and Part 5 pins tools to the authenticated server and the descriptions that were reviewed.

An in-process plugin governs what a misled model can do, and a compromised runtime can skip it.

Controls outside the process

Agent Gateway, which Google describes as the key enforcement component of Agent Platform, sits outside the agent. Its overview says that by default all connections are blocked unless an explicit IAM policy grants access, and that for MCP traffic it can parse requests so policies can restrict specific tools (Google Cloud, n.d.a). Calls that violate its Access policies are blocked in enforce mode and only logged in dry-run mode (Google Cloud, n.d.e). A destination that isn't registered in Agent Registry needs an explicit access policy, and Principal Access Boundary policies can also be enforced on the agent's identity (Google Cloud, n.d.a; Google Cloud, n.d.e). In enforce mode, that narrows what a compromised runtime can reach to what the agent's identity is allowed, provided egress can't bypass the gateway. Within that, any delegated user tokens the runtime holds remain usable.

Agent Gateway's egress policies are keyed on the agent's identity: "Agent principals are the members that you grant access to" (Google Cloud, n.d.e). The documentation we read doesn't show policy conditions on the end user, and as far as we can tell the gateway doesn't see session history, which is what the policy check works from. Google's pages also read differently on VPC Service Controls. The gateway overview lists enforcing perimeters for agent communications as a benefit (Google Cloud, n.d.a), and the release notes add that support from 9 September 2026 (Google Cloud, 2026a). It applies only to Agent Gateway deployments created after 8 September whose VPC connectivity uses the agent connectivity template in ALL_TRAFFIC egress mode. The IAM Access policies page still carries a note that Access policies don't support VPC Service Controls (Google Cloud, n.d.e). Part 6 works out what that means for a perimeter around this agent.

My read, while the documentation settles: treat Agent Gateway, and Model Armor below, as defence in depth, and treat the policy check as the control you own and can test today.

Cloud Run note. As of October 2026, Agent Gateway's documentation lists Agent Runtime (ingress and egress) and Gemini Enterprise (egress only), and doesn't list Cloud Run, although Agent Identity lists Cloud Run (Google Cloud, n.d.b; Google Cloud, n.d.a). A Cloud Run deployment keeps the in-process policy check but needs its egress controls from elsewhere, which Part 6 covers.

Where Model Armor fits

Filters that look for injected instructions are useful as a second layer. I wouldn't rely on one on its own, because it can let too much through. As of October 2026, Google's Model Armor can inspect tools/call requests and responses for Google's MCP servers, but not tools/list, and Google advises enabling its prompt injection and jailbreak filter only where MCP traffic carries natural language (Google Cloud, n.d.g). BigQuery is on the list of MCP servers it supports (Google Cloud, n.d.i). Google's BigQuery MCP page also warns that "When Model Armor is enabled with logging enabled, Model Armor logs the entire payload", so query results can end up in your logs (Google Cloud, n.d.o).

In this design, that integration would see the BigQuery traffic but not the ticket text in Figure 2, which arrives through the vendor's server.

On Agent Runtime, Model Armor can also be enabled on Agent Gateway. For egress MCP traffic to servers in general, Google's page says it sanitises tools/call and prompts/get requests and responses and tool execution errors, and it lists tools/list, resources/*, and "Streamable HTTP/SSE for MCP" among the payloads that pass without sanitization. Model Armor also has to be in the same region as the services it integrates with (Google Cloud, n.d.f). The page doesn't make clear whether a tools/call response that a server streams over SSE is inspected, so check this against the vendor's server before counting on it. Where that doesn't apply, including on Cloud Run, the application can send get_ticket results to Model Armor itself, as applications do for prompts and model responses (Google Cloud, n.d.k), for example from an after-tool callback.

Willison's point about guardrail products applies: vendors often claim to catch "95% of attacks", but "in web application security 95% is very much a failing grade" (Willison, 2025). A detector can lower the rate of successful injections, and the policy check denies the send in Figure 2 whether or not the detector flags the ticket. Part 4 tests both against the same set of tickets.

Mapping the threats to OWASP and ATLAS

I map against two catalogues, because they answer different questions. The OWASP GenAI Security Project published the Top 10 for Agentic Applications for 2026 in December 2025. It describes risk categories and mitigations, and refers to the 2025 edition of the OWASP LLM Top 10: ASI03, for example, is described as the agentic evolution of Excessive Agency (LLM06:2025) (OWASP GenAI Security Project, 2025). The project published a 2026 edition of the LLM Top 10 in August 2026 with renumbered entries, so Excessive Agency is now LLM03 (OWASP GenAI Security Project, 2026), and Table 3 uses the 2026 numbers. MITRE ATLAS describes adversary techniques and mitigations; the data release used here is 2026.09 (MITRE, 2026).

Threat to the reference agent	OWASP Agentic 2026	OWASP LLM 2026 (our mapping)	Part
Ticket or document text redirects the agent	ASI01 Agent Goal Hijack	LLM01 Prompt Injection	4
Allowed tools chained for an unintended purpose	ASI02 Tool Misuse and Exploitation	LLM03 Excessive Agency	1, 4
The agent's authority is wider than the user's or the task's	ASI03 Identity and Privilege Abuse	LLM03 Excessive Agency	2, 3
A caller poses as another employee, resumes someone else's session, or changes the task or flags in session state	ASI03 Identity and Privilege Abuse		2
A token is harvested through the agent or stolen, then replayed	ASI03 Identity and Privilege Abuse		3
A tool's description or behaviour is poisoned or changes	ASI04 Agentic Supply Chain Vulnerabilities	LLM04 Supply Chain	5
Data leaves through email, links, or rendered images	ASI02 Tool Misuse and Exploitation	LLM02 Sensitive Information Disclosure; LLM10 Improper Output Handling	6
Poisoned memory changes later sessions	ASI06 Memory & Context Poisoning		7
A misleading summary gets a human to approve an action	ASI09 Human-Agent Trust Exploitation		4
Looping, runaway cost, or an agent acting outside its purpose	ASI02 Tool Misuse and Exploitation; ASI08 Cascading Failures; ASI10 Rogue Agents	LLM06 Unbounded Consumption	8

Table 3. The reference agent's threats mapped to OWASP, with the part of the series that covers each. The LLM 2026 column is our mapping; OWASP's agentic document lists its own related LLM entries, in 2025 numbering, for each ASI item in an appendix. Blank cells are where we didn't map an entry.

Threat to the reference agent	MITRE ATLAS 2026.09
Ticket or document text redirects the agent	AML.T0051.001 LLM Prompt Injection: Indirect; AML.T0099 AI Agent Tool Data Poisoning
Allowed tools chained for an unintended purpose	AML.T0053 AI Agent Tool Invocation
The agent's authority is wider than the user's or the task's	Mitigations AML.M0026, AML.M0027, AML.M0028, and AML.M0037
A token is harvested through the agent or stolen, then replayed	AML.T0091.000 Use Alternate Authentication Material: Application Access Token; AML.T0098 AI Agent Tool Credential Harvesting
A tool's description or behaviour is poisoned or changes	AML.T0010.005 AI Supply Chain Compromise: AI Agent Tool; AML.T0110 AI Agent Tool Poisoning
Data leaves through email, links, or rendered images	AML.T0086 Exfiltration via AI Agent Tool Invocation; AML.T0077 LLM Response Rendering; AML.T0057 LLM Data Leakage
Poisoned memory changes later sessions	AML.T0080.000 AI Agent Context Poisoning: Memory
A misleading summary gets a human to approve an action	AML.T0067 LLM Trusted Output Components Manipulation
Looping, runaway cost, or an agent acting outside its purpose	AML.T0034.002 Cost Harvesting: Agentic Resource Consumption

Table 4. The same threats mapped to MITRE ATLAS. We didn't find an ATLAS technique that matches over-broad agent authority well, so that row names the ATLAS mitigations for agent and tool permissions instead, and we didn't map the session-tampering row.

Two OWASP entries get less attention here. ASI07, Insecure Inter-Agent Communication, applies once agents call other agents, and this is a single agent. ASI05, Unexpected Code Execution, is narrow while the agent's only code-like capability is read-only SQL. A query can still call a remote function, though. BigQuery documents remote functions as running code in Cloud Run or Cloud Run functions through a connection, used "in a query just like any other user-defined functions" (Google Cloud, n.d.p). I haven't found Google documentation saying whether execute_sql_readonly allows them, so the identity running the query shouldn't have access to routines, connections, or external data sources it doesn't need. I'd also test whether execute_sql_readonly accepts EXPORT DATA statements or the EXTERNAL_QUERY function before relying on it.

The mapping is also a useful artefact for governance work in the US and the EU. In NIST's AI Risk Management Framework, "The MAP function establishes the context to frame risks related to an AI system" (National Institute of Standards and Technology, 2023), and the Generative AI Profile lists information security, including direct and indirect prompt injection, among the risks it covers (National Institute of Standards and Technology, 2024).

In the EU, many internal operations agents won't fall into a high-risk category under the AI Act. Where one does, Article 15(5) requires such systems to be "resilient against attempts by unauthorised third parties to alter their use, outputs or performance by exploiting system vulnerabilities" (European Commission, n.d.), and the company that built it may count as its provider. For the rest, Article 15 is still a useful bar. The Commission's page notes that Article 15 has been amended by the Digital Omnibus on AI, so check the current text and dates; our Regulatory Stack post on Article 15 covers that side.

I'd keep the threat model as a versioned table beside the agent's code, so the engineers and the governance team work from the same file.

Launch stages as of 13 October 2026

Most of the platform features this post relies on shipped this year, so Table 5 records their launch stage as of the date we checked.

Feature	Launch stage	Source
Agent Identity	Generally available since 22 April 2026	Google Cloud, 2026c
Agent Identity auth manager	Generally available since 22 August 2026, after a preview from April	Google Cloud, 2026c
Agent Gateway	Generally available since 18 June 2026; VPC Service Controls support announced on 9 September 2026 without a stated stage, for deployments created after 8 September that use the agent connectivity template in `ALL_TRAFFIC` egress mode	Google Cloud, 2026a
IAM Unified Access Policies (Access policies)	Generally available since 31 August 2026	Google Cloud, 2026a
Model Armor for Agent Gateway	Generally available since 24 June 2026	Google Cloud, 2026a
Google's remote MCP servers	Generally available as a capability since 1 May 2026, with individual servers in Preview or GA	Google Cloud, 2026b
BigQuery remote MCP server	Not marked Preview in Google's list of supported MCP products, which the MCP release notes point to for launch status; no stage stated on the server's own page	Google Cloud, 2026b; Google Cloud, n.d.m; Google Cloud, n.d.o
Model Armor's MCP server integration	No stage stated on its page	Google Cloud, n.d.g
ADK tool confirmation	Experimental	Google, n.d.a

Table 5. Launch stages of the features this post relies on.

Where no stage is stated, I'd confirm it with Google before relying on the feature in production, since pre-GA terms and support differ. The design doesn't depend on ADK's experimental confirmation feature.

What I'd check in your own deployment

Most of what follows can't be settled in a post, because it depends on how your tenant, projects, and servers are configured. These are the checks I'd want to see pass, with evidence, before an agent like this goes near production. Four of them I'd treat as blocking a release: only the backend can call the agent, user_id comes from the validated token, the policy check's keys can't be changed after the session is created, and a tool missing from the policy check's table is denied.

Who can reach the agent

Owner: usually the platform team that runs Google Cloud IAM.

Only the company backend can call the agent, and only the backend and the agent's own identity hold permissions on its sessions. The backend gets a custom role with only aiplatform.reasoningEngines.query on the agent resource, as Google's sharing page suggests (Google Cloud, n.d.l), and nothing else inherits that permission from the project, folder, or organisation through a role such as roles/aiplatform.user or a basic role (on Cloud Run, the invoker role). Google notes that default ADK agents trust the frontend, "which gives the frontend full control of sessions and users" (Google Cloud, n.d.l).
Deploy rights on the agent are treated as the agent's authority: anyone who can change its code or configuration acts as it.
The backend keeps its own record of which session belongs to which user, and resumes a session only for the user it was created for. Anyone holding the query permission can call the agent's session operations directly, including creating sessions, so the IAM restriction above is what protects policy_task.

How the backend checks the employee

Owner: the backend's developers, with the identity team.

The backend validates the Entra token's signature, issuer, tenant, and lifetime (exp and nbf), and checks that the audience identifies its own API: in v2.0 tokens that is the client ID, and in v1.0 tokens the client ID or the resource URI the client requested (Microsoft, 2026a).
It accepts only delegated user tokens that carry its scope in scp, from client apps it expects (azp in v2.0 tokens, appid in v1.0), and rejects app-only tokens, for which the optional idtyp claim is app (Microsoft, 2026b).
It rejects guest accounts unless a task is meant for them, for example by requiring user assignment on the API's enterprise application and assigning only the intended users or groups, and by checking the optional acct claim, which is 1 for a guest (Microsoft, 2026b). Both idtyp and acct have to be configured as optional claims on the API's app registration. Once configured, idtyp is emitted on app-only tokens, which is all this check needs (Microsoft, 2026b).
It derives the ADK user_id from the tenant and object IDs (tid and oid; the client has to request the profile scope for these to be issued), never from a client-supplied field (Microsoft, 2026a). Agent Runtime accepts a user ID the caller chooses (its documentation says "[c]hoose your own user ID") (Google Cloud, n.d.n), so this binding is the backend's job.
The agent never sees the Entra token, so it takes the employee's identity entirely on the backend's word. If delegated tokens are held per user, a caller who can choose user_id can use another employee's tokens, which makes the backend's identity and code as sensitive as the tokens they can unlock. Part 2 covers this binding.

Session state and credentials

Owner: the agent's developers, with the platform team.

The policy check's keys (policy_task and the flags) can't be changed after the session is created. In practice that means:
- no initial state or events from the client, and no state_delta or extra session events on a query;
- no session PATCH through the ADK API server, and no session rewind outside the backend;
- no agent output_key, tool, or callback that writes those keys, and no user: or app: prefix on them.

Keeping the keys, and the approval records, in a store outside session state is stronger still.

Delegated tokens sit with the auth manager or another broker, not in session state that anything reading the session can see (ADK can keep them in session state, so this is a configuration choice), and each task gets its own toolsets, so credentials for servers outside the task aren't loaded.

Tools, arguments, and approvals

Owner: the agent's developers, with an application security review.

No plugin or callback that runs after the policy check modifies tool arguments, model-side built-in tools are off, and no MCP sampling or elicitation callbacks are registered on the toolsets, since server-initiated requests don't pass through the tool pipeline.
Every tool exposed by every MCP server the agent can reach appears in the policy check's table, and a tool that doesn't is denied in the deployed agent, which a test confirms.
The email tool's real argument names match SEND_FIELDS in Listing 1. A reply_to, attachment, or template field stays denied until the policy check knows how to check it, and the email server builds recipient headers only from to, cc, and bcc, and normalises line endings in the body so it can't end the message and start another (SMTP smuggling).
Approvals are bound as described above, single-use, shown by the application rather than summarised by the model, and checked again by the policy check at release.

Where data can go

Owner: the data owners and the teams that run mail and ticketing, with security.

The internal addresses that reach outsiders (intake addresses, groups with guests, and forwarding mailboxes) are listed and kept current.
The BigQuery tables that hold text written outside the company are listed, so queries against them can be classed as reading untrusted content.
Every principal that reaches BigQuery is denied read-write MCP tools and holds no BigQuery write permissions. Google documents a deny policy that limits its MCP servers to read-only tools; its example denies every principal in the project, folder, or organisation it is attached to. The denied tool still appears in the tool list, so a tool_filter on the toolset keeps it away from the model (Google Cloud, n.d.j). A VPC Service Controls perimeter around the BigQuery project, which Google documents for its own MCP servers with conditions such as read-only tool calls (Google Cloud, n.d.d), helps keep whatever statements execute_sql_readonly does allow from moving data outside it. Check how that perimeter and Agent Gateway's Access policies fit together, given the documentation notes above.

Platform support and logging

Owner: the platform team, with security operations.

Agent Gateway, Model Armor, and the Agent Identity auth manager are confirmed to support the runtime, region, and MCP servers in use, at a launch stage you have accepted for production, checked against Table 5 and Google's current documentation.
The policy check's decisions are logged with the session, task, tool, arguments, and outcome, with access and retention set as for the data the arguments carry. Part 8 builds on this.

What this series leaves out

Some common questions fall outside the reference agent. It is a single agent, so agent-to-agent delegation and multi-agent protocols are out of scope; our post on secure multi-agent protocols covers that ground. It has no code-execution tool or sandbox. Cost and resource exhaustion appear only as far as Part 8 detects and stops a looping agent, and our post on resource governance covers budgets and throttling. The model's own supply chain, meaning where its weights come from and how they were trained, is also out of scope, as are the web security of the chat front end, the model provider's data handling, and the ticketing vendor's own platform security.

The rest of the series

Part 2 follows the employee's identity into the agent: from an Entra sign-in through the company backend, how the session is bound to that employee, and how the employee's Entra identity is federated into Google Cloud for the calls that need it. Part 3 follows the agent's calls out to its tools and compares the authority each can run under: delegated user authority, the agent's own identity, or a combination. It also covers the current MCP authorization specification (version 2026-07-28), under which servers accept only tokens issued for them and never pass a client's token on to upstream APIs (Model Context Protocol, 2026a; Model Context Protocol, 2026b), and where the servers in this design depart from it.

Part 4 takes indirect injection through tickets, documents, and query results, with value-level tracking, the approval path, and a test of Model Armor and the policy check against the same tickets. Part 5 deals with MCP servers you didn't write, Part 6 with the ways data leaves, and Part 7 with memory. Part 8 is about detecting and containing an agent mid-task, and Part 9 is red-teaming before release.

Which of your agents reads text from outside the company in the same session as it reads your data, and what can it send?

References

European Commission, n.d. Article 15: Accuracy, robustness and cybersecurity. AI Act Service Desk. Available at: https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-15 [Accessed 13 October 2026].

European Union, 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation). Article 33. Available at: http://data.europa.eu/eli/reg/2016/679/oj [Accessed 13 October 2026].

European Union, 2022. Regulation (EU) 2022/2554 of the European Parliament and of the Council of 14 December 2022 on digital operational resilience for the financial sector. OJ L 333, 27.12.2022, Article 19. Available at: http://data.europa.eu/eli/reg/2022/2554/oj [Accessed 13 October 2026].

Google, n.d.a. Get action confirmation for ADK Tools. Agent Development Kit documentation. Available at: https://adk.dev/tools-custom/confirmation/ [Accessed 13 October 2026].

Google, n.d.b. Types of callbacks. Agent Development Kit documentation. Available at: https://adk.dev/callbacks/types-of-callbacks/ [Accessed 13 October 2026].

Google Cloud, 2026a. Gemini Enterprise Agent Platform release notes. Entries from 22 April to 9 September 2026. Available at: https://docs.cloud.google.com/gemini-enterprise-agent-platform/release-notes [Accessed 13 October 2026].

Google Cloud, 2026b. Google Cloud MCP servers release notes. Entry of 1 May 2026. Available at: https://docs.cloud.google.com/mcp/release-notes [Accessed 13 October 2026].

Google Cloud, 2026c. IAM release notes. Entries of 22 April and 22 August 2026. Available at: https://docs.cloud.google.com/iam/docs/release-notes [Accessed 13 October 2026].

Google Cloud, n.d.a. Agent Gateway overview. Gemini Enterprise Agent Platform documentation. Available at: https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/gateways/agent-gateway-overview [Accessed 13 October 2026].

Google Cloud, n.d.b. Agent Identity overview. Identity and Access Management documentation. Available at: https://docs.cloud.google.com/iam/docs/agent-identity-overview [Accessed 13 October 2026].

Google Cloud, n.d.c. Agent Platform overview. Gemini Enterprise Agent Platform documentation. Available at: https://docs.cloud.google.com/gemini-enterprise-agent-platform/overview [Accessed 13 October 2026].

Google Cloud, n.d.d. Control MCP use in a VPC Service Controls perimeter. Google Cloud MCP servers documentation. Available at: https://docs.cloud.google.com/mcp/control-mcp-use-vpc-sc-perimeter [Accessed 13 October 2026].

Google Cloud, n.d.e. IAM Access policies overview. Gemini Enterprise Agent Platform documentation. Available at: https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/policies/iam-overview-uap [Accessed 13 October 2026].

Google Cloud, n.d.f. Integrate Model Armor with Agent Gateway. Model Armor documentation. Available at: https://docs.cloud.google.com/model-armor/model-armor-agent-gateway-integration [Accessed 13 October 2026].

Google Cloud, n.d.g. Integrate with Google and Google Cloud MCP servers. Model Armor documentation. Available at: https://docs.cloud.google.com/model-armor/model-armor-mcp-google-cloud-integration [Accessed 13 October 2026].

Google Cloud, n.d.h. Manage sessions with Agent Development Kit. Gemini Enterprise Agent Platform documentation. Available at: https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/sessions/manage-with-adk [Accessed 13 October 2026].

Google Cloud, n.d.i. Model Armor supported products. Google Cloud MCP servers documentation. Available at: https://docs.cloud.google.com/mcp/model-armor-supported-products [Accessed 13 October 2026].

Google Cloud, n.d.j. Prevent read-write MCP tool use. Google Cloud MCP servers documentation. Available at: https://docs.cloud.google.com/mcp/prevent-read-write-tool-use [Accessed 13 October 2026].

Google Cloud, n.d.k. Protect AI applications with Model Armor. Security Command Center documentation. Available at: https://docs.cloud.google.com/security-command-center/docs/model-armor [Accessed 13 October 2026].

Google Cloud, n.d.l. Share an agent. Gemini Enterprise Agent Platform documentation. Available at: https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/share-agent [Accessed 13 October 2026].

Google Cloud, n.d.m. Supported products. Google Cloud MCP servers documentation. Available at: https://docs.cloud.google.com/mcp/supported-products [Accessed 13 October 2026].

Google Cloud, n.d.n. Use an Agent Development Kit agent. Gemini Enterprise Agent Platform documentation. Available at: https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/runtime/use-an-adk-agent [Accessed 13 October 2026].

Google Cloud, n.d.o. Use the BigQuery remote MCP server. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/use-bigquery-mcp [Accessed 13 October 2026].

Google Cloud, n.d.p. Work with remote functions. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/remote-functions [Accessed 13 October 2026].

Microsoft, 2026b. Optional claims reference. Microsoft identity platform documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity-platform/optional-claims-reference [Accessed 13 October 2026].

Microsoft, n.d.a. Developer guide to Conditional Access authentication context. Microsoft identity platform documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity-platform/developer-guide-conditional-access-authentication-context [Accessed 13 October 2026].

Microsoft, n.d.b. Conditional Access adaptive session lifetime policies. Microsoft Entra ID documentation, Microsoft Learn. Available at: https://learn.microsoft.com/en-us/entra/identity/conditional-access/concept-session-lifetime [Accessed 13 October 2026].

MITRE, 2026. ATLAS data, release 2026.09. MITRE ATLAS. Available at: https://github.com/mitre-atlas/atlas-data/blob/main/dist/v6/ATLAS-2026.09.yaml [Accessed 13 October 2026].

Model Context Protocol, 2026b. Authorization security considerations. Specification, version 2026-07-28. Available at: https://modelcontextprotocol.io/specification/2026-07-28/basic/authorization/security-considerations [Accessed 13 October 2026].

National Conference of State Legislatures, 2022. 2022 Security Breach Legislation. Available at: https://www.ncsl.org/technology-and-communication/2022-security-breach-legislation [Accessed 13 October 2026].

National Institute of Standards and Technology, 2023. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. Available at: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf [Accessed 13 October 2026].

National Institute of Standards and Technology, 2024. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1. Available at: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf [Accessed 13 October 2026].

New York State Department of Financial Services, n.d. 23 NYCRR Part 500: Cybersecurity Requirements for Financial Services Companies, section 500.17. Available at: https://www.dfs.ny.gov/cybersecurity/23-NYCRR-Part-500 [Accessed 13 October 2026].

OWASP GenAI Security Project, 2025. OWASP Top 10 for Agentic Applications for 2026. December 2025. Available at: https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/ (document: https://genai.owasp.org/download/52117/?tmstv=1765059207) [Accessed 13 October 2026].

OWASP GenAI Security Project, 2026. OWASP GenAI LLM Top 10 2026. 3 August 2026. Available at: https://genai.owasp.org/home/resources/owasp-genai-llm-top-10-2026/ (document: https://genai.owasp.org/download/56857/?tmstv=1785822482) [Accessed 13 October 2026].

Willison, S., 2025. The lethal trifecta for AI agents: private data, untrusted content, and external communication. Simon Willison's Weblog, 16 June. Available at: https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/ [Accessed 13 October 2026].

Keeping Raw Data You May Be Asked to Delete

Andrew Stevens — Tue, 13 Oct 2026 12:14:51 +0000

Your first job on an erasure request is a search: where does this person's data sit? For a customer with one online order, the example estate in this series holds it in more than twenty places across the source database, the lakehouse, and the records around it. Some are obvious, such as the orders table and its bronze copy. Others are easy to miss: the change tables that feed replication, a month-end snapshot, the text of a query someone ran to look into a refund, and the table that caught a pipeline error. Each has its own rule for how long data stays after a delete, and several keep it for weeks after the DELETE statement succeeds.

This series is for teams that have built a medallion lakehouse on Google Cloud and now have to run it. It assumes the layers are in place, as described in our post on the medallion architecture. Part 1 starts with bronze, because bronze is where replay comes from and also where personal data is often kept longest, in its least processed form. The aim is an erasure process that finishes by a date the platform can compute for its own stores. That date applies to erased data. A lot of order data is kept on purpose under legal obligations, and this post keeps the two apart.

The estate is a composite built for this series, close to the one in our post on multi-cloud retail: Microsoft Entra ID for workforce identity (which a later part on access uses), Google Cloud for the data platform, and Azure SQL Database under an online retailer's operational systems.

The storage options this series refers to

Several parts of this series depend on how bronze, silver, and gold are stored, so we give three storage options a letter here. A later part compares them in depth.

A. BigQuery native tables, stored and managed entirely by BigQuery.
B. Apache Iceberg managed tables (formerly BigLake tables for Apache Iceberg in BigQuery). Data files sit in your Cloud Storage bucket and BigQuery manages them. Google garbage-collects data files after the time travel window, but not the files of a dropped table, and recommends keeping the bucket's default soft delete policy (Google Cloud, n.d.b).
C. Iceberg in an open catalog, such as Google's Apache Iceberg REST catalog endpoint, with other engines writing the files and you running the file maintenance. BigQuery can't create or modify tables in that endpoint with DDL or DML (Google Cloud, n.d.y).

This part works through the design on options A and B, where BigQuery runs both the split job and the erasure job. On option C, those jobs run in Spark against the catalog instead. The file chain (delete files, compaction, snapshot expiry, and orphan cleanup) belongs with the storage comparison.

Option B carries one constraint that shapes the design. Google lists authorised views as unsupported for managed Iceberg tables (Google Cloud, n.d.b). In this design, readers reach bronze, the retained store, and the identity table only through authorised views. So on option B those three stay native tables, as do the stores those views read (the suppression entries, the reopen table, the hand-off table, and the kept list). Managed Iceberg holds silver and gold, whose readers have table access anyway. If you put bronze on managed Iceberg, you have to give silver and analysts read access on the tables, and that lets them read deleted rows during the time travel window. The option B rows in the ledger and the worked example cover that case's timing. That read exposure comes on top.

Where this design follows the textbook medallion, and where it departs

Google's guide to the medallion architecture describes four implementation steps (Google Cloud, n.d.ag). The first three build the layers. Ingestion moves data into bronze "without changing it", so no information is dropped or accidentally altered on the way, and bronze acts as a permanent, historical archive. Building silver uses Spark or SQL to apply data quality rules, such as converting timestamps to UTC or removing test accounts. Gold comes from SQL queries that join tables into metrics like revenue by category, often as high-performance views that a tool such as Looker can read directly.

The fourth step, orchestration and governance, names Managed Service for Apache Airflow for scheduling. Its governance example is letting only a few authorised service accounts write to silver and gold. The page's list of Google Cloud products also names Dataform for building and scheduling the transformations that turn silver into gold (Google Cloud, n.d.ag).

This series follows those steps, with SQL for every move between layers on options A and B, and Spark SQL on option C. It's one approach, and we've seen it work well where the data already sits in BigQuery: the work runs where the data is, with no separate cluster to run and no data to move out. On the estates we've worked on, it has usually been cheaper and faster to run, and quicker to build and change, than the same work in pipeline code. The right pattern depends on the data, team, and existing tooling, and another estate can call for another.

The tool that runs that SQL varies by engagement. We've used Airflow, Dataform, and dbt (scheduled by the existing orchestrator or by dbt Cloud), usually whatever you already run. The example estate uses Managed Service for Apache Airflow. It schedules every job in this post and submits each SQL script to BigQuery as a job under that job's own service account. The infrastructure account creates the gold view once. The hardening section sets out the access rules for Airflow and what changes under Dataform or dbt.

Personal data calls for one departure. Google's page doesn't cover it, and a bronze layer that keeps every customer's identifiers as a permanent archive makes the storage limitation and erasure duties hard to meet. So this series splits Google's bronze in two:

A restricted raw dataset takes the source rows as received, except the email (which in the main design never leaves the source), and keeps them only for a short retention period.
Bronze holds the longer-lived history for replay, kept for its purpose periods, with direct identifiers replaced by a token.

Google's steps leave every change to the data until silver. This design tokenises before bronze, so the history kept longest holds the least.

There are two smaller differences. Google says data scientists often want raw bronze data to train models. Here they get tokenised bronze through authorised views, and no person or analyst reads raw. Google also names Cloud Storage as the bronze landing zone. In this design raw lands in BigQuery through Datastream, and on option B, bronze's data files sit in Cloud Storage under BigQuery's management.

What bronze is kept for, and what has to be kept anyway

After launch, bronze usually serves three purposes: replaying history when a rule changes or a bug is found, answering audit questions about what arrived and when, and debugging pipelines. Each needs a different amount of history. In the estates we've seen, retention is easier to justify per purpose than as one number for the whole layer.

GDPR's storage limitation principle in Article 5(1)(e) asks that personal data be kept in a form that permits identification of people "for no longer than is necessary for the purposes for which the personal data are processed" (European Parliament and Council, 2016). In California, section 1798.100(a)(3) of the Civil Code requires businesses to disclose how long they intend to keep each category of personal information or, if that isn't possible, the criteria used to decide it (California Legislature, n.d.a), and the regulations require the same in the notice at collection (California Privacy Protection Agency, n.d.a).

Retention runs the other way too. Orders, invoices, payments, and returns are often subject to tax and accounting retention, and both regimes allow for this. Under Article 17(3)(b) and (e), GDPR's right to erasure doesn't apply to the extent processing is necessary "for compliance with a legal obligation" under Union or Member State law, or "for the establishment, exercise or defence of legal claims" (European Parliament and Council, 2016). Section 1798.105(d) of the CCPA has an exception for complying with a legal obligation (California Legislature, n.d.b). So the first step in this design is to sort event families into two groups. In the running example:

Erased: saved carts, recently viewed items stored by the application, wishlists, and reviews. Marketing preferences are erased too, with one exception covered below.
Kept under a stated obligation: orders, order lines, invoices, payments, and returns, with shipment events following their orders. They're access-restricted and minimised, and recorded in the erasure log with their retention and basis. Where the obligation allows, they lose their token-to-person mapping.

Where an obligation requires identifying fields, such as the name and address on an invoice, those fields stay in the retained record under the record's own access and retention. You can still trace retained records to the person through the source system's own retained records, so they stay personal data for as long as those exist.

That split is an illustration. Which records a business has to keep, and for how long, depends on where it trades. This post doesn't set out any country's retention periods.

Three ways to make bronze erasable

Rewrite on delete. Keep identifiers in the events and delete or rewrite the person's rows wherever they are. It needs nothing extra at landing, but every erasure scans every table that might hold the person, and a missed table means a missed erasure.

Separate identifiers with a random token. Split identifiers from events soon after landing. It costs a split job and an identity table, and each erasure then starts from a short, known list of tokens.

Per-subject encryption. Encrypt each person's fields with their own key, and delete the key to make the data unreadable. Google describes deleting a key to render data unreadable as crypto-shredding, and BigQuery's AEAD functions can use keysets wrapped by Cloud KMS (Google Cloud, n.d.a). Each person's keyset would usually sit in a table. Deleting its row is then subject to that table's time travel, with fail-safe assumed as an upper bound.

If the keysets are wrapped, destroying the wrapping KMS key version means no keyset wrapped by it can be unwrapped again. That suits shredding a whole dataset more than one person. The key material is destroyed only after a scheduled period of 30 days by default, and Google says it can then remain in its systems for up to 45 days from the scheduled destruction time (Google Cloud, n.d.j).

Encryption also covers only the copies that stayed encrypted. Values decrypted into silver, gold, caches, or extracts aren't affected. When the KMS resource name and wrapped keyset are passed to KEYS.KEYSET_CHAIN as constants, they can appear in the diagnostic query plan, though Google says the keys are never exposed in plaintext and decrypting the wrapped keyset needs permission on the Cloud KMS key (Google Cloud, n.d.e). NIST classes cryptographic erase as a purge technique for media sanitisation (National Institute of Standards and Technology, 2025). Its conditions include, as NIST says ISO/IEC 27040 specifies, being able to sanitise all copies of the key. In our reading, keyset rows held in time travel are copies. We haven't found EU guidance that treats key deletion as erasure, so this design uses crypto-shredding, where it uses it at all, alongside deletion rather than in place of it.

Each fits a different context. Where your estate has many event tables and erasure requests arrive regularly, we've seen the token approach work well, with rewrite-on-delete for the few stores that have to hold identifiers. That's the design the rest of this post builds. If yours has few tables, or identifiers that can't leave the events, rewriting on delete may serve you better. Per-subject encryption suits retiring a whole dataset more than erasing one person.

The design in brief

These terms carry most of the post.

Source key: the customer's account ID in the source system.
Customer key: a pseudonymous join key, an HMAC of the normalised email address, computed by one key service. Guest orders and carts carry one even though they have no source key. The split gives each guest customer key an identity row of its own, with no source key, so erasure finds a guest's records by customer key. Retained records carry the token in place of the customer key, for guests too.
Token: a random value that stands in for the person on bronze events.
Identity table: the mapping from token to source key and customer key. Only the split and erasure jobs, and any named scanner, read or write it directly. Analysts and silver jobs get tokens and never the mapping.
Suppression entries: the keys of people who have asked for erasure, checked by the split job before anything new is written.
Retained store: the restricted home of event families kept under an obligation.
Erasure time: when the source system's erasure of the person took effect, fixed once by the erasure job and used to tell the person's old events from a returning customer's new ones.
Request intake, queue, and erasure log: the service account that records verified requests, the table where a request's keys wait for the erasure job, and the job's record of what it did, which holds no keys or tokens.
Hand-off table and kept list: where the erasure job passes a person's tokens, with a pass number, to the jobs that erase silver and gold; and the identity rows an obligation requires to be kept, with their basis.
Horizon: the computed date by which erased copies have left the pipeline's own stores.
Cutoff and landing check: the last split run whose identity rows count as the person's own, recorded by the erasure job at its first step; and the test that the source erasure has reached the raw dataset.

Figure 1 shows where one customer's record lands, with the backfill path drawn separately from CDC.

One customer, end to end

Take one account holder with a wishlist and one order. In the application's customers table, their row holds their email next to the customer key the key service returned. Their wishlist and order rows hold their source key.

Datastream copies changes to all three rows into the restricted raw dataset, leaving out the email. From there the split job does three things:

It creates an identity row, one per source key, with a new token, the source key, and the customer key.
It writes the wishlist events to bronze with the token in place of the account ID.
It writes the order to the retained store, also with the token, because orders are kept under an obligation.

An analyst who counts wishlist items by category reads a gold metric with no token in it. That metric is built from a silver table that carries the token.

A month later, the customer asks for erasure. Request intake adds their source key and customer key to the suppression entries, so no new parent event keyed to them reaches bronze. Child rows of records already in bronze are removed in steps 1 and 3. The source system erases the customer row on its own schedule, and the erasure job works through its steps:

It deletes the wishlist events by token (step 1).
It waits for any split run already under way, and checks again (steps 2 and 3).
It hands the token to the jobs that erase silver and gold (step 4).
Once they confirm, it deletes the identity row and clears the token from the retained order (step 5).
When the source erasure has landed, it deletes the customer's rows from raw (step 6).

The order stays in the retained store, linked to its invoice by order ID. It has no token and no mapping back to the person, though it keeps any identifying fields its obligation requires. Bronze, the identity table, and raw still hold the deleted rows for their time travel and fail-safe windows, and the post works out when those windows close.

The tables, and the SQL between the layers

Table 1. Tables in the listings. It groups them by layer. We explain each column where its rule comes up: the erasure time and cutoff in stage 2, and split runs in the split job section. The names are illustrative. Each erased family, such as carts, has tables of the same shape.

Table	Layer	Holds	Written by	Read by
`raw.*`	Raw	Source rows as received, minus the email	Datastream	Split, erasure job, cleanup job
`staging.wishlist_changes`, `staging.wishlist_item_changes`	Raw	Raw rows the split hasn't processed yet	Split, as new tables each run, named per run (shown without the suffix), each with a one-day expiry	Split
`identity.identity_map`	Keys	Token, source key, customer key, and the split run that set the key (`key_set_run`): one row per source key, or for guests one per customer key and token, so a returning guest gets a second row	Split; the erasure job deletes rows	Split, erasure job, export view
`suppression.source_keys`, `suppression.customer_keys`	Keys	Erased keys with their request ID, request time (`requested_at`), and erasure time (`erasure_ts`)	Intake appends; the erasure job fixes the erasure time	Split, erasure job, export view
`suppression.record_keys`	Keys	Suppressed record keys with their family, request ID, and how far each reaches (`suppress_through`)	Split, erasure job	Split, erasure job
`erasure.queue_keys`	Erasure	The request's queue row: its keys and record keys, one table row each, every one carrying the request's cutoff (`cutoff_run`) once step 1 records it	Intake, erasure job	Erasure job
`split_marks.marked_rows`	Raw	Raw rows finished with, by row ID, reason, record key, and request ID	Split, erasure job	Split, cleanup job, erasure job
`ops.split_runs`, `ops.committed_split_runs`, `ops.merge_watermarks`	Operations	Run numbers and start times only	Split; silver merge	Split staging, silver merge, erasure job
`bronze.wishlists`, `bronze.wishlist_items`	Bronze	Events with the token, record keys, the raw row each came from (`raw_row_id`), the split run that wrote them (`split_run`), and an ordering field for changes in the same instant (`change_seq`)	Split; the erasure and bronze retention jobs delete	Split, erasure job; silver through `bronze_views`
`bronze_views.wishlist_items`	Bronze	An authorised view, storing nothing	Infrastructure account	Silver merge
`silver.wishlist_items`, `silver.orders`, `silver.products`	Silver	Cleaned rows; items and orders carry the token	Silver merges (only the items merge is shown); silver's erasure job deletes items and clears order tokens	Gold view (items and products); other gold jobs
`gold.wishlist_items_by_category`	Gold	A view of counts, storing nothing and carrying no token	Infrastructure account	Dashboards
`erasure_handoff.tokens`, `erasure_confirm_silver.confirmations`	Erasure	Tokens with request ID and pass number; confirmations with no token	Erasure job; silver's erasure job	Downstream erasure jobs; erasure job

Listing 1 traces the wishlist from bronze to gold, in the shape of Google's steps. A MERGE rebuilds each changed item in silver from its bronze history. Along the way it converts timestamps to UTC and drops test accounts. A view then joins silver tables into a gold metric. You'll find the split that writes bronze in Listing 2, in the split job section, and the erasure DELETEs in Listing 3, in stage 2. Each block runs as its own script. Silver reads and moves its watermark inside its own transaction, so it needs no per-run values.

SQL

-- Two scripts: the merge into silver, run after each committed split run,
-- and the gold view, created once.

-- Bronze to silver: recompute each changed item from its full bronze history,
-- read through an authorised view, up to the latest committed split run.
-- Timestamps become UTC, and an item whose latest change is a delete, or
-- comes from a test account, leaves silver. The watermark is read and moved
-- inside the transaction, so this script needs no per-run values.
DECLARE last_merged INT64;
DECLARE merge_to INT64;

BEGIN TRANSACTION;

SET last_merged = (SELECT IFNULL(MAX(last_merged_run), 0)
                   FROM ops.merge_watermarks
                   WHERE target = 'silver.wishlist_items');
SET merge_to = (SELECT IFNULL(MAX(split_run), 0)
                FROM ops.committed_split_runs);

MERGE silver.wishlist_items AS s
USING (
  SELECT token, wishlist_id, product_id, change_type, added_at_utc, is_test
  FROM (
    SELECT e.token, e.wishlist_id, e.product_id, e.change_type,
           TIMESTAMP(e.added_at_local, e.time_zone) AS added_at_utc,
           IFNULL(e.is_test_account, FALSE) AS is_test,
           ROW_NUMBER() OVER (PARTITION BY e.wishlist_id, e.product_id
                              ORDER BY e.source_ts DESC, e.change_seq DESC)
             AS latest
    FROM bronze_views.wishlist_items AS e
    JOIN (
      SELECT DISTINCT wishlist_id, product_id
      FROM bronze_views.wishlist_items
      WHERE split_run > last_merged
        AND split_run <= merge_to) AS changed
      ON changed.wishlist_id = e.wishlist_id
     AND changed.product_id = e.product_id
    WHERE e.split_run <= merge_to)
  WHERE latest = 1
) AS b
ON s.wishlist_id = b.wishlist_id AND s.product_id = b.product_id
WHEN MATCHED AND (b.change_type IN ('DELETE', 'UPDATE-DELETE') OR b.is_test)
  THEN DELETE
WHEN MATCHED THEN
  UPDATE SET token = b.token, added_at_utc = b.added_at_utc
WHEN NOT MATCHED AND b.change_type NOT IN ('DELETE', 'UPDATE-DELETE')
  AND NOT b.is_test THEN
  INSERT (token, wishlist_id, product_id, added_at_utc)
  VALUES (b.token, b.wishlist_id, b.product_id, b.added_at_utc);

MERGE ops.merge_watermarks AS m
USING (SELECT 'silver.wishlist_items' AS target) AS t
ON m.target = t.target
WHEN MATCHED THEN UPDATE SET last_merged_run = merge_to
WHEN NOT MATCHED THEN INSERT (target, last_merged_run)
  VALUES (t.target, merge_to);

COMMIT TRANSACTION;

-- Silver to gold: a plain view that joins silver tables into a metric for
-- dashboards. It stores nothing and carries no token or key. The
-- infrastructure account creates it once and authorises it on silver's
-- dataset, so its readers need no access to silver; no job recreates it.
CREATE OR REPLACE VIEW gold.wishlist_items_by_category AS
SELECT DATE(w.added_at_utc) AS day_added, p.category, COUNT(*) AS items
FROM silver.wishlist_items AS w
JOIN silver.products AS p
  ON p.product_id = w.product_id
GROUP BY day_added, p.category;

Listing 1. Bronze to silver and silver to gold for one wishlist, in BigQuery SQL.

On option A, the gold view is authorised on silver's dataset, so dashboard readers don't need access to the silver tables that carry the token. If silver is a managed Iceberg table, the view can't be authorised. In that case gold is a table built by its own registered job.

A plain view like this gold one stores nothing, so it needs no erasure job of its own. Silver can only be rebuilt from bronze, and by the time the token is handed off, bronze holds nothing for it. Gold is different when it's a table, or, on option A, a materialised view built for speed (our reading of Google's "high-performance views"). Then it holds aggregates computed from the person's rows until it's refreshed or rebuilt. A table has its own time travel, and a materialised view needs the same check. That gold job registers for the hand-off and confirms only after the refresh. Where aggregates are published widely, this design also suppresses small counts, because a count of one can single someone out.

Silver tables built from retained families, such as orders, keep their rows at erasure. Silver's erasure job clears their token, as step 5 does in the retained store, so revenue in gold doesn't move. A later part of this series covers what gold readers see when published numbers do change.

The rest of the post fills in each part: keeping the email out of capture, what the split writes, the two stages of erasure and their edge cases, marketing preferences, the stores to account for, and the date. The design leans on access controls throughout, so we've gathered them in one section near the end.

Keeping the email address out at source

Our post on multi-cloud retail computed the customer key in one key service, the only identity with sign and verify permission on the secret key in Cloud KMS. It also set out the choice between replicating raw addresses into the data platform and keying at source. This part takes the keying-at-source route and adds the mechanism for an estate fed by Datastream.

The email gets this treatment because it's the input to the join key. Other identifiers, such as names, phone numbers, delivery addresses, and free text, do land in the raw dataset. From there the split moves them to the identity side, or keeps them in a retained record whose obligation needs them.

The source application stores the customer key the key service returns. That post allows records to be stored first and keyed asynchronously. Neither the database nor the application holds the secret key, though the application can still request keys. Because the source database holds each email next to its customer key, its change tables and backups are on the ledger. The hardening section covers the controls that keep the key service's key to itself.

Two controls keep the email out of the lakehouse, one for each way data leaves the source.

Change data capture. Datastream supports Azure SQL Database only with the change tables CDC method, and lists it at service tier S3 and above (Google Cloud, n.d.aa). The @captured_column_list parameter limits which columns reach a change table, and Microsoft documents it for Azure SQL Database where only some columns should be tracked "for privacy or performance reasons" (Microsoft, 2025b). Primary key columns, or the columns of the unique index named with @index_name, must be captured (Microsoft, 2026h), so the email mustn't be part of either.
Backfill. A Datastream backfill streams a snapshot of the existing data (Google Cloud, n.d.o). We don't rely on the capture instance's column list to govern it. The stream's API schema allows columns to be listed under its included and excluded objects (Google Cloud, n.d.v). We'd list the allowed columns under included objects and treat that list as the control for backfills. A column added later then stays out until someone adds it.

A table over 500 million rows can be backfilled only if it has a unique index with no nullable columns and every index column is in the stream (Google Cloud, n.d.aa). That's one more reason to keep the email out of such indexes.

When a stream starts, Datastream selects the latest capture instance, doesn't switch to one created afterwards, and skips a table whose selected instance is deleted (Google Cloud, n.d.aa). So the instance that excludes the email has to exist before the stream starts.

If CDC is already running with every column, you recreate the stream against the new instance, usually with a backfill, and only then drop the original instance. Dropping it removes its change table (Microsoft, 2026g), but backups taken before then still contain it. SQL Server allows two capture instances per table (Microsoft, 2026h), so adding a source column becomes a planned change. A later part covers it.

This design leaves Datastream's enable_ddl_support_for_ct label off. With the label on, Datastream creates and switches to a new capture instance when a column is added (Google Cloud, n.d.aa), and the documentation doesn't say which columns that instance captures. Without the label, columns added after the stream starts don't replicate, and that's what this design wants. Masking the column isn't a workaround either. Datastream doesn't support tables with masked columns, and where they are replicated, the values arrive unmasked (Google Cloud, n.d.aa). Datastream's own login on the source is a privileged account, and the hardening section covers it.

If you can't change the source application, one fallback is a BigQuery remote function that calls a batch endpoint of the key service from the split job. It comes at a cost. Because the email is captured, it stays in the change tables and the raw dataset for their full retention. The schema check allowlists that one column, and the split job keys it and doesn't carry it forward. The worked example in "Computing the date" assumes keying at source. Under the fallback, its change-table and raw chains would also be chains for the email itself. The hardening section covers the endpoint's controls.

The split job

Datastream writes to BigQuery in merge mode by default, which keeps no history of change events (Google Cloud, n.d.f). A lakehouse that replays from bronze needs that history, so the example estate uses append-only mode. There, an update becomes a single UPDATE-INSERT row, an update that changes the primary key becomes an UPDATE-DELETE and UPDATE-INSERT pair, and a delete becomes one DELETE row (Google Cloud, n.d.f). Datastream applies these through the Storage Write API over gRPC (Google Cloud, n.d.f). Rows written that way can be changed by UPDATE, DELETE, or MERGE, with no waiting period stated (Google Cloud, n.d.ac). The erasure steps below rely on that.

Datastream can write every table into a single dataset (Google Cloud, n.d.f). The example estate creates that raw dataset beforehand, because otherwise a new BigQuery dataset gives READER, WRITER, and OWNER access to project Viewers, Editors, and Owners, and OWNER access to its creator (Google Cloud, n.d.c). Its access list names only the split job, the erasure job, the raw cleanup job, and any named scanner. Datastream writes through its project-level service agent. Raw, bronze, the identity table, the suppression entries, and the other restricted stores all follow one isolation rule, set out in the hardening section.

Datastream applies partitioning only when it creates a table, and you configure it in the stream (Google Cloud, n.d.r). So ingestion-time partitioning goes in before the stream first runs. Expiration alone doesn't give a date: an expired partition is removed by a background process that "can take several days to complete", and that deletion isn't recorded in audit logs (Google Cloud, n.d.p). A deleted partition's data is then kept under time travel and fail-safe (Google Cloud, n.d.p).

So the erasure job deletes the person's raw rows itself, and a scheduled cleanup job deletes everything else older than the raw retention period. The cleanup job skips rows that the split is holding back or that are waiting for a parent. A hold older than a stated threshold pages someone.

Expiration stays as a backstop, set longer than the raw retention period plus that hold threshold. The split records the raw rows it has finished with in its marks table, split_marks.marked_rows, with a reason: 'copied' for retained-family rows it has sent to the retained store, 'suppressed' for rows it dropped, 'orphaned' for erased-family children whose parent never arrived, and 'identity' for rows from identity sources, such as the customers table, once their fields are on the identity side. On its next daily run, the cleanup job removes marked rows, including the erasure job's 'erased' rows. It deletes a mark once its raw row is gone. Expiration set as a dataset default affects newly created tables (Google Cloud, n.d.ad), so it goes in before the stream creates its tables, or per table afterwards with ALTER TABLE ... SET OPTIONS (Google Cloud, n.d.p).

The split job is written in BigQuery SQL, runs from Airflow on options A and B, and writes three things. Airflow starts each split run one at a time, with a run timeout. Each run begins with a task that runs as the split account. It takes the next run number from a counter table, ops.split_runs, and records its start time there. Numbers therefore increase by start time, and a retry gets a new number. Without that, Airflow would run the retry under the same run. That task also waits until INFORMATION_SCHEMA.JOBS shows no earlier split job still pending or running, so two runs never overlap. A run that times out is cancelled when Airflow kills its task, and a job timeout set on the BigQuery script ends it even if the worker running the task dies.

The identity table maps a random token to the source key and the customer key. Clear identifiers from erased families are kept only where a named purpose needs them (for example a coarsened delivery postcode). Each has its own retention, which the erasure job enforces, and the purpose reads it through its own authorised view. Free text counts as an identifier. Erasure deletes the whole identity row, these identifiers included.
Bronze events carry the token in place of both the direct identifiers and the customer's source key. Record keys such as order and cart IDs stay, because deduplication and child lookups need them. Those keys can be joined back to the person through the source system while its records are kept. That's one more reason to treat the events as personal data. The events are clustered by token, and only the split job inserts them. A bronze retention job deletes erased-family events once they pass the longest of bronze's purpose periods, so bronze isn't kept indefinitely for people who never ask.
The retained store holds event families kept under an obligation, including any identifying fields the obligation requires. They don't go to bronze. While the mapping exists, they carry the token like bronze events. Silver reads them through a minimised view that returns the token as null for any token on the kept list or in the erasure hand-off table (the view is authorised on both datasets). That way, a retained record merged into silver after silver's erasure job has run doesn't bring the token back. A retention job deletes records when their obligation period ends.

Child records, such as cart lines, wishlist items, order lines, and shipments, get their token from the token-keyed parent event. A child that arrives before its parent waits in the raw dataset. One whose parent never arrives is counted. In an erased family, it's then marked 'orphaned' for the cleanup job. In a retained family it goes to the retained store unlinked.

When the split suppresses a parent in an erased family, it writes that parent's record key, with its family (cart or wishlist) and the request ID from the matching entry, to a table of suppressed record keys kept with the suppression entries. A source-key match suppresses every event of that record. A customer-key match covers events up to the erasure time, or up to the request while that time isn't fixed. The erasure job's own entries, from step 1, cover every event. A suppressed parent's children are then dropped and counted separately. Every suppression entry carries its request ID.

Listing 2 is the split for the wishlist family. It writes a wishlist event to bronze only when no suppression entry covers it. For each wishlist it suppresses, it records the key and marks those rows and their items for the cleanup job. The other items take their token from the wishlist's latest event in bronze. Each split run's statements share one transaction, so a failed run leaves nothing in bronze. The listing leaves out the 'orphaned' marks, the retained-store writes with their 'copied' marks, and the identity rows. The split writes those identity rows under the same suppression check.

Each bronze row carries the raw_row_id it came from. A raw row therefore counts as processed once it's in bronze, the retained store, or the marks table, from a committed split run or the erasure job. Because that column links bronze back to raw, only holders of raw access can read it. The raw retention period is set shorter than bronze's shortest purpose period, so a row the bronze retention job deletes is never copied back. A halted cleanup job breaks that ordering as well as the raw dates. So it pages someone, and its delay moves the date the same way a halted split does.

The split's run number is a named query parameter, written with @ (Google Cloud, n.d.w). The run's first task takes it from ops.split_runs and passes it to the script.

SQL

-- Raw to bronze (split job). The split's staging tables hold raw rows it
-- hasn't processed yet: rows not in bronze, the retained store, or the marks
-- table under any reason, including the erasure job's 'erased' marks, which
-- carry no split run. They're written as new tables before this transaction,
-- named per run (shown here without the suffix), each with a one-day expiry.
-- Each run's statements run in one transaction, so a failed run leaves nothing
-- in bronze. Not shown: identity rows and their 'identity' marks, guest
-- records keyed by customer key, retained-store writes with their 'copied'
-- marks, and 'orphaned' marks for children whose parent never arrives.
BEGIN TRANSACTION;

-- Parents: write a wishlist event only when no suppression entry covers it.
-- Rows it skips stay in raw, either suppressed or held back until the
-- person's erasure time is known. Identity rows are written under the
-- same check.
INSERT INTO bronze.wishlists
  (token, wishlist_id, change_type, change_seq, source_ts, raw_row_id,
   split_run)
SELECT i.token, w.wishlist_id, w.change_type, w.change_seq, w.source_ts,
       w.raw_row_id, @split_run
FROM staging.wishlist_changes AS w
JOIN identity.identity_map AS i
  ON i.source_key = w.customer_id
WHERE NOT EXISTS (
    SELECT 1 FROM suppression.source_keys AS s
    WHERE s.source_key = w.customer_id)
  AND NOT EXISTS (
    SELECT 1 FROM suppression.customer_keys AS c
    WHERE c.customer_key = i.customer_key
      AND (c.erasure_ts IS NULL OR w.source_ts <= c.erasure_ts))
  AND NOT EXISTS (
    SELECT 1 FROM suppression.record_keys AS r
    WHERE r.record_family = 'wishlist'
      AND r.record_key = w.wishlist_id
      AND (r.suppress_through IS NULL OR w.source_ts <= r.suppress_through));

-- Record the key of each wishlist this run suppressed, with the covering
-- entry's request ID and how far it reaches: a source-key match covers
-- every row (NULL), a customer-key match covers rows up to the erasure
-- time, or up to the request while that time isn't fixed. Held-back rows
-- aren't suppressed yet. A source-key match counts even after the
-- person's identity row has been deleted.
INSERT INTO suppression.record_keys
  (record_family, record_key, request_id, suppress_through, split_run)
SELECT DISTINCT 'wishlist', w.wishlist_id,
       COALESCE(s.request_id, c.request_id),
       IF(s.source_key IS NOT NULL, NULL,
          IFNULL(c.erasure_ts, c.requested_at)),
       @split_run
FROM staging.wishlist_changes AS w
LEFT JOIN identity.identity_map AS i
  ON i.source_key = w.customer_id
LEFT JOIN suppression.source_keys AS s
  ON s.source_key = w.customer_id
LEFT JOIN suppression.customer_keys AS c
  ON c.customer_key = i.customer_key
 AND ((c.erasure_ts IS NOT NULL AND w.source_ts <= c.erasure_ts)
      OR (c.erasure_ts IS NULL AND w.source_ts <= c.requested_at))
WHERE s.source_key IS NOT NULL OR c.customer_key IS NOT NULL;

-- Mark the suppressed parent and child rows once each, for counting and for
-- the cleanup job.
INSERT INTO split_marks.marked_rows (raw_row_id, reason, record_key,
                                     request_id, split_run)
SELECT raw_row_id, 'suppressed', record_key, MIN(request_id), @split_run
FROM (
  SELECT w.raw_row_id, w.wishlist_id AS record_key, r.request_id, w.source_ts,
         r.suppress_through
  FROM staging.wishlist_changes AS w
  JOIN suppression.record_keys AS r
    ON r.record_family = 'wishlist' AND r.record_key = w.wishlist_id
  UNION ALL
  SELECT w.raw_row_id, w.wishlist_id AS record_key, r.request_id, w.source_ts,
         r.suppress_through
  FROM staging.wishlist_item_changes AS w
  JOIN suppression.record_keys AS r
    ON r.record_family = 'wishlist' AND r.record_key = w.wishlist_id)
WHERE suppress_through IS NULL OR source_ts <= suppress_through
GROUP BY raw_row_id, record_key;

-- The other children take their token from the parent's latest event in
-- bronze, under the same customer-key check as the parent. Items whose parent
-- hasn't arrived stay in raw for a later run. A child written after intake but
-- before its account's identity row has a customer key is caught by steps 1
-- and 3.
INSERT INTO bronze.wishlist_items
  (token, wishlist_id, product_id, change_type, change_seq, added_at_local,
   time_zone, is_test_account, source_ts, raw_row_id, split_run)
SELECT p.token, w.wishlist_id, w.product_id, w.change_type, w.change_seq,
       w.added_at_local, w.time_zone, w.is_test_account, w.source_ts,
       w.raw_row_id, @split_run
FROM staging.wishlist_item_changes AS w
JOIN (
  SELECT wishlist_id, token
  FROM (
    SELECT wishlist_id, token,
           ROW_NUMBER() OVER (PARTITION BY wishlist_id
                              ORDER BY source_ts DESC, change_seq DESC)
             AS latest
    FROM bronze.wishlists)
  WHERE latest = 1) AS p
  ON p.wishlist_id = w.wishlist_id
WHERE NOT EXISTS (
    SELECT 1 FROM identity.identity_map AS i
    JOIN suppression.customer_keys AS c
      ON c.customer_key = i.customer_key
    WHERE i.token = p.token
      AND (c.erasure_ts IS NULL OR w.source_ts <= c.erasure_ts))
  AND NOT EXISTS (
    SELECT 1 FROM suppression.record_keys AS r
    WHERE r.record_family = 'wishlist'
      AND r.record_key = w.wishlist_id
      AND (r.suppress_through IS NULL OR w.source_ts <= r.suppress_through));

INSERT INTO ops.committed_split_runs (split_run) VALUES (@split_run);

COMMIT TRANSACTION;

Listing 2. Raw to bronze for the wishlist family, in BigQuery SQL: one split run, in one transaction.

A schema check runs on the raw tables every time. If a column's name suggests an email address, the run stops and pages someone. The same happens when a content check finds email addresses in a column not already routed to the identity side. Other unrecognised columns go to the identity side. A halted split job pages someone well inside the raw retention period, and it also holds up erasures, as step 6 below shows.

Downstream, silver jobs and analysts work with tokens. Some uses need the cross-system customer key, such as the campaign audiences our earlier post described leaving the platform as customer keys. For those, a named export job resolves tokens through one authorised view. The view returns token and customer key only, and it leaves out an erased person's own identity rows. The hardening section gives its rule. Whatever the job exports goes on the ledger with its recipient.

The events are still personal data. As our earlier post set out, Recital 26 says pseudonymised data that could be attributed to a person with additional information should be considered information on an identifiable person (European Parliament and Council, 2016). The EDPB's 2025 coordinated enforcement action found that some controllers relying on anonymisation in place of deletion applied only basic pseudonymisation or partial masking, which "would not fulfil the requirements of the GDPR regarding deletion" (European Data Protection Board, 2026).

California's regulations, by contrast, let a business comply with a deletion request by deidentifying or aggregating the information (California Privacy Protection Agency, n.d.b), so the bar differs from one regime to another. Section 1798.140(m) of the CCPA defines deidentified information as information that "cannot reasonably be used to infer information about, or otherwise be linked to, a particular consumer", provided the business also takes reasonable measures, commits publicly to keep it deidentified, and binds recipients by contract (California Legislature, n.d.d). As we read it, tokenised events with a live mapping and record keys wouldn't meet that. Where this post says a source system anonymises a record, it means the record can no longer identify the person. The split shortens the search for a person's events. The erasure job below still deletes them.

Erasure in two stages

Figure 2 shows the two stages.

Stage 1, at the request

Only the request-intake service account records verified requests. IAM can't tell whether a request is verified, so intake's application makes that check. It appends the person's keys to the suppression entries first. The queue row goes last, and only after the entries' appends are acknowledged, so the erasure job never sees a request whose entries aren't there.

The keys come from the source. At intake, the source application uses its existing path to the key service. It sends every address it holds or has held for the person and verified for the account, plus the verified address the requester supplied. It passes the returned customer keys, computed under every enabled key version, to intake. Stored addresses matter because the requester may type a different one, and older guest orders may carry an older address. The person's source key and customer keys then go on the suppression entries and in the queue. You need both, because a guest order carries a customer key but no account's source key.

The source erasure itself runs on its own clock. When it finishes, the source tells intake, and intake appends that signal to the queue. The source also clears the customer key from the records it keeps, so a later backfill can't link them to the person again.

The split job checks every event against the entries. The rules apply top to bottom, and the first match wins:

Event	Condition	Action
A change row from an identity source, such as the customers table, including its DELETE or anonymising UPDATE-INSERT row at source erasure	Any time	Identity fields go to the identity side, except that no identity row is written or updated for an erased source key or a covered customer key; the raw row is marked 'identity' either way
The DELETE row, or anonymising UPDATE-INSERT row, for the source erasure	Erased family, any timestamp	Suppressed
Other erased-family event for an erased source key	Dated after the erasure time	Suppressed, and raised with the source system's owner as a sign of a reused key or an incomplete source erasure
Erased-family event with an erased source key	Any time	Suppressed, and the source key's entry is kept while retained events for that key can still arrive
Retained-family event with an erased source key	Any time	Sent to the retained store without a token and with the source key and customer key dropped, with no new identity row
Erased-family event with an erased customer key	Dated at or before the request, or at or before the erasure time once it's fixed	Suppressed
Retained-family event with an erased customer key	Dated at or before the request, or at or before the erasure time once it's fixed	Sent to the retained store without a token, with the customer key and source key dropped
Any event with an erased customer key	Dated after the request, erasure time not yet known	Held back in raw
Event with the same customer key	Dated after the erasure time	Treated as a returning customer and given a new token

Two of these rules carry a cost. The source system's own retained records keep the account's source key for as long as the source keeps them, often years. Source-key entries last that long too, and it's worth saying so in the retention notice. Customer-key entries reveal that a returning customer was erased before, so they expire once no pre-erasure event can still land. That point depends on change-table retention, landing lag, and any backfill window. The erasure job deletes suppression entries when their retention ends.

Fixing the erasure time

The erasure job (its steps follow below) fixes the erasure time once. It does so on its first daily sweep after both of these are true: the landing check has passed for every table where the person has rows, and step 1 has recorded the request's cutoff. The job takes the latest source timestamp among the person's erasure change rows and sets it on their customer-key suppression entries, updating intake's rows. On the same entries it records the latest split run that finished before the erasure time was written. The split and the export view read that value. Both values stay on the entries after the queue row is gone.

Sometimes there's nothing to erase at source, because the source purged the person's records long ago while bronze kept them. In that case the source reports the time it checked, and the erasure job accepts that time as the erasure time once a heartbeat row has landed past it.

The landing check needs a watermark that always advances, because a table's latest source timestamp stops moving when nothing in it changes. The check runs only on tables where the person has rows. It reads Datastream's source timestamp metadata, which Google defines as the time the record changed on the source (Google Cloud, n.d.l).

Table	The check waits for
A table where the source erasure deleted the person's rows	Their DELETE rows
A table where the source erasure anonymised the person's rows	The change that clears the customer key, which touches a captured column and so produces a change row
Any other table where the person has rows	A heartbeat row, written every few minutes to a small captured table holding no personal data, landed with a source timestamp past the source's signal

Stage 2, the erasure job

The job picks up new requests on a daily sweep. From there it runs each request as a state machine that wakes when one of its waits ends. Fixing the erasure time and step 6 both happen on the daily sweep.

Check that every key in the queue row is present in the suppression entries. If any is missing, stop and page someone. Then record the request's cutoff: the latest split run that started before this step began. A run starts when its number is taken, and run IDs increase by start time. Look up the person's tokens. Copy their erased-family parent record keys (cart and wishlist IDs) from bronze into the queue and into the suppressed record keys, each with its family and no date limit, because every event of the person's own records goes. Copy their order IDs from the retained store into the queue only, since order lines and refunds must still reach the retained store. Then DELETE their events in erased families, marking their raw rows 'erased' (Listing 3).
Wait until every split run that began before step 1's record-key insert has finished. Split runs are serialised and time out. The job waits until INFORMATION_SCHEMA.JOBS shows no job by the split's service account, started before the insert, still pending or running, and until at least one split-run timeout has passed since the insert. That view's job states are pending, running, and done (Google Cloud, n.d.m). A timed-out run is cancelled, and its jobs can take a while to stop, so the wait ends only when the view shows them done. If it hasn't after a stated margin past the timeout, the job pages someone and keeps waiting. For this, the erasure job can read job metadata in the split's project.
Look up the tokens and record keys again, to catch anything a run in flight wrote. Copy any new record keys and run a confirming DELETE. If this found anything new, repeat steps 2 and 3 until a lookup finds nothing new.
Write the person's tokens to the hand-off table with a pass number. Each registered downstream erasure job (silver, gold, and feature stores) reads the table and writes its confirmation to its own table. The confirmation holds a request ID and pass number only. Without this step, those jobs would have no way to find the person once the mapping is gone.
Once every registered job has confirmed the current pass, or after a stated timeout that pages someone, delete the mapping and clear the token from the person's retained records. Where the obligation requires the mapping, it moves to the kept list with its basis instead. In that case the retained records keep their token, the minimised view that silver reads returns that token as null, and the mapping never passes through the export view. The hand-off rows go only when every job has confirmed, so a job that timed out can still find the person. They also always go after the retained token is cleared or moved to the kept list, so a merge into silver in between can't bring the token back.
Wait until step 5 has finished, the erasure time is fixed (as above), and the split has processed every row that landed before it. Then, in one transaction, DELETE the person's raw rows that steps 1 and 3 or the split have already processed, along with the marks of the rows it deletes. After that, delete the queue row. If the erasure time isn't fixed within a stated period of the request, the job pages someone, because the queue row keeps the person's keys until step 6.

Step 6 finds raw rows four ways: by source key, by customer key on rows dated at or before the erasure, by the record keys in the queue, and by the suppressed record keys carrying the request ID. That catches retained families' raw rows too, once the split has copied them to the retained store. Rows that land after the split's last processed point are left to the split. It routes retained rows to the retained store and marks suppressed ones for the cleanup job.

The job takes only a request ID as a parameter. It processes only request IDs that appear in intake's own record of verified requests, and it joins its DELETEs on the queue table, as Listing 3 shows. BigQuery reruns a DML statement that conflicts with a concurrent one up to three times (Google Cloud, n.d.ac), and the job retries beyond that. The bronze mark-and-DELETE pair runs as a transaction. BigQuery cancels it on a conflict, so the job retries the whole transaction. Silver's erasure job and the merge into silver both run in transactions, and BigQuery cancels conflicting transactions (Google Cloud, n.d.q), so each retries a cancelled transaction. The erasure log records the request ID, timestamps, row counts, and the computed horizon. It holds no keys or tokens.

Listing 3 shows two of the DELETEs erasure needs. The bronze DELETE in steps 1 and 3 is keyed by request ID alone, so no key or token appears in its query text. In the same transaction it marks the person's raw rows 'erased', so the split doesn't stage them again and step 6 can find them. Silver's DELETE is keyed by the tokens the erasure job hands off. For a reopen entry, the job runs the same bronze pair, but takes the token from the reopen table for that request in place of the identity lookup. A token found late can belong to a guest whose key was set after the cutoff, and neither branch of the lookup matches such a key. The erasure jobs run as Airflow tasks and pass the request ID and pass number as query parameters.

SQL

-- Two scripts: the erasure job's bronze DELETE, and silver's erasure job.

-- Erasure job, steps 1 and 3: delete the person's erased-family events from
-- bronze, and mark the raw rows behind them 'erased' in the same transaction,
-- so the split still counts those raw rows as processed and step 6 can delete
-- them. Raw rows that never reached bronze are left to step 6. The same pair
-- runs on bronze.wishlists and each erased-family table. The only parameter is
-- the request ID, and the keys come from the queue table, so no key or token
-- appears in the query text. Step 1 records cutoff_run before this runs; a
-- NULL cutoff would make the customer-key branch match nothing. For a reopen
-- entry the job runs the same pair with the token taken from the reopen table
-- in place of the identity lookup.
BEGIN TRANSACTION;

INSERT INTO split_marks.marked_rows (raw_row_id, reason, record_key,
                                     request_id, split_run)
SELECT e.raw_row_id, 'erased', e.wishlist_id, @request_id, NULL
FROM bronze.wishlist_items AS e
WHERE e.token IN (
  SELECT i.token
  FROM identity.identity_map AS i
  JOIN erasure.queue_keys AS q
    ON q.request_id = @request_id
  WHERE i.source_key = q.source_key
     OR (i.customer_key = q.customer_key
         AND i.key_set_run <= q.cutoff_run));

DELETE FROM bronze.wishlist_items AS e
WHERE e.token IN (
  SELECT i.token
  FROM identity.identity_map AS i
  JOIN erasure.queue_keys AS q
    ON q.request_id = @request_id
  WHERE i.source_key = q.source_key
     OR (i.customer_key = q.customer_key
         AND i.key_set_run <= q.cutoff_run));

COMMIT TRANSACTION;

-- Silver's erasure job, after step 4: delete by the hand-off tokens for this
-- request, up to the current pass, and confirm the pass in the same
-- transaction, so a failed DELETE is never confirmed. Silver tables from
-- retained families keep their rows and lose the token, so gold's revenue
-- doesn't move.
BEGIN TRANSACTION;

DELETE FROM silver.wishlist_items AS s
WHERE s.token IN (
  SELECT h.token
  FROM erasure_handoff.tokens AS h
  WHERE h.request_id = @request_id
    AND h.pass_number <= @pass_number);

UPDATE silver.orders AS o
SET token = NULL
WHERE o.token IN (
  SELECT h.token
  FROM erasure_handoff.tokens AS h
  WHERE h.request_id = @request_id
    AND h.pass_number <= @pass_number);

INSERT INTO erasure_confirm_silver.confirmations (request_id, pass_number)
VALUES (@request_id, @pass_number);

COMMIT TRANSACTION;

Listing 3. The erasure DELETEs in bronze and silver, in BigQuery SQL.

The customer-key branch of the bronze DELETE matches only identity rows whose key was set by split runs up to the request's cutoff (the latest run that started before the erasure job's first step). Rows set up to the cutoff can't belong to a returning customer. New tokens for that key wait until the erasure time is fixed, and that never happens before the cutoff is recorded. The edge cases below explain the ordering.

Late arrivals and edge cases

These cases are rare, but each one needs a defined path.

A token found late. If a row arrives before its customer key, the suppression check runs again when the key arrives, under the same date rules, so a returning customer's row isn't reopened. On a match, the split writes the reopen entry before it sets the key on the identity row, so the export view never sees the key unexcluded.

A match, or any token found after step 3, goes with its request ID to a reopen table kept alongside the suppression entries. The erasure job processes reopen entries whether or not the request is still open. If it's still open, the token joins the request under a new pass number. If step 6 has finished, the job rebuilds the request's queue row from the suppression entries that carry its request ID. If those entries have expired, the reopen pages someone and is handled by hand. Either way, the job deletes the token's events and marks their raw rows, hands the token downstream, deletes its mapping once the pass is confirmed, and logs a new horizon that includes those raw rows.

Returning customers and the customer-key lookups. Each identity row records the split run that set its customer key. The lookups match customer keys only on rows set by runs up to the request's cutoff. Intake's entries were visible before step 1 began, so every later run saw them. Each of those runs suppressed or held back the person's events, and any identity row it created for that key belongs to a returning customer. The cutoff can include a few runs that had already seen the entries. That only widens the search for the person's own rows, because the hold-back rule stops new identity rows for that key until the erasure time is fixed, and the erasure time is never fixed before the cutoff is recorded.

Rows that land after step 6. As we understand it, Datastream reads each change table separately, so the heartbeat shows that capture and landing are moving, but it doesn't prove that every other table has caught up. If a pre-erasure row lands after step 6, the split suppresses and marks it, and the cleanup job deletes it on its next run. Any such row moves the date, because its chain runs from when it landed. It's recorded against the request with a new horizon.

Backfilled rows. Datastream labels backfill events with a read_method starting sqlserver-backfill (Google Cloud, n.d.l), and as we understand it they don't carry the original change time. So the split treats backfilled erased-family rows for an erased key as pre-erasure. While customer-key entries live, that also drops a returning customer's new carts and wishlists from a backfill. You can accept that cost, or avoid it by not backfilling during that window.

Key rotation. The lakehouse holds no emails, so it can't re-key anything itself. A rotation of the customer-key secret isn't started while any request is open before step 6, because the suppression entries hold keys computed under the old version and a rotated key on a guest's identity row would no longer match them. The source recomputes its stored customer keys through the key service, and the split updates identity rows from the resulting change rows. It matches a guest's rotated key to the existing token by record key, so the row keeps its token and the erasure job can still find it. The row also keeps the run recorded as having first set the key. An old key version stays enabled until every customer-key suppression entry made under it has expired, because the addresses behind those entries are gone.

Marketing preferences

Article 21(2) and (3) give people the right to object to direct marketing at any time, after which their data "shall no longer be processed for such purposes" (European Parliament and Council, 2016). In the US, CAN-SPAM bars sending commercial email more than 10 business days after an opt-out request, unless the person later gives affirmative consent (United States Congress, n.d.). So when preferences are erased, a forward-looking do-not-contact marker stays, and we'd record a basis for keeping it. The UK ICO's direct marketing guidance describes a suppression list as keeping just enough information to respect people's preferences in future (Information Commissioner's Office, 2026).

The marker is an HMAC of the email under a separate suppression key that only the key service can use. It's stored in the key project, where only the key service can read it. The source application asks the key service to create the marker at erasure, and the consent system asks it to remove one when someone gives fresh consent. The sending system asks about an address and gets back allow or suppress. Customer-key rotation doesn't touch the marker, and the suppression key can be rotated without the erased addresses, as the hardening section describes. The marker covers email-keyed channels, and other channels need their own suppression.

The ledger

Figure 1 shows the main stores, and the ledger lists all of them across two tables. The first covers the pipeline's own stores, which the worked example below computes, with backups listed as beyond use. The second covers stores outside the computed date, each with its owner. Some are kept by design, some are owned elsewhere, and silver, gold, and other downstream stores are covered in later parts.

Stores within the horizon (backups listed as beyond use)

Where it sits	How it's removed	What sets when it's gone
CDC change tables	Azure SQL Database's CDC scheduler	Retention (three days by default) and hourly cleanup, plus measured cleanup lag (Microsoft, 2025b)
Restricted raw dataset	Erasure job, step 6; cleanup job for marked and older rows	Landing, erasure time fixed, split catch-up, job timing, then time travel and fail-safe
Identity table	Erasure job, step 5	Job timing and downstream confirmation, then time travel and fail-safe
Token hand-off table	Erasure job, step 5, once every downstream job confirms	Job timing and downstream confirmation, then time travel and fail-safe
Tokens in retained records	Erasure job, step 5 (token cleared)	Job timing and downstream confirmation, then time travel and fail-safe
Bronze events	Erasure job, steps 1 and 3; bronze retention job for the rest	Job timing, then time travel and fail-safe
Erasure queue table	Erasure job, step 6	The same as the raw dataset
Split job staging tables	Expiry set on each table	Split processing of the erasure rows, expiry, then time travel and fail-safe
Split marks	Erasure job, step 6, with the raw rows it deletes; cleanup job for the rest, once the marked raw rows are deleted	Job timing, then time travel and fail-safe
Reopen table	Erasure job, once the reopened request is done	Job timing, then time travel and fail-safe, for a request that reopens
Point-in-time restore backups	Expire on schedule (beyond use until then)	Restore window of 1 to 35 days (1 to 7 on Basic), 7 by default; files kept up to a week longer (Microsoft, 2026a)

Stores outside the horizon: kept by design, or owned elsewhere

Where it sits	Owner and rule
Source application tables	The source system's erasure or anonymisation
Original all-column change table	Removed when its capture instance is dropped (Microsoft, 2026g)
Long-term retention backups	Up to 10 years, if enabled (Microsoft, 2026f). Can't be deleted while any immutability applies: an unlocked time-based policy has to be disabled first, a locked one keeps each backup until its retention ends, and a legal hold, a preview feature, keeps it until the hold is removed. An immutable backup stays after its retention expires until every immutability is removed (Microsoft, 2026d). Beyond use throughout
Source query history and audit logs	Settings on the source database
Intake's record of verified requests	Kept by design, with its own basis, apart from the ticketing system; links the request ID to the requester
Ticketing	Kept by design, with its own basis
Suppression entries	Customer-key entries until no pre-erasure event can land; source-key entries while retained events can arrive; record-key entries until no child of that parent can land; then time travel and fail-safe
Do-not-contact markers	Until fresh consent; kept by design, with a stated basis
Kept list	Identity rows an obligation requires, for the obligation's period, then time travel and fail-safe; reached only through a named authorised view
Downstream confirmation tables	Request IDs and pass numbers only, kept with the erasure log; linkable to a person only through intake's record
Erasure log	Kept by design; holds no keys or tokens
Retained store	The obligation's period, recorded per family
Table snapshots and Iceberg tags	Google lists table snapshots as unsupported for managed Iceberg tables, so on option B this covers native tables only. Snapshot expiry, set early enough that expiry plus time travel and fail-safe, which we assume apply as for any table, ends by the computed horizon
Table clones, materialised views, search indexes, dev copies	Their owners, or rebuilt after the erasure
Query text in `INFORMATION_SCHEMA.JOBS`	Google's retention for the view, covered in a later part
Audit logs and log sinks	Log bucket retention and each sink's settings
Cached and anonymous query results	BigQuery's own expiry, covered in a later part
Error logs and dead-letter tables	Retention set per log bucket or table
Notebook outputs	Their owners
Profiling and data quality results	Retention set per tool
Sensitive Data Protection findings	Retention set per job
Silver and gold tables	Downstream erasure jobs reading the hand-off table, then time travel and fail-safe; covered in later parts
Exports, extracts, and audiences	Their owners, with each recipient listed; exported customer keys stay linkable to the person wherever they're kept
ML training sets and feature stores	Downstream erasure jobs reading the hand-off table, or their owners
Cross-region replicas	Replication of the delete, then the replica's own windows
Processors and other recipients	Notified under contract; their contractual deletion terms

A few notes apply to particular rows.

Time travel and fail-safe. BigQuery keeps changed and deleted data for a time travel window of two to seven days, seven by default, and Google says deleted data is then kept for a further seven days of fail-safe that can't be configured or extended (Google Cloud, n.d.i). Google describes fail-safe for deleted or replaced tables and deleted partitions, and doesn't list row-level DML, so for rows removed by DELETE we count it as a conservative upper bound.
Option B. Data files are garbage-collected after the time travel window (Google Cloud, n.d.b). We take it that deleted files then sit in the bucket's soft delete, since Google recommends keeping that policy for these buckets. Soft delete is seven days by default and configurable from 7 to 90 days, or disabled (Google Cloud, n.d.z). Google doesn't document fail-safe for these tables, so the worked example leaves it out pending confirmation from Google. A dropped option B table leaves its files in the bucket (Google Cloud, n.d.b). So in this design, personal-data tables on option B aren't dropped, including for rebuilds and dev copies, unless the bucket path is cleaned up afterwards.
Locks. A Bucket Lock retention policy stops objects being deleted until they reach the policy's age, and a locked policy can't be removed or shortened (Google Cloud, n.d.d). In this design, personal data stays out of locked buckets. Per Google, option B tables don't support buckets with Bucket Lock at all (Google Cloud, n.d.b).
Snapshots. Our post on multi-site pipelines suggested month-end snapshots or tags for finance. Either they expire early enough that their own time travel and fail-safe end by the computed horizon, or they're taken only of gold tables with no personal data.
Iceberg statistics. On option C, and possibly option B (Google doesn't say), Iceberg's default metrics mode stores lower and upper bounds for columns, truncated to 16 for string and binary values (Apache Iceberg, n.d.). By default it covers the first 100 columns, taking top-level fields first and then each nested struct in turn (Apache Iceberg, n.d.). A truncated bound can be enough of a name or delivery address to identify someone. The storage comparison covers the per-column setting that turns this off.
Queries. Query text can hold the values someone searched for. Google's AEAD guidance warns that query text may be logged, including any plaintext keyset in it, and recommends parameterised queries (Google Cloud, n.d.a). The erasure job's request-ID design applies the same habit.
Emails already in bronze. Many teams will already have emails in raw and bronze. Removing them means rewriting those tables without the column, or rebuilding them from a keyed copy. Each rewrite starts its own time travel and fail-safe clock, so the horizon for that one-off removal counts from the rewrite.

Computing the date

With the stores listed, you can compute the completion date for the pipeline's own stores. Listing 4 models each store as a chain of stages whose waits add up. Some chains start when the request is accepted, and others when the source system erases the record. A chain can also wait for a stage in another chain to end, and then starts at whichever comes later.

The horizon is the latest end across the chains. The result names the chain that sets it and that chain's longest adjustable stage, which is usually the place to start. Fixed windows, such as fail-safe, can't be that stage. When chains tie, the one listed first is named, so list them in a deliberate order. Backups get their own list, because each expires on its own schedule and needs re-erasure if it's restored. Records kept by design or obligation are listed with their retention rule and basis, so the date doesn't read as the moment every trace is gone.

Python

from collections.abc import Sequence
from dataclasses import dataclass
from datetime import datetime, timedelta
from enum import Enum

class Start(Enum):
    REQUEST = "request"          # clock starts when the erasure request is accepted
    SOURCE_ERASURE = "source"    # clock starts when the source system erases the record

@dataclass(frozen=True)
class Stage:
    name: str
    wait: timedelta              # worst-case wait before data moves to the next stage
    adjustable: bool = True      # False for fixed windows such as BigQuery fail-safe

@dataclass(frozen=True)
class Chain:
    store: str
    start: Start
    stages: tuple[Stage, ...]    # waits add up along a chain
    after: tuple[tuple[str, str], ...] = ()  # (store, stage): also wait for that stage to end

@dataclass(frozen=True)
class Backup:
    name: str
    retention: timedelta         # until the last backup taken before source erasure is gone

@dataclass(frozen=True)
class Kept:
    name: str
    retention_rule: str
    basis: str                   # by design, legal obligation, or fixed retention

@dataclass(frozen=True)
class Horizon:
    deleted_by: datetime         # live stores: the latest chain end
    store: str                   # the chain that sets the date
    stage: str | None            # that chain's longest adjustable stage (first on ties)
    beyond_use: tuple[tuple[str, datetime], ...]
    kept: tuple[Kept, ...]

def _stage_ends(
    chains: Sequence[Chain], requested: datetime, source_erased: datetime
) -> dict[str, list[datetime]]:
    by_store = {c.store: c for c in chains}
    if len(by_store) != len(chains):
        raise ValueError("store names must be unique")
    ends: dict[str, list[datetime]] = {}
    visiting: set[str] = set()

    def resolve(chain: Chain) -> list[datetime]:
        if chain.store in ends:
            return ends[chain.store]
        if chain.store in visiting:
            raise ValueError(f"dependency cycle at {chain.store}")
        visiting.add(chain.store)
        t = requested if chain.start is Start.REQUEST else source_erased
        for store, stage in chain.after:     # start at the latest of the clock and each dependency
            dep = by_store.get(store)
            names = [s.name for s in dep.stages] if dep else []
            if stage not in names:
                raise ValueError(f"{chain.store} waits for unknown stage {store}: {stage}")
            t = max(t, resolve(dep)[names.index(stage)])
        out = []
        for s in chain.stages:
            if s.wait < timedelta(0):
                raise ValueError(f"negative wait in {chain.store}: {s.name}")
            t += s.wait
            out.append(t)
        visiting.discard(chain.store)
        ends[chain.store] = out
        return out

    for c in chains:
        resolve(c)
    return ends

def horizon(
    requested: datetime,
    source_erased: datetime,
    chains: Sequence[Chain],
    backups: Sequence[Backup] = (),
    kept: Sequence[Kept] = (),
) -> Horizon:
    if requested.utcoffset() is None or source_erased.utcoffset() is None:
        raise ValueError("times must be timezone-aware")
    if not chains or any(not c.stages for c in chains):
        raise ValueError("need at least one chain, each with at least one stage")
    if any(len({s.name for s in c.stages}) != len(c.stages) for c in chains):
        raise ValueError("stage names must be unique within a chain")
    if any(b.retention < timedelta(0) for b in backups):
        raise ValueError("negative backup retention")
    stage_ends = _stage_ends(chains, requested, source_erased)
    ends = [(stage_ends[c.store][-1], c) for c in chains]
    end, chain = max(ends, key=lambda pair: pair[0])  # first chain wins on ties
    lever = None
    for s in chain.stages:       # first maximum wins, as in the Rust version
        if s.adjustable and (lever is None or s.wait > lever.wait):
            lever = s
    beyond_use = tuple((b.name, source_erased + b.retention) for b in backups)
    return Horizon(end, chain.store, lever.name if lever else None, beyond_use, tuple(kept))

Rust

use chrono::{DateTime, TimeDelta, Utc};
use std::collections::{HashMap, HashSet};

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Start {
    Request,       // clock starts when the erasure request is accepted
    SourceErasure, // clock starts when the source system erases the record
}

#[derive(Debug, Clone)]
pub struct Stage {
    pub name: String,
    pub wait: TimeDelta,  // worst-case wait before data moves to the next stage
    pub adjustable: bool, // false for fixed windows such as BigQuery fail-safe
}

#[derive(Debug, Clone)]
pub struct Chain {
    pub store: String,
    pub start: Start,
    pub stages: Vec<Stage>, // waits add up along a chain
    pub after: Vec<(String, String)>, // (store, stage): also wait for that stage to end
}

#[derive(Debug, Clone)]
pub struct Backup {
    pub name: String,
    pub retention: TimeDelta, // until the last backup taken before source erasure is gone
}

#[derive(Debug, Clone)]
pub struct Kept {
    pub name: String,
    pub retention_rule: String,
    pub basis: String, // by design, legal obligation, or fixed retention
}

#[derive(Debug, Clone)]
pub struct Horizon {
    pub deleted_by: DateTime<Utc>, // live stores: the latest chain end
    pub store: String,             // the chain that sets the date
    pub stage: Option<String>,     // that chain's longest adjustable stage (first on ties)
    pub beyond_use: Vec<(String, DateTime<Utc>)>,
    pub kept: Vec<Kept>,
}

type Ends = HashMap<String, Vec<DateTime<Utc>>>;

fn resolve<'a>(
    chain: &'a Chain,
    by_store: &HashMap<&'a str, &'a Chain>,
    clocks: (DateTime<Utc>, DateTime<Utc>),
    ends: &mut Ends,
    visiting: &mut Vec<&'a str>,
) -> Result<Vec<DateTime<Utc>>, String> {
    if let Some(e) = ends.get(&chain.store) {
        return Ok(e.clone());
    }
    if visiting.contains(&chain.store.as_str()) {
        return Err(format!("dependency cycle at {}", chain.store));
    }
    visiting.push(&chain.store);
    let mut t = match chain.start {
        Start::Request => clocks.0,
        Start::SourceErasure => clocks.1,
    };
    // start at the latest of the clock and each dependency
    for (store, stage) in &chain.after {
        let dep = by_store.get(store.as_str());
        let idx = dep.and_then(|d| d.stages.iter().position(|s| &s.name == stage));
        let (Some(dep), Some(idx)) = (dep, idx) else {
            return Err(format!("{} waits for unknown stage {store}: {stage}", chain.store));
        };
        t = t.max(resolve(dep, by_store, clocks, ends, visiting)?[idx]);
    }
    let mut out = Vec::with_capacity(chain.stages.len());
    for stage in &chain.stages {
        if stage.wait < TimeDelta::zero() {
            return Err(format!("negative wait in {}: {}", chain.store, stage.name));
        }
        t += stage.wait;
        out.push(t);
    }
    visiting.pop();
    ends.insert(chain.store.clone(), out.clone());
    Ok(out)
}

/// Taking `DateTime<Utc>` means a time without a zone can't be passed in.
pub fn horizon(
    requested: DateTime<Utc>,
    source_erased: DateTime<Utc>,
    chains: &[Chain],
    backups: &[Backup],
    kept: &[Kept],
) -> Result<Horizon, String> {
    if chains.is_empty() || chains.iter().any(|c| c.stages.is_empty()) {
        return Err("need at least one chain, each with at least one stage".into());
    }
    let unique = |c: &Chain| c.stages.iter().map(|s| &s.name).collect::<HashSet<_>>().len() == c.stages.len();
    if !chains.iter().all(unique) {
        return Err("stage names must be unique within a chain".into());
    }
    if backups.iter().any(|b| b.retention < TimeDelta::zero()) {
        return Err("negative backup retention".into());
    }
    let by_store: HashMap<&str, &Chain> = chains.iter().map(|c| (c.store.as_str(), c)).collect();
    if by_store.len() != chains.len() {
        return Err("store names must be unique".into());
    }
    let (mut ends, mut visiting) = (Ends::new(), Vec::new());
    let mut latest: Option<(DateTime<Utc>, &Chain)> = None;
    for chain in chains {
        let stage_ends = resolve(chain, &by_store, (requested, source_erased), &mut ends, &mut visiting)?;
        let end = *stage_ends.last().expect("stages is not empty");
        match latest {
            Some((t, _)) if end <= t => {}
            _ => latest = Some((end, chain)),
        }
    }
    let (deleted_by, chain) = latest.expect("chains is not empty");
    // first maximum wins, as in the Python version (max_by_key would keep the last)
    let lever = chain
        .stages
        .iter()
        .filter(|s| s.adjustable)
        .fold(None::<&Stage>, |best, s| match best {
            Some(b) if b.wait >= s.wait => Some(b),
            _ => Some(s),
        });
    Ok(Horizon {
        deleted_by,
        store: chain.store.clone(),
        stage: lever.map(|s| s.name.clone()),
        beyond_use: backups
            .iter()
            .map(|b| (b.name.clone(), source_erased + b.retention))
            .collect(),
        kept: kept.to_vec(),
    })
}

Listing 4. Computing the horizon from chains of stages. Waits add along a chain, a chain can wait for a stage in another, and the horizon is the latest chain end.

The table runs the example estate through Listing 4. The erasure job runs daily, split runs time out after six hours, BigQuery's time travel window is at its default, rows take about an hour to land in the raw dataset, and the source system erases the record two days after the request. Microsoft gives no SLA for when changes reach the change tables (Microsoft, 2025b), so the landing lag and the change tables' cleanup lag are measured figures, watched against thresholds. The days count from the request.

Chain	Starts at	Stages	Ends
Bronze events (native)	Request	Job cadence 1 day, split-run timeout 6 hours, time travel 7 days, fail-safe 7 days	Day 15, plus 6 hours
Bronze events (option B variant)	Request	Job cadence 1 day, split-run timeout 6 hours, time travel 7 days, garbage-collection lag (measured), soft delete 7 days (fail-safe left out pending Google)	Day 15, plus 6 hours and the measured garbage-collection lag
Identity table, hand-off table, and retained-record tokens	Request	Job cadence 1 day, split-run timeout 6 hours, downstream confirmation timeout 1 day, time travel 7 days, fail-safe 7 days	Day 16, plus 6 hours
Raw landing (timing only)	Source erasure (day 2)	Landing 1 hour, erasure time fixed on the next job run 1 day, split catch-up 12 hours (a run already under way, then one more)	Day 3, plus 13 hours
Restricted raw dataset	The later of raw landing (day 3, plus 13 hours) and the end of step 5 (day 2, plus 6 hours)	Job cadence 1 day, time travel 7 days, fail-safe 7 days	Day 18, plus 13 hours
Erasure queue table	The same as the raw dataset	The same stages as the raw dataset	Day 18, plus 13 hours
Split staging tables	The end of raw landing (day 3, plus 13 hours)	Table expiry 1 day, time travel 7 days, fail-safe 7 days	Day 18, plus 13 hours
CDC change tables	Source erasure (day 2)	Retention 3 days, cleanup interval 1 hour, cleanup lag (measured)	Day 5, plus 1 hour and the measured cleanup lag
Point-in-time restore backups	Source erasure (day 2)	Restore window 7 days, files up to 7 days more, listed as beyond use	Restorable until day 9, files gone by day 16

Here the raw dataset, the queue table, and the split staging tables tie and set the horizon. The raw dataset is named because it comes first, and time travel is its longest adjustable stage. The staging tables are there because split runs still process the person's erasure rows after the source erasure. With a two-day table expiry, they would set the horizon on their own.

Step 6 can't run until the split has processed the landed erasure rows and step 5 has finished, so the raw and queue chains start at the later of those two points. The raw landing row is there only to time the first. The erasure rows land, the next daily sweep fixes the erasure time, and then the split processes what it was holding back. The split runs continuously, with each run starting as the last ends. The stages are summed as independent upper bounds, so the table overcounts by up to the split catch-up whenever that finishes before the next sweep, as it does here.

A longer downstream confirmation timeout pushes both chains later: with three days, the horizon moves to day 19, plus 6 hours. Listing 4 names a stage only from the chain that sets the date. When a dependency drives the date, as a long confirmation timeout would, that dependency's stage is the better place to start.

Some waits fall outside the table. The hand-off table's date holds only if every downstream job confirms within the timeout. After a timeout, its rows stay until the late job confirms, and the request's horizon is logged as provisional until then. A slow cancellation can stretch even the first round's wait to the margin past the timeout, which adds the margin to the bronze and identity chains. The table takes it as zero. A halted cleanup job holds raw rows the same way a halted split does, so its delay is added to the raw chain. Each extra round of steps 2 and 3 adds a split-run timeout, and its margin if cancellation is slow, to the bronze and identity chains, and to raw when step 5 is its later start. A halted split job holds up step 6 and moves the date for every open request.

Shortening the time travel window to the two-day minimum, where the recovery trade-off allows, moves every native BigQuery chain five days earlier. A shorter window also shortens how far back you can recover from a mistake, which a later part on recovery takes up.

Getting the source system to erase sooner helps only while raw landing is the later start. Here raw landing ends a day and seven hours after step 5, so an earlier source erasure gains up to that much before the confirmation timeout and job cadence become the levers. For the same reason, fixing the erasure time as soon as the rows land, rather than on the next daily run, would take a day off whenever raw landing is the later start, as here.

A soft delete period at the 90-day maximum would make option B set the horizon.

Our tests check the horizon in the worked example, the effect of a longer confirmation timeout, and three properties on randomised chains without dependencies: a longer wait never brings the date forward, removing a chain never pushes it back, and a later source erasure never brings it forward. Both suites also run the same tie case. The Python needs version 3.10 or later, and the Rust uses chrono.

Hardening the design

The sections above lean on access controls at almost every step. This section gathers them in one place. They're written for the example estate, and a later part on access goes further into Entra ID and BigQuery.

The key service and its keys

Only the key service can use the customer-key secret, under three conditions. Its rate limits, and logging without the addresses, are rules in its own code.

No person holds basic roles, Cloud KMS Admin, or IAM admin roles on the key's project or above it as standing access, and any elevation is just-in-time with approval. Basic roles don't separate managing keys from using them, and Cloud KMS Admin can set IAM policy on keys and key rings (Google Cloud, n.d.s), so either lets its holder grant themselves use of the key.
The one automation account that manages the key and deploys the key service works only through changes that pass two-person review. No person can impersonate it, create or upload keys for it, or run its pipeline outside those changes.
No person can impersonate the key service's service account, create keys for it, or deploy code that runs as it. Service account key creation and key upload are blocked by organisation policy, and nobody holds Organization Policy Administrator as standing access.

The same conditions apply to the suppression key, and the third applies to the batch endpoint's service account under the fallback, which the deployment account deploys. The first also applies to every isolated project, every project that holds a job or intake service account, the Airflow project, and the projects holding the deployment and infrastructure accounts, at that level and above. The last applies to every job and intake service account in this post.

The source database and Datastream's login

Creating a capture instance needs db_owner (Microsoft, 2026h), and Google's setup for Azure SQL Database adds Datastream's own login to db_owner and db_denydatawriter, with a password (Google Cloud, n.d.g). That login, and the connection profile holding its password, can create an all-column capture instance or read every column. Members of db_owner can manage fixed database role membership (Microsoft, 2026e), so we'd assume the login can also remove itself from db_denydatawriter.

Google's example enables CDC with no gating role (Google Cloud, n.d.g). Naming one limits which other users can read change data. It doesn't restrict db_owner members, including Datastream's own login. db_owner members (or sysadmin on SQL Server) can read changes outright; anyone else needs column permissions and, where one is set, the gating role (Microsoft, 2025a).

So automation generates the password and stores it in Secret Manager. A SQL Server connection profile can reference a secret in place of a password (Google Cloud, n.d.t), and Google's guide grants the Secret Manager Secret Accessor role to the Datastream service account (Google Cloud, n.d.ae), which we'd grant on that one secret only. The guide doesn't show the reference for Azure SQL Database specifically, so it's one of the checks in the testing section. The secret sits in the raw project. No person knows the password, and no person holds secret-access roles on that secret or its project as standing access. The automation account that sets it uses a SQL principal that can alter Datastream's login. That makes it admin-equivalent on the source, so it works only through reviewed changes, can't be impersonated, and can't read secret versions after writing them.

Alerts on that login come from auditing configured at the Azure resource level, outside the database because, as we understand it, a db_owner member could drop a trigger inside it. The SQL Security Manager role can change server and database audit settings (Microsoft, 2026b), and Owner and Contributor can manage all resources (Microsoft, 2026c). SQL Server Contributor's actions include Microsoft.Sql/servers/*. As we read them, those cover the server's administrators, though the role's excluded actions keep it from changing audit settings (Microsoft, 2026b). As we understand it, the server's admin login and its Microsoft Entra admin can do everything Datastream's login can.

So the key project's first condition applies here too. No person holds the admin login, the Entra admin, any of those Azure roles, User Access Administrator, or Role Based Access Control Administrator (Microsoft, 2026c) as standing access, and elevation is just-in-time. If the Entra admin is a group, it has no standing members or owners. The Entra directory roles that could change that group are a matter for the later part on access. Deletion in the audit destination is restricted in the same way. Restoring a backup is also a server-level action, so the same list, with SQL DB Contributor added, is what keeps backups beyond use. A database restore raises an alert.

The isolation rule

The raw dataset, split staging, the split's marks table, the identity table, the suppression entries, the erasure queue, the reopen and token hand-off tables, the downstream confirmation tables, bronze, the retained store, intake's record of verified requests, the kept list, and every authorised view's dataset follow one isolation rule. The queue, the suppression entries, the hand-off table, intake's record, and the kept list are on that list because each amounts to a list of erased or kept people. The do-not-contact markers sit in the key project, under its conditions. The retained store is there because it holds identifying fields the obligations require.

Creation. Each dataset is created by an infrastructure-as-code service account that sets the access list at creation, so no person picks up a creator's OWNER access. That account can create, change, and grant access to tables, but a custom role leaves it without read or write access to the data, so reaching the data through it needs an access change. Impersonating it and running its pipeline are restricted, and merges to its repository need two-person review.
Projects. Each sits in a project where no principal other than the named job service accounts and the infrastructure account holds BigQuery or basic roles at or above the project. A scanner that reads these datasets, such as Sensitive Data Protection, is named in the access list like any other job.
The raw project. Raw sits in its own project, apart from the identity and suppression datasets, holding only the raw dataset and the Datastream secret. The Datastream API is enabled only there, and the Restrict Resource Service Usage organisation policy constraint, which supports Datastream (Google Cloud, n.d.x), blocks it elsewhere. That matters because each project's Datastream agent gets BigQuery access in its own project. Datastream's permissions for writing to BigQuery come from the Datastream Service Agent role (Google Cloud, n.d.f), and the isolation rule gives that agent no access in the other projects. A role held at project level applies to every dataset in the project. So the agent is the only exception there, and split staging and the remote function's dataset live in the split's own project.
Streams and connection profiles. The connection profile logs in as a db_owner member, so anyone who can create or edit a stream can send every column, the email included, to a dataset of their choosing. Those permissions follow the key project's first condition: no standing access, and just-in-time elevation only.
Deployment. One deployment account deploys the job code. It can run code as the jobs it deploys, so it effectively has their data access, though it holds no table-data permission itself. It's held under the same rules: two-person review, no impersonation, no key creation or upload, and a pipeline that runs only from reviewed merges.
Views. Every view authorised on an isolated dataset, for bronze, the retained store, the identity table, or the export job, has its SQL kept in infrastructure as code with review.
Option B buckets. On option B the data files sit in Cloud Storage, where Storage roles reach them without any BigQuery permission. Google has the table's connection service account hold Storage Object User and Storage Legacy Bucket Reader on the bucket, and recommends restricting write and delete permissions for most users (Google Cloud, n.d.b). We'd go further: only that service account holds object access, and restoring soft-deleted objects is held as tightly. Each Iceberg connection sits in the isolated project of the tables it serves, and permission to create tables through it is held by the infrastructure account alone, so nobody can define a second table over the bucket's files.

Job roles, and what IAM can and can't enforce

The job accounts get custom roles: the table-data permission where they write, read access only where they read, and no permission to create, delete, or reconfigure tables. The roles are granted at dataset level for raw and split staging, whose tables are created at run time, and at table level elsewhere. The split job is the one exception to the table rules, with table creation in its staging dataset. The gold view in Listing 1 is created once by the infrastructure account, and no scheduled job recreates it.

Airflow is an execution path, so it falls under the deployment rule. Each job, the split, silver, any gold tables, the erasure job, silver's erasure job, and the cleanup and retention jobs, runs as Airflow tasks under its own service account, and no task runs with a person's own credentials.

Airflow's environment service account can act as every job account, and impersonates each job's account for that job's tasks. So it's held like the deployment account: no person can impersonate it or create or upload keys for it, and only the deployment account writes to the environment's DAG bucket or updates the environment. It is the only principal that can act as job accounts, and it holds no BigQuery role. The Airflow roles that can trigger DAGs or edit connections are just-in-time, and so are the Google Cloud permissions that reach the same account another way: running Airflow commands through the environment's API, and any access to the cluster that runs its workers.

If you already run your SQL in Dataform or dbt, the same rules apply: one service account per job, no run under a person's own credentials, and only the deployment account changing what runs. In Dataform that means a repository and custom service account per job. Under its strict act-as mode, anyone who creates or updates a repository or workflow configuration, creates a workflow invocation, or updates a release configuration needs permission to act as the effective service account, and the Dataform service agent needs the Service Account User and Token Creator roles on each job account (Google Cloud, n.d.af).

The Dataform service agent then takes the environment account's place as the principal that can act as job accounts, and changes to repositories or configurations are alerted like environment updates. With dbt, we'd give each job its own target and service account and keep the profiles with the deployment account, and the runner's own service account takes that place.

We'd assume, pending a test, that one table-data permission covers inserts, updates, and deletes alike. So "only the split job inserts bronze rows" rests on who holds that permission, plus alerts on which statements each account runs. The table below lists them. The retention, bronze retention, and cleanup jobs need read access, permission to run jobs, and the table-data permission on their tables, because a DELETE runs as a query job and reads the rows it matches.

Request intake's role holds the table-data permission on its own tables, the queue, the suppression entries, and its record of verified requests, and nothing else. With no permission to run jobs anywhere, it can, as we understand it, append through the Storage Write API and can't read, load, copy, or run DML. Those appends go to a stream rather than a query job, so there's no query text or job parameter to hold a key. The Storage Write API's change mode, which can delete, requires a primary key on the destination table (Google Cloud, n.d.ab), so none of intake's tables declares one.

Reads during the time travel window

Reading bronze during the time travel window is also reading rows the erasure job has deleted. As we understand it, a plain query with FOR SYSTEM_TIME AS OF needs only read access and permission to run jobs. Snapshotting a table's past state needs bigquery.tables.createSnapshot among the source permissions, with BigQuery Data Viewer listed among the roles that qualify, plus permission to create tables in a destination dataset (Google Cloud, n.d.h).

So analysts and silver jobs read bronze through authorised views and hold no read access on the tables themselves. Principals with direct table access, including the jobs named in this post, can still read deleted rows until the time travel window ends.

The export view

The export view returns token and customer key only. It compares split runs, the same clock as Listing 2. It leaves out an identity row whose source key has a suppression entry, and one with a matching customer-key entry if the entry has no erasure time yet, or if the row's key was set by a run no later than the one recorded when the erasure time was fixed. It also leaves out any token in the reopen table. A returning customer's later rows are included; marketing consent is checked separately. Because the view reads the suppression entries, it's also authorised on their dataset, and uses them only to exclude. The export job can read the view and nothing else in that dataset.

The remote-function fallback

Calling a remote function needs bigquery.routines.get on its dataset and the BigQuery Connection User role (Google Cloud, n.d.ah), granted to the split job's service account on that one connection. BigQuery calls the endpoint through the connection, in batches, and each request carries documented sessionUser and caller fields (Google Cloud, n.d.ah).

Those fields sit in the request body, so the endpoint can trust them only once it knows the caller. The batch endpoint runs as its own Cloud Run service whose only invoker is the connection's service account, and it forwards each batch to the key service, so it never uses the key itself. It checks that sessionUser is the split job's service account, and caps rows per job with headroom for retries.

The marker key

The suppression key can be rotated without the erased addresses. Cloud KMS rotation leaves earlier versions active (Google Cloud, n.d.n). Automatic rotation is supported only for keys with the ENCRYPT_DECRYPT purpose, and other keys have no primary version (Google Cloud, n.d.u), so this HMAC key is rotated by hand and the key service names the version it signs with. New markers use the new version, and the check computes the HMAC under every enabled version, writing a new-version marker when it matches an old one.

Checks alone won't move every marker, because an address that's never checked keeps its old-version marker. So re-marking runs in bulk, as its own named caller with its own rate limit and a log of each run: every address on the sending system's list goes through the check after a rotation, and again before any list import. The key service can't check that a caller already holds an address it asks about, so each caller's rate limit and logging are what limit it.

Google advises disabling a compromised version and revoking access to it as soon as possible (Google Cloud, n.d.n). Disabling this one would stop checks against existing opt-outs. IAM can't be set on a key version (Google Cloud, n.d.s), so limiting the old version to checks is a rule in the key service's code. On compromise, revoke the misused access first, then run the bulk re-marking straight away.

Markers for addresses on no list can't be moved that way. Email addresses are guessable, so a leaked key plus the marker store would show which of them opted out. The choice is then between keeping the old version enabled for checks, with that exposure, and disabling it and losing those opt-outs, and whichever you choose gets recorded.

Alerted events

Detection backs every rule above that IAM can't enforce on its own. Data Access audit logs are disabled by default for every service except some BigQuery services (Google Cloud, n.d.k), so they're enabled for Secret Manager on the project that holds the Datastream secret.

Event	Where it's seen
IAM policy change on a key, key ring, the Datastream secret, the Iceberg connection, isolated project, folder, or the organisation; on an isolated table; on the option B bucket; on the key service's or batch endpoint's Cloud Run service; or on a job, intake, key-service, batch-endpoint, or deployment service account; any change to organisation policy	Cloud Audit Logs
A Datastream stream or connection profile created or changed	Cloud Audit Logs
Change to a dataset access list, or to an authorised view's definition	Cloud Audit Logs
DML, load, or copy job on an isolated table from a principal outside its named writers; table deletion, settings change, or truncating write	Cloud Audit Logs
A named writer running the wrong statement: an insert into bronze from anyone but the split job, or into raw from anyone but Datastream's service agent; an insert into the retained store from anyone but the split job; a delete from bronze by anyone but the erasure and bronze retention jobs; a delete from the identity table by anyone but the erasure job, or an insert by anyone but the split job; a delete from the retained store by anyone but the retention job, or an update by anyone but the erasure job; an update or delete in raw by anyone but the erasure and cleanup jobs; an insert into the split's marks table by anyone but the split and erasure jobs, or a delete by anyone but the erasure and cleanup jobs; an insert into `ops.committed_split_runs` by anyone but the split job; an update or delete on the suppression entries by anyone but the erasure job; an insert or append into the suppression entries, queue, or intake's record by anyone but their named writers; any update to bronze; an update to the identity table by anyone but the split and erasure jobs	Cloud Audit Logs, for job-based writes; Storage Write API appends depend on one of the checks in the testing section
A primary key added to any of intake's tables	Cloud Audit Logs
An update to the Airflow environment; an Airflow command run through the environment's API; a grant of Service Account User or Token Creator on a job account	Cloud Audit Logs
A write to the Airflow environment's DAG bucket	Cloud Storage Data Access logs, enabled for that bucket
A change to Airflow connections or variables	Airflow's own audit log
A table created in an isolated dataset, or through an Iceberg connection, by anyone but the infrastructure account, the split job in its staging dataset, or Datastream in raw	Cloud Audit Logs
A query with `FOR SYSTEM_TIME AS OF`, or a snapshot, clone, or copy job, reading bronze, raw, the identity table, the retained store, the suppression entries, the queue, or the hand-off table	Cloud Audit Logs
A read of the Datastream password secret by any principal other than Datastream's service agent	Secret Manager Data Access logs
A new version of that secret	Cloud Audit Logs (Admin Activity)
A new capture instance on a captured table, or a change to database role membership	Azure SQL auditing at resource level, with the relevant audit action groups enabled
A change to audit settings, the admin password, or the Entra admin, or a new database on the server, which is how a restore appears	Azure Activity Log
A change to the Entra admin group's members or owners	Microsoft Entra audit logs
A password change on Datastream's login	Azure SQL auditing at resource level, with the relevant audit action groups enabled
An email-like column in raw, a halted split, a hold past its threshold, landing lag past its threshold, or a downstream confirmation timeout	Pipeline monitoring

What a data engineer still has to test

Google's and Microsoft's documentation, and an AI agent working from it, can take you a long way with this design. What neither can determine is how the services behave where the documentation is unclear or covers a point only in part, and whether this design's own scripts behave as intended. That part is the data engineer's job: run each check below against real services in a non-production project before anyone relies on the design. The checks fall into five groups.

How Datastream behaves on this source. Confirm that the stream's included-objects column list governs backfills as well as CDC, and that DELETE rows carry the deleted row's values. Check that the source timestamp on SQL Server change rows is the transaction's commit time, and find which Datastream field orders changes that share a timestamp (carried into bronze as change_seq in Listing 2). See how backfilled rows can be told apart in BigQuery and how their timestamps are set. Confirm that Datastream reads change tables independently, so one table landing doesn't prove another has, and that rows it writes over gRPC can be changed by DML straight away, which depends on it using that path. Check that partition expiration behaves as set on the tables it creates.
Whether the access boundaries hold. Test that an Azure SQL Database connection profile accepts a Secret Manager password reference, with Secret Accessor granted to the Datastream service account on that secret. Point a stream at the identity or suppression projects and expect it to fail, and expect enabling the Datastream API outside the raw project to be refused. Confirm that Datastream's login can remove itself from db_denydatawriter. Check whether one table-data permission covers insert, update, and delete in BigQuery. Check that intake's role allows nothing beyond Storage Write API appends, and what audit logs record for those appends on every isolated table. Confirm that a plain FOR SYSTEM_TIME AS OF query needs nothing beyond read access and permission to run jobs, and that time travel can't be applied through an authorised view.
Whether the jobs run the way the design assumes. Check that Airflow's Google provider runs each script as a BigQuery job under the job account it impersonates, with the run number passed as a query parameter, and find the narrowest grant on the environment service account that allows it. Confirm that INFORMATION_SCHEMA.JOBS shows a running split script, by the account Airflow impersonates, within a bounded delay. A job timeout on a BigQuery script should end the whole script, cancelling a timed-out Airflow task should stop its script, and the view should show it done before Airflow's retry starts. Measure how often the erasure job's bronze transaction cancels a concurrent split run's transaction, which shows up as a failed split run and a retry. Run Listings 1 to 3, with their variables substituted, as a BigQuery dry run against real tables, and confirm they pass.
What happens to deleted data afterwards. Confirm that an expired table snapshot goes through time travel and fail-safe like any table. Fail-safe for option B tables is a question for Google.
The split's rules and its race with an erasure. Test a child arriving before its parent, a customer key arriving after its row, and the DELETE row after source erasure. Then test the races: a long split run started before step 1's record-key insert, one that creates a new identity row, a suppression append that lands after the queue row, an erasure time that would be fixed before step 1, and a key rotation requested while an erasure is open.

Proving it, and the clocks that apply

The EDPB's report recommends that controllers map personal data and its storage locations. It also describes, as good practice, systems that generate records automatically as proof of deletion (European Data Protection Board, 2026). The ledger gives the map, and the erasure log, checked against the horizon, gives the evidence. Since BigQuery doesn't record the automatic deletion of expired partitions in its audit logs (Google Cloud, n.d.p), the erasure job and the raw cleanup job write their own records.

Several clocks apply, and they measure different things. Article 17(1) requires erasure "without undue delay", and Article 12(3) separately requires the controller to tell the person what action was taken within one month of the request, extendable by two further months where necessary (European Parliament and Council, 2016). Under section 1798.130(a)(2)(A), the CCPA gives businesses 45 days to respond to a verifiable request, extendable once by a further 45 days when reasonably necessary, if the consumer is told within the first 45 days (California Legislature, n.d.c). CAN-SPAM's 10 business days apply to email opt-outs (United States Congress, n.d.). Other US state privacy laws set their own deadlines, which this post doesn't cover.

The computed horizon is about when recoverable copies are gone, so we'd report it alongside those clocks. In this part it covers the pipeline's own stores only, which makes it a lower bound for the platform. We wouldn't give it to the person as a completion date until the silver, gold, and other downstream chains are added, which later parts do.

Backups are handled differently. The UK ICO's guidance, which the ICO notes is under review, asks that backup data be put "beyond use" even if it can't be overwritten immediately (Information Commissioner's Office, n.d.). The California regulations let a business delay deletion from archived or backup systems until that system is restored to an active system or next accessed or used for a sale, disclosure, or commercial purpose (California Privacy Protection Agency, n.d.b).

For backups, the EDPB report recommends that controllers verify and be able to demonstrate that erasure has been carried out. It found that many controllers had no specific procedure for backups, and that some relied on automatic deletion measures or on retention periods applied to backups (European Data Protection Board, 2026). A beyond-use list with a re-erasure rule on restore gives that procedure a concrete form, and the response to the person can say that backups are kept beyond use until they expire.

A checklist for the next request

For a token-based design like this one, these are worth having in place before the next erasure request arrives:

Event families sorted into erased and retained, with the retained ones' basis written down.
The email kept out of capture and out of backfills, with the capture instance created before the stream.
Raw rows deleted by the erasure job, a cleanup job for the rest, and expiration as a backstop.
The restricted stores under one isolation rule, with per-entry retention for suppression, and do-not-contact markers under their own key.
The erasure job's order: suppress at the request; then delete, wait, look up again until nothing new, hand tokens downstream, delete the mapping once downstream jobs confirm, and clear raw once the erasure time is fixed and processed.
A ledger row for every store, each with an owner, and a named request-intake identity that alone records verified requests, with the erasure job acting only on those.
A computed horizon and a beyond-use list for every request, recorded in the erasure log.
The data engineer's checks run in a test project.

A useful first exercise is to take one of last quarter's erasure requests, run it through the ledger, and check that every row has an owner.

Our Data & AI practice is glad to go through a ledger like this with your platform team.

This post describes engineering approaches. It isn't legal advice, and which obligations apply to a particular business is a question for its counsel.

References

Apache Iceberg, n.d. Configuration. Apache Iceberg documentation. Available at: https://iceberg.apache.org/docs/latest/configuration/ [Accessed 9 October 2026].

California Legislature, n.d.a. Civil Code section 1798.100. California Legislative Information. Available at: https://leginfo.legislature.ca.gov/faces/codes_displaySection.xhtml?lawCode=CIV&sectionNum=1798.100 [Accessed 9 October 2026].

California Legislature, n.d.b. Civil Code section 1798.105: Consumers' Right to Delete Personal Information. Text as published by Justia, 2025 California Code. Available at: https://law.justia.com/codes/california/code-civ/division-3/part-4/title-1-81-5/section-1798-105/ [Accessed 9 October 2026].

California Legislature, n.d.c. Civil Code section 1798.130: Notice, Disclosure, Correction, and Deletion Requirements. Text as published by Justia, 2025 California Code. Available at: https://law.justia.com/codes/california/code-civ/division-3/part-4/title-1-81-5/section-1798-130/ [Accessed 9 October 2026].

California Legislature, n.d.d. Civil Code section 1798.140. California Legislative Information. Available at: https://leginfo.legislature.ca.gov/faces/codes_displaySection.xhtml?lawCode=CIV&sectionNum=1798.140 [Accessed 9 October 2026].

California Privacy Protection Agency, n.d.a. Cal. Code Regs. tit. 11, § 7012: Notice at Collection of Personal Information. Text as published by the Legal Information Institute, Cornell Law School. Available at: https://www.law.cornell.edu/regulations/california/11-CCR-7012 [Accessed 9 October 2026].

California Privacy Protection Agency, n.d.b. Cal. Code Regs. tit. 11, § 7022: Requests to Delete. Text as published by the Legal Information Institute, Cornell Law School. Available at: https://www.law.cornell.edu/regulations/california/11-CCR-7022 [Accessed 9 October 2026].

European Data Protection Board, 2026. 2025 Coordinated Enforcement Action: Implementation of the right to erasure by controllers. Adopted 10 February 2026. Available at: https://www.edpb.europa.eu/system/files/2026-02/edpb_cef-report_2025_right-to-erasure_en.pdf [Accessed 9 October 2026].

European Parliament and Council, 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation). Official Journal of the European Union, L 119, 4 May, pp. 1-88. Available at: https://eur-lex.europa.eu/eli/reg/2016/679/oj [Accessed 9 October 2026].

Google Cloud, n.d.a. AEAD encryption concepts. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/aead-encryption-concepts [Accessed 9 October 2026].

Google Cloud, n.d.b. Apache Iceberg managed tables. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/biglake-iceberg-tables-in-bigquery [Accessed 9 October 2026].

Google Cloud, n.d.c. Basic roles and permissions. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/access-control-basic-roles [Accessed 9 October 2026].

Google Cloud, n.d.d. Bucket Lock. Cloud Storage documentation. Available at: https://docs.cloud.google.com/storage/docs/bucket-lock [Accessed 9 October 2026].

Google Cloud, n.d.e. Column-level encryption with Cloud KMS. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/column-key-encrypt [Accessed 9 October 2026].

Google Cloud, n.d.f. Configure a BigQuery destination. Datastream documentation. Available at: https://docs.cloud.google.com/datastream/docs/configure-bigquery-destination [Accessed 9 October 2026].

Google Cloud, n.d.g. Configure an Azure SQL database for CDC. Datastream documentation. Available at: https://docs.cloud.google.com/datastream/docs/configure-azure-sqlserver [Accessed 9 October 2026].

Google Cloud, n.d.h. Create table snapshots. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/table-snapshots-create [Accessed 9 October 2026].

Google Cloud, n.d.i. Data retention with time travel and fail-safe. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/time-travel [Accessed 9 October 2026].

Google Cloud, n.d.j. Destroy and restore key versions. Cloud Key Management Service documentation. Available at: https://docs.cloud.google.com/kms/docs/destroy-restore [Accessed 9 October 2026].

Google Cloud, n.d.k. Enable Data Access audit logs. Cloud Logging documentation. Available at: https://docs.cloud.google.com/logging/docs/audit/configure-data-access [Accessed 9 October 2026].

Google Cloud, n.d.l. Events and streams. Datastream documentation. Available at: https://docs.cloud.google.com/datastream/docs/events-and-streams [Accessed 9 October 2026].

Google Cloud, n.d.m. JOBS view. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/information-schema-jobs [Accessed 9 October 2026].

Google Cloud, n.d.n. Key rotation. Cloud Key Management Service documentation. Available at: https://docs.cloud.google.com/kms/docs/key-rotation [Accessed 9 October 2026].

Google Cloud, n.d.o. Manage backfill for the objects of a stream. Datastream documentation. Available at: https://docs.cloud.google.com/datastream/docs/manage-backfill-for-the-objects-of-a-stream [Accessed 9 October 2026].

Google Cloud, n.d.p. Manage partitioned tables. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/managing-partitioned-tables [Accessed 9 October 2026].

Google Cloud, n.d.q. Multi-statement transactions. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/transactions [Accessed 9 October 2026].

Google Cloud, n.d.r. Partition and cluster BigQuery tables. Datastream documentation. Available at: https://docs.cloud.google.com/datastream/docs/partitioning-and-clustering [Accessed 9 October 2026].

Google Cloud, n.d.s. Permissions and roles. Cloud Key Management Service documentation. Available at: https://docs.cloud.google.com/kms/docs/reference/permissions-and-roles [Accessed 9 October 2026].

Google Cloud, n.d.t. REST Resource: projects.locations.connectionProfiles. Datastream API reference. Available at: https://docs.cloud.google.com/datastream/docs/reference/rest/v1/projects.locations.connectionProfiles [Accessed 9 October 2026].

Google Cloud, n.d.u. REST Resource: projects.locations.keyRings.cryptoKeys. Cloud Key Management Service API reference. Available at: https://docs.cloud.google.com/kms/docs/reference/rest/v1/projects.locations.keyRings.cryptoKeys [Accessed 9 October 2026].

Google Cloud, n.d.v. REST Resource: projects.locations.streams. Datastream API reference. Available at: https://docs.cloud.google.com/datastream/docs/reference/rest/v1/projects.locations.streams [Accessed 9 October 2026].

Google Cloud, n.d.w. Run parameterized queries. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/parameterized-queries [Accessed 9 October 2026].

Google Cloud, n.d.x. Services that support restricting service usage. Organization Policy documentation. Available at: https://docs.cloud.google.com/resource-manager/docs/organization-policy/restricting-resources-supported-services [Accessed 9 October 2026].

Google Cloud, n.d.y. Set up the Apache Iceberg REST catalog endpoint. Lakehouse documentation. Available at: https://docs.cloud.google.com/lakehouse/docs/set-up-lakehouse-iceberg-rest-catalog [Accessed 9 October 2026].

Google Cloud, n.d.z. Soft delete overview. Cloud Storage documentation. Available at: https://docs.cloud.google.com/storage/docs/soft-delete [Accessed 9 October 2026].

Google Cloud, n.d.aa. Stream data from SQL Server databases. Datastream documentation. Available at: https://docs.cloud.google.com/datastream/docs/sources-sqlserver [Accessed 9 October 2026].

Google Cloud, n.d.ab. Stream table updates with change data capture ingestion. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/change-data-capture [Accessed 9 October 2026].

Google Cloud, n.d.ac. Transform data with data manipulation language (DML). BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/data-manipulation-language [Accessed 9 October 2026].

Google Cloud, n.d.ad. Update dataset properties. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/updating-datasets [Accessed 9 October 2026].

Google Cloud, n.d.ae. Use Secret Manager to store sensitive data. Datastream documentation. Available at: https://docs.cloud.google.com/datastream/docs/use-secret-manager [Accessed 9 October 2026].

Google Cloud, n.d.af. Use strict act-as mode. Dataform documentation. Available at: https://docs.cloud.google.com/dataform/docs/strict-act-as-mode [Accessed 9 October 2026].

Google Cloud, n.d.ag. What is medallion architecture? Google Cloud. Available at: https://cloud.google.com/discover/what-is-medallion-architecture [Accessed 9 October 2026].

Google Cloud, n.d.ah. Work with remote functions. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/remote-functions [Accessed 9 October 2026].

Information Commissioner's Office, 2026. Respect people's preferences. Direct marketing guidance. Available at: https://ico.org.uk/for-organisations/direct-marketing-and-privacy-and-electronic-communications/direct-marketing-guidance/respect-peoples-preferences/ [Accessed 9 October 2026].

Information Commissioner's Office, n.d. Right to erasure. UK GDPR guidance and resources. Available at: https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/individual-rights/individual-rights/right-to-erasure/ [Accessed 9 October 2026].

Microsoft, 2025a. cdc.fn_cdc_get_all_changes_ (Transact-SQL). Microsoft Learn. Available at: https://learn.microsoft.com/en-us/sql/relational-databases/system-functions/cdc-fn-cdc-get-all-changes-capture-instance-transact-sql [Accessed 9 October 2026].

Microsoft, 2025b. Change data capture (CDC) with Azure SQL Database. Microsoft Learn. Available at: https://learn.microsoft.com/en-us/azure/azure-sql/database/change-data-capture-overview?view=azuresql [Accessed 9 October 2026].

Microsoft, 2026a. Automated backups in Azure SQL Database. Microsoft Learn. Available at: https://learn.microsoft.com/en-us/azure/azure-sql/database/automated-backups-overview?view=azuresql [Accessed 9 October 2026].

Microsoft, 2026b. Azure built-in roles for Databases. Microsoft Learn. Available at: https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles/databases [Accessed 9 October 2026].

Microsoft, 2026c. Azure built-in roles for Privileged. Microsoft Learn. Available at: https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles/privileged [Accessed 9 October 2026].

Microsoft, 2026d. Backup immutability for long-term retention backups in Azure SQL Database. Microsoft Learn. Available at: https://learn.microsoft.com/en-us/azure/azure-sql/database/backup-immutability [Accessed 9 October 2026].

Microsoft, 2026e. Database-level roles. Microsoft Learn. Available at: https://learn.microsoft.com/en-us/sql/relational-databases/security/authentication-access/database-level-roles [Accessed 9 October 2026].

Microsoft, 2026f. Long-term retention backups: Azure SQL Database and Azure SQL Managed Instance. Microsoft Learn. Available at: https://learn.microsoft.com/en-us/azure/azure-sql/database/long-term-retention-overview?view=azuresql [Accessed 9 October 2026].

Microsoft, 2026g. sys.sp_cdc_disable_table (Transact-SQL). Microsoft Learn. Available at: https://learn.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/sys-sp-cdc-disable-table-transact-sql?view=sql-server-ver17 [Accessed 9 October 2026].

Microsoft, 2026h. sys.sp_cdc_enable_table (Transact-SQL). Microsoft Learn. Available at: https://learn.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/sys-sp-cdc-enable-table-transact-sql?view=sql-server-ver17 [Accessed 9 October 2026].

National Institute of Standards and Technology, 2025. Guidelines for Media Sanitization (NIST SP 800-88 Rev. 2). NIST Special Publication 800-88 Revision 2, September 2025. Available at: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-88r2.pdf [Accessed 9 October 2026].

United States Congress, n.d. 15 U.S. Code § 7704: Other protections for users of commercial electronic mail. Text as published by the Legal Information Institute, Cornell Law School. Available at: https://www.law.cornell.edu/uscode/text/15/7704 [Accessed 9 October 2026].

Sakura Sky and UpCloud: A European Cloud Partnership

Andrew Stevens — Mon, 12 Oct 2026 14:53:24 +0000

If your team builds for customers in the UK and Europe, where your platform runs is increasingly part of what you sell. Your customers ask where their data lives, who can reach it, and what it will cost to grow. I am delighted to say that Sakura Sky is now an UpCloud partner, which means we can answer those questions with a European provider and a single contract.

For a long time, when a customer asked us for a European option, the answer was a European region of a hyperscaler. Those regions work well, and we will keep building on them. What I have wanted is a European provider we could recommend with the same conviction, with flat pricing and in-house engineers behind its support. UpCloud is that provider for us. From today we resell UpCloud, and our teams are ready to design, build, and run platforms on it for organisations across the UK and the Benelux.

Why UpCloud

UpCloud was formed in 2011 and is headquartered in Helsinki (UpCloud, 2026a). UpCloud's legal team states that all of its European data centres are operated by the Finnish parent company, UpCloud Oy, under the relevant Finnish and EU member state laws, and that the group's US subsidiary has no control over customer account information or over data held by the parent outside the US, and so cannot be compelled to disclose it under the US CLOUD Act (UpCloud Legal Team, 2026). Each organisation will want its own advice, but for many teams that makes UpCloud a practical step towards sovereign cloud.

UpCloud offers a 99.999% uptime SLA, holds ISO 27001 certification, and describes its infrastructure as GDPR-compliant. Support is 24/7 and run by UpCloud's own people, and UpCloud quotes under 40 seconds to reach a human expert (UpCloud, 2026b). It runs fifteen data centres, ten of them in Europe, including London and Amsterdam (UpCloud, 2026c), which puts infrastructure close to the customers we work with most.

Here is what UpCloud offers that we think matters most, and what we will be testing as we build:

Pricing you can plan around. Flat-rate pricing (UpCloud, 2026b), with outbound transfer included under UpCloud's Fair Transfer Policy: the allowance scales with your cloud spend. There are no excess fees by default; if you go over, UpCloud may limit bandwidth for the rest of the month, or you can opt in to metered transfer above the allowance (UpCloud, 2026d). That lets data-heavy products treat egress as a planned line in the budget.
Fast storage and fast starts. Premium plans include UpCloud's MaxIOPS block storage by default, which UpCloud lists at 100k IOPS (UpCloud, 2026j), and UpCloud says new servers are running in under 45 seconds (UpCloud, 2026b). We will test both in our open benchmarks.
Managed services where you need them. Managed Kubernetes is available in every UpCloud location, with a free development tier and a highly available production tier (UpCloud, 2026e; UpCloud, 2026f). Managed Databases for PostgreSQL run in every location (UpCloud, 2026g), with the pgvector extension available for retrieval workloads (Piwosz, 2026), and UpCloud describes Managed Object Storage as fully compatible with the S3 API (UpCloud, 2026h).
GPUs for open models. GPU Servers with NVIDIA L4, L40S, RTX PRO 6000, H100, and B200 cards, currently offered from Helsinki (UpCloud, 2026i), give us a European home for running open-weight models on dedicated GPUs.
Built to stay portable. UpCloud describes its pricing as open and reversible (UpCloud, 2026b), which matches how we build: infrastructure as code on open interfaces, so you can move if your needs change.

What this brings our customers

Sakura Sky is headquartered in Amsterdam, so this is a European partnership on both sides. Where we resell UpCloud, the platform and our engineering come under one contract and one monthly bill. You work with a team that knows your platform well, with UpCloud's own support team behind the infrastructure. On UpCloud, compute and managed databases can run in London or Amsterdam (UpCloud, 2026c), and object storage runs in Helsinki, Stockholm, or Frankfurt (UpCloud, 2026h), so you can tell your customers or your regulator where each part of your data lives.

What we are launching

Alongside the partnership, we are introducing four services built on UpCloud:

European Landing Zone. An UpCloud foundation delivered as code, covering account and sub-account structure, networking, access control, security baselines, and observability, so your first production workloads go live on a foundation that is already reviewed and documented.
Managed European Platform. Your platform on UpCloud, run by us: Kubernetes, managed databases, object storage, monitoring, patching, and cost management, with the same operating discipline our Managed Services customers already rely on.
Managed Open Models. Open-weight models chosen for your workload and served on dedicated GPU capacity in UpCloud's Helsinki data centre (UpCloud, 2026i), with the retrieval layer, gateway, spend controls, and upgrades run by us. By default, inference runs on GPU capacity in Finland, inside your environment, and prompts are not sent to a third-party model API.
Exit Readiness Assessment. A practical review of how portable your workloads are across providers, wherever you run today, with a plan to make future moves simpler and cheaper.

Our Professional Services teams can also work alongside your engineers on migrations, architecture, and platform engineering on UpCloud.

Where UpCloud works well

If your cloud bill is hard to forecast, data residency and third-party risk keep coming up in your GDPR, NIS2, or DORA work, or your team spends more time running Kubernetes than building on it, UpCloud is worth a look. These are the workloads we expect to bring to it most often:

Data and AI. GPU servers in Helsinki and fast storage for training and inference, in a European location you can name.
Backup and disaster recovery. Off-site copies and a standby environment in another European country, on compute that is affordable to keep warm.
Regulated industries and the public sector. Fintech, healthtech, and public sector services that need European jurisdiction and an environment they can audit.
A second cloud alongside a hyperscaler. A European home for the workloads that have to stay in-region, next to an existing AWS, Google Cloud, or Azure estate.
Moving off VMware or on-premises. Lift-and-shift where that is the right first step, or modernisation onto Kubernetes and managed services where the business case supports it.

Sharing what we learn

We will be publishing a good deal of engineering work on UpCloud over the coming months: build-along guides, a series on running open models in production, field notes from migrations, and cloud benchmarks with an open methodology and harness that anyone can rerun. We will publish the results and the raw data together, so you can check our working.

Come and meet us

We are starting user groups in the Netherlands and the UK for people running, or thinking about running, workloads on UpCloud. Both groups will meet through Meetup, and we will announce the first sessions here on the blog. We are also planning monthly online office hours, where you can bring an architecture or a migration plan and talk it through with our engineers.

My thanks to the UpCloud team for the work they have put into getting us here. If you are weighing up European cloud for your next platform, or wondering whether it suits what you run today, I would be glad to talk.

Disclosure: Sakura Sky is an UpCloud reseller and partner. Our engineering series and benchmarks are written independently: UpCloud does not pay for or direct them, and does not review or approve them before publication. Managed Open Models and the other services named here are delivered by Sakura Sky.

References

Piwosz, P., 2026. Installing pgvector on UpCloud Managed PostgreSQL. UpCloud. Available at: https://upcloud.com/resources/tutorials/installing-pgvector-upcloud-managed-postgresql/ [Accessed 12 October 2026].

UpCloud, 2026a. About UpCloud – European Cloud Provider. Available at: https://upcloud.com/about/ [Accessed 12 October 2026].

UpCloud, 2026b. Global Cloud Platform. Available at: https://upcloud.com/ [Accessed 12 October 2026].

UpCloud, 2026c. UpCloud Data Center Information. Available at: https://upcloud.com/data-centres/ [Accessed 12 October 2026].

UpCloud, 2026d. Fair Transfer Policy: Zero-cost Egress. Available at: https://upcloud.com/fair-transfer-policy [Accessed 12 October 2026].

UpCloud, 2026e. Managed Kubernetes availability. UpCloud Documentation. Available at: https://upcloud.com/docs/products/managed-kubernetes/availability/ [Accessed 12 October 2026].

UpCloud, 2026f. UpCloud Managed Kubernetes Platform. Available at: https://upcloud.com/products/managed-kubernetes/ [Accessed 12 October 2026].

UpCloud, 2026g. Managed PostgreSQL availability. UpCloud Documentation. Available at: https://upcloud.com/docs/products/managed-postgresql/availability/ [Accessed 12 October 2026].

UpCloud, 2026h. The Perfect Home For Your Data: Managed Object Storage. Available at: https://upcloud.com/products/object-storage/ [Accessed 12 October 2026].

UpCloud, 2026i. GPU Servers availability. UpCloud Documentation. Available at: https://upcloud.com/docs/products/gpu-servers/availability/ [Accessed 12 October 2026].

UpCloud, 2026j. Premium Cloud Servers. Available at: https://upcloud.com/solutions/premium-cloud-servers/ [Accessed 12 October 2026].

UpCloud Legal Team, 2026. U.S. CLOUD Act and Why Corporate Structure Matters for Data Sovereignty. UpCloud, 22 January. Available at: https://upcloud.com/blog/us-cloud-act-corporate-structure-matters-data-sovereignty/ [Accessed 12 October 2026].

Multi-Cloud Retail Without the Mess

Andrew Stevens — Fri, 09 Oct 2026 11:04:52 +0000

Many multi-brand QSR and retail estates span clouds in a recognisable way. Staff sign in with Microsoft Entra ID, some back-office systems run on SQL Server in Azure, the data platform is on Google Cloud, and a loyalty vendor or an acquired brand brings AWS. If your estate looks like that, it probably isn't anyone's plan. Each piece was a sensible decision when it was made. The cost shows up later, in the connections between them: service account keys copied into another cloud's secret store, the same customer held under three different IDs, and nightly copies of the same tables in two places.

Our earlier post Multi-Cloud Versus Consolidation made the cross-industry case that most organisations are multi-cloud whether or not they chose to be, and that the work is to make the estate coherent. This post applies that to retail and QSR. It looks at how these estates come to span three clouds, where the extra engineering falls, and the identity and data layer that holds them together, with a diagram of the layer, the token exchange behind it, and code for the customer key at its centre.

Joining one customer across three clouds

Take an estate where the catering and gift card systems run on SQL Server in Azure, and a back-office sync job moves their changes into BigQuery on Google Cloud, which is the data platform for sales, operations, and marketing. The loyalty programme runs on a SaaS platform, with an integration in the retailer's AWS account that syncs member records, and a history export that the vendor writes to an S3 bucket in the same account. Delivery orders arrive as webhooks from each aggregator at an integration service on Google Cloud, which injects them into the POS and forwards them to the data platform. In our experience, what aggregators pass on about the customer varies by aggregator and by market, and some orders carry no usable email at all, while others carry a masked relay address that won't match anything else. Marketing wants to send a customer an offer, so the data platform needs to know that a catering order, a loyalty member, and a delivery order with the same email address belong to the same person, and to treat delivery orders without one as unmatched, flagging relay-address keys so they don't inflate customer counts.

Three teams own the three parts of that question, and each has solved its own part well. The back office stores the email typed in at checkout. The loyalty platform has a member ID. The delivery integration stores the aggregator's customer email, when the order carries one. Joining them means moving email addresses, or full customer tables, between clouds, and each team has a reasonable fix that makes the overall picture a little harder to manage.

How estates end up on three clouds

Most of the clouds in an estate like this arrive with a product or a business decision. Microsoft 365 often brings Entra ID with it, and Entra ID becomes the place staff identities live. Back-office applications built on SQL Server often move to Azure, a short step from the data centre they ran in. A data team picks Google Cloud for the data platform on technical grounds. A SaaS loyalty platform runs on whichever cloud its vendor chose, and its exports tend to land nearby. An acquired brand brings its own cloud accounts, its own POS integration, and a team that knows them.

Franchise networks add another layer, because franchisees may run systems the franchisor doesn't control and send data in whatever form the franchise agreement provides for.

Most of the cost comes from the default way of connecting the pieces. A workload on one cloud that needs something on another usually gets a long-lived credential, such as a service account key or an access key, stored in its own cloud's secret manager. Each team identifies customers its own way, so matching them means exporting email addresses or customer tables to wherever the match runs. And data needed in two clouds tends to get copied on a schedule, so there are two copies to secure, two to keep current, and a transfer bill each time it moves.

Sometimes the right answer is consolidation. Sakura Sky moved Craveable Brands' data platform to Google Cloud, bringing data from across its brands and outlets into one BigQuery warehouse. Where consolidation doesn't pay, because the loyalty vendor's cloud isn't yours to choose or the SQL Server estate works well where it is, the job is to make the connections between clouds deliberate.

Where the cross-cloud work falls

For a retail estate, most of that work falls in three places.

The first is machine identity. Services in one cloud need to call services and read data in another without anyone storing a long-lived key for the other cloud. AWS, Google Cloud, and Microsoft Entra ID all document ways to exchange one platform's credentials for another's short-lived tokens (AWS, n.d.a; Google Cloud, n.d.a; Microsoft, 2025), so the stored key can usually be retired.

The second is customer identity. Every system that holds customer records needs to agree on who a customer is, and it helps if they can do that without each one holding the others' raw contact details. That usually means one pseudonymous customer key, computed by one service.

The third is data placement. Each dataset of record lives in one place, with one owner, and other clouds read it where it is. Copies are made for a reason, such as latency, data residency, or resilience, and each copy is recorded with its source and refresh schedule.

Two smaller disciplines sit alongside them. Events crossing clouds, such as aggregator orders, use one envelope format, so each consumer doesn't write its own parser for each source. And consent travels with the customer key: an opt-out recorded in the loyalty platform has to reach the data platform before the next campaign runs, which Part 1 covered from the loyalty side.

The identity and data layer

Figure 1 shows one way the pieces fit together for the estate above, with Google Cloud as the data platform and the home of the customer key service. In your estate the clouds may play different roles, with AWS as the data platform, for example, or the loyalty platform on Azure, and the same pattern should carry over, though the services and their limits differ.

Machine identity with fewer stored keys

Google Cloud's Workload Identity Federation lets workloads on Azure and AWS exchange their own environment's credentials for short-lived Google Cloud credentials, so they can call Google Cloud APIs without a service account key (Google Cloud, n.d.a). Figure 2 shows the Azure case. The back-office sync job runs on an Azure VM with a managed identity. It asks the Azure Instance Metadata Service for an access token for a Microsoft Entra ID application set up for federation, identified by its Application ID URI. In Google Cloud, a provider in a workload identity pool takes the tenant's issuer as its issuer URL and that Application ID URI as its allowed audience. Google's example issuer is https://sts.windows.net/<tenant-id>, but the value can vary, so it's copied from the iss claim of a real token. The token's subject is the managed identity's object ID, which the provider maps to the federated identity's subject, so IAM roles can be granted to that one identity, and an attribute condition on the provider can reject tokens from any other identity in the tenant (Google Cloud, n.d.a). Google Cloud's Security Token Service then exchanges the Entra token for a short-lived Google Cloud access token, referred to here as a federated token. BigQuery's API has no known limitations with federated identities (Google Cloud, n.d.g), so the sync job can be granted access to its datasets directly.

The AWS side of Figure 1 works the same way with different credentials. One of the options Google documents for AWS uses the workload's ordinary temporary AWS credentials, which Google Cloud verifies through AWS's GetCallerIdentity API with no configuration changes in the AWS account. The other uses AWS outbound identity federation, where AWS acts as the identity provider and its workloads request short-lived JSON Web Tokens to present to external services such as Google Cloud (AWS, n.d.b; Google Cloud, n.d.a). Google recommends granting the federated identity access to resources directly, and falling back to service account impersonation only for products that don't support that (Google Cloud, n.d.a).

Calls in the other direction follow the same pattern. Entra ID's workload identity federation lets an app registration or user-assigned managed identity trust tokens from an external provider, with Google Cloud and AWS among the documented scenarios (Microsoft, 2025). AWS's AssumeRoleWithWebIdentity returns temporary AWS credentials, by default for one hour, in exchange for a token from an OpenID Connect provider such as Google, provided the role's trust policy names that provider (AWS, n.d.a). We'd also restrict the subject and the audience in the trust policy, because naming only the provider trusts every identity it issues tokens for. For Google tokens there's a trap: the accounts.google.com:aud condition key takes the token's azp claim when that claim is set, and the token's aud is then checked with accounts.google.com:oaud instead (AWS, n.d.c).

What changes in practice is where trust is configured. Instead of a key in a secret store, each cloud holds a trust relationship naming the other cloud's identity provider, plus conditions on which workloads it accepts. Those relationships need the same care a key did. Revoking access happens on the receiving side, because a token that's already been issued typically stays valid until it expires. In Google Cloud, that means removing the role granted on the resource itself, such as access to the BigQuery dataset or the Cloud Run invoker role held by the impersonated service account, because IAM checks it on each call. Disabling the provider stops new token exchanges, but tokens already issued run until they expire. In AWS, revoking the role's active sessions denies credentials issued before that point (AWS, n.d.d). A provider that accepts any token from an Entra tenant, or any identity in an AWS account, trusts far more workloads than the ones that need access, so its attribute condition names only the specific managed identities or AWS roles that do.

For people, the Entra ID that head-office staff already use can also sign analysts and engineers in to Google Cloud through Workforce Identity Federation, without copying their accounts into Google (Google Cloud, n.d.d), though not every Google Cloud product supports federated identities in the same way (Google Cloud, n.d.g).

One customer key, computed by one service

The customer key in Figure 1 is a pseudonymous join key: a value derived from a customer's normalised email address that every system can store and join on in place of the address itself. Listing 1 computes it with HMAC-SHA256 and a secret key. A plain hash such as SHA-256 of the email isn't enough, because anyone with a list of email addresses can hash them and compare. With HMAC, customer keys can only be produced by whoever holds the secret key.

That's also why the key service runs in one place. If the secret key were copied into every cloud so each could compute customer keys locally, it would become exactly the kind of shared secret federation removes. Instead, the back-office sync job and the loyalty integration authenticate to the key service through federation, the delivery integration calls it with its own Google Cloud identity, and each sends an email address and stores the customer key it returns. If the key service runs on Cloud Run, the federated callers can't be granted access directly, because Cloud Run doesn't support direct resource access for federated identities (Google Cloud, n.d.g). Instead, they use their federated token to impersonate a service account that may invoke the service, and request an ID token as that account, which Cloud Run requires (Google Cloud, n.d.a). Addresses still cross clouds, but only to one authenticated endpoint that doesn't store or log them, in place of whole customer tables copied to wherever the match runs.

The secret itself can live in Cloud KMS, which supports HMAC keys for MAC signing (Google Cloud, n.d.e), so the service calls KMS to compute each HMAC; Listing 1 takes the key as a parameter to keep the example self-contained. Any workload allowed to call the service can still check whether an address matches a stored customer key, so access is limited to named workloads, rate-limited, and logged without the addresses. For the same reason, only the key service's own identity should hold permission to sign and verify with the KMS key, because anyone who can compute or check the HMAC can test addresses against stored keys. Each key also costs a network round trip and a KMS request, so order and enrolment flows don't wait on it: a record can be stored first and keyed asynchronously, and bulk jobs such as backfills and re-keying go through a batch endpoint, where the extra time and cost are easiest to absorb. Campaign audiences leave the data platform as customer keys, and the system that holds the address, such as the loyalty platform or the email service, resolves them for sending.

Python

import hashlib
import hmac
import unicodedata

_EDGE_WHITESPACE = " \t\r\n"  # trimmed identically in every implementation

def _ascii_lower(s: str) -> str:
    # bytes.lower() changes only A-Z, so non-ASCII characters are left as-is
    return s.encode("utf-8").lower().decode("utf-8")

def normalise_email(raw: str) -> str:
    s = unicodedata.normalize("NFC", raw.strip(_EDGE_WHITESPACE))
    local, at, domain = s.rpartition("@")
    if not at or not local or not domain:
        raise ValueError("not an email address")
    # Policy choice: lowercase the local part too (see RFC 5321, section 2.4)
    return f"{_ascii_lower(local)}@{_ascii_lower(domain)}"

def customer_key(raw_email: str, key: bytes, key_version: str) -> str:
    """Pseudonymous join key: the same email always gives the same key."""
    if len(key) < 32:
        raise ValueError("key must be at least 32 bytes")
    message = normalise_email(raw_email).encode("utf-8")
    digest = hmac.new(key, message, hashlib.sha256).hexdigest()
    return f"{key_version}:{digest}"

Rust

use hmac::{Hmac, Mac};
use sha2::Sha256;
use unicode_normalization::UnicodeNormalization;

type HmacSha256 = Hmac<Sha256>;

// trimmed identically in every implementation
fn is_edge_whitespace(c: char) -> bool {
    matches!(c, ' ' | '\t' | '\r' | '\n')
}

pub fn normalise_email(raw: &str) -> Result<String, &'static str> {
    let s: String = raw.trim_matches(is_edge_whitespace).nfc().collect();
    let (local, domain) = s.rsplit_once('@').ok_or("not an email address")?;
    if local.is_empty() || domain.is_empty() {
        return Err("not an email address");
    }
    // Policy choice: lowercase the local part too (see RFC 5321, section 2.4)
    Ok(format!("{}@{}", local.to_ascii_lowercase(), domain.to_ascii_lowercase()))
}

/// Pseudonymous join key: the same email always gives the same key.
pub fn customer_key(raw_email: &str, key: &[u8], key_version: &str) -> Result<String, &'static str> {
    if key.len() < 32 {
        return Err("key must be at least 32 bytes");
    }
    let message = normalise_email(raw_email)?;
    let mut mac = HmacSha256::new_from_slice(key).map_err(|_| "invalid key")?;
    mac.update(message.as_bytes());
    let digest: String = mac
        .finalize()
        .into_bytes()
        .iter()
        .map(|b| format!("{b:02x}"))
        .collect();
    Ok(format!("{key_version}:{digest}"))
}

Listing 1. A pseudonymous customer key from a normalised email address. Both implementations must return the same bytes for the same input.

Most of the listing is normalisation, and that's where the hard decisions are. The key only works as a join if every implementation produces the same bytes for the same address. That can mean two languages even inside one service, for example when the service is rewritten, or when its batch endpoint runs in a different language from the online one, and the obvious library calls don't always agree. Python's str.strip() with no arguments, for example, removes a few control characters that Rust's trim() keeps. So the listing trims a fixed set of characters, applies Unicode NFC normalisation so that "é" typed as one character or as "e" plus an accent gives the same key, and lowercases only ASCII letters. Both implementations are tested against the same set of input and output vectors, with the library versions they were generated on pinned.

Lowercasing the local part, the bit before the @, is a policy decision. RFC 5321 says the local part must be treated as case-sensitive, while discouraging anyone from relying on that (Klensin, 2008). Lowercasing it accepts a small chance of merging two people's addresses in exchange for fewer split records. Leaving non-ASCII letters alone means "ÉLODIE" and "élodie" produce different keys, which is the cost of keeping the result identical in every language.

The version prefix on each key makes rotation possible. A new secret key gets a new version. Re-keying means recomputing from the email address, so it can only run where addresses are still held, such as the back office or the loyalty platform. Systems that hold only keys, such as the data platform, are re-keyed from an old-to-new key table produced where the addresses are, and that table should be held as tightly as the secret and deleted once re-keying finishes, because anyone holding it and the old secret can link new keys to addresses. Keeping an address-to-key mapping in the key service would make re-keying easier, but it turns the service into a central store of email addresses that needs the same protection as the secret. Because of that cost, some teams rotate only when a key may have been exposed. During a rotation both versions stay live, and the service issues both for new records until every stored record has been re-keyed.

A customer key isn't anonymous. Under the GDPR, pseudonymisation means data can't be attributed to a person without additional information that is kept separately and protected (European Parliament and Council, 2016, Art. 4(5)), and pseudonymised data that could be attributed using that information is still personal data (European Parliament and Council, 2016, Recital 26). California's CCPA defines pseudonymization along similar lines (California Legislature, n.d., §1798.140(aa)). The key cuts the number of systems that need raw addresses to join records, and makes erasure and access requests easier to trace, but the same privacy obligations apply to every system that stores it.

Data read where it lives

In Figure 1, the loyalty vendor's history export lands in an S3 bucket in the retailer's own AWS account, and the loyalty integration writes a member-to-customer-key map beside it. Keeping the map beside the export lets BigQuery Omni, covered in the next paragraph, join loyalty history to customer keys inside AWS, so only keyed, filtered results cross to Google Cloud. Vendor exports are often CSV or Parquet files. If the vendor writes Apache Iceberg tables, or the retailer converts the export, engines such as Spark, Trino, and Flink can work with the same tables at the same time (Apache Iceberg, n.d.).

BigQuery Omni is one way for the data platform on Google Cloud to use that data without copying it first. It runs the BigQuery query engine in AWS and Azure regions over data in S3 or Azure Blob Storage, through BigLake tables, so processing happens where the data sits (Google Cloud, n.d.b). Results can come back to Google Cloud or be written straight to S3. The options differ in practice: cross-cloud joins suit one-off queries over data that isn't large, with a limit of 60 GB per transfer, while materialised view replicas suit repeated queries such as a dashboard. Data moved out of AWS this way still incurs AWS egress charges (Google Cloud, n.d.b), so a filtered read, such as one store-day of loyalty redemptions for one brand, should cost less than moving the whole table. For Iceberg tables in S3 that are managed by a catalog such as AWS Glue, Google now points to cross-cloud data access through its Lakehouse runtime catalog instead (Google Cloud, n.d.c), so the right route depends on how the export is catalogued.

The SQL Server systems are a different case. Their data changes constantly and the data platform needs it alongside sales from every other source, so the sync job reads SQL Server's change data capture and merges changed rows into BigQuery. A managed service can do the same job. Google's Datastream reads change tables or transaction logs from self-managed SQL Server, including SQL Server on a cloud VM, and change tables from Azure SQL Database at service objective S3 (a Standard-tier size, unrelated to Amazon S3) or above (Google Cloud, n.d.f), and can stream them to BigQuery (Google Cloud, n.d.h). For SQL Server it doesn't support Windows Active Directory authentication (Google Cloud, n.d.f), and its setup creates a SQL Server login with a password for the connection (Google Cloud, n.d.h), so there's a stored credential to manage, which is worth weighing against a sync job that uses its managed identity. Replicating the email column as it is in the source means email addresses arrive in BigQuery unkeyed and need keying and removal there, which brings raw addresses back into the data platform. Excluding the email column avoids that, but then those rows have to be keyed at source. Either way it's a deliberate copy, with a known source, owner, and cadence.

A copy is worth making when it has a specific reason. A dashboard that refreshes every few minutes, a dataset that has to sit in a particular country, a replica kept for recovery, or operational data that has to be joined with sales and other sources, like the SQL Server feed, can each justify one. Each of those copies goes in the same record of copies, so that it can be found, refreshed, and eventually removed.

Events need the same treatment at a smaller scale. Aggregator webhooks arrive in each aggregator's own format. CloudEvents, a CNCF graduated specification for describing event data in a common way, gives every order event the same envelope, and its adopters include Amazon EventBridge, Azure Event Grid, and Google Cloud Eventarc (CloudEvents, n.d.). Wrapping each aggregator's order in that envelope at the integration service means downstream consumers in any cloud read one format, and once the integration service has a customer key for an order from the key service, the event can carry the key in place of the email address.

What the estate looks like with this in place

A few things change that teams outside the platform group will notice. Head-office staff use their Entra ID identity in Azure and Google Cloud. Long-lived credentials between clouds are the exception, kept only where a service such as Datastream, or a vendor integration with the S3 bucket that only supports access keys, requires one, and then recorded and rotated; elsewhere each cloud holds trust relationships with conditions narrowed to the workloads that need them. Every system that stores customers stores the same customer key, issued by one service, and an opt-out or an erasure request can be traced across all of them by that key. Each dataset of record has one owner and one location, and every copy has a recorded reason. Aggregator events reach every consumer in one envelope.

The estate also becomes easier to change. Moving the back office off SQL Server, replacing the loyalty platform, or bringing an acquired brand's data in becomes mostly a matter of re-pointing trust relationships and readers, while the customer key and event format can stay the same. Deciding which workloads stay where they are, and which are worth consolidating, is placement work Sakura Sky's Cloud practice does with retail and QSR teams.

Trust conditions and key versions drift as vendors and teams change, and reviewing them is ongoing work Sakura Sky's Managed Services team takes on.

References

Apache Iceberg, n.d. Apache Iceberg™: The open table format for analytic datasets. Apache Software Foundation. Available at: https://iceberg.apache.org/ [Accessed 9 October 2026].

AWS, n.d.a. AssumeRoleWithWebIdentity. AWS Security Token Service API Reference. Available at: https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html [Accessed 9 October 2026].

AWS, n.d.b. Getting started with outbound identity federation. AWS Identity and Access Management User Guide. Available at: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_outbound_getting_started.html [Accessed 9 October 2026].

AWS, n.d.c. IAM and AWS STS condition context keys. AWS Identity and Access Management User Guide. Available at: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_iam-condition-keys.html [Accessed 9 October 2026].

AWS, n.d.d. Revoke IAM role temporary security credentials. AWS Identity and Access Management User Guide. Available at: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_revoke-sessions.html [Accessed 9 October 2026].

California Legislature, n.d. California Civil Code §1798.140 (California Consumer Privacy Act). California Legislative Information. Available at: https://leginfo.legislature.ca.gov/faces/codes_displaySection.xhtml?lawCode=CIV&sectionNum=1798.140 [Accessed 9 October 2026].

CloudEvents, n.d. CloudEvents: A specification for describing event data in a common way. Cloud Native Computing Foundation. Available at: https://cloudevents.io/ [Accessed 9 October 2026].

European Parliament and Council, 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). Official Journal of the European Union, L 119, 4 May, pp. 1-88. Available at: https://eur-lex.europa.eu/eli/reg/2016/679/oj [Accessed 9 October 2026].

Google Cloud, n.d.a. Configure Workload Identity Federation with AWS or Azure VMs. Identity and Access Management documentation. Available at: https://docs.cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds [Accessed 9 October 2026].

Google Cloud, n.d.b. Introduction to BigQuery Omni. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/omni-introduction [Accessed 9 October 2026].

Google Cloud, n.d.c. Create Apache Iceberg external tables. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/iceberg-external-tables [Accessed 9 October 2026].

Google Cloud, n.d.d. Workforce Identity Federation. Identity and Access Management documentation. Available at: https://docs.cloud.google.com/iam/docs/workforce-identity-federation [Accessed 9 October 2026].

Google Cloud, n.d.e. MAC signatures. Cloud Key Management Service documentation. Available at: https://docs.cloud.google.com/kms/docs/mac-signatures [Accessed 9 October 2026].

Google Cloud, n.d.f. Stream data from SQL Server databases. Datastream documentation. Available at: https://docs.cloud.google.com/datastream/docs/sources-sqlserver [Accessed 9 October 2026].

Google Cloud, n.d.g. Identity federation: products and limitations. Identity and Access Management documentation. Available at: https://docs.cloud.google.com/iam/docs/federated-identity-supported-services [Accessed 9 October 2026].

Google Cloud, n.d.h. Configure a self-managed SQL Server database for CDC. Datastream documentation. Available at: https://docs.cloud.google.com/datastream/docs/configure-self-managed-sqlserver [Accessed 9 October 2026].

Klensin, J., 2008. RFC 5321: Simple Mail Transfer Protocol. Internet Engineering Task Force. Available at: https://datatracker.ietf.org/doc/html/rfc5321 [Accessed 9 October 2026].

Microsoft, 2025. Workload identity federation concepts. Microsoft Entra Workload ID documentation. Available at: https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation [Accessed 9 October 2026].

Thousands of Sites, One Pipeline

Andrew Stevens — Thu, 08 Oct 2026 09:00:31 +0000

When your operations team and your analytics team bring different sales figures for the same day to the same meeting, the discussion tends to turn to whose figure is right. A multi-brand QSR operator with two thousand sites might run three POS systems, two loyalty platforms, four delivery integrations, and an inventory layer built in-house. In that kind of network, analytics dashboards often get checked against other sources before anyone acts on them, and operations keeps its own dashboards built from its own extracts.

The usual response targets the dashboards: a new BI tool, a semantic layer, or a working group to agree definitions. Those can help, agreed definitions especially, but they act after the steps where data is collected, assigned to a day, and matched across sources. In many networks the disagreement is created at that earlier stage. This post works backwards from one disputed number to those causes, then forwards to the pipeline and the operating disciplines that reduce them. Our earlier post Telemetry Is Becoming the Business made the cross-industry case that operational telemetry now has to be engineered as a data product, and a network's sales record is a clear case.

The competing dashboards

Take one store, the fictional store 0412, on a Saturday. It takes delivery through one aggregator, which injects orders into its POS. At about 11pm on Saturday its broadband failed. The cellular failover carried only card payments and order injection, so the back-office upload stopped until Sunday morning. The operations dashboard shows net sales of $18,420. That's the figure in the POS end-of-day report the store manager signs off. The report arrived on Sunday morning when the upload resumed. The analytics dashboard, built from transaction feeds loaded overnight, shows $19,255.

Rather than argue over whose figure is right, the two teams can build a reconciliation bridge, which walks from one figure to the other, one cause at a time. The figures are illustrative, but in our experience the line items are the kind that turn up when this is done on a real network.

Step	Amount	Running total
Operations dashboard (POS end-of-day report)		$18,420
Trading-day boundary: the POS counts sales up to its 4am cutoff as the previous day's; the warehouse uses calendar date	−$1,160	$17,260
Transactions recorded before midnight while the link was down, which missed the overnight load	−$640	$16,620
Aggregator-feed copies of orders the POS also recorded (excluding tax and fees)	+$2,175	$18,795
Tax and customer-paid delivery and service fees on those aggregator copies, counted as net sales	+$460	$19,255
Analytics dashboard		$19,255

In the warehouse, Saturday night's midnight-to-4am window drops out of Saturday and Friday night's comes in, which nets to the first line. Sales after midnight during the outage were also late; the bridge counts them under the trading-day line and leaves them out of the late-transactions line. The late-transactions line exists because the analytics model loads each day once and doesn't go back for late arrivals.

Two lines are rules that differ between teams, one is an arrival gap, and one is a duplicate the analytics model doesn't look for. At this store the fee line disappears with the duplicates. The rule still matters for every aggregator order the POS never sees. Tax is not part of net sales, and customer-paid delivery and service fees on marketplace orders usually belong to the aggregator. Resolve all four, using this POS's definition of net, and store 0412 comes to $18,420, the POS figure, because every sale here went through the POS. Where injection fails during an outage and orders are taken on the aggregator's tablet instead, those orders exist only in the aggregator feed. Orders a POS never sees are added on top and reconciled at settlement.

There's often a fifth item that never makes the bridge, because it has to be solved before any comparison can be made. The store is 0412 in the POS, carries a different code in the inventory system, and has a third identifier in the delivery platform's merchant portal. Before anyone can compare a figure, someone has to know which records belong to the same store.

How QSR data estates fragment

In our experience, networks like this grow one system at a time. Each system arrives to solve a specific problem and brings its own clock, store identifiers, and definition of a sale. An acquisition brings a second POS. A brand launched in another market brings a third. Delivery marketplaces are often integrated one at a time, some injecting orders into the POS and some not, and the choice can differ by brand, by market, or by aggregator. The loyalty platform from Part 1 adds its own view of the same transactions. Unifying data that sat in separate point-of-sale, loyalty, marketing, supply chain, and franchisee reporting silos on a single data engineering platform was a core part of Sakura Sky's work with Craveable Brands.

Franchise networks add another layer. Some systems are chosen by franchisees, and the franchisor may receive only what the franchise agreement provides for, sometimes a daily sales report rather than transaction detail. The royalty base is usually a contractual definition of gross sales, so any published definition of net also has to map back to it.

Each team then builds the extracts it needs for its own questions. Operations takes the POS end-of-day reports because they come from the same close the manager signs off and counts the tills against. Analytics builds from the transaction-level feeds because it needs baskets, dayparts, and channels. Finance usually books from the daily sales journal and closes against card settlement, aggregator payouts, and bank deposits. All three are reasonable choices, and all three produce numbers that differ for reasons like those in the bridge.

When the numbers disagree, the fix usually lands in the dashboard layer, as a filter that excludes one aggregator's feed, a calculated field that subtracts tax, or a date shift for after-midnight trading. Each fix lives in one dashboard or one team's semantic model, so dashboards built elsewhere from the same tables don't have it, and the copies accumulate.

The dashboard layer is also a poor place for some of these rules. It can't recover transactions that weren't loaded. Matching duplicate orders and assigning trading days can be done there, but every model has to repeat the work, and the copies drift apart. A shared semantic layer is a reasonable place to expose the definition of net, but matching orders and recovering late data have to happen upstream of it.

A medallion pipeline for a multi-site network

What matters most in the architecture is where each decision is made, whichever products implement it. Much of Sakura Sky's lakehouse work uses the medallion pattern for this. Bronze holds raw data as received, silver holds cleaned, deduplicated, and conformed records, and gold holds the aggregated, business-ready tables that dashboards read (Databricks, 2026; Google Cloud, n.d.d). Figure 1 shows the layers.

Ingestion jobs (API pulls from POS and aggregator platforms, file drops, webhooks, and event streams) write each source's data into the bronze layer as received. Bronze is append-only apart from retention and erasure deletes. It also holds the POS end-of-day reports and payment settlement files, which silver later parses into control totals. The one exception to "as received" is cardholder data. Incoming data is checked for full card numbers or track data at ingestion. Card numbers are truncated and track data is removed before landing, and each hit is logged (without the data) and raised with the source system to fix. The ingestion step that does this, and any drop zone or queue the raw files pass through first, handles card data, so in our designs it's treated as inside PCI DSS scope and kept small and locked down.

Bronze also needs retention and erasure rules for personal data, such as loyalty identifiers and the customer names and addresses in delivery orders. Erasure has to reach every layer. In silver and gold it only fully takes effect once BigQuery's time travel window has passed (Google Cloud, n.d.b) or old Iceberg snapshots have been expired, and in bronze once any soft-deleted or versioned objects have aged out.

That raw history is what lets the pipeline reprocess a store-day when a rule changes or a late batch arrives (Databricks, 2026; Google Cloud, n.d.d), without asking the POS vendor to resend anything. Parsing happens after landing, with one parser per source system and version, so a change in one vendor's export format is contained in one place.

Silver is where most of the bridge items get resolved, once. Databricks lists deduplication and the resolution of out-of-order and late-arriving data among the jobs of this layer (Databricks, 2026). That's where the duplicate line belongs, though matching one order across two systems is closer to entity resolution than to removing repeated rows. The late-transactions line is closed by reprocessing the day from bronze, with gold's store-day status showing it as provisional until then.

The trading day is a conforming rule applied in silver, using the store master (silver reference data). The store master maps each source system's code to one store key and records the store's time zone, trading-day cutoff, delivery integrations, and status. It keeps each attribute with the dates it applied, so reprocessing an old day uses the cutoff and codes that applied then.

Each order becomes one sales event carrying the store key, the event time, and a business date (the trading day it belongs to). Its line items are kept alongside, and refunds and post-tender voids are recorded as their own events. Money is kept as components (gross, discounts, brand-funded and aggregator-funded promotions, tax and who remits it, customer-paid delivery and service fees, aggregator commission, and tips), so net sales becomes a published definition built from those components. Networks that trade in more than one currency also need a stated conversion policy for any network-wide figure.

Orders that arrive from more than one source carry a cross-source link, typically the aggregator's order ID where the POS integration records it. Where it doesn't, a fallback match on store, time, and amount pairs orders one-to-one within a tolerance, since the two copies rarely carry identical amounts (modifiers, promotions, and tax are often recorded differently). Ambiguous candidates are left unmatched. Until it's matched, the aggregator copy of an order from an injecting integration is held out of net sales, so a missed match understates delivery rather than double-counting it. Once the store-day reconciles to its end-of-day report, the POS side is treated as complete. Any aggregator orders still unmatched are then checked against the day's POS orders from any channel, because staff often re-key tablet orders by hand when injection fails. Those still unmatched after a holding period are released into net as aggregator-only sales and flagged. That changes the store-day's net but not its reconciled status, which covers POS-sourced orders only, and they're tracked against settlement rather than counted as corrections. A rising rate of unmatched orders is treated as an integration fault to fix at source, usually by getting the POS integration to record the aggregator's order ID.

The merged order keeps the POS business date, so the two copies can't land on different trading days near the cutoff. It also keeps both sets of money components, the POS's for reconciling to the end-of-day report and the aggregator's for settlement. Net sales uses the POS's components.

Where the POS records its own business date, the pipeline usually adopts it, because that's the day the end-of-day report covers. Listing 1 is for the sources that don't, such as aggregator feeds, and assigns each event to the store's trading day from its time and the store's configured cutoff. Where managers run the POS close by hand, the cutoff approximates it. Aggregator feeds can carry several timestamps per order, so the rule uses one agreed timestamp, usually when the order was accepted.

Python

from dataclasses import dataclass
from datetime import date, datetime, time, timedelta
from zoneinfo import ZoneInfo

@dataclass(frozen=True)
class Store:
    store_key: str     # one key across POS, delivery, loyalty, and inventory
    tz: str            # IANA zone name, e.g. "America/Chicago"
    day_cutoff: time   # trading day rolls over here, e.g. 04:00; keep it
                       # clear of the hour that repeats when clocks go back
    # zone and cutoff are the store's attributes as of the event's date

    def __post_init__(self) -> None:
        ZoneInfo(self.tz)  # fail fast on an unknown zone name
        if self.day_cutoff.tzinfo is not None:
            raise ValueError("day_cutoff must be naive local time")

def business_date(store: Store, event_time: datetime) -> date:
    """The trading day an event belongs to, in the store's own terms."""
    if event_time.utcoffset() is None:
        raise ValueError("event_time must be timezone-aware")
    local = event_time.astimezone(ZoneInfo(store.tz))
    if local.time() < store.day_cutoff:
        return local.date() - timedelta(days=1)
    return local.date()

Rust

use chrono::{DateTime, Days, NaiveDate, NaiveTime, Utc};
use chrono_tz::Tz;

#[derive(Debug, Clone)]
pub struct Store {
    pub store_key: String,     // one key across POS, delivery, loyalty, and inventory
    pub tz: Tz,                // IANA zone, e.g. chrono_tz::America::Chicago
    pub day_cutoff: NaiveTime, // trading day rolls over here, e.g. 04:00; keep it
                               // clear of the hour that repeats when clocks go back
    // zone and cutoff are the store's attributes as of the event's date
}

/// The trading day an event belongs to, in the store's own terms.
/// Taking `DateTime<Utc>` means a time without a zone can't be passed in.
pub fn business_date(store: &Store, event_utc: DateTime<Utc>) -> NaiveDate {
    let local = event_utc.with_timezone(&store.tz);
    if local.time() < store.day_cutoff {
        local.date_naive() - Days::new(1)
    } else {
        local.date_naive()
    }
}

Listing 1. Assigning an event to the store's trading day, using the store's IANA time zone and cutoff.

The store carries an IANA time zone name instead of a fixed UTC offset, because offsets and daylight-saving rules change by government decision. The IANA Time Zone Database is updated to track those changes (IANA, n.d.). Its 2026 releases record British Columbia, Alberta, the Northwest Territories, and Manitoba moving to permanent UTC offsets (IANA, 2026), so a service on older rules would put most stores in those regions an hour out each winter, starting in November 2026.

The name only helps if the database behind it is current. Python's zoneinfo reads the operating system's copy or the tzdata package (Python Software Foundation, 2026), so updating those is the usual route. ZoneInfo also returns cached zone objects, so a long-running process can keep serving the old rules until it restarts or clears the cache. The chrono-tz crate builds the database into the binary at compile time (chrono-tz contributors, 2025), so a change normally arrives with a crate release that includes it, followed by a rebuild. Its latest release at the time of writing, 0.10.4 from July 2025, predates all four changes. A Rust service for stores in those regions needs either a chrono-tz release with current rules, once one ships, or a time zone source that reads the system's database at runtime. Listing 1 uses chrono-tz to keep the example short. Rules applied in SQL or in Spark typically use that engine's own copy of the database, which needs the same attention.

Daylight saving also makes some trading days shorter or longer than usual, which is worth flagging on hourly and same-day-last-year comparisons. Some POS exports record only local wall-clock time with no offset. The parser then converts using the store's zone, and both the skipped hour when clocks go forward and the repeated hour when they go back need an explicit rule. Python's zoneinfo, for example, treats a repeated time as its first occurrence (fold=0) unless told otherwise (Python Software Foundation, 2026).

Gold metrics are then computed once, upstream of every dashboard, with their definitions published alongside them. Operations, analytics, and finance can still build different views, at different grains and for different questions, but they read the same net sales for store 0412 on Saturday.

Google Cloud's medallion explainer puts bronze in Cloud Storage and silver and gold in BigQuery, with the transformations written in SQL (for example with Dataform) or in Apache Spark (Google Cloud, n.d.d).

Networks that want to stay on open formats can keep silver and gold as Apache Iceberg tables (Apache Iceberg, n.d.a). BigQuery's Iceberg managed tables (formerly BigLake tables for Apache Iceberg in BigQuery) store the data in buckets the customer owns. Writes go through BigQuery, including from Spark and Dataflow via its Storage Write API. Open-source engines such as Spark can read the tables, though Google warns that modifying the data files outside BigQuery can cause query failure or data loss (Google Cloud, n.d.a). Late store-days are reprocessed through BigQuery's DML (Google Cloud, n.d.a). Where Spark manages its own Iceberg tables through its own catalog, MERGE INTO does the same job (Apache Iceberg, n.d.d).

Neither platform's time travel is a long-term record. BigQuery's window is configurable from two to seven days, with seven by default (Google Cloud, n.d.b). Iceberg snapshots last until they're expired, and expiring them regularly is recommended, after which they're no longer available for time travel (Apache Iceberg, n.d.c). A month-end state can be kept longer with a BigQuery table snapshot of a standard table, taken at month-end (Google Cloud, n.d.c), or with an Iceberg tag where Spark or another engine manages the Iceberg catalog (Apache Iceberg, n.d.b). BigQuery's Iceberg managed tables don't support table snapshots (Google Cloud, n.d.a), so on those the month-end state is written out explicitly.

Finance and franchisees need to know what a store-day showed when a month-end report or royalty statement was produced. A pinned copy doesn't record why a figure changed, so revision history is kept explicitly, as a gold table of store-day versions with the reason for each change.

The same layering works on Databricks (Databricks, 2026) or a self-managed Iceberg catalog, and the platform choice is a question for an initial cloud assessment. Part 3 of this series will look at networks whose data already spans more than one cloud.

Operating disciplines around the pipeline

An architecture like Figure 1 can still produce numbers nobody trusts if the pipeline treats each day as finished as soon as a load has run. The Dataflow Model paper treats this as a starting assumption: for data that arrives continuously and out of order, a system can't know whether it has seen everything, only that new data will arrive and earlier data may be retracted (Akidau et al., 2015). A two-thousand-site network behaves like that kind of source. On a typical night in a network this size, some stores are likely to be offline and some POS terminals likely to forward late.

The discipline that follows is to make each store-day's state visible. Listing 2 is a simple version. A store-day is open until its trading day ends. It's provisional until the pipeline's net for POS-sourced orders agrees with the store's end-of-day report, and reconciled once it does. That report covers only what went through the POS. The comparison uses each POS's own definition of net, rebuilt from the stored components, so a definitional difference doesn't look like missing data.

Python

from enum import Enum

class DayStatus(Enum):
    OPEN = "open"                # trading day not yet ended
    PROVISIONAL = "provisional"  # ended, not yet reconciled
    RECONCILED = "reconciled"    # agrees with the store's end-of-day report

def store_day_status(
    *,
    trading_day_ended: bool,
    pipeline_pos_net_minor: int,     # pipeline net for POS-sourced orders, in minor units
    pos_eod_net_minor: int | None,   # None until the end-of-day report lands
    tolerance_minor: int,            # set per POS; line rounding and tax-inclusive
                                     # pricing can differ
) -> DayStatus:
    if tolerance_minor < 0:
        raise ValueError("tolerance_minor must not be negative")
    if not trading_day_ended:
        return DayStatus.OPEN
    if pos_eod_net_minor is None:
        return DayStatus.PROVISIONAL
    if abs(pipeline_pos_net_minor - pos_eod_net_minor) > tolerance_minor:
        return DayStatus.PROVISIONAL
    return DayStatus.RECONCILED

Rust

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum DayStatus {
    Open,        // trading day not yet ended
    Provisional, // ended, not yet reconciled
    Reconciled,  // agrees with the store's end-of-day report
}

pub fn store_day_status(
    trading_day_ended: bool,
    pipeline_pos_net_minor: i64,    // pipeline net for POS-sourced orders, in minor units
    pos_eod_net_minor: Option<i64>, // None until the end-of-day report lands
    tolerance_minor: u64,           // set per POS; line rounding and tax-inclusive
                                    // pricing can differ
) -> DayStatus {
    if !trading_day_ended {
        return DayStatus::Open;
    }
    match pos_eod_net_minor {
        Some(eod) if pipeline_pos_net_minor.abs_diff(eod) <= tolerance_minor => {
            DayStatus::Reconciled
        }
        _ => DayStatus::Provisional,
    }
}

Listing 2. Store-day status: open, provisional, or reconciled against the store's own end-of-day report.

Orders that never pass through the POS, such as those from non-injecting aggregators, aren't covered by this check; they're reconciled against the aggregator's own reports or settlement, usually later. A production version also records why a day is provisional (report missing or totals that don't match) and how long it has been in that state, and compares transaction counts as well as net, since totals alone can hide offsetting errors.

Franchised stores that send only a daily sales report have nothing at transaction level to reconcile it against, so those store-days carry their own status. They're checked against card settlement or aggregator data where the franchise agreement provides it, and against royalty submissions.

The status is recalculated whenever a store-day's events change, so a reconciled day returns to provisional if a late batch moves its POS-sourced total. In the estates we work with, refunds are usually booked to the day they're processed, matching the POS, so a reconciled day isn't reopened by them.

A change to a provisional day is a revision, and a change to a day that had already reconciled is a correction. The revision history covers both. Neither is a restatement in the accounting sense.

Once finance has closed a period, a small correction is usually carried into the open period, and a correction to a day already covered by a royalty statement becomes an adjustment on the next one. Gold keeps the corrected store-day for operational analysis, and the revision history keeps the version that finance and the franchisee were given.

Figure 2 follows store 0412 through its lost link. Saturday shows as provisional, with its end-of-day report counted as missing in the coverage figure, until the buffered transactions and the report arrive on Sunday morning and the day reconciles. The revised figure then reaches both teams' dashboards from the same table on their next refresh, with a note that the day was revised.

The status depends on a few other practices. Coverage is reported every day as store-days with an end-of-day report received against store-days expected to trade, which needs the store master to know about openings, closures, refits, and changes of trading hours. The store master itself has an owner, because a franchise transfer or a reopened site under a new code breaks joins without warning, and the first sign is often a dashboard that looks wrong weeks later.

Each parser has tests against sample files from its source system. POS and aggregator releases are also tracked; in our experience, a POS release rolled out store by store often shows up as a gradual drift in one brand's numbers. Card processor settlement and aggregator payouts give a second reconciliation point independent of the stores, usually run later and at finance's grain, while cash goes through deposit reconciliation.

What changes for operations and analytics

For operations, the most visible change is that the figure they already reconcile to becomes the control total the whole pipeline is held to. Their dashboards move onto the gold layer, and the end-of-day report keeps its standing because it's the store's own close. Analytics changes too, giving up calendar-date grouping and its own tax and fee handling for the shared conforming rules in silver and the published definition of net in gold.

Operations usually hears first about changes to trading hours, cutoffs, temporary closures, and refits. The pipeline now reads them from where operations records them, so a store that closes early for a refit doesn't show as missing. For franchised sites, coverage alerts go to the franchisee through the field team, under the reporting terms of the franchise agreement.

The daily conversation changes shape too. A morning summary might read: 1,826 of 1,839 expected transaction-feed store-days reconciled to their POS end-of-day reports, and 13 provisional. Of 150 daily-report franchise stores, 148 have reported. Two previously reconciled days were corrected since yesterday and are reconciled again, and aggregator-only sales are still to reconcile at settlement.

Disagreements still happen, but they move to definitions that can be decided once: whether staff meals and manager comps count as sales, whether operational figures follow finance in keeping gift card loads out until redemption, how loyalty redemptions and aggregator-funded promotions reduce net, which price counts for marketplace orders, and whether a delivery fee on the brand's own app counts as a sale. The decision is then applied in the pipeline, where every dashboard picks it up.

Sakura Sky's Data & AI practice builds and runs pipelines like this for restaurant and retail networks, from POS and aggregator ingestion through to the store-day status that operations, analytics, and finance read.

References

Akidau, T., Bradshaw, R., Chambers, C., Chernyak, S., Fernández-Moctezuma, R.J., Lax, R., McVeety, S., Mills, D., Perry, F., Schmidt, E. and Whittle, S., 2015. The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing. Proceedings of the VLDB Endowment, 8(12), pp. 1792-1803. Available at: https://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf [Accessed 8 October 2026].

Apache Iceberg, n.d.a. Apache Iceberg™: The open table format for analytic datasets. Apache Software Foundation. Available at: https://iceberg.apache.org/ [Accessed 8 October 2026].

Apache Iceberg, n.d.b. Branching and Tagging. Apache Iceberg documentation. Available at: https://iceberg.apache.org/docs/latest/branching/ [Accessed 8 October 2026].

Apache Iceberg, n.d.c. Maintenance. Apache Iceberg documentation. Available at: https://iceberg.apache.org/docs/latest/maintenance/ [Accessed 8 October 2026].

Apache Iceberg, n.d.d. Spark Writes. Apache Iceberg documentation. Available at: https://iceberg.apache.org/docs/latest/spark-writes/ [Accessed 8 October 2026].

chrono-tz contributors, 2025. chrono_tz (version 0.10.4). docs.rs. Available at: https://docs.rs/chrono-tz/latest/chrono_tz/ [Accessed 8 October 2026].

Databricks, 2026. What is the medallion lakehouse architecture? Databricks documentation. Available at: https://docs.databricks.com/aws/en/lakehouse/medallion [Accessed 8 October 2026].

Google Cloud, n.d.a. Apache Iceberg managed tables. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/biglake-iceberg-tables-in-bigquery [Accessed 8 October 2026].

Google Cloud, n.d.b. Data retention with time travel and fail-safe. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/time-travel [Accessed 8 October 2026].

Google Cloud, n.d.c. Introduction to table snapshots. BigQuery documentation. Available at: https://docs.cloud.google.com/bigquery/docs/table-snapshots-intro [Accessed 8 October 2026].

Google Cloud, n.d.d. What is medallion architecture? Google Cloud. Available at: https://cloud.google.com/discover/what-is-medallion-architecture [Accessed 8 October 2026].

IANA, 2026. Time Zone Database Releases. Internet Assigned Numbers Authority. Available at: https://www.iana.org/time-zones/releases [Accessed 8 October 2026].

IANA, n.d. Time Zones. Internet Assigned Numbers Authority. Available at: https://www.iana.org/time-zones [Accessed 8 October 2026].

Python Software Foundation, 2026. zoneinfo: IANA time zone support. Python 3 documentation. Available at: https://docs.python.org/3/library/zoneinfo.html [Accessed 8 October 2026].