Every estate has a boundary where the controls get serious. Production sits inside it. Access is brokered, changes are reviewed, reads are logged, and somebody owns the log.
Test environments sit outside it. Not by decision — by accretion. Someone needed a realistic dataset for a migration in 2019, took a snapshot, and the snapshot became the fixture. It got copied into a second environment for load testing, then into a third so a contractor could reproduce a bug. Nobody signed anything, because nobody was doing anything that felt like it needed signing.
That is how the least-governed part of an estate ends up holding the most production-shaped data in it.
Why the data has to be production-shaped
The obvious answer is "then don't use real data", and it is worth being honest about why teams do it anyway.
Test data has to be representative or the tests are theatre. The interesting defects live in distributional properties: the customer with 4,000 orders, the account whose name contains an apostrophe, the address with no postcode, the record migrated from the system before the system before this one. Synthetic generators produce data that satisfies the schema. Schemas are not where the defects are.
So teams reach for a subset of production, and the honest ones anonymise it. Anonymisation is real work and it helps, but it is a spectrum rather than a state — a dataset that keeps the distribution keeps a lot of what makes a record identifiable, and a dataset that does not keep the distribution has stopped being useful for the thing you wanted it for.
The practical position most teams land on is: this data is not production, but it is close enough that we would not want it in a public bucket.
Then a testing tool is pointed at it.
What that makes a testing tool
A testing tool is an unusual thing to introduce into an estate. Consider what it needs in order to work at all:
- Credentials for an application, usually more than one set, usually including a privileged one because half the interesting paths are behind an admin role.
- Network reach into the environment where that application runs.
- The ability to drive the application as a user, which means it can read whatever a user can read.
- Somewhere to keep results — screenshots, request and response payloads, database state before and after.
That last one is the part people miss when they think about test tooling as a build-time concern. The artefacts a test run produces are a copy of the data the run touched. A failing API assertion is stored with the payload that failed it. A failing UI step is stored with a screenshot of the screen it failed on, and that screen had a customer's name on it.
So a testing tool is not just a consumer of test data. It is a second store of it, in a place whose retention policy nobody wrote down.
None of this is an argument against test automation. It is an argument that the access model of a testing tool is not a datasheet appendix. It is the thing that decides whether the tool can be adopted at all in an estate that takes its boundary seriously — and it is why these tools get rejected at review by people who were never against the tool.
The four questions, and what a real answer looks like
Reviewers are not asking for reassurance. They are asking for artefacts. Four categories cover most of it.
Identity. Does the tool authenticate against the directory you already run, or does it keep its own user list? A second user list is not a preference difference — it is a population of accounts that does not get deprovisioned when someone leaves, because your leaver process does not know it exists. The answer a reviewer wants is a protocol, not a promise: SAML or OIDC against the existing identity provider, so joiners and leavers are handled by the process that already handles them.
Authorisation. Once someone is in, what can they do? The failure mode here is a tool with one effective permission level, where access control is really URL obscurity — anyone who can reach the run can see the run. Roles need to be coarse enough that people actually use them and fine enough that "can trigger a run against the payments service" and "can read the payloads that run captured" are separable, because those are genuinely different privileges.
Tenancy. If the tool is multi-tenant, what enforces separation, and at what layer? "Separate databases" and "a tenant_id column and a WHERE clause" are both answers to that question, and they are not the same answer. If the tool runs inside your own boundary, the question changes shape rather than disappearing: separation is now your own controls' problem, which is usually what a reviewer prefers, because they can already audit those.
Evidence. And this is the one worth being precise about.
"A run happened" is not the same artefact as "here is what it could see"
Most tools can produce a list of runs. Who triggered it, when, against which environment, pass or fail. That is an activity log, and it answers a question about the tool.
The question a reviewer is actually asking is about the data. Which endpoints did that run call? Which records came back? Which of those artefacts are still stored, and for how long? If a screenshot captured a customer record, is that screenshot inside the retention window that applies to the record, or the one that applies to build artefacts?
Those are different logs. The first is cheap and every tool has it. The second requires the tool to have been built with the assumption that its own outputs are sensitive — which mostly means decisions made early: artefacts stored with the same classification as their source, retention configurable per artefact class rather than globally, and redaction available at capture rather than as a cleanup job.
You can test for this in about five minutes. Ask to see the audit view for a run that touched customer data, and ask which of the artefacts from that run still exist. A tool that has thought about it will show you. A tool that has not will show you the activity log again.
A short checklist
Portable, and not about any particular product:
- Does it authenticate against our identity provider, or keep its own user list?
- Can "trigger a run" and "read what the run captured" be granted separately?
- What enforces tenant separation, and at which layer?
- For a run that touched customer data: which artefacts still exist, and under whose retention policy?
- Where are those artefacts processed, and does that answer change under load?
The fourth one is the one that finds things. It is also the one most likely to be answered with a screenshot of a dashboard rather than a policy, which is itself the answer.
Disclosure
2SD Technologies builds TAI, a testing platform, it runs inside the customer's own environment, which means most of question three becomes a question about controls the customer already audits — and the checklist .
Top comments (0)