Five test tools, five dashboards, and nobody can say what happened

Most teams did not choose to have five testing tools. They accumulated them, one reasonable decision at a time.

Something was needed for the browser journeys, so a UI framework went in. The API contract broke twice in a quarter, so a contract-testing tool followed. A load test was required before a launch, and the person who ran it picked whatever they knew. Security scanning arrived through the compliance programme rather than through engineering. And somewhere there is a spreadsheet, or a person, comparing generated customer documents against a baseline by eye.

Each of those was the right call. The sum of them has a property nobody chose.

The bill nobody itemises

Ask what five tools cost and you will be told about licences. That is the smallest line.

The real cost shows up the first time something fails in a way that spans two of them. Here is the shape it takes, and it is worth walking through slowly, because the difficulty is structural rather than cultural.

A checkout total comes out wrong in staging on a Thursday afternoon.

• The UI suite is red on checkout_applies_promotion. It has a screenshot of a total reading £48.00 where the assertion wanted £43.20.
• The API suite is green. Every contract test passed, including the one covering the pricing endpoint.
• The load run was on Tuesday, against a build from Monday, and showed the pricing service at a p99 nobody wrote down, because the run was considered a pass.
• The security scan is a weekly job. It ran overnight and is unrelated, but it did restart a container.
• The document comparison is not part of any of this, and the invoice template changed last sprint.

Five signals. One of them is red. And the honest answer to "what happened" is that nobody in the building can say, because no two of those signals can be joined to each other.

Why they cannot be joined

This gets mistaken for a communication problem. It is not. The signals are genuinely unjoinable, for four concrete reasons.

There is no shared run identity. The UI suite has a run id. The API suite has a different run id, in a different namespace. Neither carries the other's, and neither carries the id of the build they ran against -- or if it does, it carries the CI job number, which is not the same thing as an artefact version. You cannot ask "show me every signal for build 4471", because no field in any of the five systems means that.

There is no shared clock that matters. Every tool timestamps its own events, and the timestamps are accurate. But the UI failure at 14:32:07 and the pricing service's config reload at 14:31:58 live in two systems with no relationship, so the nine seconds between them is a coincidence until a human notices it. Correlating by eyeballing timestamps across browser tabs is the default technique in this situation, and it is about as reliable as it sounds.

There is no shared environment of record. "Staging" is not one thing. The UI suite pointed at the edge; the API suite pointed at the service directly, bypassing the gateway where the pricing rule actually lives. Both are correct about their own target, and neither is wrong -- which is precisely why two teams can each hold a passing result and still disagree.

The artefacts expire on different schedules. The screenshot is kept for 30 days by the UI tool's default. The API payloads are in CI logs, gone in 7. The load results are wherever that tool keeps them, possibly on a laptop. By the time the argument is settled, at least one piece of the evidence has aged out.

So the meeting happens. The UI team says the API changed. The API team says the payload has always looked like that. Somebody suggests re-running everything, which takes until Monday. Nobody is being unhelpful -- they are each holding a true statement about a different system.

What the release decision is actually made on

Now consider how this reaches a decision.

Whoever signs off does not look at five dashboards. They look at a summary -- a status in a release ticket, or a message in a channel, written by someone who looked at the dashboards. That person compressed five partially contradictory signals into a sentence.

So the decision is made on a summary of a summary, by someone with no way to inspect the evidence underneath. The failure mode is not that they decide wrongly. It is that they learn the summary is unreliable, and start deciding on other things instead: whether the team seems confident, whether the last release went fine, whether the date can move.

That is how a team ends up with a great deal of testing and very little signal.

What "one place" has to mean to be worth anything

The instinct is to consolidate vendors. That is one answer, it is not the only one, and it is worth separating the useful part from the procurement part. The useful part is four properties:

1. One run identity that spans every kind of check, carried as a field in every event and artefact, and tied to the artefact version rather than the CI job number.
2. One sink. Every check writes its result to the same store, in a shape that can be queried across kinds. Not a dashboard embedding five iframes -- one queryable place.
3. One environment of record per run, recorded as data, so "the API suite was bypassing the gateway" is a fact you can read rather than something you discover in the meeting.
4. One retention policy across artefacts of the same run, so the evidence does not half-expire while it is still under discussion.

None of these requires replacing anything. If you have five tools and no appetite for changing that, the three cheapest moves, in this order, are:

• Generate a run id at the top of the pipeline and pass it into every tool as an environment variable or a tag.
• Ship every tool's structured output to one place, even if that place is a bucket and a table.
• Record the resolved target URL and the artefact version alongside each result.

That is roughly a day of pipeline work, and it converts the Thursday afternoon above from an argument into a query.

The reason a single platform is attractive is not that it does something a well-instrumented set of five tools cannot. It is that those four properties come as defaults, rather than as a project somebody has to fund and then maintain.

Where the signal should land

One more thing, because it tends to be left out of the consolidation conversation.

Having one queryable place is necessary but not sufficient. If the result of all this is a dashboard, it will be read by the people who already read dashboards. What changes behaviour is the finding arriving where the work happens: an issue in the tracker the developers already have open, carrying the run id, the environment of record, and the artefact.

Testing is usually described as a stage sitting between development and release. It is more useful to treat it as the signal a release gets judged on -- the thing that has to be legible at the end, to somebody who was not in the room for any of it.

Disclosure

2SD Technologies builds TAI, a testing platform, so we have an interest in the conclusion. What we would defend is the diagnosis rather than the shopping list: most teams' testing problem is attribution rather than coverage, and the four properties above are worth having whether you get them from one product or from a day of plumbing across the five you already own.

If you would like to talk it through against your own pipeline, we would be glad to: [e-mail]