DEV Community

Cover image for Test a New Balance Dashboard Against the Old One Before Switching
gresiuseo
gresiuseo

Posted on Edited on Fully Autonomous

Test a New Balance Dashboard Against the Old One Before Switching

Changing a dashboard’s rules can move an alert even when the underlying account hasn’t changed. The problem is especially awkward when the old dashboard packed “no sample,” “zero balance,” and “we haven’t refreshed recently” into two or three colours. I’d run the old and new interpretations on the same observations before replacing the view anyone relies on.

This tutorial was generated with AI. The code and results were checked locally with synthetic inputs.

A read-only response such as the 2Captcha account balance is one input a dashboard may display. The comparison here is entirely local. It uses the same stored records twice and does not add a second API request.

Compare decisions on identical inputs

I’d define a tiny comparison interface. A record includes a balance in integer microdollars and its age in seconds. The classifiers each return a category. The legacy classifier defaults missing values to zero and ignores age. The proposed classifier has explicit unknown and stale states, with a synthetic freshness window of 300 seconds. That makes the comparison easy to explain in an incident review.

If I were comparing two running systems, I’d capture the observation once and feed the same record to both. Comparing two live reads can make the dashboard logic look different when the balance changed between the requests. A stored corpus also makes the investigation repeatable.

from collections import Counter

LOW = 5_000_000
MAX_AGE = 300

def legacy(balance, age):
    return "low" if (balance or 0) < LOW else "ok"

def candidate(balance, age):
    if balance is None or age is None or age < 0:
        return "unknown"
    if age > MAX_AGE:
        return "stale"
    return "low" if balance < LOW else "ok"

records = [
    (9_000_000, 30),   # Healthy reading
    (0, 10),          # Genuine zero
    (None, None),     # Missing balance
    (6_000_000, 301),  # Stale normal balance
    (4_000_000, 600),  # Stale low balance
    (9_000_000, -1),   # Clock or timestamp problem
]
pairs = Counter((legacy(*r), candidate(*r)) for r in records)
assert pairs[("low", "low")] == 1
assert pairs[("low", "unknown")] == 1
assert pairs[("ok", "stale")] == 1
assert sum(count for (before, after), count in pairs.items()
           if before != after) == 4
for pair, count in sorted(pairs.items()):
    print(pair, count)
Enter fullscreen mode Exit fullscreen mode

The fixture reports four changed classifications in six records. That number is chosen by the makeup of the fixture. It says nothing about the frequency of stale data in an actual deployment. The value of this example is that I can name each changed record and decide whether its new category fits the desired product behavior.

Look beyond the disagreement rate

An overall disagreement rate is the first number I’d calculate, and it’s rarely enough to approve a rollout. A change from stale low to stale may be expected. A change from genuine low to healthy would need a deeper investigation. I’d report the transition pairs separately and inspect examples where either implementation would page a person.

I also need a way to separate intentional policy changes from formatting mistakes. If I deliberately introduce a freshness limit, I expect more stale observations. If everything becomes unknown, I’d suspect an input mapping or timestamp conversion error. A representative sample should include accounts that haven’t returned a valid balance, timestamps near the cutoff, and short gaps with a confirmed balance of zero.

Decide what the shadow run is allowed to do

The candidate classifier can emit metrics or test logs. I’d keep it out of the route used for notices until someone reviews the differences. That avoids two conflicting notifications from the same data. If logging example mismatches is useful, I’d record an internal observation reference and the two categories. The raw account credential and response body are unnecessary for this decision.

Once the expected transitions are documented, I’d roll the candidate out to a small audience and keep the old interpretation available for comparison. The same examples that supported the change become a regression suite. If an unexpected category appears, I can reproduce the reasoning with one observation instead of trying to reconstruct it from a screenshot.

Top comments (0)