When Input Order Changes the Answer: A Hidden Assumption in Python

Sometimes a test passes because the code is correct.

Sometimes it passes because the test data arrived in exactly the right order.

The second case is more interesting. And potentially more dangerous.

Consider a small batch processor that calculates totals for each customer. Nothing complicated. Just a few records, a grouping operation, and a result.

The input is valid. The program runs without errors. The totals look correct.

Until we change the order of the records.

Three records and two different answers

Imagine we receive these records:

records = [
    {"customer": "A", "amount": 10},
    {"customer": "A", "amount": 8},
    {"customer": "B", "amount": 5},
]

Our goal is simple: calculate one total per customer.

The expected result is:

A: 18
B: 5

And that is exactly what our processor returns.

So far, so good.

Now imagine the same records arrive in a different order:

records = [
    {"customer": "A", "amount": 10},
    {"customer": "B", "amount": 5},
    {"customer": "A", "amount": 8},
]

Nothing was added. Nothing was removed. The amounts are identical.

But our processor now returns:

A: 10
B: 5
A: 8

We suddenly have three groups instead of two.

There is no exception. No invalid input. Nothing to indicate that the application has done something wrong.

The code has simply produced a different answer for the same data.

How?

A perfectly valid Python function

Here is the implementation responsible for the difference:

from itertools import groupby

def totals(records):
    return [
        (
            customer,
            sum(row["amount"] for row in group),
        )
        for customer, group in groupby(
            records,
            key=lambda row: row["customer"],
        )
    ]

At first glance, this looks reasonable.

We group records by customer, sum the amounts, and return the result.

The problem is not the syntax. It is the assumption.

Python's itertools.groupby() groups consecutive elements with matching keys.

It does not behave like SQL's GROUP BY, which combines matching values regardless of their original positions.

For the sequence:

A, A, B

Python finds two groups.

For:

A, B, A

Python finds three.

And that is exactly how the function is designed to work.

The official Python documentation explains this distinction and recommends sorting by the same key when that is the intended grouping behavior.

So this is not a Python bug.

It is a mistake in how we used a perfectly valid function.

If our input is already sorted by customer, the implementation appears correct. If it is not, the result changes.

Our algorithm has an undocumented dependency on input order.

And our original test never challenged that dependency.

The interesting question is not how to fix groupby

The immediate fix is straightforward.

We could sort the records before calling groupby().

Or, since we only want to accumulate totals, use a dictionary:

from collections import defaultdict

def totals(records):
    result = defaultdict(int)

    for row in records:
        result[row["customer"]] += row["amount"]

    return sorted(result.items())

Now the result is independent of record order for this integer-sum example.

Both approaches are valid, depending on the task.

groupby() remains useful when the input is already grouped or when we deliberately want to process consecutive runs. Sorting is appropriate when we need global groups from finite input. A dictionary is convenient when we need to accumulate totals without sorting.

But fixing the implementation was not the most interesting part of this experiment.

The more useful question was:

Could we have found the mistake without already knowing what the correct totals should be?

That question leads to a different way of thinking about tests.

Testing what should never change

Most of our tests follow a familiar pattern:

Give a function some input, calculate the expected result, and compare the two.

For example:

assert totals(records) == [("A", 18), ("B", 5)]

This is useful, of course.

But with our original sorted data, even the incorrect groupby() implementation passes.

What if, instead of checking only one expected result, we checked a property that should remain true?

In this example, the property is simple:

Reordering the same records must not change the totals calculated for each customer.

We can test that property without manually calculating the expected totals for every permutation.

For a small input, the idea looks like this:

from collections import Counter
from itertools import permutations

baseline = Counter(totals(records))

for reordered in permutations(records):
    assert Counter(totals(reordered)) == baseline

Here totals() is the implementation under test.

Using Counter matters because the order of the output rows is not important, but the values and number of occurrences are.

The test asks whether every permutation produces the same logical result.

Run it with the incorrect groupby() implementation and it fails.

Run it with the dictionary-based implementation and it passes for this small test case.

We did not need to know the correct answer for every permutation.

We only needed to know something about the relationship between their answers.

This is an example of metamorphic testing.

The idea is not new. It is especially useful when calculating an expected answer for every test case is difficult, but we know how the output should behave when the input is transformed.

And input order is only one possible transformation.

Why the smallest failing example matters

There is another detail I find interesting.

Why did this particular problem need three records?

With two records, we cannot reproduce the same mistake.

If both records belong to customer A, they remain adjacent regardless of their order.

If one belongs to A and the other to B, each customer has one group. Reversing the records changes their presentation order, but not the totals.

We need at least this pattern:

A, B, A

The second customer separates two records belonging to the first.

Now groupby() treats them as different consecutive groups.

Rearrange the same records:

A, A, B

And the two A records are combined.

Three records are enough to expose the hidden assumption.

This is one reason I like small counterexamples.

A failure involving thousands of records may tell us that something is wrong.

A failure involving three records often tells us why.

Turning the idea into a small Python experiment

This investigation led me to build ReorderWitness, a small MIT-licensed Python CLI experiment.

The tool is deliberately narrow.

It accepts JSON Lines input and a command-line program that reads from stdin and writes JSON Lines to stdout.

Then it tries different input orders, compares the resulting records, and attempts to reduce a detected difference to a smaller reproducible example.

For the groupby() scenario, I created two small processors: one with the incorrect assumption and another using defaultdict.

After cloning the repository and installing it locally, the incorrect processor can be tested with:

reorderwitness \
  --input examples/groupby/records.jsonl \
  --output-dir evidence -- \
  python examples/groupby/processor_broken.py

In the local Python 3.12 experiment, the tool reported:

DIFFERENCE: 3 records reduced to 3; 16 executions
Evidence: evidence

The report includes the two input orders, their corresponding outputs, and the differences between them.

The corrected processor produced no observed difference in the tested permutations.

I also tried the same idea with a different algorithm: finding connected components in an undirected graph.

A deliberately incorrect one-pass implementation produced different partitions when the same edges were supplied in a different order. In that example, ReorderWitness reduced the failing case to two connected edges.

These are synthetic examples, not production bug reports.

But they demonstrate the same underlying problem in two different kinds of algorithms.

A difference is evidence, not automatically a bug

There is an important limitation here.

Not every operation should be independent of input order.

A queue is order-sensitive by design. So are many ranking, sequencing, and event-processing algorithms.

If an operation is supposed to consider order, getting different results after reordering its input is perfectly normal.

A testing tool cannot decide the business meaning of an algorithm.

The developer must define which properties should hold.

There is also a difference between finding a counterexample and proving correctness.

One reproducible counterexample is enough to show that an order-independence property does not hold.

But trying 30 permutations without finding a difference does not prove that the property always holds.

That is why ReorderWitness reports no difference observed, rather than declaring the algorithm correct.

There are already mature tools, particularly Hypothesis, that support much broader property-based testing and shrinking. For Python functions, writing a small test with those tools may be the better approach.

My reason for experimenting with a separate CLI was simpler: sometimes I want to test an existing batch command without building a custom test harness around it.

There is another practical boundary: ReorderWitness runs the target program repeatedly and is not a sandbox. It is intended for trusted, isolated, side-effect-free commands using safe test data.

The lesson I am taking from this

The mistake with groupby() was small.

The larger lesson is about assumptions that never become part of our tests.

We may depend on records being sorted.

We may assume that a collection contains no duplicates.

We may assume that two events will always arrive in a particular sequence.

Sometimes those assumptions are valid and guaranteed by the surrounding system.

Sometimes they are simply true for the test data we happened to use.

And that difference matters.

A green test tells us that the application behaved as expected under the conditions we tested.

It does not tell us which untested conditions the implementation quietly depends on.

So perhaps one useful question to ask when reviewing a data-processing algorithm is:

What can I change about the input that should not change the answer?

Then change it.

If the result changes unexpectedly, there may be a hidden dependency worth investigating.

For me, that is the useful part of this Python experiment. Not the fact that we can shuffle JSONL records, or that we can find a failing permutation.

It is the habit of testing an assumption instead of trusting it.

The examples, tests, and source code are available here:

ReorderWitness on GitHub

I'd be interested to hear about other cases where input order produced surprising results, especially when the ordinary tests were passing.