The Remote Suite Stayed Green. Local pytest Was Reading a Parent Ini.

Have you ever shipped a green remote run, then watched the same pytest command fail on the laptop beside you? I spent forty-eight hours inside that gap, taking field notes instead of rewriting assertions at random. The suite was not flaky in the usual timing sense, and the assertion messages looked almost honest. The two environments were simply not reading the same configuration file, which I refused to believe.

Hour zero: the question I should have asked out loud

Why would pytest tests/test_billing.py pass on one machine and fail on another with the same git SHA? I assumed dependency drift first, because that story is easy and it is often true. I then blamed a cached .pyc file, because stale bytecode after a rename has burned me before. Neither guess is foolish in theory, and both still wasted a full evening of reinstalls.

The failure was a collection difference, not a single nasty assertion hiding in one module. Locally I collected a larger set; the remote run collected fewer tests and skipped a module I could still see on disk. Have you compared collection counts before you scrolled the traceback? I had not, and that was the first real mistake.

What I tried, in the order that actually happened

I am reconstructing this as a field notebook, not as a production postmortem with invented metrics. The commands below are the ones I would run again on a throwaway tree. If a step looks theatrical, that is because I did it theatrically the first night.

1. Re-ran the exact test node id on both machines and treated the color as evidence.
2. Wiped __pycache__ and .pytest_cache like a superstition that sometimes works.
3. Recreated the virtualenv and reinstalled from pyproject.toml without printing config.
4. Asked a coding assistant to make local match remote, with no environment fingerprint attached.
5. Only then printed pytest's chosen rootdir, inifile, and the effective addopts.

Guess which step should have been first? If you said step five, you are already ahead of where I started. The assistant did what assistants do when you omit the environment: it patched a test, then patched an import, then offered sys.path glue I did not want.

The actual bug: two config files, one invisible from my mental map

Pytest does not magically read "the" config for your repository as if the git root were sacred. It discovers a rootdir and an inifile by walking upward from the test paths you gave it. A leftover pytest.ini above the repo, or a closer pyproject.toml with [tool.pytest.ini_options], can win. A tox.ini with a [pytest] section can join that walk as well.

Would pytest walk all the way to $HOME for sport? No, and my first telling of this story was sloppy on that point. It walks from the common ancestor of the arguments, so the extra file has to sit on that chain. Mine sat in a parent of the repo because I had nested the project under ~/work/experiments/billing/ and left a config at ~/work/experiments/pytest.ini.

That parent file set addopts = -q --maxfail=1 -k "not slow". The remote checkout lived in a clean directory with no parent ini at all. Local pytest therefore dropped every test matching slow, and I spent hours "fixing" a module the other machine never skipped. Does that sound too dumb to be real? It is real enough that I now print discovery before I print tracebacks.

A throwaway layout you can copy in ten minutes

Label this as a reproducible example, not as harvested production logs. Create the nested tree on a laptop, then run the same commands in a clean directory that has no parent ini.

experiments/
  pytest.ini
  billing/
    pyproject.toml
    tests/
      test_billing.py

experiments/pytest.ini:

[pytest]
addopts = -q --maxfail=1 -k "not slow"

experiments/billing/tests/test_billing.py:

import pytest


def test_invoice_total():
    assert 1 + 1 == 2


@pytest.mark.slow
def test_invoice_slow_path():
    assert 2 + 2 == 4

From the nested project, collection lies to you in a quiet way:

cd experiments/billing
python -m pytest --collect-only -q

You should see test_invoice_total and not test_invoice_slow_path, even though both functions sit in the file you opened. Copy only billing/ to a clean directory with no parent pytest.ini, and both tests come back. That split is the entire incident.

Artifact: fingerprint both Pythons before you patch a test

Save this as scripts/pytest_fingerprint.py and run it with the same interpreter you use for tests. The script does not fix pytest. It only makes the two machines confess.

#!/usr/bin/env python3
"""Print the pytest discovery context. Run me with the test interpreter."""
from __future__ import annotations

import os
import sys
from pathlib import Path

CONFIG_NAMES = (
    "pytest.ini",
    ".pytest.ini",
    "pyproject.toml",
    "tox.ini",
    "setup.cfg",
)


def walk_configs(start: Path) -> list[str]:
    found: list[str] = []
    cur = start.resolve()
    seen: set[Path] = set()
    while cur not in seen:
        seen.add(cur)
        for name in CONFIG_NAMES:
            candidate = cur / name
            if candidate.is_file():
                found.append(str(candidate))
        if cur.parent == cur:
            break
        cur = cur.parent
    return found


def main() -> None:
    cwd = Path.cwd()
    print(f"cwd={cwd}")
    print(f"home={Path.home()}")
    print(f"executable={sys.executable}")
    print(f"version={sys.version.split()[0]}")
    print(f"PYTHONPATH={os.environ.get('PYTHONPATH', '')!r}")
    print(f"PYTEST_ADDOPTS={os.environ.get('PYTEST_ADDOPTS', '')!r}")
    print("sys.path:")
    for entry in sys.path:
        print(f"  {entry!r}")
    print("config candidates from cwd upward:")
    for item in walk_configs(cwd):
        print(f"  {item}")


if __name__ == "__main__":
    main()

Run the same block in both places and keep the dumps side by side:

python scripts/pytest_fingerprint.py
python -m pytest --collect-only -q
python -m pytest --trace-config tests 2>&1 | head -n 80

--trace-config is the command I keep forgetting, and it is the one that names the inifile. Compare rootdir, inifile, and the effective addopts before you touch an assertion. If those three lines differ, stop editing test bodies. What else is there to argue about at that point?

Decision table I now keep in the notebook

If you see this difference	Do not do this	Do this first
Collection count differs, SHA matches	Rewrite assertions	Diff --trace-config and the fingerprint script
Only local skips slow or integration	Delete skip marks	Search parent dirs for addopts and -k
Remote is green, local ImportError	sys.path.insert in tests	Print sys.path and executable on both
PYTEST_ADDOPTS set in one shell	Trust a chat patch	`env \
{% raw %}pythonpath in one ini only	Editable reinstall loops	Read the winning inifile, not the README
Agent edited pyproject.toml but pytest still odd	More TOML	Confirm which inifile won; a pytest.ini can override you

Would I paste that table into a pull request comment? Yes, because future me will skip the script otherwise. A table is not documentation theater when it stops a second night of cache wiping.

Where a second machine helped, and where it did not

I needed a checkout that did not inherit the laptop's parent-directory junk. A clean remote shell is useful for that split, because it kills the "works in my nested folder" story immediately. The method is a diff, not a vibe.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access and the free server option as that second shell, not as an oracle for pytest discovery. The model could suggest pytest --trace-config; it could not see ~/work/experiments/pytest.ini until I ran the fingerprint on both sides and pasted the two dumps. If you strip the product name out of this section, the habit is unchanged: compare discovery, then patch tests. If you already wanted a clean shell that is not your nested laptop folder, that free server option is one way to get the split, and you still have to paste the fingerprint yourself.

The clean box was empty enough that parent-directory configs did not exist. That emptiness is the feature. It is also the limitation, because emptiness will not reproduce a bug that only happens when a parent ini is present. I had to keep the laptop as the dirty case and the remote box as the clean case, then diff them on purpose.

What broke when I let the model edit first

The first assistant patch added pythonpath = src to pyproject.toml. That is a reasonable default for some layouts, and it was unrelated to my skips. The second patch renamed a test so -k "not slow" would stop matching it. That made local green without explaining remote green, which is how you grow a shadow suite.

I reverted both patches and deleted the parent pytest.ini. Collection counts matched after that, which felt almost disappointing. The original assertion in tests/test_billing.py then failed on both machines, and that is the only kind of red I trust. Have you noticed how often an agent patch hides the second environment instead of aligning it?

Here is the check I now require before I accept a test-only patch from any model:

python -m pytest --collect-only -q | tee /tmp/collect_before.txt
# apply the patch, then:
git diff --stat
python -m pytest --collect-only -q | tee /tmp/collect_after.txt
diff -u /tmp/collect_before.txt /tmp/collect_after.txt || true

If collection changed, the patch is guilty until the inifile diff says otherwise. Are you reviewing agent diffs for collection changes, or only for assertion rewrites? I was doing the latter, and it showed.

What I would repeat in the next forty-eight hours

• Fingerprint cwd, executable, PYTHONPATH, PYTEST_ADDOPTS, and config candidates before asking for a patch.
• Run pytest --trace-config on every machine that claims a color, green or red.
• Treat parent directories as hostile until they are listed in the dump.
• Keep one dirty environment and one clean environment, then diff them on purpose.
• Refuse test-body edits when collection counts already disagree.

I would not repeat wiping virtualenvs as step one, because that ritual hid the parent ini for hours. I would not repeat pasting a traceback into a chat without the fingerprint sitting above it. I would not repeat sys.path.insert(0, ...) either, because that lie has a half-life of one pull request.

Limitations, and who should not use this workflow

This notebook is for people who already run pytest on more than one machine and can execute a Python script in both places. It will not help you if the failure is a true race, a network stub, or a clock that moves. It also will not help if you cannot run --trace-config because your organization wraps pytest in a binary you do not control.

Do not use a clean remote server as proof that local is wrong. Local might be the only environment that resembles production cwd, including those leftover parent files. Do not use a model as a substitute for reading the winning ini file; the file is short and the model is guessing. If your tests must load secrets from a parent .env, a clean server will hide that dependency until deploy, which is a different incident with the same shape.

I am not claiming timings, cost savings, or model rankings, because I did not measure those things here. I am claiming a comparison habit you can rerun on a throwaway nested repo tonight. Print discovery first. Patch second. Ask the model only after both machines have spoken.