I Trusted the Remote Green Build for 48 Hours. Pytest Never Loaded conftest.py

Have you ever watched a remote coding session print a clean pytest summary while the same commit failed on your laptop? I did, and I wasted two days assuming the tests were flaky instead of asking a dumber question first. Where did pytest actually start on that machine, and which conftest.py files did the process actually load? The answer lived in the working directory of a free remote server, not in the assertion I kept rewriting.

This is a set of 48-hour field notes, not a product tour. I wrote down what I tried, what broke, and what I would repeat. If you strip every tool name out of this piece, the workflow should still save you a night.

Hour 0: the symptom looked like flakiness

The local failure was boring, which is how these things usually start. A helper in tests/conftest.py stubbed an environment variable that production code reads during import. Remote output said three hundred tests passed. Local output said the first collection already exploded.

I asked myself the usual unhelpful questions. Was the branch dirty? Was I on a stale wheel? Was the model just bad at pytest? None of those questions print a working directory, so none of them could have caught this.

Hours 1–8: what I tried first

I treated the mismatch like a dependency problem because that is my reflex. You probably have the same reflex if you live in Python services.

Here is the exact sequence I ran, twice, on both machines:

1. git rev-parse HEAD and git status --porcelain
2. python -c "import sys; print(sys.executable); print(sys.path[:3])"
3. python -m pytest --collect-only -q | tail
4. python -m pytest -q tests/test_settings.py

HEAD matched. The dirty list matched. The interpreter path looked reasonable on both sides. Collection still disagreed, and that should have been the clue. Collection is about filesystem layout, not about your favorite assertion style.

I then compared installed packages, because I did not want to admit the boring truth yet.

python -m pip freeze | sort > /tmp/freeze.local.txt
# later, on the remote shell
python -m pip freeze | sort > /tmp/freeze.remote.txt
diff -u /tmp/freeze.local.txt /tmp/freeze.remote.txt

The freeze files were not identical, and that sent me down a useless hour. A slightly newer pytest does not explain a missing fixture. A missing fixture usually means a missing conftest.py in the discovery path. Why was I still arguing with pin files?

Hours 9–18: the receipt I should have printed

I finally printed the thing pytest already knows and I keep forgetting to ask for.

python -m pytest -o addopts= --collect-only -q -vv 2>&1 | head -n 40

Look at rootdir, configfile, and testpaths before you look at a traceback. On my laptop, rootdir was the repository root and tests/conftest.py loaded during collection. On the remote session, rootdir was .../repo/src because the agent had cd src before invoking pytest. Parent conftest.py files outside rootdir are not part of that session. The suite was green because the dangerous tests were never collected.

Does that sound obvious after the fact? It always does. It is not obvious when a coding agent summarizes a log and you only read the last line.

I reproduced the layout with MonkeyCode because it offers free model access and a free server option, so the agent could run away from my laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The product was just the place the wrong working directory showed up. The bug is pytest discovery, and it will show up anywhere an agent is allowed to cd.

Hours 19–30: what actually broke

Three separate mistakes stacked, which is why the story lasted two days instead of twelve minutes.

• The generated run script used cd src && python -m pytest, which moves rootdir unless you pass an explicit path.
• The free server clone had a nested extra pytest.ini leftover from an old packaging experiment.
• I read the agent's English summary instead of the collection header, so I never saw rootdir change.

The leftover ini file was the nasty part. Pytest will happily treat a nested ini as the project config if that is where you launch it. Fixtures from the real test package then vanish without a loud error. You just get a shorter, greener suite. Who among us distrusts a shorter green suite at 1 a.m.?

I confirmed it with a throwaway layout you can recreate locally. Label this as a lab tree, not as production evidence.

repo/
  pyproject.toml
  tests/conftest.py          # defines required_env
  tests/test_settings.py     # depends on required_env
  src/
    app/__init__.py
    pytest.ini               # trap: toolsdir-style leftover

From repo, collection includes the fixture. From repo/src, collection may ignore tests/ entirely, depending on testpaths. That is not flakiness. That is two different programs that share a logo.

The artifact: a workspace receipt before you trust green

I now refuse to read a remote pytest summary until a receipt JSON is printed. The script below is small on purpose. Run it in the same shell that is about to invoke pytest.

#!/usr/bin/env python3
"""workspace_receipt.py — print cwd identity before trusting a remote green build."""
from __future__ import annotations

import hashlib
import json
import os
import socket
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path


def git(*args: str) -> str:
    result = subprocess.run(
        ["git", *args],
        cwd=Path.cwd(),
        text=True,
        capture_output=True,
        check=False,
    )
    return (result.stdout or result.stderr).strip()


def existing(paths: list[Path]) -> list[str]:
    return [str(path) for path in paths if path.exists()]


def main() -> int:
    cwd = Path.cwd().resolve()
    expected = os.environ.get("EXPECTED_ROOT")
    payload = {
        "cwd": str(cwd),
        "hostname": socket.gethostname(),
        "python": sys.executable,
        "version": sys.version.split()[0],
        "git_head": git("rev-parse", "HEAD"),
        "git_branch": git("branch", "--show-current"),
        "git_status_short": git("status", "--porcelain"),
        "markers": existing(
            [
                cwd / "pyproject.toml",
                cwd / "pytest.ini",
                cwd / "conftest.py",
                cwd / "tests" / "conftest.py",
                cwd / "src" / "pytest.ini",
            ]
        ),
        "generated_at_utc": datetime.now(timezone.utc).isoformat(),
    }
    digest_src = json.dumps(payload, sort_keys=True).encode("utf-8")
    payload["receipt_sha256_16"] = hashlib.sha256(digest_src).hexdigest()[:16]
    print(json.dumps(payload, indent=2))
    if expected and Path(expected).resolve() != cwd:
        print(
            f"ERROR: cwd {cwd} != EXPECTED_ROOT {Path(expected).resolve()}",
            file=sys.stderr,
        )
        return 2
    if (cwd / "src" / "pytest.ini").exists() and not (cwd / "tests").exists():
        print(
            "ERROR: nested src/pytest.ini with no tests/ here; refuse to trust green",
            file=sys.stderr,
        )
        return 3
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Wire it in front of the suite. Do not skip the EXPECTED_ROOT check just because the hostname looks familiar.

export EXPECTED_ROOT="/abs/path/to/repo"
python workspace_receipt.py || exit $?
python -m pytest -o addopts= --rootdir="$EXPECTED_ROOT" -q

--rootdir is the flag I should have used on hour one. Pytest still discovers tests relative to that root, and nested leftover ini files stop winning by accident. If an agent cannot pass --rootdir, the agent is not running your suite. It is running a nearby suite that happens to share files.

A second guard: fail collection when the fixture is missing

Receipts are easy to ignore under time pressure. I added a collection hook so the suite screams instead of shrinking.

# tests/conftest.py
import os
import pytest


def pytest_sessionstart(session):
    root = session.config.rootpath
    marker = root / "tests" / "conftest.py"
    if marker.resolve() != Path(__file__).resolve():  # noqa: F821 — see note
        pytest.exit(f"refusing to run: conftest loaded from {__file__}, rootdir={root}", returncode=4)


@pytest.fixture(scope="session", autouse=True)
def required_env():
    os.environ.setdefault("APP_ENV", "test")
    yield

The Path import belongs at the top of the file; I left the comment in the snippet so you notice the hook is about identity, not about environment variables. If collection cannot see this file, the fixture never runs, and that used to look like success. I would rather have a hard pytest.exit than another green lie.

Want a one-liner while you argue with an agent about whether the hook is loaded?

python -m pytest --collect-only -q -p no:cacheprovider 2>&1 | rg -n "rootdir|conftest|ERROR"

If rg is not installed, grep -E is fine. The point is to read the header, not the scoreboard.

Decision table I actually keep

Signal you see	What it usually means	What to print next	Trust the green build?
Remote pass, local fail, HEAD matches	Different rootdir or extra ini	pytest collection header	No
Collection count differs by a lot	Tests were never gathered	--collect-only -q on both sides	No
Fixture not found only on one machine	conftest.py outside rootdir	EXISTING markers from the receipt	No
Same collection count, different failures	Real product bug or env leak	required_env plus sys.executable	Maybe
Agent says "I ran pytest" with no header	You were given a summary, not a run	Refuse until receipt JSON exists	No

I keep this table in the repo under docs/pytest-rootdir.md. The table is the artifact I reuse. The prose around it changes every time I get cocky.

What I would repeat

I would repeat the boring prints before I repeat the clever theory. Working directory, rootdir, nested ini files, and a hash of HEAD beat another rewrite of the failing assertion. I would also force --rootdir in every agent-facing script, even when the script is generated by a free model on a free server. Generated shell is where cd sneaks in.

I would repeat the hard refusal in pytest_sessionstart. Soft documentation did not survive contact with a confident summary paragraph. A non-zero exit did.

Would I repeat reading only the last twenty lines of a remote log? Never. That habit is how green builds get their reputation.

Who should not use this approach

Skip the receipt script if you already pin a single tox or nox session that owns chdir and --rootdir for every job. You do not need another JSON file in that world. Skip it if your tests are a handful of modules with no parent conftest.py, because there is no hidden fixture graph to lose. Skip it if you cannot control the remote working directory at all; the script will only document a mess you cannot fix.

This is also the wrong tool if the real bug is timing, network, or data. A workspace receipt will not explain a race. It will only stop you from debugging a race that never ran.

Limitations

The receipt is only as honest as the shell that launched it. If a wrapper cds after the receipt and before pytest, you are back in the same hole. --rootdir helps, but plugins can still rewrite collection. Nested packages that ship their own conftest.py can make the hook too strict, so treat the pytest.exit as a default for apps, not for libraries.

I am not claiming the free server is slower or faster than your laptop. I did not benchmark models, quotas, or hardware, and you should not trust anyone who does that from a single anecdote. I am claiming that a green pytest summary without rootdir is not evidence. It is a rumor with formatting.

If you already have a spare remote session, the receipt script is the only piece I would actually keep. Paste it in front of the suite, print rootdir, and then decide whether the agent earned the word green.

Comments

[AiOps Community]

"A rumour with formatting" is going straight into my vocabulary.

One row I'd add to your decision table, because it's the case your own lab tree demonstrates and the receipt doesn't catch: suite shrank but stayed green. The receipt records cwd, HEAD and which marker files exist, which are all preconditions. The signature of rootdir drift isn't a bad precondition though, it's a smaller collection - you say it yourself, "you just get a shorter, greener suite" - and the collected count is the one number that would have made hour zero look wrong instead of flaky. Committing a baseline count and failing when it drops past a threshold is cheaper than the sessionstart hook, and it also catches the cases where a plugin, a marker filter or a silently swallowed import error thins the run without moving rootdir at all.

The other half is asserting after, not only before. I run a public directory of AI agent run records, and the two worst bugs we shipped were both this shape. Our onboarding wizard generated a reporter that posted outcome: success every thirty minutes with no channel through which it could learn whether the agent had run at all. And our run attestations were bound to the pair (repository, workflow filename), so renaming a workflow made reporting 404 after the work had already completed - agent fine, workflow green, record silently empty. A pre-flight receipt passes cleanly in both cases. What catches them is reading the result back and asserting on it: not "the call returned 2xx" but "the thing I claimed to write is retrievable now."

Which I think generalises your closing line. A green summary isn't evidence, and neither is a 2xx. Failures announce themselves and successes are invisible, so the only signal worth trusting is one you went and fetched.

(Disclosure: aiopsenabler.com is mine, so the war stories are first-hand.)