48-Hour Field Notes: The Job Was Green. Every Live Test Skipped in Setup.

I trusted a green pytest summary for two long days, which is a habit I keep relearning the hard way. Have you ever stopped at the word passed and ignored the skip count sitting on the same line? I did, and then I treated a remote job as proof that the live client still worked. The rest of this note is what I tried, what actually broke, and what I would run again before believing another isolated runner.

Hour 0: a summary that felt like health

I had a small HTTP client and a mixed suite that I kept calling integration, mostly because the filenames sounded serious. The laptop run looked noisy with local env leftovers, so I pushed the same commit to a free remote runner and waited. Why did I expect a stranger machine to be more honest than my own shell? Because my laptop still had cached tokens, a filled .env, and a kind of luck that never shows up in CI.

The remote job printed a line that I treated like a diagnosis instead of a table of contents:

========== 12 passed, 6 skipped in 3.41s ==========

Twelve passed felt like a working client. Six skipped felt like optional fluff that busy people are allowed to ignore. Was that a reasonable reading of a test report, or just fatigue wearing a green badge?

What I thought I was testing

The suite mixed three layers, and that mix should have scared me before any remote job started:

1. Pure unit tests around URL joining, header names, and timeout defaults.
2. Contract tests that needed a recorded fixture on disk, not a live hostname.
3. Live checks that issued a real HTTP GET only when LIVE_HTTP=1 was present.

I asked an assistant to flesh out the live checks, then I ran the job on a free server so my laptop would stay out of the story. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option were in that loop for drafting tests and executing the isolated run. I am not going to invent model names, quotas, hardware, or duration, because those details are not the failure. The failure is simpler and more repeatable: a green remote job can mean we never attempted the only behavior we claimed to verify.

Hour 6: the skip was a policy, not a flake

I finally opened the long log instead of the one-line summary, which is the moment this series usually starts telling the truth. Each live test said SKIPPED [1] LIVE_HTTP is not 1 during setup, before urllib had any chance to fail. Of course they skipped. An isolated free server should not inherit my laptop environment, and this one did not. The job followed my own policy with perfect obedience, and I still called that obedience a pass.

Here is the original pattern. It looks responsible until you count outcomes instead of reading adjectives:

# tests/test_live_headers.py — example under test, not a published benchmark
import os
from urllib.error import URLError
from urllib.request import Request, urlopen

import pytest

pytestmark = pytest.mark.live


@pytest.fixture
def live_flag():
    if os.environ.get("LIVE_HTTP") != "1":
        pytest.skip("LIVE_HTTP is not 1")


def test_example_headers(live_flag):
    request = Request("https://example.com", method="GET")
    try:
        with urlopen(request, timeout=5) as response:
            status = getattr(response, "status", 200)
            header_names = {key.lower() for key in response.headers.keys()}
    except URLError as exc:
        pytest.fail(f"live request failed: {exc}")
    assert status in {200, 301, 302}
    assert "content-type" in header_names

What could possibly go wrong with a skip message that names the missing flag so clearly? Plenty, if your runner treats skip as success and you never budget a minimum number of live tests that actually executed.

Hour 14: I applied the wrong fixes first

I did the usual thrashing before I admitted the report was lying by omission rather than by assertion.

• I reran with -v and still congratulated myself when the final line stayed green.
• I wrapped urlopen in retries, as if jitter were the villain hiding in a branch that never ran.
• I pinned the standard library, which you cannot really pin, then pinned unrelated packages anyway.
• I almost copied a .env into the free server, then deleted the idea, because secrets do not belong there.

Did any of that execute the skipped tests? No, and that is the whole joke. Retries cannot fire on setup skips. A pin cannot repair a policy. And stuffing credentials onto a shared free runner would have been a worse field note than a false green bar.

The artifact: fail the session when required work was skipped

I wanted a gate that still allows skips on a laptop smoke run, but refuses a pass when I selected live coverage and none of it ran. The pieces below are a local pattern to copy, not a claim about anyone's production metrics.

Decision table

Situation	Marker	Desired job result	Why this is the honest one
Unit assertion is wrong	none	fail	The code is wrong.
Live tests selected, env unset	live	fail	You asked for live coverage.
Laptop smoke run, live not selected	live	skip quietly	Local convenience, not a release gate.
Endpoint is down	live	fail or xfail	Skip hides outages.
Recorded fixture is missing	contract	fail	Missing data is a broken suite.
Platform cannot compile a feature	posixonly	skip without --require-live	That skip is a real capability signal.

pytest.ini and conftest.py

# pytest.ini
[pytest]
markers =
    live: test that must hit a real network
    contract: test that must read a recorded fixture
addopts = -ra --strict-markers

# conftest.py — labeled example: fail when selected live tests skipped
from __future__ import annotations

import os

import pytest

MIN_LIVE_PASSED = int(os.environ.get("MIN_LIVE_PASSED", "1"))


def pytest_addoption(parser: pytest.Parser) -> None:
    parser.addoption(
        "--require-live",
        action="store_true",
        default=False,
        help="Fail when live tests skip or too few live tests pass.",
    )


def pytest_configure(config: pytest.Config) -> None:
    config.addinivalue_line(
        "markers", "live: test that must hit a real network"
    )


def _is_live(report: pytest.TestReport) -> bool:
    keywords = getattr(report, "keywords", {}) or {}
    return "live" in keywords


def pytest_sessionfinish(session: pytest.Session, exitstatus: int) -> None:
    if not session.config.getoption("--require-live"):
        return

    reporter = session.config.pluginmanager.get_plugin("terminalreporter")
    if reporter is None:
        return

    skipped = reporter.stats.get("skipped", [])
    passed = reporter.stats.get("passed", [])
    live_skipped = [rep for rep in skipped if _is_live(rep)]
    live_passed = [rep for rep in passed if _is_live(rep)]

    if live_skipped and len(live_passed) < MIN_LIVE_PASSED:
        session.exitstatus = pytest.ExitCode.TESTS_FAILED
        reporter.write_line(
            "REQUIRED LIVE TESTS MISSING: "
            f"skipped={len(live_skipped)} passed={len(live_passed)} "
            f"min={MIN_LIVE_PASSED}",
            red=True,
        )

Commands I wish I had run at hour 0

Dump the isolated runtime before you argue with pytest. The free server is allowed to disagree with your laptop, and it usually will.

python - <<'PY'
import locale, os, platform, sys
print("python", sys.version.replace("\n", " "))
print("platform", platform.platform())
print("cwd", os.getcwd())
print("LIVE_HTTP", os.environ.get("LIVE_HTTP", "<unset>"))
print("NO_NETWORK", os.environ.get("NO_NETWORK", "<unset>"))
print("https_proxy", os.environ.get("https_proxy", "<unset>"))
print("locale", locale.getlocale())
print("TZ", os.environ.get("TZ", "<unset>"))
PY

python -m pytest -m live -ra
python -m pytest -m live --require-live
LIVE_HTTP=1 python -m pytest -m live --require-live

The first pytest command is the trap I fell into, because skips still exit zero. The second command is the gate. The third command is the only one that is allowed to talk to the network, and only when you deliberately opted in.

For contract tests, stop skipping when a fixture file is absent. Missing gold files are broken packaging, not an optional feature.

from pathlib import Path
import json
import pytest

FIXTURE = Path(__file__).with_name("headers.gold.json")

@pytest.mark.contract
def test_recorded_headers():
    if not FIXTURE.exists():
        pytest.fail(f"missing fixture: {FIXTURE}")
    payload = json.loads(FIXTURE.read_text(encoding="utf-8"))
    assert "content-type" in {k.lower() for k in payload}

Hour 30: what actually broke

The client was not proven. The isolated runtime simply never received LIVE_HTTP=1, so setup skipped every test that could have caught a real regression. Pytest did the documented thing: skipped tests do not fail the session. My assistant-generated live tests defaulted to skip for safety, which is a kind default until a release gate reads that default as success. Do you want a model that refuses to call the network without a flag, or a dashboard that cannot tell refusal from proof?

I also learned that -ra is not a gate. Extra skip reasons in the terminal are documentation for a human who is still reading. I had already stopped reading at passed.

Limitations, and who should not copy this blindly

This approach is a session policy, not a network monitor, and it will punish suites that use skip as a capability probe. If you mark CPU-specific tests as live by accident, --require-live will fail honest machines. If the free server has no outbound HTTP at all, live tests should not be your only release signal; record fixtures and keep the live marker off the default path. If you need secrets for a private host, do not pour them into a shared free runner just to make a skip count go down.

Who should not use --require-live on every invocation?

• Developers running a ten-second smoke test on a laptop with no network by design.
• Teams whose skips mean "this feature was not compiled," especially native extensions.
• Anyone hoping a green live job will replace contract fixtures and schema tests.
• Anyone about to paste production tokens into a free isolated server to force a pass.

What I would repeat in the next 48 hours

I would dump environment keys, locale, and working directory on the runner before I argue with a test helper. I would select markers explicitly, then fail the session when required markers skipped below a floor. I would keep live HTTP behind an opt-in flag, and I would fail missing fixtures instead of skipping them. I would leave secrets off the free server and keep one soft reminder for myself: if you already have an isolated runner, read the skip count before you trust the green job.

Would I still use a free remote job to get a second opinion on a suite? Yes, because the second opinion is exactly that the second machine does not share my .env. The mistake was treating silence from setup as evidence from production. Green means the assertions that ran were true. It never meant the assertions you cared about had run.