Lock Config Overlay Winners, Then Move One Loader

Messy repos hide config overlay bugs for months. Defaults, files, env, and flags collide. Pin the winner table first. Then extract one loader only.

Do not start with a clean redesign. Record current wins as they stand. A later extract must match those wins.

Why overlay order breaks first

Four sources usually feed one process. Defaults live in application code. A file overrides a subset of keys. Environment variables override the file. CLI flags override the remaining keys.

Production often sets only two sources. Tests inject a third source. Local shells inject a fourth. The merge function looks trivial. The winner table is not trivial.

A rewrite will try to simplify merge rules. Simplified merge quietly changes winners. Callers keep running without errors. Resolved values still drift under them.

Artifact: the overlay winner table

Build a table before any production diff. Rows are setting keys under test. Columns are the four sources. The final cell is the observed winner.

Do not encode intended design here. Encode observed behavior, including mistakes. Wrong winners still get pinned today. Policy cleanup belongs in a later change.

key	default	file	env	cli	observed winner
log_level	INFO	DEBUG	WARNING	unset	WARNING
timeout_s	30	10	unset	5	5
feature_x	false	true	0	unset	false
config_path	./app.json	n/a	/etc/app.json	unset	/etc/app.json

Note feature_x in that table. Env 0 beat file true. That cell is a landmine. Pin the landmine before moving code.

Decision table for ugly winners

Use this table when a cell looks wrong. It tells you the next allowed move. It forbids mixing extract and policy.

observed cell	allowed in extract PR	later PR
env 0 means false	keep the truth table	document or change policy
CLI 5 stays int	keep the type	add a schema
missing file uses defaults	keep fallback	add a hard error
unknown file keys survive	keep extra keys	add an allowlist
relative path stays relative	keep the string	resolve once

If two rows need policy changes, split them. One behavior change per pull request. The extract request changes location only.

Setup the characterization harness

Use pytest and a subprocess driver. Drive the current entrypoint as a process. Do not import internals on the first pass. Importing executes side effects in messy repos.

Follow the numbered steps in order. Stop when a step fails. Do not skip the dump probe.

1. Isolate a fixture tree

Create a temp directory per test. Place one config file inside it. Clear inherited environment keys first. Pass explicit CLI argv only.

# tests/test_overlay_char.py
from __future__ import annotations

import json
import os
import subprocess
import sys
from pathlib import Path

REPO = Path(__file__).resolve().parents[1]
ENTRY = [sys.executable, str(REPO / "app.py"), "dump-settings"]

Keep the entrypoint as the process under test. dump-settings must print JSON. If that command is missing, add a probe first.

2. Add a read-only dump if missing

Probe code must not change merge logic. Print the resolved dictionary to stdout. Exit with status zero. Refuse extra side effects.

# probe fragment for app.py
if argv[:1] == ["dump-settings"]:
    settings = load_messy_settings(argv[1:])
    json.dump(settings, sys.stdout, sort_keys=True)
    raise SystemExit(0)

Label this fragment as a probe. Do not move load_messy_settings yet. The probe only makes winners visible.

3. Record one matrix cell at a time

Each test sets exactly one conflict. Local failures stay easy to read. Name tests after the winner. Do not name them after intent.

def run_dump(env, argv, config_body, tmp_path):
    cfg = tmp_path / "app.json"
    cfg.write_text(json.dumps(config_body), encoding="utf-8")
    clean = {
        "PATH": os.environ.get("PATH", ""),
        "PYTHONPATH": str(REPO),
        "CONFIG_PATH": str(cfg),
    }
    clean.update(env)
    proc = subprocess.run(
        ENTRY + argv,
        cwd=tmp_path,
        env=clean,
        capture_output=True,
        text=True,
        timeout=10,
        check=False,
    )
    assert proc.returncode == 0, proc.stderr
    return json.loads(proc.stdout)

Timeout stays short on purpose. Nonzero status fails the pin. Stderr remains part of the evidence.

4. Pin observed winners, including ugly ones

def test_env_zero_beats_file_true_for_feature_x(tmp_path):
    data = run_dump(
        env={"FEATURE_X": "0"},
        argv=[],
        config_body={"feature_x": True, "log_level": "DEBUG"},
        tmp_path=tmp_path,
    )
    assert data["feature_x"] is False
    assert data["log_level"] == "DEBUG"


def test_cli_timeout_beats_file_and_default(tmp_path):
    data = run_dump(
        env={},
        argv=["--timeout-s", "5"],
        config_body={"timeout_s": 10},
        tmp_path=tmp_path,
    )
    assert data["timeout_s"] == 5
    assert isinstance(data["timeout_s"], int)


def test_missing_file_keeps_code_defaults(tmp_path):
    data = run_dump(
        env={"CONFIG_PATH": str(tmp_path / "missing.json")},
        argv=[],
        config_body={},
        tmp_path=tmp_path,
    )
    assert data["log_level"] == "INFO"
    assert data["timeout_s"] == 30


def test_unknown_file_key_is_preserved(tmp_path):
    data = run_dump(
        env={},
        argv=[],
        config_body={"extra_flag": "keep-me"},
        tmp_path=tmp_path,
    )
    assert data["extra_flag"] == "keep-me"

These tests describe the current mess. They are not a style guide. An extract that "fixes" FEATURE_X=0 fails here. That failure is the entire point.

5. Print the table from the same dump

Keep a helper that renders winners. Commit the text beside the tests. Humans review the table, not vibes.

KEYS = ("log_level", "timeout_s", "feature_x", "config_path")


def test_overlay_table_rows_are_explicit(tmp_path):
    data = run_dump(
        env={"LOG_LEVEL": "WARNING", "FEATURE_X": "0"},
        argv=["--timeout-s", "5"],
        config_body={
            "log_level": "DEBUG",
            "timeout_s": 10,
            "feature_x": True,
        },
        tmp_path=tmp_path,
    )
    rows = {k: data[k] for k in KEYS}
    assert rows == {
        "log_level": "WARNING",
        "timeout_s": 5,
        "feature_x": False,
        "config_path": str(tmp_path / "app.json"),
    }

No golden-file rewrite switch lives here. A model cannot hide a winner change. The assertion is the source of truth.

Smallest safe change after the pin

The extract comes last. Move one function. Keep the same winners. Add no new keys.

Numbered extract rules follow.

1. Keep the dump command working.
2. Move only load_messy_settings into overlay.py.
3. Re-export the same name from app.py.
4. Run the overlay tests and stop on fail.
5. Do not normalize types in this diff.
6. Do not add a schema library in this diff.

# overlay.py — extract target
from __future__ import annotations

import json
import os
from pathlib import Path


def load_messy_settings(argv):
    defaults = {
        "log_level": "INFO",
        "timeout_s": 30,
        "feature_x": False,
        "config_path": os.environ.get("CONFIG_PATH", "./app.json"),
    }
    path = Path(defaults["config_path"])
    file_vals = {}
    if path.is_file():
        file_vals = json.loads(path.read_text(encoding="utf-8"))
    env_vals = {}
    if "LOG_LEVEL" in os.environ:
        env_vals["log_level"] = os.environ["LOG_LEVEL"]
    if "FEATURE_X" in os.environ:
        env_vals["feature_x"] = os.environ["FEATURE_X"] in {"1", "true", "True"}
    cli_vals = parse_timeout_only(argv)
    merged = {}
    merged.update(defaults)
    merged.update(file_vals)
    merged.update(env_vals)
    merged.update(cli_vals)
    merged["config_path"] = str(path)
    return merged

The FEATURE_X truth table stays ugly. 0 remains false. false remains false. Empty string remains false. That is current behavior. Do not improve it in this move.

parse_timeout_only stays a local helper. Do not invent a full CLI parser. Unknown flags keep their current fate. Pin that fate with one extra test.

Where a free coding model helps

A model can draft the extract. It cannot invent overlay winners. Feed it the table and tests.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. Use those only after overlay tests exist. Paste the winner table into the prompt. Ask for a move of load_messy_settings with no type cleanup.

Reject any patch that edits assertions. Reject any patch that adds a schema in the same diff. The server is a place to run the harness. It is not a source of truth.

Do not treat model confidence as coverage. Coverage is the matrix. Confidence is noise.

Failure analysis checklist

When a test breaks after the extract, classify the miss. Use one class per follow-up change.

1. Type drift: "5" versus 5.
2. Truthiness drift: "0" versus 0 versus False.
3. Path drift: relative versus resolved.
4. Missing-file drift: {} versus defaults.
5. Extra-key drift: file keys not in defaults.
6. CLI parse drift: unknown flags now error.

Each class gets one extra test. Do not bundle those fixes. Type drift is a separate change. Policy change is a third change.

Sample commands for the miss classes:

python -m pytest tests/test_overlay_char.py -q
python -m pytest tests/test_overlay_char.py -q -k feature_x
python -m pytest tests/test_overlay_char.py -q -k timeout
python -m pytest tests/test_overlay_char.py -q -k missing_file

Read the assertion diff before opening an editor. If types drifted, restore the old type. If winners drifted, restore the old merge order.

Limitations

This harness does not prove correctness. It proves stability of current winners. Stable-wrong overlay still ships to production.

Subprocess tests miss in-process monkeypatches. They also miss import-time reads. If os.environ is sampled at import, pin that path first. This article does not cover import-time I/O.

JSON files only in this matrix. TOML, YAML, and INI need extra fixtures. Nested keys need dotted rows. List merge policy is undefined here. Do not invent a list policy.

Ten second timeouts hide hung network calls. If settings fetch remote flags, stub that network first. Characterization needs a local dump path.

Boolean env parsing is language specific. Python truth tables are not shell truth tables. Do not copy this matrix into another runtime.

Who should not use this

Skip this workflow for greenfield apps. Write an explicit schema first. Do not characterize fiction.

Skip it for secret material. Do not dump tokens into assertions. Redact or hash secret keys. Better: never print them.

Skip it when the process cannot boot offline. Fix boot first. Characterization needs a local dump.

Skip it if a policy rewrite sits in the same PR. Split that work. Pins go first. Policy goes second.

Run book

python -m pytest tests/test_overlay_char.py -q
git add tests/test_overlay_char.py overlay.py
git diff --stat
python -m pytest tests/test_overlay_char.py -q

The extract is done when the matrix is green. The overlay is not clean. It is pinned. Clean remains a later pull request.

If you pin a messy overlay, paste the winner table. Leave the cleanup for a second PR.