A Model Fallback PR: What to Trust, What to Revert, What to Test

A fallback PR lands at 09:12 on a Monday. CI is green and the description reads "adds model fallback for reliability." The diff spans 340 lines across four files.

The reviewer has one hour before the release branch closes. This is a scoping problem, not a coding problem.

The walkthrough below uses a constructed diff, written for teaching. It is not a real submission, and it carries no production metrics. The goal is a repeatable triage order: trust the boring parts, revert the dangerous parts, test whatever survives.

1. Slice the diff before reading it

Wide diffs hide risk in the parts nobody opens. Three commands shrink the review surface to something reviewable.

git fetch origin pull/412/head:pr-412
git diff --stat main...pr-412
git diff main...pr-412 -- 'services/*' 'tools/*' | grep -nE 'except|retry|sleep|charge|budget|fallback'

The first command isolates the branch. The second shows which files actually changed size. The third surfaces control-flow keywords, which is where fallback logic usually breaks.

A 340-line diff often reduces to 60 lines of real behavior. Review those 60 lines twice and skim the rest once.

2. The proposed fallback loop

The core hunk looks reasonable at a glance.

# tools/model_client.py (excerpt from the constructed diff)
import time

def complete(prompt, tools, models, budget):
    for model in models:
        for attempt in range(3):
            try:
                resp = model.call(prompt, tools=tools)
                budget.charge(resp.usage)      # charged after the call
                return resp
            except Exception:                  # too broad
                time.sleep(2 ** attempt)       # retry storm
    return None                                # silent failure

Four defects sit in ten lines. The bare except catches BudgetExceeded and keeps spending. The retry loop re-sends tool calls that may not be idempotent. The budget check runs after the spend. The None return erases which models failed.

3. What to trust

Some parts of an agent PR rarely need line-by-line scrutiny. Trust them provisionally, then move on.

1. Type signatures with no default values.
2. Logging fields that match existing schema names.
3. Config additions that mirror an existing block.
4. Test scaffolding that reuses the project's fixtures.
5. Renames with no behavior change, verified by git diff -M.

Trust here means "do not spend review minutes." It does not mean "approve unreviewed." Each item still needs one confirming check, usually a grep against the old call sites.

grep -rn "complete(" --include="*.py" services/ tools/ | grep -v tests/

If every call site already passes models= and budget=, the signature change is mechanical. If one call site still passes three arguments, the PR is incomplete.

4. What to revert before discussion

Revert first and debate later when a hunk violates a safety invariant. Five patterns qualify.

Pattern in diff	Why revert	Cheap replacement
except Exception around spend paths	Swallows budget and auth errors	Catch TimeoutError and ConnectionError only
Retry on non-idempotent tool calls	Duplicate side effects	Retry read-only tools, or pass an idempotency key
Post-call budget check	Overspend before refusal	Pre-flight estimate, then charge actual
return None on exhaustion	Hides failure cause	Raise AllModelsFailed(errors=[...])
time.sleep(2 ** attempt) unbounded	Retry storms under outage	Cap total wall time, not attempt count

Revert is a review action, not a rejection. The author keeps the structure and fixes three functions.

5. What to test with a fake client

The PR claims "improves reliability." That claim is testable without any network call. A fake model client makes the fallback path deterministic.

# tests/test_fallback.py
import pytest
from tools.model_client import complete, AllModelsFailed

class FakeModel:
    def __init__(self, name, failures=0, usage=10):
        self.name, self.failures, self.usage, self.calls = name, failures, usage, 0

    def call(self, prompt, tools=None):
        self.calls += 1
        if self.calls <= self.failures:
            raise TimeoutError(f"{self.name} timed out")
        return type("Resp", (), {"usage": self.usage, "model": self.name})()

def test_no_fallback_after_side_effecting_tool_call():
    primary, secondary = FakeModel("primary", failures=1), FakeModel("secondary")
    with pytest.raises(TimeoutError):
        complete("charge", tools=[charge_card], models=[primary, secondary], budget=None)
    assert secondary.calls == 0

Run the suite with pytest -q tests/test_fallback.py. Three more cases belong beside it. Budget exhaustion must not retry. Exhaustion must raise a typed error. The winning model name must appear in the response.

Write the failing test before requesting changes. A red test is a stronger review comment than a paragraph of prose.

6. Replaying the suite on free infrastructure

Cost usually blocks reviewers from rerunning a model-touching suite. MonkeyCode covers that gap with two operator-supplied options: free model access and a free server option. The operator states that access includes 10 million free tokens, and that the server option does not require a paid plan.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The relevant use here is narrow. The fake-client tests above need no model at all. The replay pass, which sends recorded prompts through real models, is what consumes tokens. Running that replay on the free tier keeps a review harness affordable for one reviewer.

Treat every quota claim as unverified until read from the vendor's own terms. This article makes no claim about model names, throughput, hardware, or duration. Those details change, and stale numbers make reviews worse.

A practical split: fake-client tests in CI, replay pass by hand before approval. Keep replay output as an artifact attached to the PR, not as a comment thread.

7. Decide with a written rule

Reviews drift when the standard lives in someone's head. Write the rule down once.

• Revert any hunk that retries a non-idempotent call.
• Revert any spend path that catches broad exceptions.
• Test any claim containing "reliability", "fallback", or "retry".
• Trust mechanical changes only after a call-site grep.
• Approve only when the failing tests from step 5 pass.

8. Limitations and who should skip this

This workflow fits PRs that touch model routing, retries, or budgets. It fits teams that can run a fake client in CI.

It does not fit three cases. Skip it when the PR is purely cosmetic. Skip it when the team has no budget abstraction to test against. Skip it when reviewers cannot attach artifacts to the PR.

Also skip the free-tier replay for anything sensitive. Recorded prompts may contain customer text. Scrub them first, or keep the entire replay local.

The triage order matters more than the tooling. Slice the diff, revert the unsafe hunks, then test the remaining claims.

Readers who want to rerun this harness can start from the project's free tier, attach their own replay artifacts, and compare notes on what their last fallback PR actually changed.