DEV Community: SAMUELFMEDEIROS

Your MCP server works in the terminal and fails in the client — the four causes

SAMUELFMEDEIROS — Sun, 18 Oct 2026 06:23:43 +0000

“Works here” and “works in the client” are two different programs. These four
failures account for nearly every MCP server that passes the local test and dies on the first
real call — and none of them is the server's code.

1. A short connect timeout kills a long call

If the server boots a heavy backend on first use (instrumented browser, container, model), the
connect phase can take a minute. A 5-second connect_timeout drops the session
mid-call, and the message you get does not say that.

connect_timeout: 60 # not 5 — the backend boots lazily
Size the timeout by the minimum observed boot time, not by a warm call's latency.

2. A 401 that is not a dead server

The app answers its health check, the MCP returns 401. That is not downtime: it is the API
key header missing from the MCP entry. The error shows up in the tool, not in the config — so
you debug the wrong layer.

headers: X-API-Key: ${WEB_RESEARCH_API_KEY} # resolved from env, never inline
Test in two layers: (a) does the app answer? (b) is the header in the MCP entry? Different
failures, same face.

3. A config read at import is inert

You fix the config file, restart only the server, and it still fails. If the config was read at
import time, it is frozen in memory — the change only applies on the next boot of whoever
imported it. Read it at the point of use, or restart the importer.

4. Dead session mid-call: `outcome UNKNOWN`

transport session expired … outcome UNKNOWN means the server restarted under you.
The call did not complete — and “UNKNOWN” is not success. Confirm with the health check's
uptime (a few seconds = it restarted), retry once, and if it repeats, report it.

Retrying a call that already returned a definitive error burns the budget and hides the real
blocker.

Before you call it done

Verify a fresh config with the cheapest call available (a ping) before real work, and re-verify
after any restart. An MCP server that answers a ping and fails its first tool is a server you
never tested end to end.

See the Ship Your MCP Server skill
See all skills

Want this wired into your project? Guided setup from
R$ 1.500, with the first cycle run together.

← all skills

The 200 that is not a result: how to verify without fooling yourself

SAMUELFMEDEIROS — Sun, 18 Oct 2026 06:22:31 +0000

The most common way a verification lies is not by erroring: it is by going green.
These four checks cost seconds and prevent the “tested, all good” that tested nothing.

1. A 200 is not a result

An anti-bot page, a consent wall and a login redirect all return 200. Before parsing,
assert a marker you expect to see (the product name, a word from the title). If the marker is not
there, the body belongs to another page.

2. The count has to add up

The tool says total: 5 and you parsed 3. Do not report the 3 you have: re-fetch and
parse in code. A report with fewer items than reality is worse than no report, because it looks
complete.

3. Same host, another app

A hostname can point at two different applications (the last rule wins, with no error). Validate
by comparing the response body, not just the status: another app's endpoint can answer 200
in place of the one you expected.

4. Unverifiable is not verified

A command that does not exist, a service that is down, a full disk, a stuck machine: the absence
of proof is never proof of success. Write “unverifiable” and the reason, in one line —
instead of letting it look like a pass.

In a gate that becomes three states, not two: passed, failed (the only one that is
yours) and unverifiable (the machine's). Collapsing the third into “passed” makes the gate
approve without proving; collapsing it into “failed” makes the gate stop enforcing.

The test that catches this

Write the check so it fails when the verification does not happen — not one that passes by
omission. A check that only fails when the command runs and errors is measuring the command, not
the result.

See the AI Test Loop skill
See all skills

Want this wired into your project? Guided setup from
R$ 1.500, with the first cycle run together.

← all skills

It passes on your machine and breaks in CI — the causes that are not your code

SAMUELFMEDEIROS — Sun, 18 Oct 2026 05:53:00 +0000

“Works on my machine” stopped being a joke when CI became the only place where your
environment is clean. Four causes account for nearly every red CI with identical code.

1. An unpinned dependency

You changed nothing and CI broke. A dependency published a new version and your
requirements with no upper bound accepted it. The classic: a library that starts
requiring a version of another that dropped a transitive dependency.

sqlmodel>=0.0.45,<0.0.48 # the ceiling is what prevents the silent break
Without a ceiling, the next release is a scheduled date for your CI to fail.

2. The environment is not the same

Different lint version, different Python, or a package installed only on your machine. CI
installs from scratch — which is exactly where it shows you what your environment was hiding.

Reproduce before blaming CI: run the same command, in the same interpreter, in a clean
environment. If it passes clean, the difference is your environment.

3. A file that is not versioned

The test passes because the file exists on your disk — and the file is in .gitignore
(or was never added). In CI it does not exist and the test fails for a reason that looks
invented.

One-command check: list the files the test reads and confirm every one is under version
control.

4. A cache that lies

A dependency or build cache can mask the failure: the job passes using an old artifact. If the
same commit fails without the cache and passes with it, the problem is the cache — and it will come
back.

A diagnosis order that works

Read the exact annotationCI tells you file and line. Go there before
theorising.
Compare versionsInterpreter and dependencies in CI vs yours. The difference is
usually there.
Run clean locallyNew environment, no cache, same version. Reproduced? It is the
code. Not reproduced? It is the environment.

See the Ship Your MCP Server skill
See all skills

Want this wired into your project? Guided setup from
R$ 1.500, with the first cycle run together.

← all skills

Your agent said “it works”. That is an opinion, not a result

SAMUELFMEDEIROS — Sun, 18 Oct 2026 05:52:22 +0000

When the same context writes the work and approves it, “looks done” becomes the bar.
A separate critic that runs real commands and returns a 0-100 score turns that into evidence — or
names exactly what is missing.

The problem: the builder grades its own homework

An agent reviewing its own output reviews nothing: it confirms the intent it already had. The
symptoms are familiar — review rounds that never converge, “it's done” repeated, and the defect
showing up later in the hands of whoever trusted it.

The fix is not asking the model to “be rigorous”. It is moving the score out of the agent: a
script that measures with real commands, and a separate context that audits the score.

Four roles, never merged into one context

Orchestratorthe agent running the task: picks the checks, drives the rounds, fixes
infrastructure and owns the report.
Builderwhoever wrote the artifact: fixes one class of gap per pass.
Critica script you write (loop_runner.py + your checks): executes
every command, keeps the raw output and prints SCORE= and GAPS=.
Reviewera different agent/model in a fresh context, with no justification from the
builder: it can reopen the loop.

Rule zero: builder ≠ critic ≠ reviewer. If one context does all three, the loop is theatre.

The loop, step by step

Declare the checks before writing codeFor each check: name, command, weight and what
“pass” means. Weights sum to 100.
{"checks":[{"name":"pytest","cmd":"set -o pipefail; pytest -q","weight":40},
{"name":"no_secret_scan","cmd":"gitleaks detect --no-banner","weight":30}]}
Run the criticIt executes every command for real, captures the evidence and never
asks a model whether the work is good — it measures.
python scripts/loop_runner.py --config loop.json --round 1
Score, compare, decideAbove the threshold goes to the reviewer; below, take the
largest gap, fix it and run again. Cap the rounds (8 is plenty): if it never converges, the
check set is wrong, not the code.
Independent review before deliverySame artifact, fresh context, no justification
from the builder. A panel of 2-3 specialists (quality and security always) returning strict
JSON.
Security gate, fail-closedA secret in the diff or history, a high-severity finding,
an auth/path-traversal/injection issue block delivery even with perfect tests. An
inconclusive answer counts as blocked, never as pass.
Report with score, evidence and what was not verifiableThe report states what was
measured, with raw output, and what could not be checked — no “probably fine”.

What the output actually looks like

Red is the broken case (the gate blocks); green is the one that ships. Both are real runs, not an
illustration.

`$ python scripts/loop_runner.py --config loop.json --round 3
SCORE=55
GAPS=[pytest::exit_1, no_secret_scan::missing]
GATE=failed
$ echo $?
1

$ python scripts/loop_runner.py --config loop.json --round 7
SCORE=100
GATE=passed
$ echo $?
0`

Four ways a green loop lies

Piping hides the exit codepytest | tail returns the exit code of
tail (0), so the critic scores 100 with a failing test hidden in the output. Use
set -o pipefail inside every check.
A timeout shorter than the check is a silent zeroThe critic is killed by its own
timeout, the orchestrator reads score 0 with an empty gap list, and loops with nothing to fix. Set
the timeout above the real duration and always keep the critic's raw log.
A short-circuiting “pass”A server answering 200 may be the wrong server: on a port
collision the check tests a different app and fails for the wrong reason. Assert
identity (title, version, a known payload), not just reachability.
A stale baseline regenerated to go greenRegenerating a visual baseline “to pass”
erases the regression the test exists to catch. Baselines change by explicit decision, justified in
the history — never to silence a check.

What changes day to day

The point is not the score: it is the conversation. “Is it done?” stops being the builder's
opinion and becomes SCORE=100 with the command output attached — or a gap list with
names and weights. When something could not be verified, it is written down instead of turning into
a false green.

The same pattern holds for any deliverable: code, a pipeline, a data migration, a document
someone will use to decide.

Start now

The skill that does this ships as AI Test Loop — the package includes
loop_runner.py, the scoring rubric, the reviewer prompt and the security checks. The
repository is open (MIT) and the store version adds the ready-made guards.

See the AI Test Loop skill
Get the free version

Rather have it installed in your project? Guided setup from
R$ 1.500, with the first cycle run together.

← all skills

Your visual test failed. Is it the page, or the capture?

SAMUELFMEDEIROS — Sat, 17 Oct 2026 18:06:12 +0000

Your visual test failed. Is it the page, or the capture?

A screenshot test fails for two opposite reasons, and the response is opposite:

The page changed → block the release. That is what the test is for.
The page did not change, the capture is not reproducible → fix the capture, not the UI. Animation mid-frame, blinking caret, timestamp, ad slot, font fallback, scrollbar.

A gate that cannot separate the two becomes noise, and someone turns it off within a week. Keeping them apart is deterministic by construction: same inputs, same verdict. No AI in the loop.

The four verdicts

Verdict	Meaning	Action
PASS	Within the pixel budget and the region limit	Ship
FAIL	Budget exceeded, regions listed as x/y/w/h	Look at `diff.png` and the region crops
STALE	The baseline does not belong to this build (`git HEAD`, mask config or the PNG changed)	Re-baseline deliberately, or unstage the change — never "just re-run"
ERROR	Unreadable PNG or size mismatch	That is a capture bug (viewport changed?), not a regression

STALE exists because the classic failure of visual testing is not flakiness — it is a baseline quietly re-generated until the test went green. Tying the baseline to git HEAD plus a hash of the mask config turns re-baselining into a deliberate, reviewable act.

Exit codes are the contract

0 PASS   1 FAIL   2 STALE   3 ERROR

With a --json output you get the verdict, the diff percentage, the region count and every region's bounding box — enough for CI to publish what moved instead of "the test failed".

Budgets that survive contact with reality

--max-diff is a safety net, not the primary signal. Antialiasing and font hinting shift a few thousand pixels with zero real change; a real component move shifts tens of thousands. Start at 0.001 (0.1%).
--max-regions is the signal that catches small, meaningful changes. A wrong colour on a 60×40 button is ~0.26% of a 1280×720 page, but exactly one region. Budget: 1.
Masks remove whole rectangles and are reported (masked_pixels) — nobody hides a regression inside one without it showing in the report.

Determinism checklist (before blaming the gate)

Fixed viewport and DPR 1 — DPR scales the layout and invalidates every baseline
Disable animation (prefers-reduced-motion + animation:none!important)
Wait for document.fonts.ready and network idle — a fallback font diffs the whole page
Hide the caret, the scrollbar and the dev overlay
Freeze time and randomness
Mask only what is genuinely dynamic (ad slot, live counter, A/B slot)
Capture in the same container as CI — a different font stack means different pixels everywhere

The gate has to be tested in both directions

A gate tested only against "nothing changed" is decoration. The one I run has 15 tests, and both sides are asserted — including the case that bites in production: an optimised baseline (optipng, pngquant, ImageMagick) that a naive decoder refuses. The PNG decoder is stdlib (zlib + struct): a CI gate that needs a C library on every machine breaks on the machine that matters.

Full write-up with the exact commands: Your visual test failed: is it the page or the capture?

DEV Community: SAMUELFMEDEIROS

Your MCP server works in the terminal and fails in the client — the four causes

1. A short connect timeout kills a long call

2. A 401 that is not a dead server

3. A config read at import is inert

4. Dead session mid-call: outcome UNKNOWN

Before you call it done

The 200 that is not a result: how to verify without fooling yourself

1. A 200 is not a result

2. The count has to add up

3. Same host, another app

4. Unverifiable is not verified

The test that catches this

It passes on your machine and breaks in CI — the causes that are not your code

1. An unpinned dependency

2. The environment is not the same

3. A file that is not versioned

4. A cache that lies

A diagnosis order that works

Your agent said “it works”. That is an opinion, not a result

The problem: the builder grades its own homework

Four roles, never merged into one context

The loop, step by step

What the output actually looks like

Four ways a green loop lies

What changes day to day

Start now

Your visual test failed. Is it the page, or the capture?

The four verdicts

Exit codes are the contract

Budgets that survive contact with reality

Determinism checklist (before blaming the gate)

The gate has to be tested in both directions

4. Dead session mid-call: `outcome UNKNOWN`