DEV Community

Morgan Xu
Morgan Xu

Posted on

Postmortem: git apply || true Accepted an HTML Error Page

#ai #testing #git #devops

Empty applies make CI lie with a straight face.
A remote agent job can return HTTP 200.
The saved body can still be an error page.

Tests then run on the unchanged tree and pass.
The merge gate records a green agent run.
Reviewers see success and stop reading the patch.

This postmortem reconstructs that failure in a lab.
It is a labeled example, not a production claim.
The durable fix inspects the tree, not the status.

Incident summary

The job asked a remote coding agent for a unified diff.
The client used curl without --fail.
A cold runner answered with an HTML waiting page.

The apply step was written as git apply || true.
That line treated a refused patch as success.
Unit tests then exercised the pinned baseline and passed.

Scope and labels

  • Label this writeup as a lab reconstruction only.
  • Label commands below as unexecuted against private prod data.
  • Control every run with a pinned pull request SHA.
  • Leave model ranking and latency claims out of scope.

The reconstruction uses public git and curl flags.
No model names are required for the gate.
No quota, hardware, or uptime numbers appear here.

Timeline

  1. Operator pins PR_SHA from the pull request head.
  2. A fresh worktree is created at that exact SHA.
  3. The agent job is posted to a remote runner URL.
  4. curl writes HTTP 200 HTML into agent.patch.
  5. git apply agent.patch || true returns a zero exit.
  6. Unit tests run and pass on the untouched baseline.
  7. The gate publishes a green status on the pull request.
  8. A later retry finally returns a real unified diff.
  9. That later diff fails git apply --check on PR_SHA.

The dangerous window is step four through step seven.
A zero exit from a guarded apply is not evidence.
It can mean the wrapper swallowed a hard git error.

What the files looked like

The first stored body started with a doctype line.
It contained no diff --git header at all.
It contained no @@ hunk header either.

git apply printed No valid patches in input.
The || true wrapper discarded that non-zero status.
git write-tree before and after matched exactly.

A later genuine patch did contain both markers.
It targeted files the pull request did not touch.
git apply --check rejected it on the pinned SHA.

Contributing factors

HTTP 200 was treated as a patch

Status codes do not classify response bodies.
A waiting page can still arrive as 200 OK.
A text/html content type is a failed agent job.

curl without --fail exits zero on many 5xx pages.
The file size looked non-empty, so the next step ran.
Non-empty is not the same as a unified diff.

The apply line failed open

git apply || true was added during a flaky week.
The intent was to survive whitespace noise.
The effect was to ignore missing patches entirely.

git apply already fails closed on HTML input.
The wrapper inverted that safe default.
Continue-on-error flags in YAML have the same shape.

Tests ran on an unchanged tree

Green tests on PR_SHA with no diff prove nothing new.
They only restate that the branch already built.
A no-op success must fail closed unless emptiness was requested.

Agent status JSON outranked git

The runner posted a small { "ok": true } object.
The merge rule scored that JSON blob.
It never scored git write-tree against PR_SHA.

The remote clone used the default branch

The runner checked out the repository default branch name.
The pull request SHA was only pasted into the prompt.
The model later diffed main, not the pinned commit.

This last factor is independent of HTML bodies.
It still produces a patch that CI cannot apply.
Both failures share one root: the gate trusted the agent.

Durable fix

The gate should ignore agent prose entirely.
It should keep status JSON only as a debug log.
It should require a real apply on a detached SHA.

Pin the tree before any fetch 

#!/usr/bin/env bash
set -euo pipefail

PR_SHA="${PR_SHA:?PR_SHA is required}"
git fetch --quiet origin "$PR_SHA"
git switch --detach --quiet "$PR_SHA"
BEFORE="$(git rev-parse HEAD)"
test "$BEFORE" = "$PR_SHA"
Enter fullscreen mode Exit fullscreen mode

Detached HEAD removes accidental default-branch drift.
The equality test catches a truncated fetch SHA.
Run this block before the model is invoked.

Fetch the body with fail-closed HTTP 

#!/usr/bin/env bash
set -euo pipefail

URL="${1:?url required}"
OUT="${2:?output path required}"

curl --fail --silent --show-error \
  --header 'Accept: text/plain, text/x-diff, text/x-patch' \
  --output "$OUT" \
  "$URL"

CTYPE="$(file -b --mime-type "$OUT")"
case "$CTYPE" in
  text/plain|text/x-diff|text/x-patch) ;;
  *) echo "refusing content type $CTYPE" >&2; exit 2 ;;
esac
Enter fullscreen mode Exit fullscreen mode

--fail turns HTTP 4xx and 5xx into a non-zero exit.
file is a second check, not a full parser.
Keep both. Either one can miss a case.

Reject non-diffs before git sees them 

#!/usr/bin/env bash
set -euo pipefail

PATCH="${1:?patch path required}"

if grep -Ei -q '<!doctype html|<html' "$PATCH"; then
  echo "refusing HTML body" >&2
  exit 2
fi

if ! grep -q '^diff --git ' "$PATCH"; then
  echo "refusing body without diff --git" >&2
  exit 2
fi

if ! grep -q '^@@ ' "$PATCH"; then
  echo "refusing body without hunk headers" >&2
  exit 2
fi
Enter fullscreen mode Exit fullscreen mode

These checks are cheap and boring on purpose.
They exist because apply wrappers get too clever.
Do not restore || true after they land.

Apply, then compare tree hashes 

#!/usr/bin/env bash
set -euo pipefail

PATCH="${1:?patch path required}"
BEFORE="$(git write-tree)"

git apply --check --whitespace=nowarn "$PATCH"
git apply --index --whitespace=nowarn "$PATCH"

AFTER="$(git write-tree)"
if [[ "$BEFORE" == "$AFTER" ]]; then
  echo "apply produced an empty tree delta" >&2
  exit 3
fi

git diff --cached --stat
Enter fullscreen mode Exit fullscreen mode

--check must run without a fallback operator.
--index updates the index so write-tree can see it.
Equal hashes mean the job did not change the tree.

Decision table

Observation Treat as Gate result
HTML, bad MIME, or missing diff --git invalid artifact fail closed
git apply --check non-zero wrong base or drift fail closed
apply ok, tree hash unchanged no-op or swallowed error fail closed
apply ok, tests fail real patch, bad change fail closed
apply ok, tree changed, tests pass candidate human review

The last row is not an autoland rule.
It is only permission to review the candidate.
Autoland still needs owners, size limits, and path rules.

Minimal fixture test

The following fixtures exist only as lab files.
They are not harvested from a private outage.

fixtures/waiting.html:

<!DOCTYPE html>
<html><body>model warming</body></html>
Enter fullscreen mode Exit fullscreen mode

fixtures/ok.patch:

diff --git a/README.md b/README.md
index 1111111..2222222 100644
--- a/README.md
+++ b/README.md
@@ -1,2 +1,3 @@
 # demo
+# gate
Enter fullscreen mode Exit fullscreen mode

Run this plan on a throwaway clone.

  1. Feed waiting.html into the validator and expect exit 2.
  2. Feed an empty file into the validator and expect exit 2.
  3. Feed ok.patch on a README that lacks the added line.
  4. Expect a new tree hash and a non-empty git diff --cached.
  5. Feed ok.patch again and expect git apply --check to fail.
  6. Confirm tests are skipped on validator exits 2 and 3. 
# labeled example commands, not a recorded outage
chmod +x gate_http.sh gate_validate.sh gate_apply.sh
./gate_validate.sh fixtures/waiting.html; echo "html=$?"
./gate_validate.sh fixtures/ok.patch; echo "diff=$?"
Enter fullscreen mode Exit fullscreen mode

Do not pipe the HTML fixture into git apply || true.
That command is the bug under test.
Keep it in a broken-control branch for regression proof.

Where a free remote runner fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A lab still needs a place to run the agent job.
MonkeyCode offers free model access and a free server option.
Those two options help reproduce the fetch-and-apply path.

They avoid standing up private GPU capacity for the lab.
They do not classify HTTP bodies as diffs.
They do not pin PR_SHA or replace git apply --check.

The gate above stays useful if product names disappear.
The failure mode is git, curl, and fail-open wrappers.
Remote free runners simply make cold HTML bodies more common.

Limitations

This fix does not catch semantic bugs in a valid diff.
A well-formed patch can still delete the test file.
A tree-hash change is necessary, not sufficient.

git apply also struggles with rename-heavy patches.
Some agents emit git diff --no-prefix output.
The validator must document the expected diff dialect.

Shared free servers add queue delay and cold starts.
This article does not measure that queue delay.
Do not treat availability as a capacity plan.

HTML detection remains a heuristic on the body text.
A patch could mention <html> inside a fixture.
Prefer MIME type plus diff --git plus hunk count.

Who should not use this approach

  • Teams that never fetch agent output over HTTP in CI.
  • Pipelines that land model output without human review.
  • Jobs that cannot check out the pull request SHA cleanly.
  • Orgs that need a contractual SLA instead of a free runner.

Do not send secrets to a free remote server.
Do not upload proprietary patches to an unreviewed endpoint.
Do not skip path allowlists because the apply succeeded.

Closing

Score the tree, not the status object.
Refuse HTML, empty bodies, and swallowed apply errors.
Pin the SHA before the remote job starts.

Teams that already pin SHAs can reuse this gate on a free remote runner.
The remote runner remains optional for this gate.
The apply check is required on every agent patch.

Top comments (0)