A remote apply sits on the next maintenance window. The dry-run log is long enough that nobody reads it linearly, so an assistant is asked for a summary. The first summary names a missing region and a one-line config change. A retry comes back with the same region and the same line.
In the channel, agreement is treated as confirmation. It is not confirmation. Both calls read the same excerpt, under the same instructions, with the same missing tail. Two readers who shared one notebook will often tell the same incomplete story.
Stable agreement can be a stable omission. This note is a 48-hour evaluation plan for that failure mode, not a diary of a finished run. No timings, token totals, or model names are stated, because no primary source for those figures was attached to this draft. The commands below are a proposal. Run them before you trust any blank they leave.
What the second read actually is
A retry feels like a control. In practice it is a second draw from the same setup, conditioned on the same truncated input. If the last 200 lines never reached the model, both summaries can omit the line that contradicts the one-line fix.
Neighboring misses have the same shape: a predicate dropped on parse, a header never forwarded, a dry run aimed at the wrong database. This gate is narrower than those postmortems. It only decides whether two summaries may count as evidence.
Evidence here means files, not a feeling. The log bytes are frozen and hashed before either call. Each summary is JSON on disk, not a scrollback paste. Required fields are present, or the run fails closed.
Disagreement on a required field blocks any apply recommendation. Agreement does not unlock apply. It only means the samples did not contradict each other on the keys you named. A free assistant can be a second reader of a frozen fixture. It still cannot be the hand on the apply button.
Where a disposable host fits
The comparison should not run on the production runner. The host that holds apply credentials is the wrong place to iterate on prompts. A separate machine, with no cloud keys and no write path to the target account, is enough. Disk for one fixture, a Python 3 install, and a directory of summary files will do.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode enters the method only as an operator-supplied option for free model access and a free server, which can serve as that disposable machine. This draft does not name models, quotas, hardware sizes, or duration. Those details change, and they were not checked against a primary document here. Read the current project documentation before you reserve a day around them. If the docs do not state a limit, do not write one into the runbook from memory.
A team that needs a contractual capacity number should skip the free option and use a billed environment. Free access is a convenience for a bounded experiment. It is not a capacity plan, and it is not a reason to move secrets onto a host you do not administer long term.
Freeze the bytes before anyone summarizes them
Start the clock when the log is copied, not when the assistant replies. Hour 0 is a hash. If the hash check fails, stop. A summary of a file you cannot name byte-for-byte is a story about a moving target.
The block below builds a synthetic 12-line fixture so the drill can be repeated without a production log. It is not a captured incident.
install -d -m 700 ./evidence
python3 - <<'PY'
from pathlib import Path
lines = [f"plan line {i}: ok\n" for i in range(1, 12)]
lines[0] = "plan line 1: command terraform plan -input=false\n"
lines.append("plan line 12: Error: region is required\n")
Path("./evidence/dry-run.log").write_text("".join(lines), encoding="utf-8")
PY
sha256sum ./evidence/dry-run.log | tee ./evidence/dry-run.log.sha256
sha256sum -c ./evidence/dry-run.log.sha256
From hour 0 to hour 2, write the schema you will accept. Keep it small enough to verify by hand. A useful summary names the failing command, the exit code as printed, the first contradicting line number, and whether the log tail was in the context window. It does not propose a patch. Patches belong in a later review, with a diff, after this gate has passed.
Between hour 2 and hour 8, write two summaries to summary-a.json and summary-b.json. Use only model access you have confirmed in the current docs. Do not paste credentials to buy a cleaner answer. If the log holds tokens, connection strings, or customer rows, redact first, or do not send it to an external model at all.
A synthetic pair that should agree looks like this. Swap one field in the second file when you want to watch the gate fail.
{
"failing_command": "terraform plan -input=false",
"printed_exit_code": 1,
"contradicting_line": 12,
"tail_included": true
}
The harness that fails closed
The script does not open a socket and does not apply a change. It checks the hash, required keys, a boolean tail flag, integer line bounds, and disagreement. It does not parse the log to prove that failing_command or printed_exit_code actually appear. That proof stays manual in this version, so the automation cannot outrun the reader. The example is unexecuted in this draft.
#!/usr/bin/env python3
"""Proposal: compare two assistant summaries of one frozen log."""
import hashlib
import json
import sys
from pathlib import Path
REQUIRED = (
"failing_command",
"printed_exit_code",
"contradicting_line",
"tail_included",
)
def sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(65536), b""):
digest.update(chunk)
return digest.hexdigest()
def load_summary(path: Path) -> dict:
data = json.loads(path.read_text(encoding="utf-8"))
if not isinstance(data, dict):
raise SystemExit(f"{path}: summary must be a JSON object")
missing = [key for key in REQUIRED if key not in data]
if missing:
raise SystemExit(f"{path}: missing {missing}")
if data["tail_included"] is not True:
raise SystemExit(f"{path}: tail_included is not true")
if not isinstance(data["contradicting_line"], int):
raise SystemExit(f"{path}: contradicting_line must be an int")
return {key: data[key] for key in REQUIRED}
def line_exists(log_path: Path, line_no: int) -> bool:
if line_no < 1:
return False
with log_path.open("rb") as handle:
for n, _ in enumerate(handle, start=1):
if n == line_no:
return True
return False
def main() -> int:
if len(sys.argv) != 5:
print(
"usage: compare_summaries.py LOG SHA_FILE SUMMARY_A SUMMARY_B",
file=sys.stderr,
)
return 2
log_path, sha_path, a_path, b_path = map(Path, sys.argv[1:])
expected = sha_path.read_text(encoding="utf-8").split()[0]
actual = sha256(log_path)
if actual != expected:
print(f"hash mismatch: {actual} != {expected}")
return 1
left, right = load_summary(a_path), load_summary(b_path)
for label, summary in (("A", left), ("B", right)):
if not line_exists(log_path, summary["contradicting_line"]):
print(f"{label}: contradicting_line out of range")
return 1
diffs = [key for key in REQUIRED if left[key] != right[key]]
if diffs:
print("disagreement on: " + ", ".join(diffs))
return 1
print("samples agree on required fields; this is not an apply approval")
return 0
if __name__ == "__main__":
raise SystemExit(main())
Invoke it only after both files exist.
python3 compare_summaries.py \
./evidence/dry-run.log \
./evidence/dry-run.log.sha256 \
./evidence/summary-a.json \
./evidence/summary-b.json
echo "exit=$?"
Exit 0 means the samples matched on the required keys, both claimed the tail was included, and both line numbers exist. It does not mean the summary is true. Exit 1 means you do not carry this summary into an apply review. Exit 2 means the command was invoked wrong, which is also not a pass.
From hour 8 to hour 24, read every disagreement yourself. If the files agree, open the log at contradicting_line and check the words. On the synthetic fixture, line 12 should be the region error. If a live log's cited line talks about something else, the samples are jointly wrong, and the script will not save you. That is a limitation, not a footnote to skip.
Break the fixture on purpose
Hours 24 to 36 are the negative tests. Copy the frozen log, delete the last 50 lines of a real fixture, or delete line 12 of the synthetic one, and request a new summary without saying you truncated it. If tail_included comes back true, that field is theater. Drop it, or redefine it as a byte offset the harness can check against file size.
Then edit one required value in summary-b.json and confirm the script exits 1. A gate you have never seen fail is a slogan. After that, restore the fixture from the hash, not from memory. If sha256sum -c fails, you are no longer testing the same incident.
Hours 36 to 48 belong to the host. If you used a disposable server, wipe it. Confirm the evidence directory has no live credentials. Write down what you would repeat. Repeat the hash, the schema, the two-file compare, and the truncation drill. Do not repeat the habit of pasting a chat reply into a ticket and calling it confirmation.
A lab notebook for the two days should stay boring. Record the hash, the clock time of each summary file, the script exit code, and the line you opened by hand. Leave out adjectives about the model. If those four columns are empty, you do not have a result, and the apply review should say so.
What the protocol will not decide
It will not rank models. It will not estimate token cost, and this draft will not publish a quota it cannot cite. It will not catch a wrong but shared reading of a line that does exist. It will not replace the person who can read the failing command in context.
Think of the gate as two matching checksums before you call a download intact. The match rules out one class of accident. It does not prove the file is the one you wanted.
Leave this approach alone if the log holds secrets you cannot redact, if apply credentials would have to live on the same host, or if your process already demands a named reviewer and a signed diff. Do not treat a free server as a long-lived environment. Do not schedule a production apply on the hope that a free tier will still answer on Friday. If you need prompt text, model version, and retention in an audit log, use a stack where those fields are actually recorded.
A disposable machine is still the right shape for the comparison, because the production runner is the wrong place to learn how your schema fails. If that machine is the only missing piece, MonkeyCode's free server option is one place to try the harness after you read what the option includes today. Keep the apply on a different host, with a different credential. The second summary can inform the review. It does not get to close it.
Top comments (0)