Score the Host Before You Promote a Vibe-Coded Agent

A green chat is not a merge signal, because the host can hide the failure you actually care about. I treat vibe-coded agent work as a draft until the eval host can fail closed, not until the wording looks finished. Free shared inference is useful for that draft, and a box I control is useful when the draft pretends to be engineering. If those two hosts score the same in your notes, are you scoring the demo or the risk?

This is a decision guide, not a pricing rant, and not a prompt-tuning diary. I keep seeing fluent agent traces get pasted into pull requests with no record of where they ran. The interesting question is not whether the model sounded confident. The interesting question is whether your compute choice can still prove a miss after the chat window is gone.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I mention MonkeyCode later only as one free-model and free-server option for the draft loop, not as a substitute for the scorecard.

The failure I am trying to prevent

Vibe coding is a fast way to sketch tool calls, retries, and glue code you would not type by hand. Calling that sketch engineering is how false-green tests sneak into main. Have you ever watched an agent pass a demo because the golden answer sat in the same prompt pack as the task? That is not intelligence. That is leakage with good manners.

Shared free runtimes make that leakage cheaper to produce, which is fine while you are still exploring. Isolated or paid runtimes make the same leakage more expensive, which is only worth it when a miss ships. I do not pick a host by leftover GPU time. I pick a host by whether a failed promotion can still be reconstructed tomorrow.

Five questions before the host gets a vote

I score every agent spike with the same five questions, in this order, before I argue about cost. If you cannot answer them from artifacts, the chat log does not count.

1. Can a reviewer replay the eval suite without reading the original chat?
2. Does the prompt pack hash differ from the eval-suite hash on disk?
3. Are tool credentials and network egress denied unless the host is a box you control?
4. Will a failed run leave a structured trace, or only a screenshot of a happy path?
5. If the model changes under you, can you freeze the comparison or must you shrug?

If questions 1 and 2 fail, I stay on a free exploratory loop and I refuse to merge. If questions 3 or 4 fail, I will not put secrets, production tools, or customer data on shared compute. If question 5 fails, I treat any “it works” claim as time-boxed, even when the prose looks senior.

A numbered workflow I actually run

I keep the workflow boring on purpose, because excitement is how demos skip gates. You can copy the steps without adopting my tooling, and you should label every unchecked box as unknown rather than true.

1. Write the task and the eval fixtures in two files, then refuse to paste fixtures into the system prompt.
2. Hash both files before the first agent call, and store the hashes next to the run id.
3. Run the draft on free shared models only if the tools are stubs and the data is synthetic.
4. Score the host with the table below, then promote to a self-hosted or paid box when any isolation row trips.
5. Re-run the same fixtures on the new host, and compare traces, not vibes.
6. Merge only when a stranger can fail the suite without sitting in your original session.

Would I skip step 3 to go faster? Sometimes, when the spike is a throwaway. Would I skip step 6 because the chat felt done? That is the whole bug this article is about.

Artifact: a demo-to-merge scorecard

The following script is a proposal you can run locally. It does not benchmark models, and it does not prove production safety. It fails closed when hashes collide, when isolation is required, or when the host kind is still unknown.

#!/usr/bin/env python3
"""Demo-to-merge scorecard. Proposal: treat unknown fields as fail."""
from __future__ import annotations

import hashlib
import json
from pathlib import Path

ISOLATING_TOOLS = {"shell", "browser", "payments", "prod_db", "secret_store"}


def sha256_file(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def score_run(
    prompt_pack: Path,
    eval_suite: Path,
    host_kind: str,
    tools: list[str],
    has_structured_trace: bool,
    can_freeze_model: bool,
) -> dict:
    prompt_hash = sha256_file(prompt_pack)
    eval_hash = sha256_file(eval_suite)
    needs_isolation = any(t in ISOLATING_TOOLS for t in tools)
    host = host_kind.lower().strip()

    checks = {
        "eval_outside_prompt": prompt_hash != eval_hash,
        "host_declared": host in {"free_shared", "self_hosted", "paid_isolated"},
        "isolation_matches_tools": (not needs_isolation) or host != "free_shared",
        "structured_trace": has_structured_trace,
        "model_can_be_frozen": can_freeze_model or host != "free_shared",
    }
    promote = all(checks.values()) and host != "free_shared"
    stay_draft = host == "free_shared" and checks["eval_outside_prompt"] and not needs_isolation
    return {
        "prompt_hash": prompt_hash,
        "eval_hash": eval_hash,
        "checks": checks,
        "decision": "promote" if promote else ("keep_draft" if stay_draft else "fail_closed"),
    }


if __name__ == "__main__":
    result = score_run(
        prompt_pack=Path("prompt_pack.txt"),
        eval_suite=Path("eval_suite.json"),
        host_kind="free_shared",
        tools=["retriever_stub"],
        has_structured_trace=True,
        can_freeze_model=False,
    )
    print(json.dumps(result, indent=2))

Create two tiny fixtures so the script is reproducible on a laptop. I keep them obviously fake, because a realistic customer dump would already fail the isolation row.

printf 'You are a summarizer. Never read eval_suite.json.\n' > prompt_pack.txt
printf '%s\n' '{"cases":[{"id":"t1","expect":"ok"}]}' > eval_suite.json
python3 scorecard.py

If you change tools to "shell" and leave host_kind as free_shared, the decision should become fail_closed. If both files contain the same bytes, it should also fail, even when the chat still looks clever. That is the point of hashing before arguing about prompts.

Tradeoffs I write down, not vibes I remember

I use this table when a teammate wants to stay on free shared compute because the demo already “works.” Working in a chat is one column. Promoting is another column. If you cannot fill a cell, the cell is a no.

Decision axis	Free shared models / free server	Self-hosted or paid isolated box
Fit	Synthetic data, stub tools, throwaway spikes	Secrets, egress, production tools, merge candidates
Speed to first trace	Usually faster to start	Slower until images, queues, and IAM exist
Eval honesty	Easy to contaminate the prompt pack	Easier to keep fixtures off the model path
Drift	Model and neighbor load can move under you	You can freeze runtime and often the weights
Failure evidence	May be a scrollback window	Can be files, metrics, and access logs
Cost shape	Cash is low; review time is not free	Cash is higher; incident cost should be lower
Wrong fit	Shipping a vibe as a contract	Isolating a spike that should have stayed a sketch

Notice that “free” only wins the cash row. Review time, incident time, and contaminated evals are still costs. Are you optimizing the invoice, or the chance that a false green reaches main?

MonkeyCode is an open-source project with free model access and a free server option, which I use only for the draft side of that table. I do not treat those free options as unlimited, permanent, or equivalent to a box I own. If the scorecard returns keep_draft, a free loop is enough. If it returns fail_closed or you want promote, move the same fixtures onto compute you can isolate.

What this method refuses to claim

This scorecard does not name models, quote throughput, or promise that any vendor will keep a free tier. It does not replace security review, license review, or load tests. Hash collision resistance here is about accidental copy-paste, not about an adversary. A structured trace can still lie if you log the wrong fields.

I also do not claim that self-hosting makes vibe-coded agents honest. Isolation only preserves the evidence. You still have to write evals that the agent cannot see. If your suite is three happy-path strings, both hosts will flatter you.

Who should not use this approach

Skip this if you are pasting secrets into a shared chat to “just try something.” Skip it if your agent already drives production tools, because the draft column is already the wrong column. Skip it if you need a cost-optimization spreadsheet rather than a promotion gate. Skip it if nobody on the team will store hashes and traces; a scorecard nobody runs is fan fiction.

Students and weekend spikes can stay on free shared inference for a long time, and they should. Teams that merge agent output into services should not. Which group are you in when the demo looks good at 5 p.m.?

A closing check I use on my own drafts

Before I call a vibe-coded agent “done,” I ask whether a reviewer can fail it without me in the room. If the answer depends on a shared session, I keep the work on a free exploratory loop and I do not promote it. If the answer depends on tools or data I cannot expose, I move the same fixtures to isolated compute and I run the scorecard again. The host is part of the test. Pretending otherwise is how engineering gets renamed after the fact.

If you want a draft-side place to run that loop, try MonkeyCode’s free model access and free server option on synthetic fixtures only, then promote with evidence instead of a fluent screenshot.