Rank Agent Runtimes by the Contract You Can Enforce

Price tags still lie about agent runtimes, and I keep learning that lesson the hard way. The useful question is not which box is free today, but which contract I can actually enforce tomorrow. Can I bound latency, keep secrets off shared disks, own the transcript, and retry without burning the budget? If those four checks fail, I refuse the complimentary endpoint, even when the model quality looks fine.

Why sticker price is a weak ranking signal

Shared free compute looks attractive when a prototype still changes shape every single afternoon, right? Paid APIs and self-hosted boxes look expensive until a leaked token or irreproducible trace appears. I do not treat those three options as a quality ranking; I treat them as contracts I can or cannot test. The public argument keeps circling model IQ, while my incidents usually start in the runtime contract instead.

A free shared workspace can still be the correct runtime for a throwaway exploration job today. It just has to survive the same checks I would write for a paid or isolated cluster. If I cannot express a requirement as a failing test, I do not get to call it a requirement. That sounds pedantic until a teammate cannot replay last Tuesday's tool call from memory.

Four surfaces I actually write down

I keep the contract tiny on purpose, because a twelve-page policy never actually runs in CI. Each surface maps to one executable check, and I reject runtimes that cannot fail those checks loudly. Does that feel too strict for a weekend prototype that nobody else will supposedly touch? Maybe so, but the prototype is exactly when secrets and logs get copied onto the wrong disk.

1. Latency bound. The run must terminate or abort inside a declared wall-clock budget.
2. Secret boundary. Values marked secret must not appear in traces, shells, or shared volumes.
3. Transcript ownership. I must export a hashed, timestamped log that I still control after exit.
4. Retry budget. A flaky tool call may retry N times, then stop, without silent amplification.

Those four checks do not cover safety, evaluation quality, fairness, or any prompt injection defense at all. They cover the boring failures that make free compute look cheap until a Friday rollback. If a provider cannot help me test even one of them, I stop arguing about tokens per dollar.

A numbered workflow I actually follow

Here is the sequence I use before I attach any model endpoint, free or not.

Step 1: Freeze the contract as YAML

I write the contract first, in a file the agent runner can parse without ceremony. The numbers below are placeholders for your team, not measurements I am claiming as evidence. Would I ship those numbers into production without measuring my own jobs against them first?

# agent_compute_contract.yaml — proposal, not a measured SLA
latency:
  wall_clock_seconds: 90
  on_timeout: abort
secret_boundary:
  deny_patterns:
    - "API_KEY"
    - "PRIVATE"
  scan_targets: ["stdout", "trace.json", "workspace"]
transcript:
  export_path: "./artifacts/transcript.json"
  hash_alg: sha256
  retain_locally: true
retry:
  max_attempts: 2
  backoff_seconds: 3
  amplify: false

Those placeholder values exist so the next step has something concrete to fail against later. Change them when your jobs are longer, chatty, or allowed to retry with side effects. Please do not copy them into a status page and then call the result an SLA. The file is a test fixture, and that is the entire point of writing it down.

Step 2: Run a contract probe before the agent loop

I treat the probe as a gate for promotion, not as a benchmark of model quality. The script below is a labeled example you should adapt rather than paste into production. It does not claim any vendor result, quota, hardware profile, or any measured latency distribution either.

# contract_probe.py — example probe, not executed against a vendor here
from __future__ import annotations

import hashlib
import json
import os
import time
from pathlib import Path

CONTRACT = {
    "wall_clock_seconds": 90,
    "max_attempts": 2,
    "export_path": Path("artifacts/transcript.json"),
}


def hash_file(path: Path) -> str:
    digest = hashlib.sha256()
    digest.update(path.read_bytes())
    return digest.hexdigest()


def probe_timeout(run_agent, payload: dict) -> dict:
    started = time.monotonic()
    try:
        result = run_agent(payload, timeout=CONTRACT["wall_clock_seconds"])
        status = "ok"
    except TimeoutError:
        result = {"error": "timeout"}
        status = "timeout"
    elapsed = time.monotonic() - started
    return {"status": status, "elapsed": elapsed, "result": result}


def probe_secrets(trace_text: str) -> list[str]:
    leaks = []
    for key, value in os.environ.items():
        if not value:
            continue
        if "KEY" in key or "TOKEN" in key or "SECRET" in key:
            if value in trace_text:
                leaks.append(key)
    return leaks


def probe_transcript(payload: dict, raw: bytes) -> str:
    CONTRACT["export_path"].parent.mkdir(parents=True, exist_ok=True)
    CONTRACT["export_path"].write_bytes(raw)
    return hash_file(CONTRACT["export_path"])


def decide(leaks: list[str], elapsed: float, attempts: int) -> str:
    if leaks:
        return "reject-shared-or-fix-redaction"
    if elapsed > CONTRACT["wall_clock_seconds"]:
        return "reject-or-move-isolated"
    if attempts > CONTRACT["max_attempts"]:
        return "reject-retry-amplification"
    return "accept-candidate"


if __name__ == "__main__":
    # Example driver. Wire run_agent to your runner; this stub stays local.
    artifacts = Path("artifacts")
    artifacts.mkdir(parents=True, exist_ok=True)
    fake_trace = "ok"
    leaks = probe_secrets(fake_trace)
    elapsed = 0.0
    attempts = 1
    decision = decide(leaks, elapsed, attempts)
    raw = json.dumps({"trace": fake_trace, "decision": decision}, indent=2).encode()
    digest = probe_transcript({"demo": True}, raw)
    payload = {
        "decision": decision,
        "leaks": leaks,
        "elapsed": elapsed,
        "transcript_sha256": digest,
    }
    (artifacts / "probe.json").write_text(json.dumps(payload, indent=2))
    print(json.dumps(payload, indent=2))

I run that probe against every candidate runtime, including the free one in my bookmarks. If the probe cannot even start, that absence is already a decision about the contract. Why would I debug an agent loop on a box that cannot export a hashed transcript? I would not, and that early refusal is cheaper than almost any later migration story.

Step 3: Score runtimes with a fit matrix, not a vibe

I map each failed probe onto a runtime change instead of delivering another motivational speech. The table is the artifact I want you to copy, argue with, and then keep in git. Can a vibe in a Slack thread replace a row that already failed in CI? Not on my team, because vibes do not leave a hash next to the transcript.

Contract check	Passes on free shared compute	Typical paid API fit	Typical self-hosted fit	Move when it fails
Latency bound	Only if queue delay stays inside the budget	Useful when an abort path is documented	Strongest when you own the queue	Isolated runtime or smaller jobs
Secret boundary	Risky if the workspace is shared	Better if the vendor redacts traces	Strongest if secrets never leave the box	Self-hosted or a local runner
Transcript ownership	Weak if logs remain trapped in a UI	Medium if a real export path exists	Strong if your process writes the files	Any runtime that cannot export
Retry budget	Weak if hidden retries already exist	Medium if you cap attempts yourself	Strong if you wrap the whole loop	Disable auto-retry, then retest

Notice I still keep a column for free shared compute rather than deleting it in panic. I am not allergic to free resources, but I am allergic to untestable complimentary runtimes. If every row passes, the cheap box wins until a new secret or side effect appears. If one row fails, the discussion is about that row, not about brand loyalty.

Step 4: Attach a model only after the runtime survives

Only now do I care which model endpoint sits on the other side of the runner. If the contract passes on a free shared server, I use it for exploration I can throw away. If secret boundary or transcript ownership fails, I stop sending extra prompts to that box. I move the same probe to a paid API or a machine I control, then I compare hashes.

That sequence is the whole decision guide, and everything after it is only supporting detail. People still want a winner among free, paid, and self-hosted, and I do not have one. I only have a function that maps each failed check onto the next runtime class I will try. Is that less exciting than a leaderboard? Yes, and that is why it survives contact with incidents.

Where a free model and free server actually help

I still want a cheap runtime for developing the contract itself, before any invoice exists. Writing probes only against a billable API teaches invoice anxiety instead of teaching actual isolation. A complimentary model endpoint plus a free server helps when the work is breaking the probe on purpose. The agent loop should remain a small repo while those failures are still cheap to repeat.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is one candidate I put in that cheap runtime, because it offers free model access and a free server option. I do not treat that offer as a permanent SLA, a hardware spec, or a ranking against paid labs. I treat it as a place to fail the contract quickly while the runner still fits in git. If you need a complimentary model-and-server target for the probe, that is the only role I am describing.

Fit criteria I use in review meetings

When a teammate asks why we skipped the free box, I do not answer with a speech. I answer with the matrix row that failed, plus the probe output sitting in artifacts. Does that feel cold in a planning meeting that really wanted a narrative instead of a table? It keeps the conversation technical, which is the only reason I write contracts as tests.

Use the free shared option when all of the following are true:

1. The job has no production secrets, or secrets are injected only on your side of the probe.
2. You can export a transcript you hash and store in your own artifacts directory.
3. A timeout abort is acceptable, and a noisy-neighbor delay will not page anyone.
4. Hidden retries cannot duplicate side effects such as emails, charges, or issue comments.

Prefer a paid API when you need a documented export path or a rate limit you can quote. Prefer self-hosted when the secret boundary is non-negotiable, or when transcripts are a legal record. Those are fit criteria for a specific job, not moral rankings of vendors or of free tiers. Can one team hold all three runtimes at once? Yes, if the probe stays the promotion gate.

Limitations, and who should skip this

This framework will annoy people who want a single winner and a tweet-sized recommendation after lunch. It also underfits teams that already banned shared workspaces through a real, written security review. If you train models, serve regulated data, or drive irreversible actuations, skip free shared servers. A green probe on a toy repository does not authorize those jobs, even when nobody is watching.

The probe is not a security audit, and I need to say that without softening it. Pattern-scanning environment variables will miss encoded leaks, side channels, and writes outside the workspace path. The YAML numbers are not evidence, and I have not included vendor benchmarks, model names, or quotas. If your organization needs those figures, measure them yourself and store the results beside the transcript hash.

I also would not use this on a one-hour demo where the agent cannot touch the network. In that case the contract is simply an airgapped laptop, and the matrix is unnecessary ceremony. Use judgment, then write the smallest contract that would have caught your last painful incident. If you have never had an incident yet, start with secrets and transcript ownership anyway.

A compact command path you can copy

Once the YAML file and probe exist, I keep the operator loop boring on purpose. Excitement belongs in the agent task, not in the promotion path that guards production secrets. The commands below are a labeled local example, not a claim about any hosted fleet.

# labeled example — run locally against YOUR runner
mkdir -p artifacts
export AGENT_CONTRACT=./agent_compute_contract.yaml
python contract_probe.py
python - <<'PY'
import json
from pathlib import Path
probe = json.loads(Path('artifacts/probe.json').read_text())
print('decision:', probe.get('decision', 'missing'))
print('leaks:', probe.get('leaks', []))
print('elapsed:', probe.get('elapsed'))
print('transcript_sha256:', probe.get('transcript_sha256'))
PY
test -f artifacts/transcript.json && sha256sum artifacts/transcript.json

If the printed decision is not accept-candidate, I do not pass the model a production prompt. That habit has saved more money than discount hunting, and it has saved more sleep than dashboards. Can you ignore the gate for a demo on synthetic data? Yes, and you should label it a demo. Unlabeled demos are exactly how complimentary runtimes accidentally become unofficial production dependencies overnight.

Closing

I started with the conclusion, and I will end on the same sentence shape for a reason. Rank agent compute by the contract you can enforce, then spend free shared capacity only on passing jobs. Paid and self-hosted runtimes exist for the rows that fail, not for a vague sense of professionalism. Keep the matrix in git, keep the hashes next to it, and let the failed row choose the next box.