Killed Work Still Hits the Meter

Free capacity is a discount on the next token, not a refund on the last one. If the box dies, the queue expires, or the call is cut off, you still spent the prompt. You still spent the partial answer. You still spent the minutes a human waited. The kill switch does not roll the meter back.

That is the part most teams skip when they park production-shaped work on a scavenger lane. They see a zero in the rate column and stop reading. The invoice is not the rate. The invoice is every token that never became a finished, accepted result.

Think of a taxi the dispatcher can recall mid-ride. The fare looks cheap until the car vanishes at the third stoplight. You do not get those kilometers back. You stand on the curb with a half-finished trip and you pay again to start over. Model jobs behave the same way. Input tokens are prepaid. Partial output is often billed. A restart repeats the prefix you already paid for.

So you need a different unit. Stop asking whether the lane is free. Ask how many times this job can die before the discount costs more than paying now. That number is an abort budget. Without it, spare capacity is not a strategy. It is a leak with a friendly name.

Two lanes, one promotion rule

You already know the shape of the work. Drafts, eval sweeps, prompt archaeology, and throwaway summaries can die. The nightly batch that writes a customer-facing changelog cannot. The classifier that gates a merge cannot. The job that holds a lock in your deploy graph cannot.

Put killable work on scavenged capacity. Put must-finish work on capacity you actually control. The rule is boring, which is why it survives contact with a real queue. A scavenger lane is still useful. You want somewhere to burn exploratory tokens without contaminating the committed path.

MonkeyCode's free model access and free server option can sit in that lane: a box for work you are willing to lose. Disclosure: This article was prepared as part of MonkeyCode's product outreach. That is not a reserved machine, and it is not a claim about quota, hardware, model names, or how long a server stays yours. Treat it as spare capacity you might not get back. If the job cannot tolerate a restart, promote it before the first token. If you want a place to park killable drafts off the paying queue, that free lane is there to try. Measure aborts before you trust it.

An abort budget you can run

The following is a labeled example, not a production scheduler and not a benchmark. You supply the paid token rate and the cost of your own time. The function answers one question: after this many kills, is waiting on free capacity still cheaper than paying?

# abort_budget.py
# Proposal: promote a job off scavenged capacity when restart
# waste meets or exceeds the paid alternative.
from dataclasses import dataclass

@dataclass
class Job:
    name: str
    must_finish: bool
    max_aborts: int
    input_tokens: int
    expected_output_tokens: int
    queue_s_expected: float
    engineer_usd_per_hour: float
    paid_usd_per_million: float

def tokens_burned_on_kill(job: Job, partial_output: int) -> int:
    # Prompt is spent. Partial completion is spent.
    # A restart pays the prompt again.
    return job.input_tokens + max(partial_output, 0)

def waste_usd(job: Job, kills: int, avg_partial_out: int) -> float:
    token_waste = kills * tokens_burned_on_kill(job, avg_partial_out)
    token_usd = token_waste * job.paid_usd_per_million / 1_000_000
    hours = kills * job.queue_s_expected / 3600.0
    time_usd = hours * job.engineer_usd_per_hour
    return token_usd + time_usd

def paid_path_usd(job: Job) -> float:
    total = job.input_tokens + job.expected_output_tokens
    return total * job.paid_usd_per_million / 1_000_000

def should_promote(job: Job, kills: int, avg_partial_out: int) -> bool:
    if job.must_finish:
        return True
    if kills >= job.max_aborts:
        return True
    return waste_usd(job, kills, avg_partial_out) >= paid_path_usd(job)

The comparison uses your paid rate as the opportunity cost of tokens you already threw away. That is deliberate. A free prompt you later repeat on a paid model was never free. It was a loan. You repay it with a second prefix and whatever waiting you did the first time.

Wire a test so the rule cannot drift into folklore.

# test_abort_budget.py
from abort_budget import Job, should_promote, waste_usd, paid_path_usd

def sample_job(**kwargs) -> Job:
    base = dict(
        name="draft-summary",
        must_finish=False,
        max_aborts=2,
        input_tokens=8_000,
        expected_output_tokens=1_200,
        queue_s_expected=180.0,
        engineer_usd_per_hour=75.0,
        paid_usd_per_million=3.0,
    )
    base.update(kwargs)
    return Job(**base)

def test_must_finish_promotes_immediately():
    job = sample_job(must_finish=True, name="release-notes")
    assert should_promote(job, kills=0, avg_partial_out=0) is True

def test_killable_job_stays_until_budget_breaks():
    job = sample_job()
    assert should_promote(job, kills=0, avg_partial_out=400) is False
    assert should_promote(job, kills=2, avg_partial_out=400) is True

def test_long_queue_promotes_before_token_waste_does():
    job = sample_job(queue_s_expected=2400.0, max_aborts=9)
    # Forty minutes of blocked waiting, twice, is not a discount.
    assert waste_usd(job, 2, 0) > paid_path_usd(job)
    assert should_promote(job, kills=2, avg_partial_out=0) is True

Those dollar figures are illustrations you must replace. They are not product prices, not market rates, and not a claim about your bill. If you do not know the paid rate, stop and export one day of usage from the provider you actually use. A budget with a made-up rate is a story.

python -m pip install pytest
pytest -q test_abort_budget.py

You can also poke the function from a shell without standing up a framework. Keep the numbers in environment variables so a teammate can disagree with the rate instead of editing the rule.

python - <<'PY'
from abort_budget import Job, should_promote, waste_usd, paid_path_usd
job = Job(
    name="eval-sweep",
    must_finish=False,
    max_aborts=3,
    input_tokens=12_000,
    expected_output_tokens=800,
    queue_s_expected=90.0,
    engineer_usd_per_hour=75.0,
    paid_usd_per_million=3.0,
)
print("paid_path", round(paid_path_usd(job), 6))
print("waste_after_2", round(waste_usd(job, 2, 300), 6))
print("promote", should_promote(job, kills=2, avg_partial_out=300))
PY

Read the incomplete calls

The budget only works if you can see kills. Most traces hide them under a generic timeout. You want one line per attempt: job id, attempt number, tokens in, tokens out, terminal state. Terminal state is the whole point. complete, timeout, preempted, and cancelled are different invoices.

# attempt_log.py
# Proposal: parse JSONL attempts and split spent tokens into
# finished work versus killed work.
import json
import sys

def summarize(path: str) -> None:
    spent = 0
    finished = 0
    killed = 0
    with open(path, encoding="utf-8") as handle:
        for raw in handle:
            row = json.loads(raw)
            used = int(row["tokens_in"]) + int(row["tokens_out"])
            spent += used
            if row["state"] == "complete":
                finished += used
            else:
                killed += used
    print(f"spent_tokens={spent}")
    print(f"finished_tokens={finished}")
    print(f"killed_tokens={killed}")
    if spent:
        print(f"killed_ratio={killed / spent:.3f}")

if __name__ == "__main__":
    summarize(sys.argv[1])

Start with a fixture so the parser has something honest to chew on.

cat > attempts.jsonl <<'EOF'
{"id":"draft-summary","attempt":1,"tokens_in":8000,"tokens_out":410,"state":"preempted"}
{"id":"draft-summary","attempt":2,"tokens_in":8000,"tokens_out":0,"state":"timeout"}
{"id":"draft-summary","attempt":3,"tokens_in":8000,"tokens_out":1180,"state":"complete"}
EOF
python attempt_log.py attempts.jsonl

If killed_ratio climbs while must_finish jobs are still on the scavenger lane, you are not saving. You are rehearsing the same prompt. That is cousin to a retry storm, except the trigger is the machine going away rather than the model returning an error. The fix is promotion, not a longer timeout. A timeout is how you donate a larger partial completion to the killed column.

Watch the shape of the waste. A huge prompt with a tiny completion is a bad scavenger candidate, because every kill repeats the expensive half. A short prompt with a long, optional draft is a better one. The log will tell you which job you actually have. Your intuition about "cheap experiments" will not.

When free capacity is the wrong bet

Use the scavenger lane when the result is optional, the prompt is cheap to replay, and nobody is blocked on the output. Eval harnesses, prompt diffs, and personal drafts fit. A free server is a fine place to learn whether a prompt is even worth paying for.

Keep committed work off that lane. If the output enters a customer path, a compliance record, or a deploy gate, the job already has an SLO whether you wrote it down or not. If a human is sitting in a meeting waiting on the result, their time will beat a token discount in a single missed slot. If the prompt is large and the completion is small, a kill repeats the costly half and the discount evaporates on the first preempt.

Do not use spare capacity as capacity planning. A box that can vanish is not a headcount substitute and not an SLO. If you need the job tomorrow morning, you already know the answer. Pay, or cut the scope until the job can die without consequences.

The approach also fails if you cannot attribute tokens per attempt. Shared keys, one giant chat session, and dashboards that hide usage behind a monthly total will lie to you. Fix the log first. Then set max_aborts. Then promote.

Limitations are blunt. This model ignores cache hits, ignores provider-side retries you cannot see, and ignores the chance that the paid path is congested too. It assumes you can move a job between lanes without rewriting it. If your orchestration cannot redirect a single task, the function is a comment, not a control. It also assumes the scavenger lane still costs you nothing per token, which may not remain true, and which you should re-read in the product's own terms rather than in a post.

You should not adopt this if you need a guarantee. You should not adopt it if you cannot kill the job. You should not adopt it if the work is someone else's data you are not allowed to send to a shared box. You should not adopt it if the only number you track is requests, because requests are how incomplete work hides.

The core move is small. Tag the job. Cap the kills. Promote when the waste crosses the paid path. Everything else is decoration on a meter that already started.