A Zero-Budget Tool-Call Smoke Test: One Small Project, End to End

Tool-call failures are the most expensive bug class in agent apps. The model answers confidently, the tool never runs, and the user sees a silent gap. A scheduled smoke test catches that class early.

This case study walks one small project end to end: a $0, always-on tool-call probe built on MonkeyCode's free model access (10M tokens) and free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness and scheduler below are the real artifact; the product is only the transport.

Background

The pattern started with a recurring observation: single-turn tool-call tests pass locally, then break in production. Model updates change argument formatting. Prompt drift changes tool selection. A test that runs once, on your laptop, tells you nothing about next week.

Recent DEV threads made the same point from different angles. One argued that AI badges do not measure what developers think they measure. Another proposed a reasoning ledger: record decisions, not just data. Both point at the same engineering habit — trust your own runtime measurements over third-party scores.

The gap was operational, not conceptual. Smoke tests existed, but nobody scheduled them. Cloud runners cost money. A free server with free model access removes the excuse.

Goal

The project goal was narrow:

• Run three tool-call probes every hour.
• Host the runner on a free server.
• Spend zero dollars.
• Keep a permanent pass/fail log.

No dashboards. No alerts. Just a CSV file that grows.

Implementation

The harness

Each probe is a single chat completion with tools attached. The model must call the right tool, with the right argument names — or decline to call a tool when none is needed.

# probe.py — tool-call smoke test, one model, three probes
import json
import os
import time

from openai import OpenAI

client = OpenAI(
    api_key=os.environ["MONKEYCODE_API_KEY"],
    base_url=os.environ["MONKEYCODE_BASE_URL"],
)
model = os.environ["MONKEYCODE_MODEL"]

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a city.",
            "parameters": {
                "type": "object",
                "properties": {"city": {"type": "string"}},
                "required": ["city"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "calculate",
            "description": "Evaluate a math expression.",
            "parameters": {
                "type": "object",
                "properties": {"expression": {"type": "string"}},
                "required": ["expression"],
            },
        },
    },
]

PROBES = [
    {
        "name": "weather_city_arg",
        "messages": [{"role": "user", "content": "What is the weather in Tokyo?"}],
        "expect_tool": "get_weather",
        "expect_args": ["city"],
    },
    {
        "name": "calculate_expression_arg",
        "messages": [{"role": "user", "content": "Compute 17 * 23"}],
        "expect_tool": "calculate",
        "expect_args": ["expression"],
    },
    {
        "name": "no_tool_for_smalltalk",
        "messages": [{"role": "user", "content": "Hello, who are you?"}],
        "expect_tool": None,
        "expect_args": [],
    },
]


def run_probe(probe):
    response = client.chat.completions.create(
        model=model,
        messages=probe["messages"],
        tools=TOOLS,
        tool_choice="auto",
        temperature=0,
    )
    message = response.choices[0].message
    call = message.tool_calls[0].function if message.tool_calls else None
    if probe["expect_tool"] is None:
        return call is None
    if call is None or call.name != probe["expect_tool"]:
        return False
    args = json.loads(call.arguments)
    return all(key in args for key in probe["expect_args"])


def main():
    passed = 0
    for probe in PROBES:
        start = time.time()
        ok = run_probe(probe)
        passed += ok
        print(f"{'PASS' if ok else 'FAIL'}  {probe['name']}  {time.time() - start:.2f}s")
    print(f"score: {passed}/{len(PROBES)}")


if __name__ == "__main__":
    main()

The scheduler

One process, one loop, one CSV row per run. That is the whole server.

# runner.py — hourly loop, appends one CSV row per run
import datetime
import subprocess
import time

INTERVAL_SECONDS = 3600
CSV_PATH = "smoke_results.csv"


def run_once():
    stamp = datetime.datetime.now(datetime.timezone.utc).isoformat()
    result = subprocess.run(["python", "probe.py"], capture_output=True, text=True)
    summary = result.stdout.strip().splitlines()[-1] if result.stdout else "no output"
    with open(CSV_PATH, "a", encoding="utf-8") as fh:
        fh.write(f"{stamp},{result.returncode},{summary}\n")


while True:
    run_once()
    time.sleep(INTERVAL_SECONDS)

Wiring it to the free tier

MonkeyCode currently advertises free model access (10M tokens) and a free server option. The server is what hosts this loop. Set three environment variables and run:

export MONKEYCODE_API_KEY="your_key"
export MONKEYCODE_BASE_URL="https://..."   # from the current MonkeyCode docs
export MONKEYCODE_MODEL="model-name"       # from the current model list

python probe.py        # one manual run
python runner.py       # hourly loop, runs forever

Use the exact values from the project's current docs. Free tiers change; verify the terms before you build a workflow on them.

Results

Each run appends one line. A healthy run ends with score: 3/3. A regression shows up as score: 2/3 with a FAIL line above it. For example:

2026-08-20T08:00:01+00:00,0,score: 3/3
2026-08-20T09:00:02+00:00,0,score: 3/3
2026-08-20T10:00:03+00:00,1,score: 2/3

That FAIL line is the signal. Something changed — the model, the prompt, or the tool schema. The CSV gives you the exact hour it happened.

The design outcomes matter more than any single run:

• Cost stays at zero for the smoke-test workload.
• The probes are deterministic: temperature 0, fixed messages, fixed tools.
• The log is replayable. Any FAIL can be reproduced by running probe.py again.

When this pattern fits

Approach	Cost	Coverage	Setup	Best for
One-off local smoke test	$0	one snapshot	10 minutes	pre-deploy sanity check
Scheduled free-tier runner	$0	continuous, shallow	~1 hour	catching regressions early
Full eval harness with golden sets	model spend	broad, multi-turn	days	release gating

Limitations

This approach is deliberately shallow. Each probe is single-turn; it does not test multi-step reasoning, tool output handling, or retry logic. If your agent chains five tools per task, this harness will miss the failure mode.

Who should not use it:

• Teams that need release gating. Use a proper eval harness with golden sets.
• Anyone putting a production SLA on a free server. Free tiers can change without notice.
• Agents with complex tool schemas. Argument-name checks are not schema validation.

Lessons learned

• Keep probes tiny. Three focused probes catch more real regressions than a 50-case suite nobody runs.
• Log the raw output, not just pass/fail. The last line of probe.py is the alert.
• Temperature 0 is non-negotiable for smoke tests. Nondeterminism turns every failure into a debate.
• Treat the free tier as a floor, not a ceiling. 10M tokens is plenty for hourly probes; it is not a budget for batch evaluation.

Conclusion

The project's real output is not the CSV. It is the habit of watching your own runtime instead of someone else's leaderboard. If you want to adapt the harness to your own tools, the code above is the starting point — MonkeyCode's docs have the current setup details.