Everyone is shipping AI coding agents now. Few teams have a repeatable way to test them. Your team adopted one last month. The demo looked flawless. The first real task failed silently. No error message. No diff. Just a wasted afternoon.
You need a repeatable smoke test. Not a benchmark. Not a sales demo. A tiny task that proves the agent can install, run, and fix something real.
This tutorial builds that test in three stages. Every stage has commands and a verification step. You need a server and a token budget. Both are free in this workflow.
Why a free server matters
Agents run arbitrary commands. They install packages. They edit files. They sometimes delete things. You do not want that on your laptop.
A disposable server contains the blast radius. When the test ends, you destroy the server. Nothing touches your local environment.
MonkeyCode is an open-source project that offers free model access and a free server option. The free tier includes a 10-million-token allowance. That is enough for many evaluation runs. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Check the project docs for current limits. Model lists change. Token allowances change. Treat this article as a workflow, not a contract.
Stage 1: Provision the free server
Create a clean machine first. The commands below are examples. Your CLI flags may differ.
# Example: create a throwaway server
monkeycode server create --name smoke-lab --free
# Example: connect to it
ssh root@<server-ip>
Verify the machine before you continue.
uname -a
df -h / | tail -1
Both commands should return clean output. If they fail, stop here. A broken server invalidates every later result.
Keep the server isolated. Do not add your SSH keys. Do not mount your home directory. The agent only needs the repository you give it.
Stage 2: Install the agent and cap the budget
Install the CLI on the server. Then authenticate. Then check your allowance.
# Example: install the CLI
curl -fsSL https://get.monkeycode.dev | sh
# Example: authenticate and inspect the allowance
monkeycode auth login
monkeycode tokens status
Set a hard token cap before the first run. Ten million tokens sounds huge. One runaway loop can burn through it fast.
# Example: cap each task at 50,000 tokens
monkeycode config set max_tokens_per_task 50000
Verify the setting.
monkeycode config get max_tokens_per_task
It should print 50000. If it prints something else, fix the config. A missing cap turns a smoke test into a cost experiment.
Stage 3: Run a real task
Pick a tiny repository with one failing test. The agent must find the failure. Then it must fix it. Then the test must go green.
Choose a task you can verify by hand. A failing test is ideal. A refactor is not. Refactors have no objective pass signal. Your smoke test needs a binary outcome.
The script below is a template. Adjust the flags to match your CLI.
#!/usr/bin/env bash
# smoke_test.sh — run one agent task and record the result
set -euo pipefail
LOG="smoke-$(date +%s).log"
# Put your real agent command here.
# Example:
# monkeycode agent run --repo <url> --task "fix the failing test"
bash -c "$1" > "$LOG" 2>&1
if grep -q "tests passed" "$LOG"; then
echo "PASS"
else
echo "FAIL"
tail -20 "$LOG"
fi
Run it with a concrete task.
./smoke_test.sh 'monkeycode agent run --repo https://github.com/example/parser-demo --task "Fix the failing test in tests/test_parser.py"'
Record four signals from every run.
| Signal | Pass | Fail |
|---|---|---|
| Exit code | 0 | non-zero |
| Test result | green | red |
| Tokens used | under cap | cap hit |
| Wall time | under 10 minutes | timeout |
Run the same task three times. Agents are stochastic. One pass proves nothing. Three passes show a pattern.
What the results tell you
Two passes and one fail means flakiness. Investigate before you trust the agent. A clean cap hit means the agent is looping. Raise the cap or simplify the task. A green test with a huge token bill means the fix works but costs too much.
Compare runs across weeks. Model updates change behavior. Your smoke test becomes a drift detector. That is the real value of a repeatable task.
What can go wrong
The clone fails. Check the repository URL and network access. The agent never edits files. Give it a smaller task. The test stays red. Read the log before you blame the model. The cap hits instantly. Your prompt is probably too vague.
Every failure is data. Record it. A smoke test that fails is still a successful experiment.
Limitations
The free server is an evaluation environment. Do not run production workloads on it. Do not store secrets there. The model list and token allowance can change. Verify current numbers in the project docs.
Who should skip this
Teams with strict data residency rules should not send code to a free tier. Teams that need guaranteed uptime should look at paid options. This workflow is for evaluation. It is not a production platform.
The 30-minute plan
Provision the server. Install the CLI. Cap the budget. Run one task three times. Record the results. Destroy the server. That is the whole workflow.
If you want to run this test yourself, the MonkeyCode docs walk through the free server setup. Start with a tiny repo. You will learn more in 30 minutes than in a week of demos.
Top comments (0)