Every project has a pile of work that is important but never urgent. For
Wanaku, that pile was the test plans.
We have hundreds of test units written in Markdown: set up a namespace,
deploy the operator, call the CLI, check the output. A person can follow
them. Nobody wants to follow them every night.
So I tried something. I wrote down what I wanted, handed it to coding
agents, and kept the role of reviewer and product owner. Six days later I
have Midnight Saci: an AI test
engineer that runs Markdown test plans against real environments and shows
the evidence for every verdict.
This post covers what Saci does, and what building it taught me about
working with coding agents.
The name
The Saci-Pererê is a trickster from Brazilian folklore. He hides things,
ties knots and causes small chaos for fun. Our Saci uses that mischief for
good: he looks for bugs at night so you do not have to.
What Saci does
You write a test plan the way you would write it for a colleague: phases,
tests, commands and the results you expect to see.
### Test 1.1: Read an existing order
```bash
curl -sS -w '\nstatus %\{http_code}\n' "$SERVICE_URL/orders/1"
```
**Expected result:**
- The output ends with `status 200`.
- The order's `status` is `shipped`.
Saci carries out the plan step by step. An LLM decides what to do, and a
small, bounded set of tools (shell, HTTP and MCP) does it. Each run gets
its own isolated worker, as a Podman or Docker container or a Kubernetes
Job. Credentials are redacted from the evidence. Any OpenAI-compatible
model works, from OpenRouter to a local Ollama.
The important part is what Saci does not do. It follows the plan. It
does not invent repair steps, and it does not patch your code. Every
verdict ties each expected result to the command output that supports or
contradicts it:
1.2 Create an order [test]: failed
expected result 1: contradicted (mechanical; ev-2): the output ends with status 500, not 201
expected result 2: contradicted (mechanical; ev-2): the body is {"error":"orders database is read-only"} and has no id
You check the reasoning instead of taking it on faith.
Watching a run
The web console shows a run while it happens. Here Saci runs the Wanaku MCP
CLI test plan: 29 units, against a real Wanaku instance.
The Activity tab is the live log. Every shell command links to its
evidence, and every attempt reports how many model requests and tokens it
used.
The Plan performance tab answers the question I asked most often in
the first days: where did the time go? In this run, the model took 97% of
the time and the tools took 2%. That tells me where to optimize, and it is
not the shell.
The Environment tab records the worker image, the OS and tools in the
worker, and the exact Saci build. When a run from last Tuesday fails, I
know what it ran on.
After the run
A test report is only useful if someone acts on it. Saci adds three
things after a run.
Outcome actions. After a run finishes, the backend can call MCP tools:
open a ticket, post to chat, send an email. Actions can wait for a person
to approve them in the console, and they never change the verdict.
Investigations. When a unit fails, Saci can start a second agent that
clones the source code, reads the code that serves the failed step, and
ends with hypotheses that cite file and line. It can file a bug, or say why
it did not. Other agents can request an investigation over
A2A, so Saci fits into a larger agent setup.
Example investigation using the documentation’s sample data.
The script runner. saci-script-runner exposes administrator-defined
scripts as MCP tools. A Saci worker, Wanaku or any other MCP client can
call them. The first example emails run results with a plain bash script.
Saci works with your coding agent too
Hundreds of existing Wanaku test units did not have the exact format Saci
expects. Most of them had their expectations in echo "PASS ..." lines
instead of an Expected result section. Turning those into criteria
needs judgment, so a converter that only applies rules would have to guess.
So the conversion is a skill for your coding agent, plus a validator:
saci skill install --all --agent claude # or codex, cursor, copilot, gemini, opencode
saci check --strict plans/operator-basic-functionality.md
The agent converts the plan, and saci check tells it, offline, what is
still wrong. Other skills start a backend, run a plan and investigate a
failure. Saci was built by agents, and it is designed to be driven by them
too.
An experiment with small models
Most steps in a test plan are simple: did the command exit 0, does the
output contain this line? Asking a frontier model about each of them is
slow and expensive. Saci can use a "System One" judge, a small and fast
decision model, to pass the obvious steps. It falls back to the LLM for
everything else.
We tried Julia 1, a 144M
parameter open model, served by a small Rust module with ONNX Runtime. To
be honest: on real Wanaku runs it was rarely confident enough, so the LLM
still judged almost every step. The plumbing is in place, and fine-tuning
is the next experiment. I include this because not every idea works on the
first try, and that is fine.
How it was built
The repository started on 2026-10-17. By 2026-10-23 it had about 40,000
lines of Rust across eight crates, a React web console, an OpenAPI
contract, Helm charts, multi-arch container images, and 130 issues and
pull requests. I did not write that code. Coding agents (Claude Code,
running in cloud sessions) did. My job was different, and these are the
things that made it work.
Write the plan first. Before any code, I wrote a system plan: the
goal, the principles (simplicity, trustworthiness, adaptability), the
technical baseline and the shape of the system. The agents broke it into
tickets. Each change traces back to an issue.
Write the rules down. A CLAUDE.md file holds the standing rules: no
unwrap or panic in production code, no Python or Perl for tooling,
meaningful tests for every major feature, documentation in Simplified
Technical English, no abstractions before a second real use. Agents follow
written rules well. They do not read minds.
Let CI be the judge. make check runs format, Clippy with warnings
as errors, and the full test suite. Agents fix their own CI failures
before I look at a pull request. My reviews are about design, not about
lint.
Run many threads at once. Each piece of work got its own agent thread:
the investigations design, the Julia 1 judge, the script runner, the
console pages. They ran in parallel and opened pull requests that I
reviewed and merged.
Use the product to test the product. Saci has test plans for Saci
itself, and it runs them.
Stay the product owner. The agents asked good questions: should plan
conversion be a CLI or a skill? Should the judge run in the worker or as a
separate service? Those decisions were mine, and they shaped the project
more than any line of code.
What is next
Saci is a new project, and it shows. Some areas are rough, and the Julia 1
judge needs work. But it already runs real Wanaku test plans, and it
already saves me time.
More than that, it shows what is possible now. One person with a clear
plan, written rules and good CI can build a real system in a week. The
skill that matters is not typing code. It is knowing what you want, and
saying it clearly.
The code is available on Midnight's Saci Github.
The walkthrough runs everything on one machine with a scripted model, so you do not need an API key to try it.
Top comments (0)