Ship or Don't: An Executable Demo Contract for a Weekend AI Side Project

A weekend side project ships when a script can prove it works, not when the author stops finding bugs. Everything below is built around that claim: a contract file, a checker, and a written record of what was deliberately left out. The demo is the small part. The proof is the deliverable.

The project itself is deliberately boring — an HTTP endpoint that takes an exported support inbox and returns a weekly digest, with the summarization step going through a hosted model endpoint and the service running on a free server instance. Boring is good here: it has an input, an output, one external dependency, and one failure mode worth catching.

The failure mode this fixes

Weekend builds die at minute ninety. The feature list was written on Friday against an imaginary week; by Sunday it is four items long and the last one is half-wired. The demo then "works" on a laptop, gets pushed to a free server, and the first outside request returns a 500 that nobody sees until Monday.

Two habits prevent this: deciding what to cut before writing code, and writing the acceptance criteria as data the machine reads. Prose acceptance criteria get reinterpreted under deadline pressure. A JSON file does not.

Step 1 — Sort the backlog into Ship, Prove, Skip

Every candidate feature gets exactly one bucket, and the bucket rules are fixed in advance:

Bucket	Rule	Artifact required
Ship	Without it, the demo has no input or no output	A check in the contract
Prove	It is the reason the demo is interesting (degraded mode, empty input)	A check in the contract
Skip	It protects against a user who does not exist yet	An entry in skipped.json with a re-open trigger

Auth lands in Skip for a single-user demo. Multi-format ingestion lands in Skip. Retry-with-backoff lands in Skip, and that one surprises people — see below.

For this build the contract ended up with four checks: a health endpoint, the happy-path digest shape, empty input, and a forced model timeout that must return a partial digest flagged degraded rather than a 500.

Step 2 — Write the contract as data

demo_contract.json, roughly forty lines:

{
  "version": 1,
  "target": { "base_url": "https://REPLACE-ME.example.dev" },
  "checks": [
    {
      "id": "health",
      "kind": "http",
      "path": "/healthz",
      "expect_status": 200,
      "timeout_ms": 5000,
      "max_ms": 2000
    },
    {
      "id": "digest_shape",
      "kind": "json",
      "path": "/api/digest?source=sample-export",
      "expect_status": 200,
      "timeout_ms": 20000,
      "max_ms": 15000,
      "require_keys": ["week", "items", "omitted"],
      "assert": { "path": "week", "op": "eq", "value": "2026-W45" }
    },
    {
      "id": "empty_input",
      "kind": "json",
      "path": "/api/digest?source=empty-export",
      "expect_status": 200,
      "timeout_ms": 15000,
      "max_ms": 10000,
      "require_keys": ["items"],
      "assert": { "path": "items", "op": "len_eq", "value": 0 }
    },
    {
      "id": "model_timeout_degrades",
      "kind": "json",
      "path": "/api/digest?source=sample-export&simulate=model_timeout",
      "expect_status": 200,
      "timeout_ms": 20000,
      "max_ms": 15000,
      "require_keys": ["degraded"],
      "assert": { "path": "degraded", "op": "eq", "value": true }
    }
  ]
}

The simulate=model_timeout flag is a fault-injection hook that has to be implemented in the app. If it does not exist yet, delete that check from the contract instead of leaving a check that can never pass — a permanently red check trains everyone to ignore the output.

Step 3 — The checker stays boring too

Standard library only, no test framework, no fixtures. It runs from a CI job or a laptop shell with the same result.

#!/usr/bin/env python3
"""Executable demo contract. Python 3.9+, stdlib only.
Exit 0 = every check passed (ship). Exit 1 = at least one check failed."""

import json
import sys
import time
import urllib.error
import urllib.request


def fetch(url, timeout_s):
    started = time.monotonic()
    try:
        with urllib.request.urlopen(url, timeout=timeout_s) as resp:
            body, status = resp.read(), resp.status
    except urllib.error.HTTPError as err:
        body, status = err.read(), err.code
    except urllib.error.URLError as err:
        return None, str(err), (time.monotonic() - started) * 1000
    return status, body, (time.monotonic() - started) * 1000


def walk(payload, dotted):
    node = payload
    for part in dotted.split("."):
        node = node[int(part)] if part.isdigit() else node[part]
    return node


OPS = {
    "eq": lambda actual, want: actual == want,
    "len_eq": lambda actual, want: len(actual) == want,
}


def run_check(check, base_url):
    url = base_url.rstrip("/") + check["path"]
    status, body, elapsed_ms = fetch(url, check["timeout_ms"] / 1000)
    problems = []

    if status != check["expect_status"]:
        problems.append(f"status={status} want={check['expect_status']}")
    if elapsed_ms > check["max_ms"]:
        problems.append(f"{elapsed_ms:.0f}ms > {check['max_ms']}ms")

    payload = None
    if check["kind"] == "json" and isinstance(body, bytes):
        try:
            payload = json.loads(body or b"{}")
        except json.JSONDecodeError:
            problems.append("body is not JSON")

    if payload is not None:
        for key in check.get("require_keys", []):
            if key not in payload:
                problems.append(f"missing key '{key}'")
        rule = check.get("assert")
        if rule and not problems:
            try:
                actual = walk(payload, rule["path"])
                if not OPS[rule["op"]](actual, rule["value"]):
                    problems.append(
                        f"{rule['path']} failed {rule['op']}={rule['value']!r}"
                    )
            except (KeyError, IndexError, TypeError):
                problems.append(f"assert path '{rule['path']}' not reachable")

    return problems


def main():
    path = sys.argv[1] if len(sys.argv) > 1 else "demo_contract.json"
    with open(path, encoding="utf-8") as handle:
        contract = json.load(handle)

    results = []
    for check in contract["checks"]:
        problems = run_check(check, contract["target"]["base_url"])
        results.append((check["id"], problems))
        print(f"[{'FAIL' if problems else 'PASS'}] {check['id']}")
        for problem in problems:
            print(f"       - {problem}")

    failed = [cid for cid, problems in results if problems]
    print(f"\n{len(results) - len(failed)}/{len(results)} checks passed")
    return 1 if failed else 0


if __name__ == "__main__":
    sys.exit(main())

Timing is measured inside the checker, so a slow network path shows up as a failure rather than a footnote. That is intentional: the free server instance is part of the system under test, not an excuse.

Step 4 — Run it against the deployed URL

Point base_url at the running service, then:

# cold start on a free instance: hit once, then measure
curl -s -o /dev/null -w "%{http_code}\n" "$BASE_URL/healthz"
python3 check_demo.py demo_contract.json
echo "exit=$?"

The warm-up request matters on any instance that sleeps when idle. Measuring the first request after a cold start produces a latency failure that says nothing about code quality; hit the health endpoint once, discard that number, then run the contract. If the checks still exceed max_ms while warm, the budget is wrong or the model call needs a shorter timeout — not a bigger number in the JSON.

A passing run prints four PASS lines and exits 0. A failing run exits 1, which is the entire integration story for CI:

- run: python3 check_demo.py demo_contract.json

Step 5 — The skipped list earns its keep with re-open triggers

skipped.json is not a graveyard. Each entry carries the condition that promotes it back into scope, which turns "we didn't have time" into a decision with an owner and a trigger.

[
  {
    "item": "multi-format ingestion (CSV, HTML, chat export)",
    "why": "three parsers, one demo, zero extra proof",
    "reopen_when": "a second real export format appears in a bug report",
    "cost": "half a day per format"
  },
  {
    "item": "authentication",
    "why": "single-user demo, no data worth protecting yet",
    "reopen_when": "the URL is shared outside the author's own browser",
    "cost": "2 hours for a signed cookie"
  },
  {
    "item": "retry with backoff on model calls",
    "why": "one blind retry hides the timeout the contract exists to catch",
    "reopen_when": "the degraded path passes twice in a row without manual help",
    "cost": "1 hour"
  }
]

The retry entry is the one worth arguing about. A retry feels like resilience, but during a weekend build it usually converts a visible failure into an invisible slow one, and the model_timeout_degrades check stops meaning anything. Ship the degraded path first; add retries once the degraded path is provably correct.

Where the free model access and free server fit

MonkeyCode is the open-source project this build used for both halves of the constraint: free model access for the summarization call and a free server option for hosting the endpoint. The grant quoted for this outreach is 10 million free tokens — treat that as a ceiling to measure against rather than a reason to add features, and confirm the current terms and limits on the project page before planning around them, since published quotas change faster than blog posts do.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Practical consequences for the contract: the model call must have an explicit timeout shorter than the check's max_ms, and every run should log token usage for the request. If a single digest run consumes an amount of the grant that would not survive a few dozen demo hits, cut the input size in the sample export instead of cutting the contract.

Limitations and who should not use this

• The contract proves observable behavior only: status codes, JSON shape, coarse latency. It says nothing about digest quality, and a plausible-looking but wrong summary passes every check.
• Timing thresholds are environment-specific. Copying max_ms from another project guarantees false failures.
• The fault-injection check requires app-side support for simulate=model_timeout. Without it, the most valuable check in the file cannot exist.
• This is not a load test, a security review, or a replacement for unit tests around the parsing code.
• Teams with real SLOs, paying users, or regulated data should skip the free-tier weekend approach entirely and start from a monitoring stack.

The pattern is worth keeping even when the project grows: a contract file in the repo, a checker with a single exit code, and a skipped list with re-open conditions.

If the free model access and free server path is the constraint you are working under this weekend, start by writing the four checks — then go look at what the project currently offers and confirm the numbers yourself.