Grade the Free Model Like a Take-Home

You know the moment. A teammate drops a chat log into Slack and says the new free model basically nailed the take-home. You skim it. The prose is confident, the function names are tidy, and then you notice there are no tests, the clock is time.sleep, and the retry loop can spin forever if the callee returns None.

Chat is an interview. A take-home is a work sample. If you are about to let a free model touch a script that sits next to production, you should grade it the way you grade a human candidate: a sealed prompt, a rubric you wrote first, a sample solution you keep private, and a short memory of how these submissions usually fail.

This packet is small on purpose. You can run it in an afternoon against any free model endpoint. Nothing here assumes a paid plan, a named model, or a particular machine.

Chat hides the same holes a live interview hides

A fluent model can narrate a rate limiter it cannot implement. It will talk about tokens and refill rates, then mutate a global dict and call the result thread-safe. Human candidates do this too when they talk longer than they code. The fix is not a warmer prompt. The fix is a take-home that is small enough to grade in ten minutes and sharp enough that bluffing gets expensive.

You hand the model one file of instructions. You do not negotiate in the thread. You score the returned tree against a rubric that already existed. That order is the whole method. If you write the rubric after you see a pretty NOTES.md, you will grade the story, not the work.

Treat the model like a contractor who cannot come onsite. You would not hire that person from a hallway conversation. You would send a sealed assignment, hide the answer key, and read the tests before you read the cover letter.

Seal the prompt, then change the seed

Save the following as TAKEHOME.md. Change the seed line every time you reuse the packet so a memorized blog solution cannot coast through. Classic puzzles leak. Your seed is how you notice.

# Take-home: freeze-clock token bucket

Seed: 2026-09-03-devio / do not paste public blog solutions.

Implement a token-bucket rate limiter in Python 3.11 as `bucket.py` plus `test_bucket.py`.

Constraints:
- Standard library only.
- No `time.sleep` inside the limiter. Time comes from a `Clock` protocol
  with `now() -> float` (seconds) and `advance(seconds)` for tests.
- `TokenBucket(rate_per_sec: float, capacity: float, clock: Clock)`
- `allow(n: float = 1.0) -> bool` must be deterministic on the same clock trace.
- Refill is continuous: tokens += rate * elapsed, capped at capacity.
- Reject `n > capacity` without consuming tokens.
- `allow` must not throw on normal inputs. Raise `ValueError` for
  non-finite or negative rate, capacity, or n.
- Tests must cover: burst up to capacity, refill after exact elapsed time,
  rejection when empty, no token leak on oversize reject, freeze clock
  never calling `time.time`.

Deliver:
1. `bucket.py`
2. `test_bucket.py` passing under `python -m unittest`
3. `NOTES.md` with one paragraph on a race you did *not* solve (threads,
   process forks, etc.) and why you left it unsolved.

Do not add features. Do not install packages. Do not call the network.

The prompt is boring on purpose. Take-homes that sound like product pitches invite the model to invent architecture. You want a work sample, not a pitch deck. If the model returns Redis, asyncio, and a dashboard, that is not ambition. That is a candidate who did not read the brief.

Write the rubric before any files arrive

You score four lanes. Each lane is 0, 1, or 2. The total is 8. You only keep using the endpoint if the score is at least 6 and neither Correctness nor Tests is a zero. A charming notes file cannot rescue a broken allow.

Correctness asks whether allow matches continuous refill on a freeze-clock. A 2 means burst, refill, empty, and oversize-reject all behave. A 1 means the happy path works and one edge is fuzzy. A 0 means tokens leak, time comes from the wall clock, or oversize rejects still drain the bucket.

Tests asks whether the suite would go red if you sabotaged refill or the oversize rule. A 2 means the tests pin those behaviors. A 1 means the tests only prove that something returned True. A 0 means the file is missing, skipped, or asserts nothing that could fail.

Discipline asks whether the module stayed in the standard library and avoided sleep. Judgment is the notes file. If NOTES.md claims the bucket is thread-safe without a lock, that lane is a zero. If it names a race and declines to pretend it was solved, that lane is a 2. Apology without a named race is a 1.

Keep that table in a file you do not send to the model. You are not discussing taste. You are applying a key.

Hide a sample solution the way you hide an answer key

Do not paste the next module into the prompt. You keep it private, the same way you would hide a human take-home key from the candidate. The code below is a proposed answer key, not a claim that any particular endpoint produced it.

from __future__ import annotations

from dataclasses import dataclass
from typing import Protocol


def _finite_nonneg(value: float) -> bool:
    return value == value and value not in (float("inf"), float("-inf")) and value >= 0


class Clock(Protocol):
    def now(self) -> float: ...
    def advance(self, seconds: float) -> None: ...


@dataclass
class FreezeClock:
    _t: float = 0.0

    def now(self) -> float:
        return self._t

    def advance(self, seconds: float) -> None:
        if not _finite_nonneg(seconds):
            raise ValueError("seconds must be finite and >= 0")
        self._t += seconds


class TokenBucket:
    def __init__(self, rate_per_sec: float, capacity: float, clock: Clock) -> None:
        if not _finite_nonneg(rate_per_sec):
            raise ValueError("rate_per_sec must be finite and >= 0")
        if not _finite_nonneg(capacity) or capacity == 0:
            raise ValueError("capacity must be finite and > 0")
        self._rate = rate_per_sec
        self._capacity = capacity
        self._tokens = capacity
        self._clock = clock
        self._last = clock.now()

    def allow(self, n: float = 1.0) -> bool:
        if not _finite_nonneg(n):
            raise ValueError("n must be finite and >= 0")
        if n > self._capacity:
            return False
        now = self._clock.now()
        elapsed = now - self._last
        if elapsed > 0:
            self._tokens = min(self._capacity, self._tokens + elapsed * self._rate)
            self._last = now
        if self._tokens >= n:
            self._tokens -= n
            return True
        return False

A short test module belongs next to it. Notice the freeze-clock never imports time. If a submission calls time.time inside allow, the suite should still pass only if your tests are too weak. That is the point of writing the key first.

import unittest
from bucket import FreezeClock, TokenBucket

class TokenBucketTests(unittest.TestCase):
    def test_burst_then_empty(self):
        clock = FreezeClock()
        bucket = TokenBucket(rate_per_sec=1.0, capacity=2.0, clock=clock)
        self.assertTrue(bucket.allow(2.0))
        self.assertFalse(bucket.allow(1.0))

    def test_refill_after_exact_elapsed(self):
        clock = FreezeClock()
        bucket = TokenBucket(rate_per_sec=2.0, capacity=2.0, clock=clock)
        self.assertTrue(bucket.allow(2.0))
        clock.advance(0.5)
        self.assertTrue(bucket.allow(1.0))
        self.assertFalse(bucket.allow(0.1))

    def test_oversize_reject_does_not_consume(self):
        clock = FreezeClock()
        bucket = TokenBucket(rate_per_sec=1.0, capacity=1.0, clock=clock)
        self.assertFalse(bucket.allow(2.0))
        self.assertTrue(bucket.allow(1.0))

    def test_invalid_n_raises(self):
        clock = FreezeClock()
        bucket = TokenBucket(1.0, 1.0, clock)
        with self.assertRaises(ValueError):
            bucket.allow(-1.0)

Run the key on your machine once so you trust it:

python -m unittest test_bucket

If your own key fails, you are not ready to grade anyone else. That rule is older than models.

Let a local harness do the first pass

The script below is a proposed grader. Treat it as an unexecuted example until you run it on files you actually collected. It does not score Judgment for you. It only catches the cheap zeros: missing notes, a failed suite, or a sleep that turned a freeze-clock into a nap.

# grade_takehome.py — proposed local grader, unexecuted example
import json, pathlib, re, subprocess, sys

root = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else ".")
report = {"zeros": [], "notes": False, "unittest": False, "sleep": False}

bucket = root / "bucket.py"
tests = root / "test_bucket.py"
if not bucket.exists() or not tests.exists():
    report["zeros"].append("missing bucket.py or test_bucket.py")
    print(json.dumps(report, indent=2))
    sys.exit(1)

src = bucket.read_text(encoding="utf-8")
if re.search(r"\btime\.sleep\b", src):
    report["sleep"] = True
    report["zeros"].append("discipline: time.sleep")

notes = root / "NOTES.md"
report["notes"] = notes.exists() and notes.stat().st_size > 40
if not report["notes"]:
    report["zeros"].append("judgment: NOTES.md missing or empty")

proc = subprocess.run(
    [sys.executable, "-m", "unittest", "test_bucket"],
    cwd=root,
    capture_output=True,
    text=True,
)
report["unittest"] = proc.returncode == 0
if proc.returncode != 0:
    report["zeros"].append("tests: unittest failed")
    report["stderr_tail"] = proc.stderr[-800:]

print(json.dumps(report, indent=2))
sys.exit(0 if not report["zeros"] else 1)

You still read NOTES.md with your own eyes. A model can write a paragraph that names "threads" and still ship a global dict. The harness is a bouncer. You are the interviewer.

Failure modes you will see on the first try

The first collapse is sleep. The model cannot figure out how to test time, so it pauses the wall clock and hopes. That submission can look concurrent in a demo and still be untestable. If time.sleep appears in bucket.py, Discipline is a zero and you stop. You are not grading patience.

The second collapse is integer tokens. Continuous refill becomes if elapsed >= 1. A half-second advance then does nothing, and the test you wrote for rate_per_sec=2.0 goes red. This is the model rounding a math problem into a counter because counters feel like code.

The third collapse is consuming on reject. allow(2) against a capacity of 1 returns False and still drains the bucket. Demos rarely catch it. Your oversize test exists for that exact theft.

The fourth collapse is a wall clock hiding in the tests. The suite passes on a quiet laptop and flakes in CI. If test_bucket.py imports time and calls time.time, the freeze-clock was theater.

The fifth collapse is extra credit. Redis, Prometheus, an async context manager, a CLI flag parser. It reads like seniority. It is a candidate who failed to follow a one-page brief. In a human loop you would worry about that person on a team. You should worry here too.

The sixth collapse is notes that apologize. "This would need more work in production" is not Judgment. Name the race. Say you left threads alone because the brief forbade locks and extra files. Then stop talking.

Where a free endpoint actually helps

You do not need a paid plan to run this packet. The work sample is small, the grader is local, and the only network call you should make is the one that asks the model for three files.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you already have MonkeyCode's free model access or free server option, point this sealed prompt at that endpoint, save the raw files, and keep the JSON scorecard next to them. That is the only product-specific step. The rubric still works if you swap the endpoint tomorrow.

You are not measuring intelligence. You are measuring whether this endpoint, on this day, can sit in a loop that writes a tiny library with tests. Store the date on the scorecard. Free endpoints move. A score without a timestamp is a rumor.

Who should not use this, and what it will not tell you

Do not use this packet to grade human candidates by proxy. A model take-home is not a lawful or kind stand-in for a person's work sample. Do not use it as a security review, a load test, or an SLA. A token bucket that passes unittest can still be wrong the moment two threads share it, which is exactly why the notes file must refuse that claim.

Do not use it if you will change the rubric after you see a fluent paragraph. Do not use it if your prompt is a famous puzzle with no seed. Memorized answers are not work samples. Do not use it as a reason to skip reading the code. The harness catches sleep and missing tests. It will not catch a bucket that refills from capacity instead of from remaining tokens unless your tests say so.

One take-home is not production readiness. It is a filter for bluffing. If the endpoint scores a 7 and you still would not let it open a pull request unsupervised, believe that feeling. The packet told you the model can follow a brief. It did not tell you the model can own an incident.

When someone pastes the next glowing chat log, open the scorecard instead of arguing about vibes. Run the grader on the files you already have. Keep the JSON. That file is quieter than a demo, and it ages more honestly.