My Syllabus Bot Ran a Week on a Free Server. 9 Answers Were Confident Lies.

The ledger entry stopped me cold.

id=198  verdict=FAIL
q: What does the paper claim about dropout rates?
a: The paper claims dropout rates fell by 12% after the intervention.
reason: Source says "roughly one in eight participants left". No 12%. No "fell".

Nine of those failures in 214 entries. Every one of them confident. None of them crashed anything. That is exactly what scared me.

Here is the question I wanted to answer: can a free model endpoint and a free server run a weekly study loop for one student without me babysitting it? Not a product. Not a demo. Just my own reading habit, automated for seven days.

Background

I am a student with one course that piles on dense weekly readings. My old study method was copy-paste flashcards, which is just rewriting the textbook with extra steps. Useless. So I decided to build a tiny bot that reads a chapter outline, writes five questions with short answers, stores them, and then double-checks its own answers against the source text.

The constraints were simple. Cost: zero. Setup: under an hour. Every answer: verifiable after the fact.

MonkeyCode, an open-source project, gave me two things for this experiment: a free model endpoint and a free server to run the loop on. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I did not benchmark their hardware or measure their quotas. I used the tier the way a broke student actually would.

Goal

Three requirements, in order. First, a weekly job that turns an outline into questions. Second, a ledger that records every question, answer, and verdict. Third, a checker that re-asks the model whether its own answer is supported by the source.

The ledger mattered more than the bot. A bot that answers is a toy. A bot that answers and leaves a paper trail is an experiment.

Implementation

Three small files. A ledger, a generator, a checker.

Prerequisites: Python 3.9+, the requests library, a free model endpoint that speaks OpenAI-style JSON, and any server where you can run a long-lived process. The exact endpoint schema may differ; adjust the ask() function to match what you are given.

ledger.py:

import sqlite3

DB = "quiz.db"

def init():
    con = sqlite3.connect(DB)
    con.execute("""
        CREATE TABLE IF NOT EXISTS entries (
            id INTEGER PRIMARY KEY,
            question TEXT,
            answer TEXT,
            verdict TEXT,
            reason TEXT,
            created_at TEXT DEFAULT CURRENT_TIMESTAMP
        )
    """)
    con.commit()
    return con

def save(con, question, answer, verdict, reason):
    con.execute(
        "INSERT INTO entries (question, answer, verdict, reason) VALUES (?, ?, ?, ?)",
        (question, answer, verdict, reason),
    )
    con.commit()

quizgen.py:

import os, requests
from ledger import init, save

SYSTEM = (
    "Turn the chapter outline into 5 questions with short answers. "
    "Answer only from the outline. If the outline lacks an answer, "
    "reply exactly: NOT IN SOURCE"
)

def ask(messages):
    r = requests.post(
        os.environ["MODEL_URL"],
        json={"messages": messages},
        timeout=30,
    )
    r.raise_for_status()
    return r.json()["choices"][0]["message"]["content"]

def main():
    outline = open("outline.txt").read()
    con = init()
    raw = ask([
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": outline},
    ])
    for block in raw.split("\n\n"):
        if not block.startswith("Q:"):
            continue
        q, a = block.split("A:", 1)
        save(con, q.strip(), a.strip(), "PENDING", "")

checker.py:

def verify(question, answer, source):
    prompt = (
        f"Question: {question}\n"
        f"Claimed answer: {answer}\n"
        f"Source: {source}\n"
        "Is the claimed answer supported by the source? "
        "Reply PASS or FAIL and one short reason."
    )
    return ask([
        {"role": "system", "content": "You are a strict fact checker."},
        {"role": "user", "content": prompt},
    ])

Deployment took longer than the code. I copied the files to the free server, set MODEL_URL in the environment, and started a loop with nohup.

scp *.py outline.txt user@server:~/quizbot/
ssh user@server "cd ~/quizbot && MODEL_URL=$MODEL_URL nohup python3 loop.py > loop.log 2>&1 &"

loop.py is the dumbest part of the system, and that is a compliment:

import time

while True:
    run_once()
    time.sleep(86400)

Results

Over seven days the bot generated 214 questions. The checker marked 9 as FAIL. I manually reviewed all 9. The checker was right every time.

The failure pattern was consistent. When the source text was ambiguous, the model ignored its own instruction. It never said "NOT IN SOURCE" when it could guess instead. And its guesses sounded like the source even when they contradicted it.

The server never went down. The model endpoint stalled once for about 30 seconds, and the loop silently skipped that run. No error. No log line. Just a missing entry. That silent skip taught me more than the 9 failures did.

Here is the exact failure input that produced the worst answer:

Outline: "Section 4 reports attrition. The authors note that roughly one in eight
participants left before the final assessment. Attrition was not correlated with
treatment assignment."

Q: What does the paper claim about dropout rates?

The model answered: "The paper claims dropout rates fell by 12% after the intervention." There is no 12% in the outline. There is no "fell" in the outline. The model built a plausible sentence out of the shape of the source. That is the failure mode you design for, not the outage.

Lessons

Free tiers fail softly. They do not crash loudly; they return confident nonsense, or they skip a run and leave a hole in your data. If you build on one, build the ledger first.

Re-asking the model to check its own answer is cheap and catches most of the nonsense. It is not a proof. It is a filter. Two wrong guesses can agree with each other, and my checker would have missed that.

Who should not use this approach? Anyone serving other people. Anyone with a latency requirement. Anyone whose wrong answer has a real cost — a grade, a medical claim, a customer. For a personal study loop, a free model and a free server are enough. For anything else, they are a starting point, not a foundation.

What you should understand after this

A free model will not fail loudly. It will fail confidently. A free server will not fail loudly either; it will just drop a run and move on. So your job is not to prevent failures. Your job is to make them visible. A 30-line SQLite ledger did that for me, and it will do it for you.

Extension

Try a second model as the judge instead of the same model. Feed it the 9 failures and see how many it catches. My guess: fewer than you hope. That gap is the real lesson.