This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady
The fable says slow and steady wins the race. I wanted to know if that's true for a language model, so I made one race itself. Same open model, same 4 GB laptop, same fifty questions. One answers immediately. The other gets the whole night.
The hare won. Then I fixed what I thought was wrong with the tortoise, gave it a second night, and the hare won again.
TL;DR: 50 two-hop questions, Gemma 3 4B on a 4 GB laptop. One quick call scored 42%. The all-night version (829 calls) scored 34%, then 30% after my fix. The accuracy experiment failed twice. The durable-overnight part worked perfectly: one night ran 17.5 hours, survived 14 hours of downtime and 91 failed model calls, and lost nothing.
What I Built
Tortoise is that race, built so I can do nothing while it runs. I queue the questions before bed, close the laptop if I want to, and read the result at breakfast.
The hare reads all ten paragraphs at once and answers in one call. Fifty questions took 14 minutes.
The tortoise reads each paragraph on its own, then re-reads the ones it skipped now that it knows what to look for. It writes three drafts, checks each one against its notes, and plain code votes. Fifty questions took 829 model calls.
Both are Gemma 3 4B through Ollama on an RTX 2050 with 4 GB of VRAM. The questions are 50 seeded two-hop questions from HotpotQA ("who directed the film that starred…"), where the answer hides across two of ten paragraphs. The answers are known, so every score below is real exact match, not a vibe.
A night is long and laptops get closed, so every one of the tortoise's 829 steps is a Temporal activity.
Dashboard
Demo
Watch the real night replay: https://tortoise.vaidehi-k.tech Press Replay the night, then click any hexagon to see what the tortoise read, wrote and answered for that question.
Code
tortoise
The same small open model, racing itself on one laptop. The hare answers each question in one call. The tortoise takes all night: it reads every paragraph on its own, re-reads the ones it skipped, writes three drafts, checks each against its notes and lets plain code vote. Temporal makes every one of those steps durable, so you can kill the worker, close the lid or stop Ollama and it carries on from the exact step it was on.
The questions are 50 seeded two-step questions from HotpotQA (distractor setting): each answer needs facts from two of ten paragraphs. Answers are known, so the score is real exact match, not vibes.
Everything runs on your machine. Nothing leaves it.
Setup (once)
- Install Ollama and pull the model
ollama pull gemma3:4b - Install the Python dependency (Python 3.10+):
pip install -r requirements.txt - Download the questions:
python -m tortoise fetch
Run the
…How I Built It
hare 1 call: all 10 paragraphs -> {reasoning, answer}
tortoise one Temporal workflow per question, every step an activity:
read x10 one paragraph at a time -> {relevant, facts}
re-read xN the skipped ones, with pass-1 facts in hand
draft x3 one careful (temp 0), two exploring (temp 0.7)
check xM each distinct answer against the notes
vote plain code: +1 per agreeing draft, +2 if the check passed
Both get the same answer rules and the same "reason, then answer" format. The only difference is time.
Night one: the hare won
Exact match F1 Model calls Compute
Hare 42% 0.57 50 14 min
Tortoise 34% 0.43 829 1 h 8 min
About five times the compute, eight points worse. The tortoise got 5 questions the hare missed, and missed 9 that the hare got.
So I measured each slow step on its own:
Step Exact match
Hare, one call 42%
Tortoise, first draft only 36%
- majority of 3 drafts 34%
- checked against its notes 34%
The damage was done before any voting. Reading slowly wasn't the problem: the second read found 43 facts the first read had skipped. My hypothesis was that the drafts were written from short extracted notes and never saw the paragraphs again, so for a two-hop question the link between the two facts got lost. The hare, sloppy as it is, has the actual text in front of it.
Night two: I tested my own explanation, and it was wrong
I changed exactly one thing. When the tortoise drafts and checks, it now sees the full text of the paragraphs it flagged, not only its notes. Same questions, same model, same hare, same 829 calls.
Night 1 Night 2
Hare 42% 42%
Tortoise, first draft 36% 32%
Tortoise, final answer 34% 30%
Tortoise F1 0.43 0.42
Tortoise tokens 335,033 520,480
More context made it slightly worse, not better. The lossy-notes story, which I believed and wrote down, doesn't explain the gap. If it did, giving the drafts the real text would have closed it.
What the data does show:
Long prompts cost it. The tortoise now read 55% more tokens, and 10 of its 150 drafts came back completely empty. In night one that number was 0. I haven't proven the two are linked, but a 4B model on 4 GB of VRAM has little room to spare.
The loss is in multi-hop questions. On bridge questions (40 of the 50) the tortoise dropped from 35% to 28%. On the 10 comparison questions it held at 40% in night two, level with the hare.
Voting still doesn't help. Majority of three drafts lowered the score again (32% → 30%), because the two exploring drafts at temperature 0.7 are mostly worse than the careful one.
So my honest read: at 4B, more steps don't buy accuracy. Every step is another chance to be confidently wrong, and a vote across drafts from the same model mostly agrees with its own mistakes. Slow and steady only wins when each step is reliable, and I couldn't make them reliable here.
Making it durable (the part that did work)
A tortoise that runs all night can't start over every time something goes wrong. Each step is a Temporal activity with a heartbeat every 5 seconds, a 30-second heartbeat timeout, and unlimited retries with backoff. If Ollama goes down, the night waits for it instead of failing.
Night two is the proof. It took 17 h 31 min of wall clock for 1 h 59 min of actual compute, because the machine spent 14 h 17 min asleep or stopped. During that time Temporal retried 91 real failures (the model was unreachable) and I killed the worker mid-question twice. Not one finished step ran again, and all 50 questions completed with every result intact.
In night one, one step that had finished in the instant before a kill ran a second time. That's Temporal's at-least-once guarantee working as documented: the result hadn't been recorded yet, so it ran again, and because each step's output depends only on its input, a repeat changes nothing.
The method is written once, as pure code with no clock and no randomness, and both Temporal and a plain --local loop run that same code.
What I'd do differently
Exact match is harsh. The tortoise often wrote "Ed Wood film" for "Ed Wood". But F1 (0.42 vs 0.57) and a lenient "contains the gold answer" check (22 vs 31 of 50) agree: the hare really was better, not just tidier.
Fifty questions is a small sample. A few questions either way moves the score by several points, so I'd call night one and two "about the same, both below the hare" rather than trusting the 4-point difference between them.
The next experiment isn't more steps. It's making the single best look better: a better reader prompt, or a bigger context on a bigger GPU.
Why Does Open Innovation Matter?
Being slow is only free when you own the model. The tortoise used 335,033 tokens one night and 520,480 the next. On my GPU that cost nothing. On a paid API, "think all night, twice" is a bill, so most people would never run this experiment, or learn that it doesn't work, which is the useful part.
I could open up every step. JSON schemas, temperature per draft, the exact prompt each step saw, the notes it took. That's how I found the empty drafts and falsified my own explanation. With a closed endpoint I'd only have seen "34%".
It survives the real world. A local model that sleeps with the laptop needs a durable runner. Open-source Temporal made the 14 hours of downtime a non-event.
Open all the way down: open weights, an open benchmark, and an open-source durability engine.
Where a closed model would win: a frontier model would probably beat both of mine in one call. It just wouldn't run on my laptop, overnight, for free.
The Lazy Payoff
Queue your questions, go to sleep, close the lid, wake up to answers. The part I was proudest of is the boring one: nothing I did during those 17 hours mattered, and nothing broke.
Prize Categories
Best Use of Temporal: every tortoise step is a durable activity with heartbeats and unlimited retries. Night two survived 14 h 17 min of downtime and 91 failed model calls with zero finished steps redone, from a deterministic method shared with a local runner.
Best Use of Gemma: Gemma 3 4B is both contestants, running locally through Ollama with structured JSON output.
Best Use of a .tech Domain: the replay lives at tortoise.vaidehi-k.tech.
Best Use of ElevenLabs: the narration of the demo video is generated with ElevenLabs text-to-speech, timed line by line to the captions, which are themselves written from the real run data.
Top comments (0)