My submission for the Hacktoberfest Open-Source AI Challenge: Week 3 — "Slow and Steady." 🦥
Repo: https://github.com/garanews/slothctf
The lazy payoff
Writing reverse-engineering CTF challenges by hand is tedious: you invent a scheme, write the C, compile it, write a solver, double-check the flag actually comes out, make sure strings doesn't give it away… for every single task.
I wanted to type one sentence and get a finished, verified challenge:
python main.py generate --name xor_demo \
--prompt "Create an easy reverse engineering challenge with XOR encryption" \
--model qwen2.5-coder:32b
SlothCTF does exactly that. A local open-weight model writes the whole challenge in C; the tool compiles it, runs it, and proves it works before I ever look at it. I fed it the 20 challenge ideas I'd been meaning to build and got 20/20 valid challenges out — source, compiled binary, player README, and an organizer walkthrough each. The slow, boring part now happens while I get a coffee. Very on-brand for International Sloth Day.
The twist nobody warns you about: LLMs can't do byte math
My first design asked the model to do everything, including computing the hidden bytes — "store the flag XORed with 0x42 as an array." Every model I tried got the arithmetic wrong, from Gemma 3 4B up to Qwen2.5-Coder 14B. It would declare the flag sloth{easy_reverse} and then emit an array that decoded to LCTE…. An LLM simply cannot reliably compute flag[i] ^ key across 20 bytes in its head.
This is the same "LLMs confidently hallucinate in reverse engineering" problem others have hit — see Stopping LLM Hallucinations in Reverse Engineering: Meet Reverify by qisanxi and, for the CTF-authoring angle, NiceTryGPT: Less pattern matching. More actual hacking. by aleff.
The fix: let the machine do the arithmetic
So I stopped asking the model to compute anything. Instead it writes two C programs, and the machine runs them:
model → encoder.c (organizer tool) + challenge.c (has a placeholder for the bytes)
→ compile & run encoder on the secret → EXACT bytes (machine math)
→ inject bytes into challenge.c, compile, run → behaviour check
→ decode: stdout == flag | check: right input accepted, wrong rejected
→ strings(binary) must NOT contain the plaintext secret
↺ any failure is fed back to the model for another attempt
The model invents the scheme and writes the code; the compiler does the exact byte math it's bad at. No templates, no per-scheme if ladders. Here's a real generated pair (XOR):
/* organizer/encoder.c — run on the flag to produce the bytes */
int main(int argc, char **argv) {
for (char *p = argv[1]; *p; ++p) printf("0x%02x,", (unsigned char)(*p ^ 0x42));
}
/* src/challenge.c — the player's binary; the machine filled in the array */
static unsigned char data[] = { 0x31, 0x2e, 0x2d, 0x36, 0x2a, 0x39, /* … */ };
int main(void) { for (size_t i = 0; i < sizeof data; ++i) putchar(data[i] ^ 0x42); }
Two kinds of challenge
-
decode— the binary decodes the embedded array and prints the flag. -
check— the binary reads stdin and accepts only the right secret (password checks, hashes, ciphers, state machines, anti-debug gates…).
That second mode covers the classic "crack the binary" tasks. The verifier feeds the real secret (must be accepted) and a wrong one (must be rejected), and confirms the plaintext never shows up under strings.
See it run
All 20 of my prompts, re-validated independently — each one is compiled, run, its flag checked, and strings-checked:
$ for n in generated/*; do python main.py validate --dir "$n"; done
xor_demo OK state_machine_demo OK
obfuscation_demo OK func_pointers_demo OK
anti_debug_demo OK constructor_demo OK
custom_b64_demo OK fnv1a_hash_demo OK
tea_crypto_demo OK sbox_crypto_demo OK
rc4_cipher_demo OK dynamic_imports_demo OK
junk_code_demo OK math_equations_demo OK
multithread_check_demo OK custom_vm_demo OK
timing_attack_demo OK anti_vm_demo OK
self_modifying_demo OK seh_exception_demo OK
VALID 20/20
A decode challenge, end to end:
$ cat generated/xor_demo/challenge.json
{ "title": "XORed Flag", "difficulty": "easy",
"flag": "sloth{easy_xor}", "challenge_kind": "decode" }
$ ./generated/xor_demo/bin/challenge
sloth{easy_xor}
A check challenge (anti-debug) — correct input wins, wrong input loses, and the secret is never in the binary:
$ echo 'sloth{DebugMeNot}' | ./generated/anti_debug_demo/bin/challenge
Access Granted
$ echo 'wrongpass' | ./generated/anti_debug_demo/bin/challenge
Denied
$ strings ./generated/anti_debug_demo/bin/challenge | grep DebugMeNot
(nothing — the plaintext secret never appears in the binary)
A couple of lessons that cost me attempts
-
Tell the model where it compiles.
ptrace(Linux) and MSVC__try/__exceptdon't exist on Windows/MinGW. Adding a one-line "BUILD TARGET: Windows/MinGW, no ptrace/SEH" to the prompt flipped the anti-debug and exception challenges from fails to valid on the first try (the model reached forIsDebuggerPresentinstead). - Hashes and block ciphers are the hard case. FNV-1a and TEA need two byte-identical implementations (encoder + challenge) to agree. The fixes: emit the target one byte per token (don't print a 32-bit word as a single token — a pitfall also worth reading Clean JSON Extraction with Ollama and Python by nearshi about), and tell the model to copy the transform function verbatim into both programs. That got me the last 2 of 20.
Why open innovation matters here
This only works because it's open and local:
- It runs offline on open-weight models via Ollama. No cloud API, no per-call cost, no challenge ideas leaving my machine — which matters when the "challenges" are security tasks.
- I can swap models freely. I watched the exact same pipeline fail on a 4B and succeed on a 32B; that experiment is only possible when you control the model. (For picking a local coder model by VRAM, Best Local LLM for Coding: 8GB to 24GB VRAM Picks by mrsaynothing is a good map.)
- "Slow and Steady" is the whole point. The model is allowed to take minutes and several tries, because a deterministic verifier catches every mistake. I don't need it fast; I need it correct, and I'm not the one waiting.
Results
With Qwen2.5-Coder 32B on a borrowed A100: 20/20 of my prompts produced valid, verified challenges — XOR, custom Base64, S-box, RC4, TEA, FNV-1a, a bytecode VM, state machines, function-pointer checks, anti-debug, timing side-channels, and more. Small models follow the two-program protocol poorly; 32B is the sweet spot.
Which brings me to my one serious piece of engineering advice: the cheapest way to make a local 32B model pleasant to iterate on is, regrettably, to just buy yourself a local A100. 🦥💸 (I'm told a workstation with 512 GB of RAM also works, slowly. The sloth approves.)
Try it
git clone https://github.com/garanews/slothctf
cd slothctf
python main.py generate --name demo --prompt "a reverse challenge with a custom cipher" \
--model qwen2.5-coder:32b --ollama-url http://localhost:11434
python main.py validate --dir generated/demo
Everything — the generator, the self-verifying loop, the two challenge kinds, remote-Ollama support, and tests — is in the repo: https://github.com/garanews/slothctf
Next up: a GUI with a small database to browse generated challenges by type and difficulty, and support for the other classic CTF genres (pcap, memory dumps, disk forensics, logs, OSINT).
Update — October 22, 2026
Thanks to the discussion in the comments, a few things changed since I first
published this. Two came straight out of reader feedback:
-
--roundtrip— after the chosen flag verifies, the harness re-runs the real pipeline (encode → inject → compile → decode) on random inputs and requiresdecode(encode(x)) == x. It's a property-based check over a few samples that gives extra confidence the transform is invertible across different inputs — not a proof that it inverts every possible input. -
--adversary— a budgeted reference attacker (a dictionary of common secrets + a short brute-force) runs againstcheckchallenges and rejects a medium/hard one it cracks. It's a first automatic filter against trivially guessable secrets, not a global difficulty assessment: a random 20-char secret passes it yet could still be reversed in seconds. Judging true reversing difficulty is out of scope.
To be precise about guarantees: the solid part is the independent oracle —
we compile and run the model's actual code and check the real output, plus a
strings leak check, so the model is never both author and sole verifier. The
two gates above are opt-in, experimental, and run only at generation time.
The repo has also moved past what this article describes. Beyond reverse/crypto
it now generates log-analysis (forensics) challenges — the model writes a
synthetic-log generator and per-question extractors, verified by re-running them
(each kept question is proven, broken ones are dropped) — and it can also turn a
log you already have into questions. There's a small SQLite catalog and a
dependency-free web GUI to browse, generate, and re-validate challenges.
Code (ahead of this post): https://github.com/garanews/slothctf
Top comments (7)
"LLMs can't do byte math" is the most useful engineering insight in this article, and it applies way beyond CTF generation.
I've seen the same problem in code generation: the model writes a perfect-looking function, but the hardcoded constants are off by one. The fix is always the same — don't ask the model to compute, ask it to write code that computes.
Curious if you've tried the "let the model write a test that computes it" pattern. Instead of injecting the bytes directly, ask the model to write a unit test that asserts
decode(encode(x)) == x. If the test passes, the math is correct.Thanks — that principle ("don't ask the model to compute, ask it to write code that computes") is the backbone of the whole thing, and your off-by-one constant example is the exact failure I hit.
Your unit-test idea nudged me to add a gate, and it's live now. One twist first: rather than trust a model-written assert decode(encode(x)) == x, the harness already runs the model's encoder to produce the real bytes, injects them, compiles and runs the actual challenge, and compares to the declared flag — an independent oracle, because if the same model writes both the code and the test, a wrong constant and a wrong assertion can agree with each other.
What your idea added: I was only verifying the single instance (the one flag). So I wired in a --roundtrip gate — after the flag passes, it re-runs the real pipeline (encode → inject → compile → decode) on several random inputs and requires decode(encode(x)) == x for each. No model-written test; it reuses the actual encoder + challenge, so it tests the real transform.
It immediately earned its keep: I fed it a decode that uses one key for the first 8 bytes and a different one after — it passes for an 8-char flag, so the single-instance check was happy, but the round-trip caught it instantly on a longer random input. Exactly your "perfect-looking function, constant off" bug.
The other half I'd add, from the CTF side: a negative test too — I also run the solver against a decoy and require it to fail, which catches the "model just hardcoded the answer" cheat that a pure round-trip wouldn't.
"The harness runs the model's encoder to produce the real bytes, injects them, compiles and runs the actual challenge, and compares to the declared flag" is the right architecture, and it's the part most people get wrong.
Trusting a model-written test is the same mistake as trusting a model-written assertion. The independent oracle is the only way to know the math is actually correct.
The negative test (decoy solver must fail) is also a great addition. It catches the "hardcoded answer" cheat, which is the most common shortcut models take when they can't solve the problem.
Curious how you'd handle the "the model writes a challenge that's too easy to solve by brute force" case. Does the round-trip gate catch that, or is that a separate check?
Great question — and no, the round-trip gate doesn't catch that. Round-trip only proves correctness (the transform is a true inverse); it says nothing about how hard the result is to solve. Brute-forceability is a separate axis.
It's also subtle, because for some challenges brute force is the intended solve — single-byte XOR is "easy" precisely because you try 256 keys. So the real question isn't "is it brute-forceable?" but "does its brute-forceability match the declared difficulty?"
Your comment nudged me to build it, so it's in now: a --adversary gate that's the same independent-oracle move pointed at difficulty. It runs a budgeted reference attacker against check challenges — a dictionary of common secrets plus a short brute-force — and rejects a medium/hard challenge it cracks within budget. decode challenges solve by being run, so the gate skips them.
It earned its keep immediately: a challenge whose secret was sloth{password}, labelled hard, got rejected as too-easy (the dictionary found it in a handful of tries), while the same setup with a random secret passed. So "too easy to brute-force" becomes a measurable property — again by making an automated attacker try, not by asking the model how hard it is.
And you're right that the hardcoded-answer negative test is the degenerate case of exactly this: an attacker that does nothing and still wins.
"Does its brute-forceability match the declared difficulty?" is the right question, and it's the one most CTF authors never ask.
A challenge can be technically correct (round-trip passes) and completely trivial to solve (dictionary attack). The
--adversarygate catches that mismatch, which is exactly what a difficulty label should mean.Curious how you handle the "the attacker is too good" case. If the reference attacker is too strong (e.g., a huge dictionary), it might reject challenges that are actually fine for humans. Do you have a way to tune the budget?
Exactly the right worry — a too-strong attacker over-rejects, and that's the failure mode I care about more than missing a weak one. A false "too easy" on a challenge that's actually fine for humans is worse than letting a borderline one through.
Two design choices keep it from being "too good":
The dictionary is deliberately small and lazy — common passwords plus a short brute-force, modelling a first-5-minutes guesser, not a determined reverser. The gate's job is only to catch trivially guessable secrets, so I keep the attacker intentionally weak.
The budget scales with the declared difficulty, and it's tunable. hard must resist a stronger attacker than medium (defaults: medium 800 tries/8s, hard 2000/20s), and you can override it with --adversary-budget ATTEMPTS,SECONDS.
I just wired the tuning in, and it behaves as you'd expect: I took one secret reachable only around the ~1000th brute-force attempt — with a 900-try budget it passes, with 1500 it's rejected as too-easy. So the boundary is an explicit knob, not a hidden constant.
I'd frame it honestly though: this is a first filter against guessable secrets, not a difficulty meter. A random 20-char secret sails through the gate and could still fall to five minutes of reversing — judging real reversing difficulty is a different (and much harder) problem.
"A false 'too easy' on a challenge that's actually fine for humans is worse than letting a borderline one through" is the right priority.
Most automated graders optimize for "catch everything", which means they reject 30% of valid submissions. Yours optimizes for "don't reject valid submissions", which means it misses some bad ones. The second failure is much easier to live with.
The
--adversary-budgetknob is also a good design. It turns a hidden constant into an explicit policy decision.