I built StacksNG for the Africa Deep Tech Challenge 2026: an offline coding assistant grounded in the docs of four Nigerian fintech APIs: Paystack, Flutterwave, Monnify, Termii. It runs on retrieval, not memory (RAG, if you know the term): pull the real doc passage before answering, cite the source. 780 scraped chunks, running entirely on-device via llama.cpp, the engine that runs the model locally. No cloud, no API keys, no hallucinated endpoints in payments code.
I didn't make the semifinals.
No feedback came with that. So I went looking myself.
StacksNG shipped qwen2.5-coder-7b, stock, compressed down to fit in memory (Q4_K_M quantization). That's the largest model that fits the challenge's 8GB envelope, and it's a deliberate bet against the scoring formula, not an oversight. I wrote the ablation into the report myself. A smaller, 1.5B version of the same pipeline wins the formula by about 35 points: faster, lighter, better on the speed and memory-efficiency scores. I rejected it anyway, because on one of my two registered test prompts, the 1.5B opened with:
import flutterwave
client = flutterwave.Client(...)
That library doesn't exist. The 7B, same retrieval context, wrote real requests calls against real documented endpoints. I picked correctness over the formula and said so in the report. Fine engineering decision. Turned out to be the wrong thing to be defending.
I cloned all 20 published semifinalists (full git history, in case anyone privatizes their repo after judging) and read every technical report. SME-Ledger, Jamii Afya, TaxSabi, CodeFellow, ARIS, Homa, all of them: LoRA, QLoRA, distillation, or a merge. Every single one touched the base model somehow.
StacksNG was the only stock-model architecture in the batch. I noticed the pattern, filed it under "interesting," and moved on. That was the first mistake.
My own submission.json:
"accuracy": []
Empty array. Not a low number. No number. git log on that file shows exactly one commit, never touched again before the deadline. The adtc-profiler run that produced it either skipped the accuracy gate or lm_eval wasn't installed at the time. Same failure mode either way: silent, not an error.
Sacc is 50% of the scoring formula. I'd shipped half the score as zero without meaning to.
Trying to fix that, I installed lm-eval-harness and pointed it at my own model through a local llama-server. It crashed. Every one of 200 requests came back "Invalid logprobs data."
Two bugs, both confirmed by reading the actual source:
-
adtc_profiler's ownaccuracy.pycallslm_evalwithbase_url=local. Not a URL. lm-eval's GGUF backend needshttp://host:portand POSTs to{base_url}/v1/completions. As shipped, that line fails for anyone who runs it literally. -
lm-eval-harness's GGUF backend expectsecho=trueto return logprobs for the whole echoed prompt, the old OpenAI completions behavior. Currentllama-serveronly returns logprobs for newly generated tokens. The harness hasn't caught up.
I patched both locally. Instead of trusting the broken echo behavior, I forced the model to generate the exact answer text (a grammar rule constraining what it's allowed to output) and read its confidence scores off that generation directly. Ran the hackathon's own default benchmark, a standard general-knowledge multiple-choice test. Got a real number:
arc_easy acc_norm = 0.74
Competitive. Ahead of one fine-tuned semifinalist, just behind two others. Plugged into my own formula: Stotal goes from the submitted 10.23 to 47.23. A 4.6x swing from one missing number, no architecture change.
I thought that was the end of it.
Your submission's originality score did not meet the threshold required to advance.
Originality Score (0–10): 3
Model Origin (Originality Review): Stock model, used as-is: Qwen2.5-Coder-7B-Instruct-GGUF from lmstudio-community's official quantization.
Round 1 runs an originality gate before any technical scoring even happens. Template compliance: fine. The RAG architecture, the citation grounding, the 780-chunk corpus: not mentioned. The problem was never Sacc. The problem was that I never touched the model.
All the archaeology (the broken base_url, the echo/logprobs mismatch, the 0.74) was real, reproducible, and completely beside the point. I'd spent a week debugging the wrong layer.
The pattern was sitting in my own data the whole time. Eighteen other teams fine-tuned. I read that fact, found it interesting, and filed it as a scoring-formula tradeoff instead of what it actually was: everyone else had cleared a bar I didn't know existed.
That's the uncomfortable part. The evidence was already in front of me before the rejection email arrived. I reasoned right past it anyway.
I was looking at the parts of the system I could inspect: the formula, the profiler source, the tooling. The part I couldn't see was the one that mattered: a review that runs above all of it, undocumented in any code or rule I had access to.
A RAG layer grounding a stock model is a defensible product decision. It is not, to an originality reviewer, evidence that you built anything. Those are different bars, and clearing one says nothing about the other.
If you're entering a hackathon with a scoring formula published anywhere: read it. Optimize against it. But ask what isn't in the formula too. A judging process can have a gate that never shows up in the rubric or the tooling. The only way to find it is to assume it exists and ship something that survives it anyway.
Ship a real modification. Even a light LoRA pass on your own corpus would have cleared this. The application layer was good work: retrieval, citations, the deterministic-gate bugs I fixed after a DEV.to reader found them in production. None of it was ever going to be evidence that the model itself was mine.
Good luck to the eighteen who advanced. You did the part I skipped.
StacksNG: github.com/dannwaneri/stacksng. Two tooling bugs found during this postmortem, filed upstream: adtc-profiler#4 and lm-evaluation-harness#4158.
Top comments (8)
The empty accuracy array and the originality rejection look like two different failures that a judging system should never collapse into one score. I’d want the result to expose at least three states: not measured because the evaluator failed, measured and failed a metric, or rejected by an earlier eligibility gate. Otherwise participants debug the wrong layer, as this example shows, while the actual decision rule remains invisible. A decision trace per gate would be more useful than a single final score.
The 3 states are exactly what my submission.json couldn't distinguish. Empty accuracy array looked identical whether the tool wasn't installed, silently errored on a bad base_url, or I now know, was never going to matter because Round 1's originality gate had already rejected the submission before any of it ran.
The gate-reason-plus-evidence part is what I'd push hardest on. Even one line, "rejected at originality gate, reason: stock model," would've saved a week debugging the wrong layer. Is that the trace you're picturing, or something closer to a full audit log per stage??
Closer to a compact decision trace by default, with the full audit log behind it. For each stage I’d expose gate/version, timestamp, verdict, reason code, evidence refs, and which downstream stages were skipped. In your case the visible result could have been: originality_gate: rejected — stock model; accuracy: not_run — prerequisite gate failed. That distinction is the product requirement. The full event log is useful for appeal and debugging, but entrants shouldn’t have to reconstruct the decision from it.
This is a good example of why evaluation needs a decision trace, not just a final score. I’d make the preflight contract explicit: validate that every required metric has a producer, record whether it was measured or skipped, and fail closed when an evaluator returns an empty field. Then keep eligibility gates separate from technical scores, with the gate reason and evidence attached to the result. That would have surfaced both “accuracy not measured” and “originality gate failed” before submission, instead of making the debugging path look like a model-quality problem.
That's the gap exactly: my adtc-profiler run had three completely different causes (lm_eval not installed, the base_url bug, and presumably nobody running the accuracy step at all) collapsing into the identical "accuracy": []. No way to tell which from the output alone.
The originality gate is the real finding here, not the two bugs. I've hit that same silent-zero pattern testing eval harnesses against quantized local models. A mismatched field doesn't throw, it just returns a number that looks plausible and is wrong. You only catch it if you go looking for the zero that shouldn't be there. Same story with the RAG-on-stock-model verdict. The retrieval and citation work is real engineering, but if the reviewer only checks whether you touched the weights, none of the application layer counts as evidence. The scoring formula and the actual bar aren't always the same document.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.