Ophyx is a 350M-parameter Python code model that I trained from scratch, and every step ran on Kaggle's free TPU, GPU and CPU quota. The finished model scores 43.9 on HumanEval+. Among the open instruct models under 1B that I tested, it leads on BigCodeBench and on my own 42 hands-on tasks.
You can chat with it in this site's terminal (type ophyx), call it from your own code through an OpenAI-compatible API, or download the open weights, which are Apache-2.0.
This post covers what's inside the model, how each training stage changed it, where it wins, where it loses, and how far it is from the frontier models people use every day.
Why I built it
I wanted to know how good a code model one developer can train without paying for compute. The goal was a model from scratch, with my own tokenizer, training loop and data pipeline, rather than a fine-tune of someone else's.
I kept the scope narrow on purpose. One language keeps the data focused. And at 350M parameters, the free TPU could pretrain on 12 billion tokens in about two weeks of quota.
What's inside
Every number here is from the model's config.json. The layout is Qwen3's, so standard tools load it without custom code.
Ophyx is a decoder-only transformer, the same kind of model as GPT or Claude, only far smaller. It reads text as tokens and predicts the next one, over and over, until it writes an end marker.
- Tokenizer: I trained a byte-level BPE tokenizer with 32,768 tokens on the training mix. It splits numbers into single digits, so 1024 is four tokens, never one opaque chunk.
- Embedding: each token becomes a list of 1,024 numbers. The same table is reused at the very end to turn vectors back into token scores, which saves 33.5M parameters.
- 28 layers: each layer has an attention block and a feed-forward block. Attention lets every token look back at the ones before it. Its 16 query heads share 4 key-value heads, which makes the memory a long answer needs four times smaller. RoPE, with a base of 100,000, tells the model where each token sits. The feed-forward block is SwiGLU, going from 1,024 to 2,816 numbers and back.
- Norms: RMSNorm before each block, and on the queries and keys inside attention, which keeps attention scores from blowing up during training.
- Next token: the last layer gives a score to each of the 32,768 tokens. The model picks one, appends it and runs again.
All of that adds up to 349M parameters and a 4,096-token context. I borrowed Qwen3's shape instead of inventing my own, so Hugging Face transformers, vLLM and llama.cpp run it as is. The weights, tokenizer and training code are mine.
The prompt format is just two markers: <|user|>, the task, then <|python|>, and the model writes Python until <|end|>. There's no system turn, which matters later.
Training it, stage by stage
Four stages, all on Kaggle's free quota. Every reward and every score comes from running the code.
Pretraining
The data was filtered Python files, Stack-Edu, the CPython library and its docs, programming text from FineWeb-Edu and synthetic exercises. Before training, I removed anything that matched the benchmarks (HumanEval, MBPP, BigCodeBench, CRUXEval and LiveCodeBench) with 10-gram and MinHash matching.
The trainer is my own JAX loop with the Muon optimiser, running at about 108k tokens a second on a TPU v5e-8. Twelve billion tokens at a 2,048-token context took five sessions and about 31 hours, and the base model scored 31.7 on HumanEval+.
Longer context made it worse at first
Two billion more tokens at 4,096 context, 11.7 hours on the TPU, dropped HumanEval+ from 31.7 to 28.0. The model started to ramble: 51 of 378 MBPP answers hit the length limit, against 19 before.
Fine-tuning taught it to answer
The first round, 600M tokens where every answer is one Python module, took HumanEval+ to 35.4. The second, 800M tokens at 4,096 context in 4.7 hours, added problems from my own bank of 8,729 with tests, plus 24,099 bug-and-fix pairs. It reached 37.8 on HumanEval+ and lost ground on MBPP+.
Averaging fixed most of the trade
Averaging the two fine-tunes weight by weight, 70% of the second and 30% of the first, scored 41.5: better than either one alone, for the cost of one CPU session. It didn't win everywhere. On CRUXEval input prediction and the hands-on tasks, it stayed below the first round.
Reinforcement learning closed it out
I used GRPO: the model writes eight answers to a problem, the sandbox runs each against the tests, and answers that pass are pushed up relative to the others. I picked the 1,399 problems from the bank where it solved between one and six of eight, because problems it always or never solves teach nothing. One pass took 5.4 hours on a single T4 and lifted HumanEval+ to 43.9.
A few things didn't ship. A DPO pilot, an earlier GRPO round and a fine-tune on contest problems all stayed within noise, so the release is the model above.
Living inside the free tier
Kaggle gives 20 TPU hours and 30 GPU hours a week, one TPU session at a time, and a TPU session could wait about nine hours in the queue. Most of the engineering went into living with that. Every session saves its checkpoint to the Hugging Face Hub, the next one resumes from it, and a watchdog queues the next session and pings my phone when a chain stops.
Benchmarks and the RL reward both run code a model wrote. That code only ever runs in Kaggle sessions with the internet off and no keys attached, in a child process with time, CPU, memory and file limits. In the RL reward, each answer also runs as its own Unix user.
Where it's strong
I ran 38 open models from other teams, from 0.14B to 8B, through the same setup: the same tasks, each model in its own prompt format, greedy decoding on Kaggle T4s, and scoring in the same offline sandbox.
Base models before any fine-tuning, pass@1 on EvalPlus. Green marks the best score in each column, even where that isn't Ophyx.
On library-heavy tasks and my hands-on set, Ophyx leads every open instruct model under 1B I tested.
- HumanEval+ at 43.9: ahead of Qwen3-0.6B (36.6) and SmolLM2-360M-Instruct (15.9). Behind Qwen2.5-Coder-0.5B-Instruct (54.9).
- BigCodeBench instruct at 21.0, on the 1,037 of its 1,140 tasks whose reference solution passes offline. These tasks need real libraries. That's about twice Qwen2.5-Coder-0.5B-Instruct (11.3) and Qwen3-0.6B (10.1), and close to deepseek-coder-1.3b-instruct (22.3), a model nearly four times its size. One caveat: 12 of its passing answers look close to the reference solution, all short tasks whose description spells out each step.
- BigCodeBench-Hard instruct at 4.6: more than deepseek-coder-1.3b-instruct and Qwen2.5-Coder-1.5B-Instruct, at 3.8 each.
- Hands-on tasks, 9 of 42: my own small tasks closer to real work, like data handling, classes and async code. Qwen3-0.6B solves 6 and Qwen2.5-Coder-0.5B-Instruct 5.
- The base model: before any fine-tuning, Ophyx scored 31.7 on HumanEval+. Qwen3-0.6B-Base scores 23.8 after 36 trillion tokens, 3,000 times more data than Ophyx saw in pretraining.
- Against 7B base models: on HumanEval+ the finished Ophyx is ahead of three of four, deepseek-coder-6.7b-base (38.4), CodeLlama-7b-Python (34.1) and starcoder2-7b (31.1). That compares a fine-tuned model with plain completion, so read it as perspective, not a win.
Where it's weak
- Only Python: it scores 0 on JavaScript, C++ and Bash. Ask for another language and it writes Python anyway.
- Library details and puzzles: 3.4 on DS-1000, 2.0 on LiveCodeBench (Qwen3-0.6B gets 7.9) and 21.9 on CRUXEval output prediction (about 25 for the Qwen models).
- Habits: I read all 42 hands-on answers by hand. It adds its own tests to answers, often returns quiet bugs unfixed and misses details in long prompts. On 33 everyday tasks (logs, CSV, SQLite, async), I count 1 right.
How far it is from frontier models
Frontier models follow the same recipe in outline: a tokenizer, a transformer that predicts the next token, pretraining, fine-tuning and RL on answers that can be checked. The differences are scale and what's built on top.
- Size: closed labs don't publish their sizes. DeepSeek-V3, an open model in that class, has 671B parameters, of which 37B work on each token: a router sends every token to a few expert blocks out of many. That's about 1,900 times Ophyx's total.
- Data: by their model cards, DeepSeek-V3 pretrained on 14.8 trillion tokens and Qwen3 on 36 trillion. Ophyx pretrained on 12 billion, and about 15 billion counting every stage.
- Thinking before answering: many frontier models write a long chain of reasoning before the answer, trained with RL on maths and code that can be checked. It's the same reward idea as my GRPO stage, run at a far larger scale.
- Reach: contexts of hundreds of thousands of tokens, dozens of languages, and tool use, so they can work through a whole repository as an agent.
Same tasks and the same offline sandbox. Frontier models answered through Kaggle Benchmarks in October 2026.
I ran seven frontier models through the same sandbox. On HumanEval+, Claude Sonnet 5.5 scores 95.7 and GPT-5.5 94.5: more than twice Ophyx. On the hands-on tasks GPT-5.5 solves 41 of 42 and Ophyx 9. On the 33 everyday tasks, five frontier models solve all 33. On SWE-bench Verified, where a model fixes real GitHub issues, the best score on its public bash-only leaderboard is 76.8%; Ophyx can't take that test at all with a 4,096-token context.
So, on one well-described Python function, Ophyx gets about half as many right as a frontier model. On anything bigger, the gap is most of the distance. For 0.35B parameters and 12 billion tokens on free hardware, half is more than I expected.
Running it on this site
The chat in this site's terminal runs the f16 build on four CPU cores with llama.cpp. The container shuts down 90 seconds after the last question, so the first question of a visit waits for a cold start.
The missing system turn matters here. A model this small, trained only on task-then-Python pairs, can't follow a written system prompt: text placed in front of the task just reads as part of the task. So the rules live in the serving code instead. Questions about Ophyx or me get fixed, true answers. Harmful requests and attempts to change its role get a refusal. Everything else reaches the model, general questions included, though a model trained only on Python answers those poorly. A test in the repo runs all 340 prompts from my own two task sets through those rules, and only one, the greeting “hi! what can you do?”, is answered without the model. A loop guard ends answers that start repeating themselves, and an optional concise mode strips comments and the test block it likes to add.
What I'd do next
- Fix its habits. The hands-on review says what to teach: answer without self-written tests, and follow every detail of a long prompt.
- More RL: one pass over 1,399 problems added 2.4 points on HumanEval+. More passes and harder problems are the cheapest next step.
- A second language: every non-Python benchmark is at zero, so even a little JavaScript data would show up.
The weights and GGUF files are on Hugging Face, and the code, data jobs and every benchmark result are on GitHub:
- Model: huggingface.co/abhash-chakraborty/ophyx-350m-v2
- GGUF for llama.cpp: huggingface.co/abhash-chakraborty/ophyx-350m-v2-gguf
- Code and results: github.com/Abhash-Chakraborty/ophyx
Originally published at abhashchakraborty.tech.
Top comments (0)