an attempt at hypernetworks on an RX 6600

Embrel Grim — Tue, 10 Nov 2026 08:06:39 +0000

Where the Idea Came From

I realized that my hardware couldn't run anything decent, so I decided to play it safe—buy proper components? Of course not—build my own architecture!

Full of confidence, I sat down at my desk and started hacking away. In short, the idea was this: train a small model that takes our prompt as input and prints a first layer tailored to it, runs the input through it, discards the layer, and prints the second one. In my crazy head, this was supposed to make the neural network more "suited" to the query (after all, it's generated specifically for it), and most importantly—drastically reduce the need for tens of gigabytes of fast memory.

To compensate for the difficulty of transferring general knowledge from the "printer model" to the layers, I chose an exquisite approach—I threw knowledge out entirely and plugged Wikipedia into the runtime. The strangest thing is that the idea actually seems to have worked.

What Came Out in the End

In practice, several more decisions were made—such as taking one base layer and generating a delta layer, then summing the hidden states, generating weights head-wise (each attention and MLP head gets its own matrices), adding a trainable depth gate (sigmoid(depth_scale[l]), so that each iteration enters the residual with its own weight), zeroing out W_o and W_out at initialization so the generated layer starts as a no-op, plus a byte morph-patcher for rare words and typos, and a compact vocabulary with prefix fallback. And more.

Ultimately, the idea took the following shape:

Base layer. A single real Qwen2 decoder layer—specifically layer #6 from Qwen2.5-0.5B.
Layer factory (CanonicalHeadwiseFactory). A small network that generates weights for the next layer based on the current hidden states and iteration number:
- AttentivePooler — attention pooling over hidden states → context vector;
- concatenation with the "frontier" (last position / anchor) → proj_in;
- depth_emb[l] — iteration index embedding;
- all this feeds into a GRUCell, which carries state across iterations;
- from the GRU output, two MLPs generate per-head matrices: W_q, W_k, W_v, W_o (d_head × d_head) for attention and W_in (d_head × gen_mlp), W_out (gen_mlp × d_head) for MLP.
Generated block (CanonicalGeneratedBlock). The same structure as a Qwen2 layer (pre-norm → attention → residual → pre-norm → MLP → residual), but the attention and MLP weights come from the outside, via the factory. A single physical block is reused across all depth iterations—only the weights change.
Assembly.

   h = embed(tokens) + 0.1 * morph(bytes)
   g = g_init
   for l in 0..depth-1:
       g, W_attn, W_mlp = factory(h, g, l)
       base_out = base_layer(h)
       gen_out  = gen_block(h, W_attn, W_mlp)
       h = h_prev + σ(depth_scale[l]) · (base_out + gen_out − h_prev)
   logits = lm_head(norm(h))

Anchor. The factory sees only the prompt (positions < anchor_len), not the response. This eliminates future leakage during training, makes training mode equivalent to generation, and allows KV-cache reuse within a segment.
Compact vocabulary. Instead of Qwen's 151k tokens—11,264 tokens selected by coverage (0.999 of the corpus). Rare tokens map to the longest prefix in the dictionary. Tool markup (<tool_call>, arguments, expression, etc.) is forcibly included to prevent JSON from breaking at inference time.
Byte patcher. For each token, its 8 bytes are taken → Embedding(256, 32) → MLP → d_model. Added to the regular embedding with a 0.1 weight. Helps with typos, rare words, and OOVs.

How It Was Trained

The qwen36m preset in build_neuroset.py (available on GitHub, link below):

init: layer #6 from Qwen2.5-0.5B;
depth=8, base_layers=1, d_model=896, n_heads=14 × d_head=64;
compact vocab: 11,264, strategy coverage, 0.999 coverage;
max_seq_len=512, batch_size=4 × grad_accum=4;
lr=1.5e-5 for the factory, lr_base=3e-6 for the base layer;
budget: ~8 hours on an RX 6600 (8 GB), ~60M tokens;
masking on assistant tokens, boosted loss weight on tokens inside <tool_call>…</tool_call>.

Demo

After many sleepless nights of failure and tormenting my RX 6600, I finally got a working 36-million parameter network, which is what we'll discuss next. Let me brag a little with some test runs:

usr> how much is 47 * 13?
agt> <tool_call>
{"name":"calculate","arguments":{"expression":"47 * 13"}}
</tool_call>
<tool_response>{"content":"611"}</tool_response>
Total: 611.

usr> what is the capital of France?
agt> <tool_call>
{"name":"web_search","arguments":{"query":"capital of France"}}
</tool_call>
<tool_response>{"content":"France is a sovereign country in Western Europe. Its capital is Paris…"}</tool_response>
Per Wiki: France is a sovereign country in Western Europe. Capital — Paris…

usr> hello!
agt> Hello. I'm listening.

The model decides on its own when to call a tool, and synthesizes the answer itself based on the retrieved fact.

Comparison with Same-Weight Competitors

============================================================
Model: veyra-ai/Veyra-30M-Instruct  (prompt language: en)
============================================================
usr> What is 47 * 13?
agt> Course Unit: Football Festival in the Himalayas

Hello there! I hope you're enjoying your meal and having fun. I'm sure you're going to do great things together! I'm an AI and I can't wait to share with you that. Would you like me to do that?

usr> What is the capital of France?
agt> In the realm of the bustling city of Paris, where the city's rich history and diverse culture intertwine the serene and serene landscape of the city, I stand before the grandeur of the bustling metropolis of Lisbon. This bustling metropolis, where the city's rich history and diverse culture intertwined, was born, is celebrated for its stunning landscapes and vibrant culture. As a result, the city's rich history and diverse culture are celebrated for its stun

usr> Hello!
agt> In the realm of technology and technology, the concept of "technology" is crucial for understanding the intersection of technology and technology. This concept is crucial in the field of technology and engineering, as it provides a framework for understanding how technology can be used to enhance or modify processes and systems. By examining the interactions between technology and engineering, we can gain insights into how these systems are developed and developed. This understanding can lead to more effective and effective technologies that are both practical and innovative. Moreover, it highlights the importance of interdisciplinary collaboration in address

============================================================
Model: VGribAI/Tensor-2-40m-instruct  (prompt language: ru)
============================================================
usr> Сколько будет 47 * 13?
agt> Чтобы найти вопрос 90 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15  15  1

usr> Какая столица Франции?
agt> Это задача про китайского мира, который вы можете использовать для создания и уважения. Вот несколько шагов, которые могут помочь:1. Установка: Затем выполните следующие шаги:1. Программа на создание типа: Законы искусственного интеллекта, котор

usr> Привет!
agt> Как я могу помочь тебе сегодня?Я здесь, чтобы помочь тебе сегодня?Я помогу вам сегодня?Я здесь, чтобы помочь тебе сегодня.Я здесь, чтобы помочь вам сегодня.Пользователь: Я могу помочь вам сегодня. Я могу помочь вам сегодня.Пользователь: Я

As you can see, our architecture performs much better. And in terms of speed:

Model	Params	Prompt	ms	tok/s	byte/s	Bytes	All 10 runs in ms
Veyra-30M-Instruct	34,611,712	22	355.2	140.75	579.87	2060	3552.5
Tensor-2-40M-Instruct	40,036,832	3	787.7	63.47	224.69	1770	7877.4
Neuroset assistant8	35,849,352	11	579.2	86.33	169.20	980	5791.9

We pay for intelligence and low parameter count with speed; however, this is primarily driven by aggressive tokenizer chunking, and it is also compensated for by huge weight compression, allowing one to buy less memory with higher bandwidth.

Benchmarks

Instruction Following

Each model responds to 6 equivalent tasks in its native language, and the results are compared directly. Correctness is evaluated (for arithmetic/capital/person — presence of the fact; for the rest — a coherent response in its language without repetitions).

Task	ours_36m	TinyStories-Instr	Stentor-30M	Veyra-30M	Tensor-2-40M
Hello!	✅	❌	❌	❌	⚠️ 50/50
How much is 47 × 13?	✅	❌	❌	❌	❌
What is the capital of France?	✅	❌	❌	❌	❌
Who is Pushkin?	✅	❌	❌	❌	❌
What is gravity?	✅	❌	❌	❌	❌
What is the largest planet?	❌	✅	❌	❌	❌
TOTAL	5/6	1/6	0/6	0/6	0.5/6

Our 36M model handles almost all tasks—including exact arithmetic via tool use and facts from Wikipedia. Competitors of the same size failed on these specific prompts: they fell into repetition loops or produced irrelevant text.

Tool Use Skill (Our Model Only)

Deterministic evaluation (91 checks; tool responses are fixed to isolate model intelligence from search noise):

Metric	`assistant8` (36M)	`assistant_big6` (129M)
Query formulation (query)	27/39	29/39
Fact marker in final answer (answer)	22/39	35/39*
Arithmetic (calculate)	7/8	8/8
Calibration (do not call tools for chitchat)	5/5	5/5
Synthesis faithfulness (faith)	0.42	0.75

129M was evaluated using an older script and has not been recomputed with the new eval_det, see GitHub for details.

Scaling: 129M on Very Little Data

runs/assistant_big6 (128.99M: base layer from Qwen2.5-1.5B, d_model=1536, vocab 32,768) is an example showing that even with a tiny amount of training data, the architecture can already produce coherent speech. It was trained for ~45 hours on a single RX 6600 on the same data mix (~60M tokens; open-source models of this size are typically trained on trillions).

usr> tell me about Mars
agt> Per Wiki: Mars is the fourth planet from the Sun in the Solar System…
usr> what is gravity?
agt> From Wiki: Gravity is a universal fundamental interaction…
usr> hello!
agt> Doing well, thanks! What's the question?

This is not a head-to-head comparison with competitors, but rather a demonstration that the architecture scales and remains conversational at ridiculously low compute costs.

Limitations / What Didn't Work

Entity copying into queries is unreliable on rare words (occasional typos in the query)—a capacity limitation of 36M.
Synthesis sometimes distorts facts (faithfulness 0.42 in 36M); a short tool response helps.
Lead-paragraph retrieval does not always answer attribute-based questions.
English is not the target language (the model is Russian-speaking).

Reproduction Commands

# stage 1 — full SFT
python build_neuroset.py --preset qwen36m --data-dir dataset1 --out runs/base \
  --hours 5 --max-seq-len 768 --lr 2e-4 --lr-base 4e-5 --warmup 400 --yes

# stage 2 — continuation on new mix
python build_neuroset.py --preset qwen36m --data-dir dataset2 --out runs/base2 \
  --init-ckpt runs/base/checkpoints/best.pt \
  --vocab-pack runs/base/compact_vocab.pt \
  --hours 1.5 --max-seq-len 768 --lr 6e-5 --lr-base 1.5e-5 --warmup 120 --yes

# stage 3 — assistant: tools + RAG
python build_neuroset.py --preset qwen36m --data-dir dataset3 --out runs/assistant8 \
  --init-ckpt runs/base2/checkpoints/best.pt \
  --vocab-pack runs/base2/compact_vocab.pt \
  --max-tokens 40000000 --hours 1 --max-seq-len 768 \
  --lr 4e-5 --lr-base 1e-5 --warmup 200 --yes

Conclusion

The model turned out surprisingly capable—honestly, I didn't expect this, as it was built (vibe-coded in parallel with other projects) and slapped together on the fly.

This is my first article on Habr, so I'd appreciate any feedback on the presentation and methodology. I understand that the benchmarks here serve as a sanity check rather than a systematic evaluation: the sample size is small, and competitor models were tested without prompt tuning. More comprehensive metrics can be found in the project's README—I didn't include them in the article to keep it concise.

For those interested in exploring the code, digging deeper, running it locally, or maybe even training it—welcome to GitHub: https://github.com/RUD9/Grimoire.

DEV Community: Embrel Grim