I trained my own 150M non-Transformer language model from scratch on 300M tokens — WarpState

zerdovzad — Sun, 30 Aug 2026 21:17:32 +0000

Hi everyone,

I’ve been experimenting with alternative language-model architectures for a while, and I recently finished the first complete pretraining run of a new architecture I’m calling WarpState.

This is still an experimental proof of concept, not a claim that it beats Transformers or existing state-space models.

The model has 150.13M parameters and was trained from scratch on roughly 300 million English tokens from Ultra-FineWeb L2.

The full run completed successfully:

Parameters: 150.13M
Training tokens: ~300.02M
Optimizer steps: 9,156
Sequence length: 1,024
Vocabulary: 32,768
Peak VRAM: ~4.52 GB

Final sampled validation:
Loss: 3.4309
Perplexity: 30.90

Training was done locally on a laptop GPU.

I’m attaching screenshots of the training logs and some generations from the final checkpoints.

What is WarpState?

WarpState is not a standard Transformer stack.

The basic idea is to combine three things:

Local tiled attention

Instead of global self-attention across the entire sequence, tokens are divided into fixed 128-token chunks.

Inside each chunk, the model uses normal causal scaled-dot-product attention.

All chunks can be processed as a large batched GPU workload during training, rather than running attention token by token.

So the local path is roughly:

tokens
↓
128-token chunks
↓
causal local attention
↓
local representation

Fast + slow tensor memory

Completed chunks are compressed into a persistent tensor memory.

For every attention head, WarpState maintains two matrices:

Fast State
Slow State

The fast state is initialized with a relatively short memory timescale, while the slow state is initialized to retain information much longer.

Conceptually:

current chunk
↓
K and U
↓
bounded tensor write
↓
┌───────────────┐
│ Fast memory │
│ Slow memory │
└───────────────┘
↓
future chunks

The memory write is based on a bounded outer-product-like update:

write = tanh(K)^T × tanh(U) / chunk_size

and the states are updated approximately as:

Fast = decay_fast × Fast + (1 - decay_fast) × write

Slow = decay_slow × Slow + (1 - decay_slow) × write

The decay rates are learned independently per head.

They start around:

Fast decay ≈ 0.90
Slow decay ≈ 0.99

The model also learns how much fast versus slow memory to read.

Learned routing between local attention and memory

For every token, the model produces a gate deciding how much information should come from:

local chunk attention
vs
long-range tensor memory

Approximately:

output =
gate × local_attention
+
(1 - gate) × memory_read

So the model can use precise local token relationships while relying on the compressed state for information from previous chunks.

Shared recurrent depth

Another unusual part of WarpState is that it does not have 16 completely separate large layers.

The current model contains only 4 physical WarpState cores, but they are reused across 16 logical depth passes:

Core 0
Core 1
Core 2
Core 3
Core 0
Core 1
Core 2
Core 3
...

Each logical depth has a small learned scale and bias, so the same physical core can behave somewhat differently depending on which depth pass it is being used for.

In simplified form:

x = x × (1 + depth_scale) + depth_bias

x → shared WarpState core

The intention is to get deeper iterative computation without duplicating every large weight matrix.

During autoregressive generation, every logical depth also receives its own independent memory cache, even when two depths share the same physical core weights.

Other details

The current version uses:

d_model: 1280
heads: 20
head_dim: 64
physical cores: 4
logical depth: 16
FFN hidden: 4480
chunk size: 128
RMSNorm
SwiGLU
RoPE inside each local chunk
tied input/output embeddings

The input projection is fused and produces:

Q
K
V
local/memory gate
memory U

from one projection.

Training results

The part I was most interested in was simply whether this architecture could survive a real pretraining run.

It did.

I trained it through the full ~300M-token run without NaNs, gradient collapse, or an obvious optimization failure.

Near the end of training, gradient norms were still sitting around roughly:

0.65 – 0.75

while the learning rate had already decayed to approximately:

3e-5

Peak allocated VRAM stayed around 4.52 GB.

The model also clearly learned language structure during training.

Very early checkpoints mostly produced English-shaped noise.

Later checkpoints started forming recognizable semantic clusters and reasonably structured paragraphs.

For example, when asked about Facebook, the final model associates it with things like:

online platform
social media
sharing content
sharing information
interaction with other people
community

It is definitely not a good chatbot yet.

There are still obvious failure modes:

repetition loops
semantic attractors
weak factual recall
occasional role confusion
long-generation degeneration

The model is also only base-pretrained.

There has been no instruction tuning, SFT or RLHF, so the chat screenshots I attached should be treated as qualitative probes rather than a chatbot benchmark.

Another important limitation is the training budget.

A 150M-parameter model trained on only 300M tokens has seen roughly:

~2 training tokens per parameter

so I consider this run primarily a proof that the architecture can train, rather than a fully trained 150M language model.

What surprised me most

The interesting part for me is that the architecture appears capable of learning meaningful language representations despite:

having only four large physical cores,
repeatedly reusing those cores,
restricting attention to local 128-token windows,
and moving information between chunks through fixed-size tensor states.

The long-range memory size therefore does not grow linearly with context in the same way as a conventional full KV cache.

There is still a lot I want to test before making any strong claims.

My next steps are probably:

deterministic evaluation over the entire validation set;
a parameter-matched Transformer baseline on exactly the same data;
analysis of the fast/slow memory states;
measuring long-context behavior;
investigating the repetition/attractor problem;
eventually testing a larger training budget.

For now I mainly wanted to share the first complete run because this was the point where the architecture stopped being only an idea and became an actually trained language model.

Feedback on the architecture is welcome, especially criticism of the memory update or shared-core design.

I scaled a pure Spiking Neural Network (SNN) to 1.088B parameters from scratch. Ran out of budget, but here is what I found

zerdovzad — Wed, 17 Jun 2026 10:26:23 +0000

Hey everyone. I’m an 18yo indie dev, and I’ve been experimenting with Spiking Neural Networks (SNNs) for language modeling. A lot of papers (like SpikeBERT) mention that training 1B+ SNNs directly from random initialization fails due to vanishing gradients, so people usually do ANN-to-SNN conversion or distillation. I wanted to see if I could force it to converge purely in the spike domain. I had to stop at 27k steps because my wallet is literally empty lol, but the loss converged to 4.4.

Here are the most interesting things that happened:

Massive Sparsity: It maintains ~93% sparsity. Only about 7% of neurons fire per token. It's incredibly cheap on memory during inference compared to dense models.

Cross-lingual emergence: Around step 25K, it randomly started generating structurally correct Russian text, even though it wasn't explicitly targeted/weighted for it in the dataset mix.

Memory routing shift: As I scaled the architecture past 600M to 1B, the model spontaneously shifted 39% of its activation routing into the persistent memory module. It basically learned on its own that memory is more valuable at a larger scale.

Limitations (Being honest):
The text generation is still janky and nowhere near GPT-2 fluency yet. The loss (4.4) is high, mostly because I couldn't train it longer. But proving that a 1B pure SNN can converge from random init feels like a solid milestone.

I'm sharing this because I'd love some harsh technical feedback.

Does anyone here have experience with neuromorphic hardware? Would an architecture like this map well to Loihi?

If anyone has tips on pushing SNN loss lower or stabilizing surrogate gradients further, I'm all ears.

The code, architecture details, and the 12GB full training checkpoint (weights + optimizer states) are on my GitHub:https://github.com/gtausa197-svg/-Project-Nord-Spiking-Neural-Network-Language-Model.git

We built a fully client-side 3D Animation Studio on Next.js (Three.js + WASM Physics).

zerdovzad — Mon, 01 Jun 2026 14:42:43 +0000

Hey everyone,

My friend and I built AnimaStage because we wanted MMD staging to feel zero-friction — open a tab, drop PMX/PMD and VMD, preview with physics, export without fighting desktop setup or weak hardware.

Live demos (no download, no login):

Lite (9:16 / Shorts-ready): https://animastage-lite.app

Pro (RTX-style lighting, bone editor): https://animastagepro.dev

Under the hood: Ammo.js WASM for physics and deterministic MP4 export. We're aiming to open-source more of the stack this summer.

We're building this for creators who want a smoother day-to-day workflow — and for people on weaker devices where full desktop MMD is rough. This isn't just a promo pitch; the goal is to keep it open and easy to use.

Would love your feedback on what to prioritize next: What's still missing for browser MMD vs desktop? Shorts export vs timeline/VMD editing — what matters more?

Anyone here using web tools for dance covers or VTuber prep?

Happy to answer technical questions. Not trying to spam — genuinely curious what this sub would actually use.

Thanks!

Demo video https://youtu.be/TUKa7emqVow

DEV Community: zerdovzad

I trained my own 150M non-Transformer language model from scratch on 300M tokens — WarpState

I scaled a pure Spiking Neural Network (SNN) to 1.088B parameters from scratch. Ran out of budget, but here is what I found

We built a fully client-side 3D Animation Studio on Next.js (Three.js + WASM Physics).