Extreme DQT with an FP4 Optimizer

Training an INT1 Transformer with a Novel FP4 Optimizer

Abstract

Impeto is a from-scratch, CPU-only transformer framework built to test how far weight quantization can actually be pushed, down to INT1 (single-bit) weights, trained directly rather than quantized after the fact (Direct Quantized Training, DQT). Everything here is written in C++ with hand-written AVX2 kernels: no PyTorch, no CUDA, no external ML libraries.

Direct training of 1-bit weights isn't new on its own. BitNet and its successors have shown this works at GPU scale. What Impeto adds is a specific combination I haven't seen elsewhere:

1. An FP4 optimizer. The model's weights are INT1, but the optimizer's internal error-accumulation buffer is also quantized, to FP4 (E2M1) with stochastic rounding, rather than kept in full precision. In direct comparisons at matched scale, this FP4 optimizer state consistently outperformed an FP8 version, especially early in training. My reasoning for quantizing the error buffer this aggressively is that since the weights themselves are binary (-1/+1), the optimizer only ever needs a rough sense of direction. Precise gradient magnitude may be wasted information for a decision that's ultimately just "flip or don't."

2. DDReGLU (Decoupled Double Rectified Gated Linear Unit). A gated activation function with learnable per-neuron thresholds, designed with a built-in dying-neuron recovery mechanism.

3. A from-scratch attention stack. Grouped-query attention (GQA) and Rotary Positional Embeddings (RoPE), implemented from scratch including backward passes, with no autodiff framework.

4. A pure C++/AVX2 CPU implementation. No PyTorch, no CUDA. Every kernel, every optimizer step, every backward pass hand-written and dispatched to AVX2 where available.

For tokenization, I used YouTokenToMe (YTTM, MIT licensed) rather than writing my own BPE implementation. That part of the stack isn't novel, and reimplementing a solved problem wasn't a good use of my time.

Why Quantize to INT1?

Quantization generally shrinks the memory a model uses, makes it more efficient, and lowers deployment cost. Quantizing to INT1 (1-bit integer) takes this further than usual: the weights can only be +1 or -1, which changes how the model behaves far more than a milder form of quantization would. This project exists to explore what training actually looks like under that constraint: INT1 models, direct quantized training, and quantized optimizer states.

Model Overview and Training

Impeto's flagship configuration for this article is a transformer with:

• 8 layers
• d_model = 192
• 4 attention heads, head_dim = 48 (n_heads * head_dim = d_model, as it should)
• 2 KV heads (GQA, 2:1 ratio)
• d_ff = 768 (4 * d_model)
• vocab_size = 2048 (custom BPE, trained on TinyStories directly)
• seq_len = 256

That works out to about 5.26 million parameters.

Training budget was set using the Chinchilla scaling-law formula (tokens needed is roughly 20 times the parameter count for compute-optimal training), with the multiplier raised for INT1 specifically, since a 1-bit model needs more updates per parameter to converge than a full-precision one would. The training loop computes this budget automatically at startup: parameter count, needed tokens, total steps, and how often to validate, all derived from the model's actual dimensions rather than hardcoded.

The model trained on TinyStories, sampling training windows randomly, running validation passes periodically, and saving the best checkpoint by validation loss automatically as training progressed.

Attention

Attention uses standard grouped-query attention (GQA) with rotary positional embeddings (RoPE), implemented from scratch including the backward pass, with no autodiff. The interesting part isn't the attention mechanism itself, it's well understood at this point. What matters here is that it works correctly end to end on top of INT1 weights: the projection weights (W_q, W_k, W_v, W_o) are all 1-bit, while the Q/K/V/O activations that flow between them stay INT8. RoPE's rotation still applies cleanly on top of that, with no special-casing needed for the reduced precision.

DDReGLU

DDReGLU (Decoupled Double Rectified Gated Linear Unit) is Impeto's activation function, defined as:

DDReGLU(x) = max(0, xW_gate - a) ⊙ max(0, xW_up - b)

It is a gated linear unit that attempts to mitigate the dying-neuron problem by introducing two independent, learnable thresholds, a and b, one per branch. Rather than a fixed cutoff, each neuron learns where its own activation threshold should sit, allowing it to stay at whatever level of activity is optimal for the task rather than collapsing into permanent inactivity.

This mechanism has a known limitation: the more inactive a neuron becomes, the smaller its gradient signal, and so the slower its thresholds adjust. Recovery isn't guaranteed for a sufficiently dead neuron. This is a soft corrective mechanism that helps in the common case, not a hard guarantee against the problem it targets.

DDReGLU's closest relative in the literature is dReLU, introduced in TurboSparse. Given that relationship, I suspect DDReGLU also promotes sparsity in the network, though I haven't measured this directly and treat it as an open hypothesis rather than a confirmed result.

In practice, Impeto uses DDReGLU6_int8: DDReGLU clamped to a maximum of 6, both to keep the output range predictable and to make INT8 quantization of the activation well-behaved. The name reflects the function's structure: two rectified (ReLU-style) branches, each with its own decoupled, learnable threshold, combined through a gating mechanism into a single linear unit.

FP4 Optimizer

The FP4 optimizer replaces the error-accumulation buffer, normally kept in full precision, with FP4 (E2M1), using stochastic rounding when writing values into it.

Stochastic rounding differs from deterministic round-to-nearest. Rather than always rounding to the closest representable level, it rounds up or down with a probability proportional to how close the true value sits to each neighboring level. Deterministic rounding at this precision would silently discard small, consistent gradient signals every step, since they would keep rounding to the same nearby level and never accumulate. Stochastic rounding preserves the true value in expectation over many steps instead, at the cost of some added per-step noise.

Why does an FP4 optimizer work at all? Since the weights are already fixed to +1 or -1 (INT1), the optimizer's only real decision is whether to flip a weight, not how to adjust a continuous value. This means it only needs a rough sense of direction, not the precise gradient magnitude a full-precision optimizer would normally track.

The optimizer also includes a threshold mechanism, currently a fixed value set in the configuration, that acts as the price of a flip: the accumulated error must cross this threshold before a weight actually flips. In addition, the design includes a lambda regularization term intended to increase over training as a flip-resistance schedule, stabilizing the model as training progresses, and mechanically separate from the threshold even though both influence flip rate. In practice, this scheduling was never implemented: lambda remains hardcoded at 0 in every run described here, meaning this regularization term has not yet been active in any experiment.

I also built an FP8 variant of this optimizer for comparison, identical in every respect except that the error-accumulation buffer is quantized to FP8 instead of FP4. Across matched-scale experiments (same model dimensions, byte-level tokenization), the FP4 optimizer consistently reached a lower loss than the FP8 version, with the largest gap appearing early in training. As training progressed, this gap narrowed considerably, though it did not fully close within the length of these experiments.

Tokenizer

Impeto uses YouTokenToMe (YTTM) for BPE tokenization rather than a custom implementation. Reimplementing tokenization would not have added anything novel to the project, and YTTM is a solid, well-built tool: fast, straightforward to integrate, and reliable across every run in this project.

Vocabulary size was chosen deliberately rather than defaulting to a large, generic vocabulary. A larger vocabulary compresses text into fewer tokens, which can lower loss simply by making each prediction cover more ground, independent of whether the model is actually better. It can also outpace what a small d_model can meaningfully represent at the output layer. For a small model, an oversized vocabulary risks spending a large share of the parameter budget on the embedding and output projection alone, leaving little capacity for the transformer body doing the actual reasoning. Vocabulary size in this project was chosen to stay proportionate to model size, for exactly this reason.

Training Behavior

Across every run in this project, loss showed the same pattern each time: a sharp drop at the start of training, followed by a long, slow flattening, where each additional step improved the model less and less. This shows up clearly in the flagship model's training curves below.

Validation loss kept inching down across the run, but validation accuracy stayed completely flat at 0.0397 from step 23,620 onward, all the way to the end. Between step 20,000 and the final recorded step (145,600), the model barely improved at all. This happened despite training only reaching about 30% of its full Chinchilla-based training budget before I stopped it. In other words, the model had flatlined well before it was anywhere close to done, at least by that budget's standard.

Results and Limitations

Impeto proves that direct INT1 training works end to end on ordinary CPU hardware, with no GPU, no PyTorch, and no external ML libraries: a full transformer, including backward passes for every custom piece (attention, DDReGLU, the FP4 optimizer), trained from scratch on binary weights and produced a steadily decreasing loss curve throughout.

It does not prove that this model reached anywhere near its full potential. The training run described in this article was stopped at roughly 30% of its Chinchilla-derived token budget, and the loss curve had already flattened out well before that point. This flattening pattern lines up with something already documented in published research on low-bit language models: models trained at very low precision (1-bit, 1.58-bit, and similar) tend to look strong early in training, then fall behind and plateau earlier than higher-precision models as training continues. Whether Impeto's flagship model is showing exactly this pattern, or something else specific to this codebase, is an open question I haven't fully answered. I see this as a limitation worth stating plainly rather than a failure to hide: it's a genuine, currently unresolved question about how far this specific approach can go, and closing it would need a longer run, a direct comparison against a higher-precision model at the same scale, or both.

Beyond the training curve itself, this project has real, known gaps. Batching (across independent training samples, not the sequence-length batching added late in development) was attempted once, made training both slower and less stable, and was reverted rather than debugged to completion. The optimizer's threshold is fixed rather than scheduled. The lambda regularization term exists in the code but was never actually activated.

What's Next

A number of things were deliberately left out of this version of Impeto, either because they were out of scope for proving the core idea, or because there wasn't time to do them properly:

• Proper batching across training samples, done carefully this time, with the gradient accumulation and optimizer-step timing worked out in advance rather than discovered through a failed attempt.
• A learning-rate and flip-threshold schedule, replacing the current fixed values.
• A CUDA implementation, to make training larger models actually practical, since CPU training time becomes the limiting factor well before parameter count does.
• A longer training run, ideally to the full Chinchilla-derived budget, to properly test whether the plateau seen here is temporary or a hard ceiling for this approach at this scale.

Acknowledgments and License

This project would not exist without the work of others. Thanks to the authors of the TinyStories paper for a dataset genuinely suited to training tiny models, to the BitNet and Direct Quantized Training papers for showing that 1-bit and low-bit training from scratch is viable at all, and to the Chinchilla paper for the scaling-law reasoning behind this project's training budgets.

Tokenization is handled by YouTokenToMe (YTTM), by Ivan Belonogov, licensed under the MIT License. YTTM's bundled flat_hash_map implementation is licensed separately under the Boost Software License. Both licenses are included in this repository.

Impeto is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0).

Code was written by hand. AI was used for debugging assistance, technical discussion, and help refining this writeup.

Github repo: https://github.com/M1-15/Impeto