Chimera: Crossbreeding Transformer and Mamba with a Five-Phase Attention Line
Heterogeneous crossbreeding
Chimera is an architecture research program, not a shipped product. Where the Darwin family merges models that share a Transformer backbone, Chimera goes further and crosses heterogeneous architectures, for example Transformer with Mamba-style state-space layers. The motivation is that different families fail differently: attention captures long-range structure but is quadratic and can hallucinate confidently; state-space models are linear-time and stable over long sequences but weaker at some retrieval patterns. Combining them is an attempt to keep the complementary strengths in one network.
The Oheng (five-phases) attention line
Alongside the cross-architecture work there is an in-house attention line called Oheng, from the Korean term for the five phases. It is a 5-way attention design intended to add robustness and suppress hallucination by routing or combining multiple attention behaviors rather than relying on a single mechanism. Framed honestly, it is a structured research bet on attention diversity, not a proven module with published leaderboard wins.
Why heterogeneous mixing is hard
Crossbreeding architectures is not free. Transformer and state-space blocks have different inductive biases, different memory dynamics, and different training-stability profiles. Naive stacking often underperforms either parent. The research questions are concrete: where to place state-space layers relative to attention, how to share or separate the residual stream, and how to initialize so that neither branch dominates early training. These are open problems, and Chimera is positioned as a place to study them rather than a finished recipe.
The hallucination-suppression angle
The stated aim of both the hybrid and the Oheng line is robustness, including reduced hallucination. That is a reasonable hypothesis: ensembling distinct attention or sequence-mixing behaviors can lower the chance that a single failure mode propagates unchecked. But robustness claims only mean something with measurement, and at this stage the work is architectural exploration. No single-benchmark number should be read as a settled result.
Where it fits in the stack
Chimera is best understood as the upstream research feeding future model families. Darwin proves that merging homogeneous models works; AETHER proves that a fully open from-scratch model is buildable at this footprint. Chimera asks the harder, longer-horizon question of whether mixing fundamentally different sequence-modeling primitives yields models that are both efficient and harder to fool. It is early, and it is meant to be.
Honest scope
Chimera is research-stage. There is no claim here of a released, benchmarked product or of state-of-the-art results. Transformer x Mamba hybrids and the 5-way Oheng attention line are architecture experiments aimed at robustness and hallucination suppression; whether they beat strong Transformer baselines at scale is exactly the open question the program exists to answer. Read this as a description of a research direction and its rationale, not as validated performance. Any future numbers should be treated as preliminary until they appear with reproducible setups and public baselines.
More: https://vidraft.net
Top comments (0)