Two experiments on the same world model, both of which came out the opposite way to how I expected. One is now an IJCNN 2026 paper, the other an ICPR 2026 paper. Both are really about the same thing: the scaling intuitions most of us carry around are older than the hardware we run on.
What a world model has to do here
A dynamic scene reconstruction model takes sparse multi-view camera observations across time and predicts a full 3D scene, including how things are moving. Point a few cameras at a street for two seconds, get back a 3D representation of that street with the cars in it moving correctly.
This is the perception half of a world model for driving and robotics. If you want to simulate "what happens if I turn left here," you first need a model that can build the scene at all.
The architecture we worked from is STORM (Yang et al.), a transformer that predicts 3D Gaussian primitives and their motion in a single forward pass. Feed-forward, so no per-scene optimization: a big deal if you ever want this on a vehicle rather than in an offline pipeline. That's prior work, not ours. What follows is what we found building on top of it.
Setup throughout: a subset of the Waymo Open Dataset, 2-second clips at 10 fps, up to three synchronized camera views, images downscaled to 160x240, 150K-300K training iterations.
Experiment 1: we allocated the parameters differently and the smaller model won
The default move when a transformer underperforms is to make it deeper. More layers, more capacity, better results. That instinct comes from language modelling and image classification, and it is mostly right there.
Dense spatio-temporal prediction is not those tasks.
We built a width-dominant, shallow variant. Concretely: fewer transformer layers, larger embedding dimension, and fewer attention heads each with a bigger head dimension. Then we ran it in a smaller configuration against the standard baseline in a larger one, under an identical training and data pipeline.
| Model | PSNR | SSIM | Depth RMSE |
|---|---|---|---|
| STORM baseline (B8) | 26.82 | 0.770 | 12.59 |
| Ours, width-dominant (V6, S8 config) | 27.57 | 0.806 | 15.59 |
+0.75 PSNR and +0.036 SSIM, from the smaller model.
The training dynamics were the more interesting part. The width-dominant model converged faster, to a higher plateau, along a visibly smoother trajectory, and its gradient norms were substantially more stable. It was not squeezing out a marginal win at the end of a hard optimization. It was simply an easier model to optimize.
Why this is plausible: self-attention already provides global mixing within every single layer. Stacking more layers buys you more sequential refinement, but sequential depth is exactly what makes optimization harder. Widening instead gives each attention head a richer representation to work with while leaving global information exchange intact. For a task where every token needs to talk to every other token anyway, spending your parameter budget on width rather than depth is not obviously the wrong call, and here it was the right one.
Now the part I want to be honest about: look at the depth RMSE column. The baseline is clearly better on geometry, 12.59 against our 15.59. We got prettier images and worse depth.
That trade-off is real and we reported it as a finding rather than burying it. Photometric quality and geometric accuracy are not the same axis, and if you are building a world model whose output feeds a planner rather than a display, the column we lost on may be the column you actually care about. Anyone reading "+0.75 PSNR" as an unqualified win has read the paper badly.
The practical upshot is deployability. A smaller model that trains on consumer GPUs and beats a larger one on photometric quality changes who can work on this at all.
Experiment 2: attention reuse, and a result I did not want
Self-attention costs O(N² · d_h · H). For spatio-temporal input the token count N is brutal, because you are multiplying pixels by views by timesteps. So the obvious efficiency target is attention itself.
There is a neat line of work on attention reuse, built on the observation that attention maps stabilize as you go deeper into a transformer. If layer 9's attention pattern looks a lot like layer 8's, why recompute the query-key interaction at all? Reuse and transform the previous map instead. You keep the global token interaction and drop a chunk of the compute.
We integrated Less-Attention layers into the STORM-B/8 backbone and evaluated across a range of reuse ratios, measuring runtime, memory, parameter growth and reconstruction quality, under both naive and optimized implementations.
Attention reuse did not improve wall-clock time over optimized full attention. At any reuse ratio. Even with careful implementation.
The reason is not that the theory is wrong. The FLOPs really do go away. The reason is that fused attention kernels on modern GPUs have already moved the bottleneck somewhere else. When full attention runs as a single fused kernel that never materializes the attention matrix in HBM, the thing you were paying for was mostly memory traffic, not arithmetic — and "skip some of the arithmetic" optimizes the wrong resource. You also add parameters and complexity for the reuse machinery, and you now have two kernels where the fused path had one.
This is the whole lesson: an algorithmic optimization derived from a FLOP count is a hypothesis about the hardware, not a fact about it. The hardware moved. FlashAttention-style kernels changed which operations are expensive, and a good chunk of the "efficient attention" literature was implicitly costed against a machine that no longer exists.
I would rather this had worked. Publishing a null result is less fun than publishing a speedup. But a null result that saves other people from re-implementing the same thing is worth more than another 3% on a benchmark.
What I take from both
Capacity allocation matters more than total capacity. Where you put the parameters is a real design decision with real consequences, and "make it bigger" is the answer you give when you have not measured which axis is actually binding.
Measure wall-clock on your target hardware. Always. Not FLOPs, not parameter counts, not asymptotic complexity. Those are proxies, and proxies drift as hardware evolves. Every efficiency claim has a silent "on the machine I tested" attached to it.
Report the column you lost on. Our depth RMSE got worse. Someone building on this needs to know that far more than they need another decimal place of PSNR.
The intuitions are load-bearing and mostly untested. "Deeper is better" and "fewer FLOPs is faster" are both reasonable priors that happened to be wrong in our setting. I only found that out by running the ablation instead of assuming.
If you work on neural rendering, world models, or transformer efficiency, I would genuinely like to hear whether the width-over-depth result holds in your setting. My prior is that it generalizes to dense prediction tasks and not much further, but that is a prior, and this post is largely about how those go.
Papers: "Rethinking Transformer Design for Dynamic Scene Reconstruction: An Efficient, Width-Dominant Approach" (IJCNN 2026) and "Evaluating Attention Reuse in Dynamic 3D Gaussian Reconstruction" (ICPR 2026).
I write about world models, robotics data pipelines, and the ways measurement quietly fails. linkedin.com/in/rickeshnatarajan
Top comments (0)