DEV Community

Siddharth Singh Bhadauria
Siddharth Singh Bhadauria

Posted on

UMM-JEPA: Predicting What the Model Understands, Not What the Decoder Wants

Unified multimodal models stop fighting themselves when the generation objective targets the same continuous representation that understanding already uses and UMM-JEPA does this through masked, coarse-to-fine prediction of those features.

The Real Problem / Why Naive Approaches Fail

The pitch for a unified multimodal model is simple. One backbone reads images, writes images and answers questions about them. You expect the two skills to help each other: a model that can draw a dog should understand dogs better.

That isn't what usually happens. Generative training often adds little or nothing to understanding. Teams that bolt an image-generation objective onto a vision-language model frequently find their VQA and captioning numbers flat or slightly worse, even though the generation samples look good.

The UMM-JEPA abstract gives a plausible diagnosis. The two tasks optimize different representation spaces. Understanding runs on perceptual features, the semantic patch embeddings from encoders like SigLIP or DINOv2. Generation, in most unified designs, optimizes a synthesis-oriented space: discrete VQ codes (Chameleon-style token prediction) or the latents of a VAE that a diffusion head denoises (Transfusion-style). Those spaces exist to reconstruct pixels. They spend capacity on texture, high-frequency detail and exact color and the understanding pathway doesn't need most of it.

So the gradients from generation train the backbone to be good at a representation the understanding head never reads. The two objectives share weights but not targets. At best they coexist. At worst they compete for capacity.

UMM-JEPA's answer, per the abstract, is to reformulate generation as progressive prediction of continuous visual-understanding representations. The model doesn't predict VQ indices or VAE latents. It predicts the features an understanding encoder would produce for the target image, coarse scale first, then finer scales, with masking inside each scale. The name comes from that combination: JEPA-style prediction in representation space, plus masked next-scale structure.

A caveat about scope before going further. The abstract I'm working from is truncated. It gives the motivation and the core reformulation and the title supplies "masked next-scale representation prediction." The concrete mechanics below (pooling scheme, loss, decoder) are my reconstruction from that framing and from the prior work it clearly builds on. They aren't claims about the paper's exact implementation. Check the paper before you treat any of it as canonical.

Architecture & Inner Mechanics

Three lineages meet here and each one solves a different piece of the problem.

JEPA (Joint-Embedding Predictive Architecture). I-JEPA and V-JEPA showed that predicting the representations of masked regions, rather than their pixels, produces features that work well for downstream understanding. The loss lives in an abstract space, so the model isn't penalized for failing to reproduce unpredictable texture. UMM-JEPA keeps that idea and points it at generation.

Next-scale prediction (VAR). VAR replaced raster-order token prediction with coarse-to-fine prediction. It produces a 1×1 map, then 2×2, then 4×4 and so on. Each scale is conditioned on all coarser scales and all tokens within a scale are produced in parallel. Global structure comes first and detail last, which matches how semantic features are organized anyway.

Masked prediction (MaskGIT, MAR). Inside a scale, you don't need to emit every token at once. You can mask a subset, predict it from the visible tokens plus the coarser context and unmask iteratively at inference. MAR also showed that continuous tokens (no vector quantization) work for autoregressive generation if you handle the per-token distribution properly. That point comes back in the gotchas section.

Put together, the training target becomes a pyramid of understanding features. A frozen encoder (or an EMA copy, as in classic JEPA) embeds the image. Its patch grid gets pooled into several resolutions. The unified backbone, conditioned on text, learns to predict masked tokens at scale k given scales 1..k-1 and the visible tokens at scale k.

flowchart TD
    subgraph Targets["Target construction (no gradient)"]
        IMG[Training image] --> ENC[Frozen understanding encoder<br/>e.g. SigLIP / DINOv2 ViT]
        ENC --> PATCH[Patch features 16x16xD]
        PATCH --> P1[Scale 1: 1x1]
        PATCH --> P2[Scale 2: 2x2]
        PATCH --> P3[Scale 3: 4x4]
        PATCH --> P4[Scale 4: 8x8]
        PATCH --> P5[Scale 5: 16x16]
    end

    subgraph Backbone["Unified backbone (trainable)"]
        TXT[Text prompt tokens] --> SEQ
        CTX[Coarser scales 1..k-1<br/>teacher-forced from targets] --> SEQ
        UP[Upsampled scale k-1<br/>as coarse prior] --> CUR
        MASK[Mask tokens at random<br/>positions of scale k] --> CUR
        CUR[Scale k input tokens] --> SEQ[Token sequence]
        SEQ --> TF[Transformer blocks]
        TF --> HEAD[Prediction head]
    end

    P5 -.target at scale k.-> LOSS
    HEAD --> LOSS[Loss on masked positions only<br/>in representation space]

    subgraph Inference["Generation path"]
        G1[Text] --> G2[Predict scale 1..K<br/>iterative unmasking per scale]
        G2 --> G3[Predicted feature map]
        G3 --> G4[Feature-to-pixel decoder]
        G3 --> G5[Feed directly to understanding head]
    end

The bottom-right box makes the case for the whole design. The generated output is a feature map in the same space the understanding pathway consumes. Generation gradients shape the backbone's sense of what a semantic feature map of "a red bicycle against a brick wall" looks like and understanding reads from exactly that space. The two tasks share a target.

The cost is that you no longer have pixels at the end. Turning a semantic feature map into an image needs a separate decoder that inverts the encoder's features. It might be a diffusion model conditioned on the features or a learned upsampling decoder. The abstract doesn't say how UMM-JEPA renders pixels, so I'll treat that as an open design slot.

Production Implementation

Below is a self-contained PyTorch reference for the training objective. It's a sketch of the idea and I wrote it to be readable, not to match the paper's code. It runs on CPU with random tensors and on a GPU with real encoder features. To use real data, swap the fake features for the patch tokens of a frozen ViT (with timm or open_clip you can pull forward_features and drop the CLS or register tokens).

# umm_jepa_sketch.py
# Reference sketch: masked next-scale prediction of frozen understanding features.
# Requires: torch>=2.1
import math
import random
from typing import List, Sequence

import torch
import torch.nn as nn
import torch.nn.functional as F

SCALES: Sequence[int] = (1, 2, 4, 8, 16)


@torch.no_grad()
def build_target_pyramid(feats: torch.Tensor, grid: int,
                         scales: Sequence[int] = SCALES) -> List[torch.Tensor]:
    """feats: [B, N, D] patch tokens from a frozen encoder, N == grid * grid.
    Returns a list of [B, s*s, D] tensors, one per scale, layer-normalized."""
    if feats.dim() != 3:
        raise ValueError(f"expected [B, N, D], got {tuple(feats.shape)}")
    B, N, D = feats.shape
    if N != grid * grid:
        raise ValueError(f"N={N} is not grid^2 with grid={grid}; "
                         "did you forget to strip CLS/register tokens?")
    x = feats.float().transpose(1, 2).reshape(B, D, grid, grid)
    pyramid = []
    for s in scales:
        t = x if s == grid else F.adaptive_avg_pool2d(x, s)
        t = t.flatten(2).transpose(1, 2)                  # [B, s*s, D]
        # Per-token normalization keeps the regression target well scaled and
        # damps the high-norm outlier tokens many ViTs produce.
        t = F.layer_norm(t, (D,))
        pyramid.append(t)
    return pyramid


class NextScalePredictor(nn.Module):
    def __init__(self, feat_dim: int = 1024, model_dim: int = 768, depth: int = 12,
                 heads: int = 12, text_dim: int = 768,
                 scales: Sequence[int] = SCALES):
        super().__init__()
        self.scales = tuple(scales)
        self.in_proj = nn.Linear(feat_dim, model_dim)
        self.text_proj = nn.Linear(text_dim, model_dim)
        self.out_proj = nn.Linear(model_dim, feat_dim)
        self.mask_token = nn.Parameter(torch.zeros(1, 1, model_dim))
        self.pos = nn.ParameterList(
            [nn.Parameter(torch.randn(1, s * s, model_dim) * 0.02) for s in self.scales]
        )
        self.scale_emb = nn.Embedding(len(self.scales), model_dim)
        layer = nn.TransformerEncoderLayer(
            model_dim, heads, 4 * model_dim, dropout=0.0,
            activation="gelu", batch_first=True, norm_first=True,
        )
        self.blocks = nn.TransformerEncoder(layer, depth)
        self.norm = nn.LayerNorm(model_dim)
        nn.init.normal_(self.mask_token, std=0.02)

    def _embed_scale(self, tokens: torch.Tensor, k: int) -> torch.Tensor:
        return self.in_proj(tokens) + self.pos[k] + self.scale_emb.weight[k]

    def _coarse_prior(self, prev: torch.Tensor, k: int) -> torch.Tensor:
        """Upsample scale k-1 features onto the scale-k grid (VAR-style prior)."""
        B, n, D = prev.shape
        s_prev, s_cur = self.scales[k - 1], self.scales[k]
        g = prev.transpose(1, 2).reshape(B, D, s_prev, s_prev)
        g = F.interpolate(g, size=(s_cur, s_cur), mode="bilinear", align_corners=False)
        return g.flatten(2).transpose(1, 2)

    def forward(self, text: torch.Tensor, pyramid: List[torch.Tensor],
                k: int, mask_ratio: float):
        B = text.shape[0]
        target = pyramid[k]                               # [B, s*s, D]
        n_cur = target.shape[1]
        n_mask = max(1, int(round(mask_ratio * n_cur)))

        # Random mask per sample at the current scale.
        noise = torch.rand(B, n_cur, device=target.device)
        mask = torch.zeros(B, n_cur, dtype=torch.bool, device=target.device)
        mask.scatter_(1, noise.argsort(dim=1)[:, :n_mask], True)

        cur = self._embed_scale(target, k)
        cur = torch.where(mask.unsqueeze(-1), self.mask_token.expand_as(cur), cur)
        if k > 0:
            cur = cur + self.in_proj(self._coarse_prior(pyramid[k - 1], k))

        parts = [self.text_proj(text)]
        parts += [self._embed_scale(pyramid[j], j) for j in range(k)]
        parts.append(cur)
        seq = torch.cat(parts, dim=1)

        h = self.norm(self.blocks(seq))
        pred = self.out_proj(h[:, -n_cur:])               # current-scale slots only
        return pred, target, mask


def masked_rep_loss(pred, target, mask, cos_weight: float = 0.5):
    p, t = pred[mask], target[mask]
    l1 = F.smooth_l1_loss(p, t, beta=1.0)
    cos = 1.0 - F.cosine_similarity(p, t, dim=-1).mean()
    return l1 + cos_weight * cos


def sample_scale(scales: Sequence[int]) -> int:
    # Weight by token count so fine scales get gradient proportional to their size,
    # but keep a floor so the 1x1 and 2x2 scales are never starved.
    w = [max(s * s, 4) for s in scales]
    return random.choices(range(len(scales)), weights=w, k=1)[0]


def train_step(model, opt, text, enc_feats, grid=16, device="cpu"):
    model.train()
    pyramid = [t.to(device) for t in build_target_pyramid(enc_feats, grid)]
    k = sample_scale(model.scales)
    mask_ratio = random.uniform(0.5, 1.0)  # 1.0 == predict whole scale from coarser context
    use_amp = device.startswith("cuda")
    with torch.autocast(device_type="cuda", dtype=torch.bfloat16, enabled=use_amp):
        pred, target, mask = model(text.to(device), pyramid, k, mask_ratio)
        loss = masked_rep_loss(pred.float(), target, mask)
    if not torch.isfinite(loss):
        opt.zero_grad(set_to_none=True)
        raise FloatingPointError(f"non-finite loss at scale {k}: {loss.item()}")
    opt.zero_grad(set_to_none=True)
    loss.backward()
    gnorm = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    opt.step()
    return loss.item(), k, gnorm.item()


if __name__ == "__main__":
    torch.manual_seed(0)
    dev = "cuda" if torch.cuda.is_available() else "cpu"
    model = NextScalePredictor(feat_dim=1024, model_dim=256, depth=4, heads=4,
                               text_dim=768).to(dev)
    opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.05)
    for step in range(20):
        text = torch.randn(8, 32, 768)       # stand-in for text encoder states
        feats = torch.randn(8, 256, 1024)    # stand-in for frozen ViT patch tokens
        loss, k, g = train_step(model, opt, text, feats, device=dev)
        print(f"step={step:02d} scale={model.scales[k]:>2}x loss={loss:.4f} gnorm={g:.2f}")
Enter fullscreen mode Exit fullscreen mode

A few design choices in there deserve explanation.

mask_ratio ranges up to 1.0. At 1.0 the model predicts an entire scale from coarser context alone, which is exactly the situation at inference when a new scale begins. If training never shows fully masked scales, the first unmasking step at inference falls outside the training distribution.

The coarse prior is added to every slot at scale k, masked ones included. That way a masked token isn't a blank. It starts from a blurred estimate and only has to predict the residual detail, which is how VAR frames each new scale.

The target pyramid comes from average pooling the finest encoder grid. That's a shortcut. Average-pooled ViT features aren't what the encoder would emit for a downscaled image and they get washed out at 1×1. Running the encoder at several input resolutions gives "honest" multiscale targets at several times the extraction cost. I'd start with pooling and measure whether it matters.

The inference loop follows directly. For each scale, initialize all slots as masked plus the coarse prior, run T refinement passes that commit a growing fraction of tokens, then move up a scale. With deterministic regression you have no per-token confidence to rank by, so commit order is random unless you add a variance head. That limitation leads to the biggest gotcha.

Production Gotchas, Edge Cases & Failure Modes

Regression to the mean kills diversity. The L1/cosine loss above trains the model to predict the conditional expectation of each masked feature. For understanding-style pretraining that's fine and is what I-JEPA does. For generation it's a real problem. Given "a dog on a beach," the expected feature map averages over every plausible dog and a decoder fed that average tends to produce mushy, low-diversity images. MAR handles this for continuous tokens with a small per-token diffusion head that samples from the distribution instead of predicting its mean. Any representation-space generator needs some stochastic mechanism (diffusion head, flow-matching head or explicit noise injection). The abstract doesn't say which one UMM-JEPA uses and I'd look for that first in the full paper.

The frozen encoder sets your ceiling. If the target space is SigLIP features, the model can't generate distinctions SigLIP doesn't encode. Contrastive image-text encoders are known to be weak on exact counts, small text rendering and precise spatial layout. A generator trained in that space inherits those blind spots and so does the decoder.

High-norm outlier tokens. Many large ViTs produce a handful of artifact tokens with huge norms (the "Vision Transformers Need Registers" paper documents this). Without per-token normalization, those tokens dominate an L1 loss. The layer_norm in build_target_pyramid handles this, but it also throws away norm information the understanding head might use. Check that your understanding pathway consumes features under the same normalization you trained generation against. A mismatch there quietly brings back the representation gap the method exists to close.

You still need a pixel decoder. Feature maps don't show up in a demo. Training a decoder from semantic features back to pixels is its own project and semantic features are lossy by design, so reconstructions will drift on fine detail. If your product needs pixel-faithful editing (inpainting a logo, preserving a face's identity), a representation-space generator is the wrong primary tool.

Teacher forcing across scales. In training, coarser scales come from ground truth. At inference they come from the model's own predictions and errors at 2×2 compound into 16×16. The usual mitigations are scheduled sampling or noise injection on the context scales.

Evaluation is two-sided. The whole claim is that generation helps understanding, so a run is only interesting if you measure both: understanding benchmarks against an understanding-only baseline at matched compute, and generation quality through the decoder. Reporting only one side hides exactly the regression this method is supposed to fix.

When I would not use this. If your goal is pure text-to-image quality, a mature latent diffusion stack is the shorter path. If you don't have a strong pretrained understanding encoder for your domain (medical imaging, satellite multispectral), you have no good target space. If your compute budget can't cover both the unified backbone and a separate feature-to-pixel decoder, don't start. The method's value is the coupling between the two tasks. Without a need for both, it's overhead.

Benchmark Comparison & Cost Tradeoffs

I don't have the paper's reported numbers and I won't make some up. What I can give is structural math that applies to any implementation of this design, plus a qualitative comparison of objectives.

| Generation target | Space type | Shares space with understanding? | Needs pixel decoder | Diversity mechanism |
|, -|, -|, -|, -|, -|
| VQ code prediction (Chameleon-style) | Discrete, synthesis-oriented | No | VQ decoder (cheap) | Softmax sampling |
| VAE latent diffusion (Transfusion-style) | Continuous, synthesis-oriented | No | VAE decoder (cheap) | Diffusion |
| Representation alignment (REPA-style) | Diffusion latent plus auxiliary feature loss | Partially | VAE decoder | Diffusion |
| Masked next-scale representation prediction (UMM-JEPA) | Continuous, understanding-oriented | Yes | Feature-to-pixel decoder (expensive) | Must be added (see gotchas) |

Sequential steps at inference. With a 16×16 final grid, raster autoregression needs 256 sequential steps. A five-scale pyramid (1, 2, 4, 8, 16) needs 5 scale steps times T refinement passes per scale. With T = 4 that's 20 forward passes, each over a sequence of at most 32 text tokens + 85 coarser tokens + 256 current tokens = 373 tokens. Fewer, wider passes are GPU-friendly, though the pixel decoder adds its own latency on top.

Target pyramid size. Total tokens across scales: 1 + 4 + 16 + 64 + 256 = 341. At feature dimension 1024 in bf16 that's 341 × 1024 × 2 bytes ≈ 698 KB per image.

| Dataset size | Precomputed targets (bf16, D=1024) | Same in fp8/int8 |
|, -|, -|, -|
| 100K images | ~70 GB | ~35 GB |
| 1M images | ~698 GB | ~349 GB |
| 10M images | ~6.98 TB | ~3.49 TB |

That table drives the main FinOps decision: precompute targets or run the frozen encoder on the fly. Precomputing removes encoder FLOPs from every epoch but turns into a storage and I/O problem fast. Under a pooled pyramid you only need to store the finest grid (256 tokens, ~524 KB per image) and pool on load. That cuts storage by about 25% for a few cheap pooling ops per batch. On-the-fly extraction costs one frozen-encoder forward per image per epoch. For a ViT-L-class encoder that's often a meaningful fraction of the predictor's own forward cost, so for multi-epoch runs precomputing usually wins once storage is cheaper than GPU time.

A simple budgeting formula, with an assumed GPU price you should replace with your provider's current rate:

cost ≈ (images × epochs / throughput_img_per_sec) / 3600 × gpus × price_per_gpu_hour
Enter fullscreen mode Exit fullscreen mode

At an assumed $2 per GPU-hour, every 1,000 GPU-hours is $2,000. Whatever throughput you measure in a short profiling run, plug it in before committing to a full run. Remember to budget the pixel decoder's training separately.

Pragmatic Checklist for Monday Morning

  1. Pick the understanding encoder first, because it is your generation target space. Run your understanding eval suite on it alone to learn the ceiling.
  2. Strip CLS and register tokens, confirm the patch count is a perfect square and record the exact normalization you apply. Use the same normalization in the understanding pathway.
  3. Extract finest-grid features for a 10K-image subset and inspect per-token norms. Flag outliers before they hit the loss.
  4. Train the sketch above at small scale and log loss per scale separately. A flat 1×1 loss with a noisy 16×16 loss is normal. The reverse means a pooling or indexing bug.
  5. Add a stochastic head (a small per-token diffusion or flow head) before judging generation diversity. Don't evaluate samples from a mean-regression model and conclude the idea fails.
  6. Set up the matched-compute baseline: same backbone, understanding-only training. The method's claim lives or dies on that comparison.
  7. Train a feature-to-pixel decoder on the frozen encoder's ground-truth features first. If it can't render clean images from real features, it won't render them from predicted ones.
  8. Decide on precomputed vs on-the-fly targets with the storage table above and profile throughput for one hour before scheduling anything long.

References & Further Reading

  • Assran et al., Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture (I-JEPA), arXiv:2301.08243
  • Bardes et al., Revisiting Feature Prediction for Learning Visual Representations from Video (V-JEPA), arXiv:2404.08471
  • LeCun, A Path Towards Autonomous Machine Intelligence, OpenReview, 2022
  • Tian et al., Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (VAR), arXiv:2404.02905
  • Chang et al., MaskGIT: Masked Generative Image Transformer, arXiv:2202.04200
  • Li et al., Autoregressive Image Generation without Vector Quantization (MAR), arXiv:2406.11838
  • Chameleon Team, Chameleon: Mixed-Modal Early-Fusion Foundation Models, arXiv:2405.09818
  • Zhou et al., Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model, arXiv:2408.11039
  • Yu et al., Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (REPA), arXiv:2410.06940
  • Zhai et al., Sigmoid Loss for Language Image Pre-Training (SigLIP), arXiv:2303.15343
  • Oquab et al., DINOv2: Learning Robust Visual Features without Supervision, arXiv:2304.07193
  • Darcet et al., Vision Transformers Need Registers, arXiv:2309.16588

Top comments (0)