Muse Glimmer 30B: An Architecture and Hands-On Look at Meta's New Mid-Range Model

In the world of generative AI, Meta's LLaMa is often considered the model that kicked off the open-weight era. Its successor, LLaMa-4, however, fell short of expectations and was widely regarded as a disappointment, leaving Meta in a difficult position over the past year.

Then, in early 2026, Meta released a new lineup called the "Muse" series. Muse Glimmer 30B, the mid-range model in that lineup covered here, appears to have drawn considerable attention in the local-LLM community.

This article walks through Muse Glimmer 30B's architecture and then verifies it hands-on on GPU hardware: standard text inference, its combination with the speculative-decoding technique DFlash, and its Vision capability. We look at what the model can actually do, based on the results below.

Meta's Muse Series

LLaMa, the Model That Started the Open-Weight Era

One of Meta AI's most significant contributions was releasing the LLaMa models as open weights.
At the time, high-performance chat models like GPT-3.5-Turbo existed, but they were closed models that couldn't be run on a typical local GPU.

To address that gap, Meta released LLaMa free of charge.
Version 1 came with restrictive licensing terms — even Alpaca, a derivative model, had commercial-use restrictions — but Version 2 relaxed those terms considerably.
That change made it much easier for a broad range of users to adopt LLMs locally, and the model drew significant attention as a result.

LLaMa 2 in particular was notable for being the first major model to adopt RoPE (Rotary Positional Embedding)¹ and Group Query Attention (GQA)², both novel techniques at the time.

Input tokens -> Embedding -+-> [ RMSNorm -> Self-Attention (GQA + RoPE) ] -+
                            |                                              |
                            +<-----------------------------(residual add)--+
                            |
                            +-> [ RMSNorm -> SwiGLU FFN ] -+
                            |                               |
                            +<------------(residual add)----+
                                   |
                            (x N decoder blocks)
                                   |
                            RMSNorm -> Linear -> Softmax -> Output tokens

Figure 1: LLaMa-2's architecture — a standard pre-norm decoder block, with GQA (fewer key/value heads shared across query heads) and RoPE-based position encoding applied at every attention layer.

These architectural choices remain standard across many open-weight models today, and models such as llm-jp-4-8b-thinking still carry that lineage forward.³

However, LLaMa-4 — the successor released after the success of LLaMa-3 — underperformed relative to expectations and was widely characterized as a disappointment. Meta itself acknowledged the setback, which was accompanied by organizational disruption within its research team and the departure of several researchers — an unusually visible decline for a lab of its standing.

From LLaMa to Muse

In April 2026, Meta announced a new model called Muse Spark.

Muse Spark launched as a closed model and initially drew little attention. Versions 1.1 and 1.2 followed in July and August respectively, and as each new version shipped, its performance gradually became better known.

Around the release of Muse Spark 1.2, Meta also introduced Muse Glimmer 30B, the mid-range model covered in this article. Its release appears to have generated notable interest within the local-LLM community.

Muse Glimmer's Architecture

Muse Glimmer uses a hybrid structure similar to several currently popular open-weight models. The architecture is shown below.

Input strings -+- Glimmer Vision (image input) -----------------+
               |                                                 |
               +- tok -> Emb -----------------------------------+-(+)-+
                                                                       |
                                RoPE (SA layers: theta=50,000)  <------+
                                RoPE (FA layers: theta=0 = NoPE) <-----+
                                                                       |
     [ SA -> SA -> SA -> FA ]  x13 groups  ------------------------->
            (52 blocks total = 4 layers x 13 groups)
                                                                       |
                                                          LNR -> SoftMax -> Output Probabilities

Figure 2: Muse Glimmer 30B's architecture. SA = Sliding Attention (local context, RoPE applied); FA = Full Attention (aggregates and consolidates, RoPE disabled / NoPE).

At a glance, the block layout resembles Qwen's more than it does Google DeepMind's Gemma-4 — it's a comparatively simple structure. It's a 29.6B-parameter, dense model that uses GeLU for its activation function.

Attention Structure

The attention structure is a hybrid built on top of Group Query Attention.

Layers labeled SA use Sliding Window Attention, which handles local context analysis. By explicitly bounding the context range each layer looks at, this reduces both memory usage and compute cost.

Layers labeled FA use Full Attention. These layers aggregate and consolidate what the preceding SA layers produced, running as standard GQA — which makes them comparatively more compute-heavy than the SA layers.

Blocks are arranged in groups of three SA layers followed by one FA layer, repeated 13 times for a total of 52 blocks.

The 3:1 SA-to-FA ratio resembles the structure used since Qwen3.5, but looking at what each layer type actually does, the closer functional match is Gemma-4's "local + global" design.⁴

A more distinctive detail is that the RoPE theta value can be set independently per layer: SA layers use 50,000, while FA layers use 0.

Looking at the config.json in the Hugging Face repository shows this setting directly.

{
  "architectures": [
    "MuseGlimmerForConditionalGeneration"
  ],
  "dtype": "bfloat16",
  "image_token_id": 200092,
  :
  :
  :
  "hidden_size": 6656,
  "initializer_range": 0.02,
  "intermediate_size": 19968,
  "layer_rope_theta": [
    500000.0,
    500000.0,
    500000.0,
    0,
    500000.0,
    500000.0,
    500000.0,
    0,
    500000.0,
    500000.0,
    500000.0,
    0,
    500000.0

In the modeling source code, a theta value of 0 causes the RoPE module to be treated as NoPE — meaning positional encoding is skipped entirely for that layer.

This behavior is confirmed in Hugging Face's transformers library:

(https://github.com/huggingface/transformers/blob/main/src/transformers/models/muse_glimmer/modeling_muse_glimmer.py)

In a separate file, modular_muse_glimmer.py, the code sets position_embeddings to None whenever the RoPE theta parameter is 0. The class that defines the model's actual forward pass then branches on that: if position_embeddings is None (i.e., NoPE), it skips the positional-encoding step entirely.

class MuseGlimmerTextAttention(nn.Module):
    :
    :
    def forward(
        self,
        hidden_states: torch.Tensor,
        position_embeddings: tuple[torch.Tensor, torch.Tensor] | None = None,
        attention_mask: torch.Tensor | None = None,
        past_key_values: Cache | None = None,
        **kwargs: Unpack[TransformersKwargs],
    ) -> tuple[torch.Tensor, torch.Tensor | None]:
        :
        :
        # NoPE layers receive `position_embeddings=None` from the model.
        if position_embeddings is not None:
            cos, sin = position_embeddings
            query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
        :
        :

This suggests a design choice to keep FA layers focused purely on the aggregation role, stripping out anything unrelated to that job and drawing a clear division of labor between the two layer types.

Vision Encoder

The vision model, called Glimmer Vision, is a custom encoder rather than the widely used SigLIP vision transformer that most models rely on.

It has 50 blocks — 12 groups of (3 SA + 1 FA), plus one more (1 SA + 1 FA) — and its structure is nearly identical to the text decoder's, differing only at the very end. The resulting vector is treated as 1,024 tokens' worth of information and fed into the text decoder.

This design is described in Meta's own paper, "Perception Encoder: The best visual embeddings are not at the output of the network" (arXiv:2504.13181v2, published April 28, 2025), and Glimmer Vision appears to be built on the Perception Encoder architecture proposed there.

Vision encoder design has diversified across model vendors this year — some follow the approach above, while others, like Gemma-4-12B-it, fold nearly all of the weight data into the text encoder itself (aside from CLIP). It's possible this custom approach delivers stronger analytical capability than a straightforward SigLIP-based encoder, though that would need to be verified directly — which is one of the things we test below.

Running It Ourselves

For this verification, we used a GPUSOROBAN High-Speed Computing instance equipped with an NVIDIA RTX A4000.

Overall Setup

The setup is as follows: we open an SSH tunnel to an access server, then reach the target instance through that tunnel.⁵ llama.cpp's service port is relayed to localhost via port 8001 on both ends.

(The network diagram accompanying this section is omitted here — it uses several Japanese-only labels for "home," "client," "access server," and "target instance," and the setup is already fully described in the text above.)

Instance Used

The GPU used for this test is an NVIDIA RTX A4000. The instance specifications are as follows.

Item	Spec
Instance type	s16-1-a-standard-ubs24-v
GPU	NVIDIA RTX A4000
GPU memory	GDDR6 16GiB
GPU memory bandwidth	448.0 GB/s
FP32 compute	19.17 TFLOPS
BF16 compute	38.34 TFLOPS
INT8 compute	153.4 TOPS
INT4 compute	306.7 TOPS
vCPU	11 cores
System memory	50 GiB
Storage	Persistent 100 GiB
CUDA version	13.2
NVIDIA driver version	580
OS	Ubuntu 24.04 Server

Table 1: Instance specifications

Model Files Used

Even a 4-bit quantized version of Muse Glimmer exceeds 16GiB. Factoring in KV cache usage as well, we used a 2-bit quantized model optimized with Unsloth Dynamic 2.0.

We downloaded the following files from Unsloth's Muse Glimmer repository:

File	Size	Description
Muse-Glimmer-30B-UD-IQ2_M.gguf	12.3 GB	UD2.0, 2-bit quantized model
mmproj-kquant.gguf	1.40 GB	Quantized vision encoder
dflash-kquant.gguf	1.63 GB	Quantized DFlash draft model

Table 2: Files used with llama.cpp

We used llama.cpp version 10380 (0b1bad14f).

Support for Muse Glimmer had already landed via model: Muse Glimmer Support – #26841, and this build includes that change.

Verification: Standard Mode

First, we ran text inference using the main model on its own.

Launch Command

We used the following command to launch the web frontend for testing.

For this run, the KV cache was kept in FP16 rather than quantized to 8-bit. We also raised the log verbosity from the default of 3 to 4 to check memory usage.

build/bin/llama-server --model ./models/Muse-Glimmer-30B-UD-IQ2_M.gguf -t 12 -np 1 --prio 2 --temp 1.0 --top-p 0.95 --top-k 64 --port 8001 --host 0.0.0.0 --fit off --no-warmup --no-cache-prompt -fa on --cache-ram 0 -c 131072 --reasoning on --log-verbosity 4

Memory Usage

Memory usage came out as follows. Thanks to the sliding-window design, the footprint looks smaller than what you'd typically see from a model relying solely on standard Grouped Query Attention.

Component	CUDA0	CPU
Weight data	10,623.10	1,052.08
KV cache data	1,664.00	—
Sliding-window KV cache	97.50	—
Gated DeltaNet compute buffer	273.52	156.52
Total	12,658.12	1,208.60

Table 3: Memory usage in standard mode, main model only (units: MiB)

Output

We asked the following question and examined the response.

Given the recent Kumamoto earthquake, this is a question we've been asking various models out of curiosity.

Describe the characteristics of the ground/soil conditions in Kyushu.

The response came out as follows. Our first impression was that it looked surprisingly well-informed.

Kyushu's ground conditions are among the most affected by volcanic activity
and crustal movement anywhere in the Japanese archipelago, and the defining
characteristic is a foundation dominated by volcanic sediment and soft
alluvial deposits.

1. Geological and Topographic Features
- An island arc ringed by active volcanoes: Located at the southwestern end
  of the Japanese archipelago, on the boundary between the Philippine Sea
  Plate and the Eurasian Plate. Some of Japan's most active volcanoes are
  concentrated here — Mt. Aso, Sakurajima, the Kirishima range, and Mt.
  Unzen — and volcanically derived ground extends across a wide area.
- The Median Tectonic Line runs through the region: A roughly east-west
  fault line crosses nearly all of Kyushu, and active fault zones are
  extremely common. A chain of active faults runs through Fukuoka,
  Kumamoto, and Oita, including the Futagawa and Hinagu fault zones, which
  were the epicenters of the 2016 Kumamoto earthquakes.
- Contrast between mountainous terrain and narrow plains: Inland areas are
  ringed by the Kyushu mountains, interspersed with alluvial lowlands such
  as the Ariake Sea coast, the Chikugo Plain, Kitakyushu, and the
  Miyazaki Plain. The plains have thick sedimentary layers, while the
  mountainous areas are dominated by hard igneous rock.

2. Ground Material Characteristics
- Extensive volcanic ash and ejecta layers: Ash from the Aso caldera's
  eruptions covers central to northern Kyushu in deposits reaching tens of
  meters, and in some places over 100 meters. Pumice, scoria, and
  tuffaceous ash are known for high water content, high compressibility and
  permeability, and a tendency toward consolidation settlement.
- Thick, soft alluvial layers: Lowlands such as the Ariake Sea coast, the
  Chikushi Plain, the Yatsushiro Plain, and the Miyakonojo Basin consist of
  thick alluvial clay, sand, and silt layers. Low N-values and poor bearing
  capacity are common across broad areas of soft ground.
- A mix of volcanic and sedimentary rock: Northern Kyushu's bedrock
  consists of Paleozoic sedimentary rock, Tertiary igneous rock, and hard
  rock formations such as the Mifune Group. Southern Kyushu is dominated by
  more recent volcanic rock, which is hard but weathers easily and is prone
  to collapse.

3. Geotechnical Characteristics
- High liquefaction risk: Reclaimed land and coastal alluvial sand layers —
  particularly around Hakata Bay in Fukuoka, Nagasaki Bay, Yatsushiro Bay
  in Kumamoto, and Kagoshima Bay, as well as sandy ground near river
  mouths — are prone to liquefaction. Widespread liquefaction damage was
  reported during the 2016 Kumamoto earthquakes.
- Ground compressibility and differential settlement: Volcanic ash layers
  combined with organic alluvial clay contribute to differential
  settlement of buildings, road subsidence, and long-term settlement
  following liquefaction. Groundwater tables are shallow and sensitive to
  fluctuation.
- Ground amplification characteristics: Thick, soft volcanic ash and
  alluvial layers amplify seismic motion in many areas. Long-period ground
  motion amplification has been specifically noted around the Aso caldera
  and the Chikugo Plain.
- Cities with extensive reclaimed and improved land: Port cities such as
  Kitakyushu, Nagasaki, Kagoshima, and Sasebo are historically built on
  reclaimed land, and ground improvement is essential across much of these
  areas.

Regional Trends
- Northern Kyushu (Fukuoka, Saga, northern Oita): Dominated by alluvial
  clay in the Chikushi Plain and reclaimed land along the Genkai Sea coast.
  Ground is soft with high liquefaction risk.
- Central Kyushu (Kumamoto, southern Oita): Thickly covered by Aso volcanic
  ash. This ash compacts poorly and is prone to settlement.
- Southern Kyushu (Miyazaki, Kagoshima): A mix of volcanic ejecta directly
  beneath active volcanoes and marine-terrace deposits. Ground consists of
  volcanic soil over shallow bedrock, giving locally high bearing capacity
  in places, but with significant heterogeneity from collapse and ejecta
  deposits.

Overall, Kyushu's ground can be characterized as "heterogeneous, multi-
layered deposits from volcanic activity plus thick, soft alluvial layers,"
making earthquake resistance, ground subsidence, and liquefaction
countermeasures major challenges for urban development in the region.

We had a DeepSearch agent built on GPT-5.6-Luna (powered by Dify) evaluate this response. The results were as follows.

The evaluation flagged a recurring issue: the response tends to generalize localized Kyushu characteristics as if they applied uniformly across the whole region. Only one point — the claim about the Median Tectonic Line — was flagged as factually incorrect outright; the rest of the issues are mostly about over-generalization.

Smaller models tend to produce noticeably off-base answers to this kind of question, so getting a response of this quality without RAG suggests fairly strong reasoning ability. That's particularly notable given that we normally test with 4-bit quantized models, and this run used a 2-bit quantized model instead.

The overall thrust of the response is reasonable, but it should be treated
with caution given that it describes all of Kyushu with a single, uniform
set of ground characteristics.

In particular, the following statements need correction:
- "The Median Tectonic Line crosses all of Kyushu": there are multiple
  views on whether and where the Median Tectonic Line runs through Kyushu,
  so this cannot be stated as fact.
- "Located on the boundary between the Philippine Sea Plate and the
  Eurasian Plate": Kyushu is affected by plate motion, but describing it as
  simply sitting on a plate boundary is inaccurate.
- "Aso volcanic ash deposits reach over 100m in central-northern Kyushu":
  this does not distinguish between airfall ash, pyroclastic-flow deposits,
  and redeposited material, and reads as if a uniform 100m+ layer covers
  the entire area.
- "Volcanic ash has high compressibility and permeability": ash, pumice,
  and pyroclastic-flow deposits each have different properties and
  shouldn't be grouped together.
- "Southern Kyushu is dominated by volcanic rock": Shirasu, sedimentary
  rock, alluvial deposits, and marine terraces are also widely distributed
  there.
- "Port cities are built on reclaimed land, making ground improvement
  essential": reclaimed land is only part of the picture, and the need for
  ground improvement varies by ground conditions and structure.

On the other hand, the following points are largely correct:
- Kyushu has many active volcanoes and active faults.
- Soft alluvial layers are distributed along the Ariake Sea coast, the
  Chikushi Plain, and the Kumamoto Plain, among other areas.
- Liquefaction should be considered for reclaimed land, former riverbeds,
  and low-lying areas near river mouths.
- During the Kumamoto earthquakes, the Futagawa and Hinagu fault zones
  were active, and liquefaction damage did occur.
- Consolidation settlement due to soft ground is a recognized issue along
  the Ariake Sea coast.

A more accurate summary, then, would be: Kyushu's ground varies regionally
across volcanic sediment, bedrock, alluvial deposits, and reclaimed land.
Rather than treating volcanic ash and alluvial layers as characteristic of
all of Kyushu, subsidence, liquefaction, slope failure, and ground-motion
amplification should be assessed separately for each region based on its
specific terrain and geology.

Performance

Speed performance came out as follows, with an output rate of about 22.52 tokens/sec. Given the model's size, that's on the lower end compared to 9B–12B models, though it's not slow enough to be a practical problem. That said, raising the reasoning effort further might noticeably increase response time.

prompt eval time =     262.18 ms /    68 tokens (    3.86 ms per token,   259.37 tokens per second)
       eval time =   82874.28 ms /  1866 tokens (   44.41 ms per token,    22.52 tokens per second)
      total time =   83136.46 ms /  1934 tokens

Verification: Using DFlash

Muse Glimmer supported DFlash from release, and Unsloth's repository includes a dedicated draft model (dflash-kquant.gguf). We loaded it via --model-draft and re-ran the same question as in standard mode.

The result was underwhelming: output speed dropped from 22.52 tps to 17.80 tps, and the token acceptance rate came in at only 15.7% (mean length 1.94).

We plan to dig further into the conditions under which this speculative-decoding technique does or doesn't pay off in a follow-up article.

Verification: Using Vision

Muse Glimmer also ships with a vision encoder, with a dedicated mmproj file available in Unsloth's repository. Loading this vision encoder together with the DFlash draft model resulted in an out-of-memory error, so we tested each capability separately.

Command Line

build/bin/llama-server --model ./models/Muse-Glimmer-30B-UD-IQ2_M.gguf -t 12 -np 1 --prio 2 --temp 1.0 --top-p 0.95 --top-k 64 --port 8000 -c 131072 --reasoning on --log-verbosity 4 --mmproj models/mmproj-kquant.gguf

Memory Usage

Memory usage came out as follows, showing an increase of roughly 1,336 MiB.

Component	CUDA0	CPU
Weight data	10,623.10	1,052.08
KV cache data	1,664.00	—
Sliding-window KV cache	97.50	—
Gated DeltaNet compute buffer	273.52	156.52
Vision weight data	1,335.41	—
Vision metadata	0.28	—
Total	13,993.81	1,208.60

Table 4: Memory usage with the vision encoder enabled (units: MiB)

Prompt and Output

Rather than text-only inference, we tested the model's analytical ability by attaching an image and asking it questions about that image.

For this, we had the model analyze the same Muse Glimmer architecture diagram shown earlier in this article⁶ and asked it to "analyze this in detail and describe everything written in it."

Note: since this article's author writes in Japanese, this test was conducted using a Japanese-language diagram as the input image. The model's response below — translated into English — conveys the diagram's content, so the image itself is kept as-is here rather than redrawn in English.

Figure 3: The image fed into the Vision Encoder

The result was as follows.

This slide, titled "Reference: MUSE GLIMMER 30B Structure," is a diagram of
the MUSE GLIMMER 30B model architecture's specifications and overall data
flow.

[Spec list, left side]
- Dense-type 29.6B model
- Hidden size: 6,656 (activation function: SwiGLU, dimension 19,968)
- LNR: Linear Layer
- RoPE: theta = 50,000. Notably not applied to Full Attention layers — for
  FA layers, theta is set to 0, which signals NoPE (no positional encoding)
- Max context: 128k tokens (131,072 tokens)
- Includes a DFlash-based drafter; supports MTP

[Terminology, right side]
- SA: Sliding Attention
  - GQA-based, with a sigmoid output gate
  - Handles local context analysis via a sliding window
  - Low compute cost, fast
  - Low memory usage
- FA: Full Attention
  - GQA-based, with a sigmoid output gate
  - Aggregates and consolidates information from Sliding Attention layers
  - Maintains global structure and handles complex reasoning
  - Uses no positional encoding — positional information is left entirely
    to Sliding Attention

[Central block diagram flow]
Input strings are tokenized (tok) and embedded (Emb). RoPE is added to the
embeddings. Separately, Glimmer Vision receives input from an image icon,
and its output is added immediately after the embedding + RoPE step.

The language model body that follows is described as having 52 blocks
total: 52 blocks = 4 layers x 13 groups.

Each group runs SA → SA → SA → FA in sequence, with SA shown in blue and FA
in red. Thirteen of these groups are chained together.

At the end of each group's output, LNR normalizes the result, which then
passes through Softmax to produce the output probabilities.

In short: taking embeddings + vision features + RoPE as input, the model
runs SA three times for local analysis, then FA once to aggregate and
consolidate — a 4-layer unit repeated across 13 groups for 52 blocks total,
forming a dense causal model. The vision encoder follows a similarly dense
50-block structure, with a 128k context window, SwiGLU activation, and the
same ability to disable RoPE for Full Attention layers.

The content is largely accurate, with only minor issues, and we judged the analytical quality to be high. The response correctly captures what each element means, how the pieces connect, and even the detail that Full Attention layers use NoPE.

External Benchmark: How the Model Is Rated Elsewhere

We checked Muse Glimmer's standing against other models using Artificial Analysis, a site that publishes third-party benchmark data. The chart below is organized around their Intelligence score.

Figure 4: Artificial Analysis Intelligence scores (models shown are a custom selection for this comparison)

Muse Glimmer 30B scores 35, which is high relative to other open-weight models in the roughly-30B parameter range. Since this chart excludes coding- and agent-specialized models (a category we don't test as heavily), the comparison is limited to that scope, but within it, Muse Glimmer scores above both Qwen3.5 and Gemma-4.

What stood out more was the score for Muse Spark, the closed model this architecture is presumably derived from — its score is close enough to Claude 5 Fable's that the gap is fairly small. Reaching this level of performance within a short update cycle, combined with reports that Meta may release this model line as open weights in the future, makes it a lineup worth continuing to watch. The Muse series took a long and difficult path to reach this point, and it will be worth seeing how the rest of the lineup performs going forward.

Conclusion

This article analyzed the architecture of Meta's Muse Glimmer 30B and evaluated its behavior through hands-on testing, including basic chat inference. Based on that evaluation, Muse Glimmer 30B looks like a strong option in the mid-range model category.

Its architecture builds solidly on the "local + global" pattern that has proven effective across recent hybrid-attention models, while going a step further by omitting RoPE from the aggregation (FA) layers — a refinement that lets those layers focus purely on their consolidation role.

The vision encoder is also a custom design based on Meta's own Perception Encoder architecture, and its analytical performance held up well in testing.

The model is released under the Apache 2.0 license, which makes it a viable option for developers considering a Tokens-as-a-Service offering — a space where Gemma-4 has had little competition until now.

Muse Spark 1.2, the closed model this architecture appears to derive from, also scores well on third-party benchmarks, suggesting Meta's more recent models have made real progress since the LLaMa-4 generation. Whether Meta continues to gain ground in the frontier-model space is something worth watching going forward.

On a practical note, this test again highlighted the value of having a GPU with more VRAM on hand. GPU prices keep climbing, but without the ability to freely run mid-range models like this one locally, there's a lot that's hard to verify firsthand — a recurring frustration in this kind of investigation.

References

Muse Glimmer 30B repository (Hugging Face)
https://huggingface.co/meta-models/Muse-Glimmer-30B

Perception Encoder: The best visual embeddings are not at the output of the network
Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Choi, et al.
https://huggingface.co/papers/2504.13181
https://github.com/facebookresearch/perception_models

Muse Glimmer model code source (GitHub)
https://github.com/huggingface/transformers/tree/main/src/transformers/models/muse_glimmer

llama.cpp PR26841
https://github.com/ggml-org/llama.cpp/pull/26841

Artificial Analysis
https://artificialanalysis.ai/

---

This article is an English adaptation of the original Japanese post published on Zenn: "Metaの意地を見せたか　Muse Glimmer 30Bの実力を見てみる", by Yuichi Tominaga.

---

1. RoPE was first introduced in RoFormer, a Transformer-based language model similar to BERT. (https://arxiv.org/pdf/2104.09864) ↩

2. GQA was first introduced by Google Research's T5-XXL, in the paper that proposed the GQA approach. (https://arxiv.org/abs/2305.13245) ↩

3. llm-jp-4-8b-thinking's config.json lists its architecture as "LlamaForCausalLM," indicating it descends from the LLaMa-2 line, though it uses its own custom tokenizer. Its vocabulary is substantially larger, and its performance far exceeds the original LLaMa-2. ↩

4. In Qwen's case, the Gated Delta Network captures global information all at once, while Gated Attention acts as a global noise filter that smooths the information afterward — a structurally different approach from Muse Glimmer's. Looking across recent architectures, despite some structural variation (including how NVIDIA's Mamba series and LFM acquire local information), the "local + global" pattern used by Muse Glimmer and Gemma-4 appears to be the more dominant trend. ↩

5. For more detail on this setup, see our earlier article on using GPUSOROBAN. (https://www.bluecore.net/archives/242) ↩

6. This image uses an earlier version of the model diagram shown previously in this article. ↩