Mixture of Experts LLM: How MoE-Lens Reveals Expert Specialization

What Is Mixture of Experts in LLM Architecture

The mixture of experts LLM architecture represents one of the most significant advances in scaling large language models efficiently. Rather than activating every parameter for each input token, MoE models route tokens through a learned gating mechanism to a small subset of specialized sub-networks—called experts—that process the input in parallel. This approach enables models to scale to trillions of parameters while keeping computational costs manageable during both training and inference.

At its core, the mixture of experts paradigm addresses a fundamental tension in deep learning: larger models generally perform better, but the computational cost of running every parameter on every input grows linearly with model size. MoE architectures break this relationship by introducing conditional computation, where different parts of the network activate depending on the input. A router network learns to assign each token to the most relevant experts, typically selecting the top-k from a pool of dozens or even hundreds of available specialists.

Modern implementations like DeepSeekMoE deploy 64 routed experts with top-k=6 activation per layer, meaning only about 9.4% of expert parameters process any given token. This sparse activation pattern has enabled models like Mixtral 8x7B, DeepSeek-V2, and Switch Transformer to achieve performance comparable to much denser models at a fraction of the inference cost. Understanding how these experts specialize—and whether all of them are truly necessary—is the central question that MoE-Lens seeks to answer.

For teams building AI-powered products, the mixture of experts architecture is becoming increasingly relevant. As organizations explore ways to transform complex research into accessible interactive experiences, understanding MoE efficiency has direct implications for deployment costs and latency requirements.

The MoE-Lens Framework for Analyzing Expert Behavior

MoE-Lens, presented at the ICLR 2025 Workshop on Sparsity in LLMs, introduces a systematic three-pronged framework for understanding how experts in mixture of experts LLM architectures actually behave. Developed by researchers from Penn State University, the University of Maryland, and Harvard University, the framework combines expert specialization analysis, extended LogitLens decoding, and quantitative validation through cosine similarity and perplexity measurements.

The first component measures expert specialization by computing the fraction of tokens from a given domain that are routed to each expert. Formally, the specialization score for expert E_i on domain D equals the number of tokens from D routed to E_i divided by the total tokens in D. An expert is considered specialized when its routing share significantly exceeds the uniform baseline—for DeepSeekMoE with 64 experts and top-k=6, this baseline sits at approximately 9.4%.

The second component extends the LogitLens interpretability technique. Standard LogitLens projects intermediate hidden states to vocabulary space to reveal what the model would predict at each layer. MoE-Lens adds a crucial innovation: it separately projects individual expert outputs, the weighted combination of all top-k expert outputs, and the top-weighted expert combined with the residual stream. This decomposition reveals exactly how each expert contributes to the final prediction.

The third component provides quantitative validation through two metrics: cosine similarity between single-expert and full-ensemble hidden states across all layers, and perplexity analysis with varying numbers of active experts. Together, these three methods paint a comprehensive picture of expert behavior that has profound implications for how we design, deploy, and optimize mixture of experts models.

The framework was tested across seven distinct domains—including English text, French text, code, mathematics (GSM8K), competitive math (AIME), academic papers (arXiv), and Chinese educational content—ensuring the findings generalize across languages and task types.

How Expert Specialization Emerges in Mixture of Experts Models

One of the most striking findings from MoE-Lens is the dramatic concentration of expert specialization. In DeepSeekMoE’s 64-expert architecture, the researchers found that for any given domain, only a handful of experts handle the majority of routing decisions—often with a single expert processing far more tokens than the uniform baseline would predict.

Consider the concrete data: at Layer 23 for arXiv academic text, one expert captures nearly 100% of routing decisions, while most other experts fall at or below the 9.4% uniform baseline. Similarly, for French-QA data at Layer 17, a single expert receives a dramatically disproportionate share of tokens. For GSM8K mathematical reasoning at Layer 22, one expert dominates with over 70% of routing weight. This pattern—extreme concentration in 1-3 experts per layer per domain—persists across all 27 MoE layers.

This emergent specialization mirrors patterns observed in vision model interpretability research, where different network branches develop specialized feature detectors. Just as early convolutional layers in AlexNet learned edge detectors while deeper layers recognized complex objects, MoE experts develop linguistic and computational specializations during training. The key difference is that in MoE models, this specialization happens at the granularity of entire sub-networks rather than individual neurons.

The identity of dominant experts varies across layers, suggesting a sophisticated division of labor. The expert handling French text at Layer 17 is typically not the same one handling it at Layer 5 or Layer 25. This layer-wise variation indicates that different aspects of language processing—from low-level token patterns to high-level semantic reasoning—are delegated to different specialists at different depths in the network.

Router Dynamics and Token Routing in Mixture of Experts LLM Systems

The router is the critical decision-making component in any mixture of experts LLM architecture. It determines which experts process which tokens, and its behavior fundamentally shapes model performance. MoE-Lens provides new insights into how routers actually operate in practice versus how they are designed to work in theory.

In DeepSeekMoE, the router is trained alongside the model using a combination of the standard cross-entropy language modeling loss and two auxiliary balance losses: an expert-level balance loss that discourages the router from sending all tokens to the same expert, and a device-level balance loss that ensures computational load is distributed across hardware. These balance losses are necessary because without them, routers tend to collapse—routing all tokens to a single expert while ignoring the rest.

Despite these balance mechanisms, MoE-Lens reveals that natural concentration still emerges. The router learns to preferentially select certain experts for certain domains even when penalties push toward uniform distribution. This tension between engineered balance and emergent specialization is one of the most important findings: the model’s optimization pressure toward accuracy consistently overcomes the regularization pressure toward uniformity.

The researchers also observed that the router tends to select experts with larger output norms, consistent with prior findings in MoE interpretability research. Experts that produce stronger modifications to the hidden state—those with larger norm outputs—are preferentially selected, suggesting that the router has learned to identify which experts will most significantly alter the prediction for a given input.

Additionally, DeepSeekMoE includes two shared experts that process every token regardless of routing decisions. These shared experts capture common knowledge that applies across all domains and contexts, functioning as a stable baseline upon which the routed experts add specialized modifications. The interplay between shared and routed experts creates a hierarchical specialization structure that MoE-Lens makes visible for the first time.

One Expert Is All You Need: The Core Mixture of Experts LLM Finding

The headline finding from MoE-Lens challenges a core assumption of mixture of experts LLM design: that activating multiple experts per token is essential for quality. The data tells a different story. When the researchers compared the hidden state produced by the single top-weighted expert plus the residual stream against the full ensemble of six experts plus the residual stream, they found cosine similarity as high as 0.95 across all 27 layers.

This means that in the high-dimensional space where these models operate, the direction of the hidden state vector produced by one expert is nearly identical to the direction produced by all six working together. The additional five experts contribute only marginal adjustments to what the top expert already provides. In mathematical terms, the additional experts are refining the prediction in a subspace that has diminishing returns on the final output.

The perplexity analysis reinforces this conclusion. Moving from top-k=6 (all six routed experts) to top-k=1 (only the highest-weighted expert) increases perplexity by approximately 5%. While this is a measurable degradation, it is remarkably small considering that 83% of the expert computation is being eliminated. The perplexity curve also reveals an important pattern: the steepest improvement comes from adding the second expert, with sharply diminishing returns from experts 3 through 6.

These results were consistent across all seven tested domains—English, French, code, mathematics, academic text, competitive math problems, and Chinese educational content. Whether the model is processing natural language, formal mathematical notation, or programming syntax, the single most important expert captures the essential prediction signal.

This finding has implications that reach beyond academic interest. For practitioners deploying mixture of experts models in production, it suggests that significant inference cost reductions may be achievable through intelligent expert selection without meaningful quality degradation.

LogitLens Analysis of Mixture of Experts Hidden States

The extended LogitLens analysis in MoE-Lens provides the most detailed view yet of how individual experts in a mixture of experts LLM contribute to token prediction across layers. By projecting hidden states from different points in the computation to the vocabulary space, the researchers could literally watch the model “think” and see where each expert’s contribution matters most.

In one illustrative example, the model processes the sentence: “When datasets are sufficiently large, increasing the capacity of neural networks can give much better prediction.” The task is predicting the next token after “these.” At early layers (1-7), predictions are scattered and uncertain. By Layer 20, both the full layer output and the single top expert converge on similar predictions. At Layer 27, both predict “performance” as the next token—and crucially, the top-1 expert plus residual stream produces the same prediction as the full six-expert ensemble.

The extended LogitLens technique adds the post-attention residual stream to the analysis, providing a more complete picture than standard LogitLens. The formulation projects LayerNorm of the hidden state combined with the attention residual through the unembedding matrix to produce token probability distributions. This extension is particularly important for MoE models because the residual stream carries forward information from all previous layers, and the expert’s contribution is a targeted modification to this accumulated representation.

Cross-lingual analysis revealed equally compelling results. For a French input passage from literary text, the model converges from diverse early predictions to the correct next token “temps” by the final layers, with the single top expert matching the full ensemble. For English scientific text, similar convergence patterns appeared. This cross-domain consistency demonstrates that concentrated expertise is not an artifact of any particular language or domain but a fundamental property of how MoE architectures organize knowledge.

The researchers also tracked how expert weights are distributed at each layer. Typical weight values range from 0.025 to 0.270, with the top expert consistently receiving the highest weight. The gap between the top expert’s weight and the second expert’s weight varies by layer and domain, but the prediction quality of the top expert alone remains consistently high regardless of the weight distribution.

Perplexity and Cosine Similarity Results Across Domains

The quantitative backbone of MoE-Lens rests on two complementary metrics that together provide compelling evidence for concentrated expertise in mixture of experts LLM architectures. The cosine similarity and perplexity analyses were conducted across all 27 layers of DeepSeekMoE and validated on two additional architectures.

Cosine Similarity Analysis

Cosine similarity between the single top-expert hidden state (H_t^{ℓ1}) and the full six-expert ensemble (H_t^{ℓ6}) was measured at every layer for seven domain datasets. Early layers (1-3) showed slightly lower similarity in the range of 0.85-0.90, which is expected since early representations are less refined. Middle and later layers consistently exceeded 0.90, with several layers reaching 0.95 or higher. All seven domains—AIME, Chinese Fineweb Edu, English Gutenberg, French-QA, FQuAD, GSM8K, and arXiv—tracked remarkably close together, with minimal inter-domain variation.

This cross-domain consistency is particularly noteworthy. One might hypothesize that code processing requires more diverse expert collaboration than natural language, or that mathematical reasoning demands more expert contributions than literary text. The data refutes these hypotheses: the single top expert approximates the ensemble equally well across all tested domains.

Perplexity Analysis

The normalized log perplexity curve as a function of active experts reveals a characteristic diminishing-returns pattern. Moving from top-k=1 to top-k=2 produces the steepest improvement—roughly half of the total gain from using all six experts comes from adding just the second expert. From top-k=2 to top-k=6, each additional expert contributes progressively less. The total perplexity penalty for using only one expert instead of six is approximately 5% across all domains.

This 5% figure demands careful interpretation. In absolute terms, a 5% perplexity increase is small—it would be imperceptible in most practical applications including text generation, summarization, and question answering. However, in autoregressive generation where each token’s prediction builds on previous ones, small per-token errors can compound. The practical impact depends heavily on the specific use case, sequence length, and quality threshold required.

Cross-Architecture Validation

The researchers extended their analysis to OLMoE from Allen Institute for AI (64 experts, top-k=8, uniform baseline ~12.5%) and Qwen 1.5 MoE (60 experts, top-k=4, uniform baseline ~6.67%). Both architectures showed similar concentration patterns, with French-QA exhibiting particularly extreme concentration in certain layers where a single expert received 80-90% of routing decisions. This cross-architecture consistency strongly suggests that concentrated expertise is an emergent property of the MoE training paradigm itself, not a quirk of any particular implementation.

Practical Implications for Mixture of Experts LLM Inference

The findings from MoE-Lens translate directly into actionable strategies for reducing the computational cost of deploying mixture of experts LLM systems in production. For organizations running MoE models at scale, these insights open several optimization pathways.

Selective Expert Pruning

The most immediate application is interpretable expert pruning. Since the top-weighted expert closely approximates the full ensemble, non-essential experts can be selectively removed from specific layers. Unlike black-box pruning methods that remove parameters without understanding their function, MoE-Lens enables targeted removal based on measured contribution. A pruned model that activates only 1-2 experts per layer instead of 6 could reduce MoE-layer computation by 67-83% while maintaining prediction quality within the 5% perplexity envelope.

Memory Optimization

GPU memory is often the binding constraint in LLM deployment. A DeepSeekMoE layer with 64 routed experts requires storing all expert weights in accessible memory, even though only 6 are used per token. With knowledge of which experts are essential for a given domain, operators could implement domain-aware expert offloading—keeping only the most important experts in GPU memory while storing others in CPU memory or on disk. For domain-specific deployments (e.g., a French-language service), the model could be drastically compressed by retaining only the relevant experts.

Dynamic Expert Budgeting

Not all inputs require the same computational investment. Simple, predictable tokens (“the”, “of”, “is”) may need only the top expert, while rare or ambiguous tokens might benefit from the full ensemble. A dynamic expert budget that adapts top-k based on router confidence or token entropy could reduce average computation without sacrificing quality on difficult tokens. The steep diminishing-returns curve observed in the perplexity analysis suggests that such an approach could achieve significant speedups.

These optimization strategies align with broader industry trends toward efficient AI deployment. Teams looking to make AI research accessible through interactive content platforms can leverage these insights to reduce the infrastructure costs of serving intelligent applications.

Implications for MoE Training

MoE-Lens findings also raise questions about how MoE models should be trained. If most experts contribute marginally, perhaps training with fewer experts but stronger specialization incentives would produce equally capable models at lower training cost. The tension between balance losses (which push toward uniform routing) and the model’s natural tendency toward concentration suggests that current training recipes may be suboptimal.

Future Directions for Mixture of Experts Research

MoE-Lens opens several compelling research directions that could reshape how the community approaches mixture of experts LLM architecture design, training, and deployment over the coming years.

Factual Knowledge Localization

One of the most promising extensions involves analyzing the internal representation sparsity of individual experts to localize where factual knowledge is stored. If specific experts in specific layers encode particular facts or reasoning capabilities, this would enable surgical model editing—correcting misinformation or updating knowledge without retraining the entire model. Early work on knowledge editing in dense models at MIT has shown promising results, and MoE architectures may offer a more natural framework for such interventions.

TunedLens and Learned Projections

The current MoE-Lens framework uses standard LogitLens projections, which assume the unembedding matrix accurately decodes intermediate representations. TunedLens, which learns affine transformations for each layer, could provide more robust decoding and reveal expert contributions that standard LogitLens misses. Incorporating learned projections into the MoE-Lens framework is a natural next step that could refine the precision of expert contribution measurements.

Scaling Analysis

The current study examines models with 60-64 experts. As architectures scale to hundreds or thousands of experts, does the concentration pattern persist? If a model with 256 experts still relies primarily on one or two per token, the implications for efficient scaling would be even more dramatic. Conversely, if larger expert pools lead to more distributed routing, this would suggest that current architectures are undersized relative to the diversity of knowledge they need to encode.

Multimodal MoE

Emerging architectures like DeepSeek-VL2 apply MoE to multimodal models that process both text and images. Understanding how experts specialize across modalities—whether visual and linguistic knowledge is cleanly separated or interleaved—could inform the design of more efficient multimodal systems and reveal fundamental insights about how neural networks organize cross-modal representations.

The MoE-Lens codebase is available on GitHub, enabling researchers worldwide to build on these findings and extend the framework to new models and domains.