**Hongye Jin, Linwei Li, Xiaotian Han, Xin Liu, Haoyang Wen, Sha Li, Chia-Yuan Chang, Tuo Zhao, Qingyu Yin, Binxuan Huang**
Update Date: August 22, 2026 · 40 min read
<aside>
An ultra-sparse MoE can look perfectly healthy while quietly wasting entire banks of experts. In our training of 60B & 180B models with 768 experts and top-8 routing, lower-layer experts norms progressively collapse, yet training loss, validation loss, and load-balance metrics look healthy; And masking all routed experts in these layers barely changes model quality. In this blog, we describe our attempts to rescue these dead experts, show how we successfully prevent their norm collapse
As the optimizer’s direct connection to expert death, we first examine the AdamW optimizer used in our training, particularly its epsilon. At the tiny gradient scales of lower-layer experts, AdamW’s epsilon materially suppresses adaptive updates while decoupled weight decay keeps shrinking the parameters. Muon largely avoids the visible collapse because its epsilon acts at a very different, matrix-level scale. In this blog, we show that lowering AdamW’s epsilon or switching to Muon can rescue dead experts—but these numerical fixes reveal a deeper issue: healthy weight norms do not necessarily imply useful experts, as downstream task performance does not improve accordingly.
After extensive experimentation, our results point to a critical window early in MoE training, during which lower-layer pathways establish their functional roles. We therefore introduce LM Loss as Auxiliary Loss (LLAL), a simple, temporary LM objective attached to an early MoE layer. It creates a prediction demand that upper layers cannot satisfy on the lower layer’s behalf, allowing the pathway to become useful before the auxiliary loss is removed. At 180B scale, using LLAL for only 4,000 of 135,000 steps prevents collapse, improves validation loss and load balance, and raises MMLU and MMLU-Pro by 2.8 and 4.8 points. LLAL prevents norm collapse, establishes a lasting functional role for lower-layer experts, and improves validation loss, downstream performance, and load balance— with applied only briefly at the beginning of training.
Beyond LLAL, we show that router learning rate, load balancing, Megatron’s gradient-buffer semantics, and residual design can all reshape expert learning, while our audit of open-source MoEs uncovers similarly hidden structures.
Findings:
1. Silent expert death can hide behind healthy training dashboards. [Section 2.1]
Training and validation losses can decrease normally, and load-balance metrics can eventually look healthy, even when entire lower-layer expert banks contribute almost nothing. Masking experiments are therefore essential: parameter presence, balanced assignment, and causal utilization are not the same thing.
2. Regarding this issue, Muon and small-epsilon AdamW are more closely related than they appear. [Section 3.2, Section 3.3]
For lower-layer routed experts, the second-moment scale can fall far below the conventional $10^{-8}$ epsilon. In this regime, epsilon no longer merely prevents division by zero: it materially caps the adaptive update, while decoupled weight decay continues shrinking the parameters. Epsilon can therefore determine whether weak experts visibly survive. Muon applies epsilon to the Frobenius norm of an entire momentum matrix before orthogonalization, whereas AdamW applies it independently to every coordinate. Their numerical epsilon values are therefore not directly comparable. For the expert-matrix shapes in our model, Muon’s commonly used epsilon of $10^{-7}$ has an intervention scale closer to an AdamW epsilon around $10^{-11}$, rather than the usual $10^{-8}$. This helps explain why Muon and AdamW with $\epsilon = 10^{-12}$ produce similar expert-norm dynamics.
3. MoE expert training has a critical early stage. Leveraging this, the proposed LM Loss as Auxiliary Loss creates a stable signal for all layers and signifanctly improves training efficacy. [Section 2.1, Section 3.4]
The eventual layer-wise expert pattern can be determined within the first few thousand training steps. After this stage, once downstream modules stop depending on the lower pathway, its learning signal collapses and later recovery becomes difficult. LLAL set language modeling loss as an aux loss and apply it to the first MoE layer in this critical stage. It asks an early MoE representation to predict the next token through the LM head. Upper layers cannot reduce this auxiliary error on the lower layer’s behalf, so the lower pathway has an opportunity to establish useful features and downstream consumers. Once that dependency forms, the main loss continues to preserve it after LLAL is removed. Such stable signal improves layer dynamics, as well as validation loss and load balance, since routers and experts are also benefit from stable signals.
4. Open-source MoEs contain hidden structures that ordinary evaluation does not reveal. [Appendix F]
MiMo-v2.5-pro, Qwen3.5 and Qwen3.8-Max also show collapsed layers, while Qwen3.8-Max exhibits similarity between two large blocks of layers, suggesting possible layer extension or copying. In Kimi-K3, the learned balancing bias can override even the router’s top-1 choice for more than 70% of tokens in some layers, occasionally promoting experts ranked below 400 by the router’s own scores. These observations reveal how much of an MoE’s internal behavior can remain hidden behind its final benchmark scores.
</aside>
Training framework: Our experiments use Megatron-Core as the base framework, with the necessary modifications.
Model architecture: Unless otherwise specified, most experiments in this article use the following architecture:
The model has 32 layers, with MLA and 16 attention heads. The first layer is dense; each of the remaining 31 MoE layers contains 768 routed experts plus one shared expert, with 8 routed experts activated per token (top-8). The router uses a sigmoid scoring function and DeepSeek-style aux-loss-free load balancing, with the expert-bias update rate fixed at 0.001. The activation function is SwiGLU.
To make the failure mode easier to expose, we use a model that is substantially sparser and relatively narrower and deeper than mainstream designs. It has roughly 60B parameters, with about 1.0B active parameters per token, excluding the LM head and embeddings. [For some small experiments, we use a 5B, 32-layer model with 512 experts and top-8 routing. We also have an experiment on a 180B model at the end. The relevant details are provided where those results are discussed.]
Optimizer: Unless otherwise noted, we use AdamW with a weight decay of 0.1. Muon experiments are identified explicitly. We use a cosine learning-rate schedule over 90,000 training steps, with 2,000 warm-up steps.
Training data: We use a mixture of open-source datasets, including high-quality web, code, math, and STEM data, broadly following a standard pretraining recipe. Sequences are packed to a length of 8K.
During ultra-sparse MoE training, we observe that after an initial period of growth, the weight norms of routed experts in lower MoE layers begin to decline, eventually even falling below $\frac{1}{10}$ of their initial values, while the shared expert remains within a normal norm range. This form of silent expert collapse leaves virtually no trace in the usual training signals: both training and validation losses continue to decrease normally, no unusual spikes appear, and no distinctive warning appears in commonly monitored load-balancing metrics. As their weight norms shrink, the routed experts become functionally inactive. Masking every routed expert in the affected MoE layer causes almost no degradation in model quality. In other words, nearly all computation spent on the routed experts in such a layer is wasted; the layer has effectively degenerated into a dense layer (the shared expert) or even a null layer.
As shown below, in a long real training, routed-expert collapse is particularly severe in the first MoE layer. Higher MoE layers remain much healthier, with their expert weight norms concentrated within a relatively narrow range.
Figure 1. Routed-expert norm dynamics (left) and layer-wise expert-norm distributions after collapse (right) of a long real training.
We can reproduce similar behavior with our experimental configurations in Section 1:
Figure2: Expert norm dynamics and layer-wise expert norm distribution of Exp1 at the end of training.
[Exp1] This run also serves as the baseline for most subsequent 60B experiments. It uses a batch size of 384 and approximately 280B training tokens. Its early MoE layers collapse and the expert norm is ~ $\frac{1}{10}$ of that in higher healthy layers.