Watch each token choose a small part of a giant model.

Sparse Mixture-of-Experts layers activate only a few expert networks per token. Change routing, capacity, balancing, and noise to see specialization collide with throughput.

Open the routing floor
ROUTER SCORES EXPERTS · TOP-K SELECTS · CAPACITY LIMITS TOKENS · ALL-TO-ALL DISPATCHES · EXPERTS COMPUTE · OUTPUTS COMBINE · LOAD BALANCING PREVENTS COLLAPSE · ROUTER SCORES EXPERTS · TOP-K SELECTS · CAPACITY LIMITS TOKENS ·
3%dropped tokens
0.84load balance
2/8active experts
expert 3hottest expert

Sparse activation separates parameter count from compute per token.

Router

A learned projection scores experts for each token. Top-k gating selects a small subset and supplies combination weights.

g(x) = top-k softmax(Wᵣx)

Capacity

Each expert accepts a bounded number of token assignments per batch. Overflow is dropped, rerouted, or sent through a shared fallback.

C = factor · tokens · k / experts

Auxiliary balance

A balancing loss discourages routing collapse, where a few experts receive nearly every token and become throughput bottlenecks.

L = Ltask + λLbalance

Routing quality and systems efficiency are coupled.

Top-k increases compute and redundancy

Routing to more experts can improve capacity and gradient coverage, but increases communication, memory traffic, and active FLOPs per token.

Expert popularity creates stragglers

Distributed MoE inference waits for loaded devices. Balanced counts are not merely aesthetic; the slowest expert can determine layer latency.

Dropped tokens are missing computation

When capacity is exceeded, dropped assignments skip expert transformation or use fallback behavior. Track drop rate by token type, language, and sequence position.

Routing questions

Does each expert learn a human-readable skill?

Sometimes clusters emerge, but specialization can be distributed, overlapping, and difficult to label. Expert IDs are not guaranteed semantic modules.

Why add router noise?

Noise encourages exploration and can improve load distribution during training. Too much noise destabilizes specialization and routing consistency.

Is an MoE model cheaper than a dense model?

Per-token compute can be lower relative to total parameters, but communication, memory, batching, and serving utilization determine actual cost.

Primary sources

Shazeer et al. (2017) introduced sparsely gated MoE layers at scale. Switch Transformers studies top-1 routing and capacity. GShard scales conditional computation with sharding. ST-MoE studies stable transfer and routing design.