Mixture-of-Experts Routing: Visually Explained

Mixtral “8×7B” can have ~47B total parameters, yet only a small slice activates per token—because a router sends each token to a top-K set of experts and combines their outputs. But MOE isn’t “pick two experts and you’re done.” We’ll walk through the real engineering story: routing math (softmax → top-K → weighted combine), why early MOE suffered expert collapse + load imbalance, and what MOE 2.0 changed with load-balancing loss and shared experts. Then we get practical: the all-to-all communication overhead that can wipe out theoretical speedups, the capacity/overflow tradeoff (and what “…