Why Mixture-of-Experts Took 30 Years to Take Off

Mixture-of-Experts (MoE) models weren’t invented yesterday — they were proposed in 1991 by Jacobs, Jordan, Nowlan, and Hinton. So why did they sit on the sidelines for 30 years… and why are they suddenly powering today’s largest AI models? In this conversation, Daria Soboleva, Head Research Scientist at Cerebras, walks through the history of MoEs. You’ll learn: Why early MoEs were theoretically brilliant but impossible to run How hardware limitations (not ideas) stalled progress for decades Why dense models have now hit a scaling wall How MoEs introduce sparsity in the most compute-effi…