MOE Explained in 150 seconds

Key Takeaways

In the modern era of deep learning, we have become obsessed with a single powerful correlation. We call it the scaling law. The premise is simple. If you want a smarter model, you make it bigger. The data shows that as you increase the number of parameters, the billions of weights inside the network, the model's performance on language tasks reliably improves. But there is a trap. As we push these models to trillions of parameters, the computational cost doesn't just grow, it explodes. We hit a compute wall. So, we need a mathematical trick that gives us the intelligence of a massive model, but the speed of a small one. This brings us to the mixture of experts. When a token like the word apple enters a layer, it is processed by a massive matrix of parameters. Here lies the inefficiency. In a traditional model, every single parameter is active for every single word, whether the concept is simple or complex. The mixture of experts approach changes this topology. Instead of one giant monolithic block, we split the parameters into specialized groups or experts. This router decide to look at the incoming data and decide which specific expert is best suited to handle this. This allows the model to be huge in terms of knowledge. Every models were made with MO architecture to chat GPT to current state-of-the-art models. Structurally, we take the standard feed forward layer of the transformer and replace it with this bank of experts. But this introduces a new challenge. How do we ensure the router doesn't just send all the work to one popular expert? This is the switch mechanism. We train a gating network to make a top one choice. It calculates the probability of which expert is best and routes the token only there. The result, we can build models with a staggering 1.6 trillion parameters, giving them immense capacity while they run four times faster than their dense counterparts. It turns out you don't need to use your whole brain at once. You just need to know which part to use.