Softmax function - Explained

Key Takeaways

Imagine a neural network just told you the answer is cat 2.0, dog 1.0, bird 0.1. But what do those numbers actually mean? Is 2.0 a 200% chance? Clearly not. These row scores called logits are just arbitrary numbers. They could be negative, huge, or tiny. So how do we turn them into something meaningful like probabilities? Let's say we have three classes, cat, dog, and bird. And our model outputs the scores 2.0, 1.0, and 0.1. Now, we need these to become probabilities, which means they have to be positive, and they have to add up to one. A naive approach would be to just divide each score by the total. But here's the problem. What if some scores are negative? you'd get negative probabilities and that's nonsense. And this is where the exponential function comes in. If we take e to the power of each score, something nice happens. Every result is guaranteed to be positive. No matter what the input was, e to the 2.0 gives us about 7.39. E to the 1.0 gives 2.72 and e to the 0.1 gives 1.11. But there's a subtlety here. The exponential doesn't just make things positive. It amplifies the differences. The gap between 2.0 and 1.0 was just one point. But after exponentiation, 7.39 is almost 3 * 2.72. The winner wins by more. Now we have positive numbers, but they don't sum to one yet. Here's the softmax formula. We take each exponential and divide by the sum of all of them. So e to the 2.0 is 7.39. E to the 1.0 0 is 2.72 and e to the 0.1 is 1.11. The sum is 11.22. And now 7.39 / 11.2 gives 0.66 a 66% probability for cat. 2.72 over 11.2 gives 0.24 for dog. And 1.11 over 11.2 gives 0.10 for bird. Does the softmax function exponentiate the normalize. There's one more knob we can turn the temperature. If we divide the logits by a temperature parameter t before exponentiating, we control how confident the output is. At t= 1, we get the standard softmax we just computed. With a low temperature like 0.5, the differences get amplified even further. The model becomes very confident in its top choice. But with a high temperature like 2.0, So the differences shrink and the probabilities spread out more evenly. At extreme temperatures, softmax either becomes a hard maximum or a uniform distribution. So here's the full story. We start with raw logits, exponentiate them to make everything positive and amplify differences, then normalize so the values sum to one and outcome proper probabilities. That's softmax. It's the bridge between a neural network's raw output and a proper probability distribution. And that's basically it. Thanks for watching and I hope I'll see you in the next one. Bye-bye.