Generative vs Discriminative Models - Explained

Here's a scatter of points in two classes, and your job is to tell a new points class. There are two completely different ways to go about it. The first way is to study what each class actually looks like, learning the shape of the blue cloud and the shape of the red cloud so completely that you could sketch new points that belong to either one. The second way ignores all of that and just asks where the dividing line goes. One approach models the whole world. The other only draws the border. Both of them classify, but underneath they are solving two very different problems, and that difference shapes everything about how they behave. Let's make the split precise. The border drawer learns the conditional probability of the label given the features, P of Y given X. In plain words, given what I see, which class is it? That's all it ever models, and we call it discriminative. The world modeler, instead, learns the joint probability, P of X and Y, how the features and the label occur together. We call that generative, and here's why the joint is the richer object. Once you have it, the definition of conditional probability hands you P of Y given X for free. You just divide the joint by P of X. So, the generative side can always recover the border, but it goes the other way, too. The joint also describes the features on their own, which means you can dream up new data from it. The conditional can't do that. It knows the border, but it has no idea what an X even looks like. So, how does the generative side build that joint? It models each class separately. For every class, it learns a class conditional density, P of X given Y, the bell-like shape that says where that class tends to live. Then, it learns the prior P of Y, simply how common each class is. Multiply those together, class by class, and you've assembled the whole joint. And because you now have a model of X itself, you really can sample from it, generating fresh points that look just like training data. To classify a new point, Bayes' rule combines the pieces, and you pick the class that scores highest. Naive Bayes is exactly this recipe, with one shortcut to keep the densities cheap. The discriminative side refuses to do any of that. It never asks what a class looks like. It points all of its effort at one thing, the function P of Y given X, and it shapes that function directly, so the boundary lands in the right place. Logistic regression is the classic example. It squashes a linear score through a sigmoid, slides the boundary until the labeled points sit on the correct sides, and stops. Notice what it never learned. It has no model of the blue cloud or the red cloud, no sense of how the features are distributed. Ask it to generate a believable new point, and it simply can't. All it owns is the border. So, which one wins? This is the beautiful part. Picture the test error as we feed in more and more training data. The generative model carries strong built-in assumptions, so it has fewer things to estimate, which lets it learn fast and do better when data is scarce. But those same assumptions are usually a little wrong. So, its error flattens out at a higher floor. The discriminative model assumes less, so it needs more data to get going. But with enough of it, it settles to a lower error. The two curves cross. With a handful of examples, generative is ahead. Pour in enough data, and discriminative pulls in front and stays there. And here's the connection that ties it all together. Naive Bayes and logistic regression are not random opponents. They're a matched pair, both ending up with the very same logistic form for P of Y given X. They just reach it differently. Naive Bayes gets there indirectly by modeling the whole world and letting the border fall out. Logistic regression carves the border directly and never looks back. So, the real question was never which algorithm is better. It's a question of philosophy. Do you model the whole world, or do you just draw the border? And that's basically it. If you found this helpful, hit that like button, subscribe for more, and drop a comment if you have any questions. See you in the next one. Bye-bye.