53 AI BASICS Overfitting

When we talk about artificial intelligence systems, especially modern machine learning systems, we are fundamentally talking about models that learn patterns from data. They do not understand in the human sense. They do not reason from first principles unless explicitly designed to do so. Instead, they approximate functions. Given inputs, they produce outputs. The entire game is about how well that approximation generalizes beyond what it has already seen. To frame today's discussion, imagine we are trying to build a system that predicts house prices. We collect data square footage, number of bedrooms, age of the house, location, and historical sale price. We feed this data into a model. The model adjusts its internal parameters so that its predictions are close to the actual prices in the data set. The optimization algorithm reduces some loss function perhaps mean squared error by tuning weights. At first glance, it may seem simple. The lower the error on our data set, the better the model. But this is the first conceptual trap. Performance on the training data is not the goal. The goal is performance on new unseen data drawn from the same underlying distribution. The gap between these two objectives is where overfitting and underfitting emerge. Underfitting happens when a model is too simple to capture the underlying structure of the data. Overfitting happens when a model is too complex and captures not only the structure but also the noise in the training set. Both represent failure modes of generalization but they occur for opposite reasons. Let us begin with underfitting. Imagine we try to model house prices using a very simple linear regression with only one feature square footage. Suppose the real world is more complex. Price depends heavily on location, proximity to transport, neighborhood crime rates, and school quality. If we ignore all these features, our model will systematically miss patterns. Even on the training data, the error will remain high. The model does not have enough expressive power to approximate the true function. In terms of bias and variance, underfitting is typically associated with high bias. Bias refers to the error introduced by approximating a complex real world problem with a simplified model. A linear model trying to approximate a nonlinear relationship will consistently deviate in the same direction cannot bend enough to capture curvature in the data. Even if we gave it infinite data, it would still be wrong because the hypothesis class is too restricted. Now imagine the opposite extreme. Suppose we use a highly flexible model such as a deep neural network with millions of parameters to fit a data set with only a few thousand examples. If we allow it to train long enough, it may drive the training error nearly to zero. It might learn the training data so precisely that it memorizes it. However, when we evaluate it on unseen data, the error increases significantly. The model has learned patterns that are specific to the training data set, including random fluctuations and measurement noise. This is overfitting. Overfitting is often associated with high variance. Variance refers to how much the model's predictions would change if we trained it on a different data set sampled from the same distribution. A high variance model is sensitive to small changes in training data. Slight differences in input data produce large differences in learn parameters and therefore predictions. In extreme cases, the model is effectively memorizing the data set rather than learning generalizable structure. To understand these ideas more formally, consider the notion of a hypothesis space. In machine learning, we select a family of functions from which our model can choose. A linear model defines a hypothesis space of all linear functions of the input features. A deep neural network defines a much larger hypothesis space that includes highly nonlinear functions. If the true underlying relationship lies outside the hypothesis space, we will necessarily underfit. If the hypothesis space is too large relative to the available data and constraints, we risk overfitting. Another useful way to think about this is in terms of error decomposition. For supervised learning, the expected prediction error can be decomposed into bias squared, variance, and irreducible error. The irreducible error comes from noise in the data generation process. For example, even if we know all features of a house, there may be random factors influencing its sale price. No model can eliminate this component. The bias and variance, however, are properties of the model and learning procedure. Underfitting corresponds to high bias, low variance. Overfitting corresponds to low bias, high variance. Let us examine a concrete visual example. Suppose we have data points distributed roughly along the quadratic curve. If we fit a straight line, we underfit. A line cannot capture the curvature. Training error is high. If we fit a polomial of degree 15, we might pass exactly through every data point. the training error is zero. However, between the points, the curve oscillates wildly. When we evaluate on new points, predictions are erratic. This is classic overfitting. The important insight is that minimizing training error alone is insufficient. We must consider generalization error. This is why we split data into training and validation sets. The training set is used to optimize parameters. The validation set is used to estimate generalization performance. If training error is low but validation error is high, we are likely overfitting. If both training and validation error are high, we are likely underfitting. Now, let us connect this to modern deep learning systems. It may seem paradoxical that large neural networks with far more parameters than data points can generalize. Well, classical statistical learning theory suggests that models with higher capacity should overfit. Yet in practice, deep networks often perform remarkably well even in overparameterized regimes. >> This has led to active research into implicit regularization, optimization dynamics, and the geometry of loss landscapes. >> Where are you? >> Nevertheless, overfitting remains a real risk. For example, if we train a model on a small data set of medical images, it may memorize patient specific artifacts such as scanner noise patterns or hospital specific labeling conventions. When deployed in a different hospital, performance may collapse. The model has not learned disease features. It has learned data set quirks. To mitigate overfitting, we use regularization techniques. Regularization constrains the hypothesis space or penalizes complexity. In linear models, L2 regularization adds a penalty proportional to the squared magnitude of weights. This encourages smaller weights and smoother solutions. L1 regularization encourages sparsity by penalizing the absolute magnitude of weights. Both methods reduce variance at the cost of slightly increasing bias. In neural networks, we use techniques such as dropout where random neurons are temporarily deactivated during training. This prevents co-addaptation of features and acts as a form of ensemble averaging. Data augmentation is another powerful technique. By artificially increasing the diversity of training data through transformations such as rotations, cropping, or noise injection, we reduce the model's ability to memorize specific examples. Early stopping is also common. We monitor validation error during training and stop when it begins to increase, preventing the model from fitting noise. If you find this video exciting, please like, subscribe, and turn on notifications. See you in the next AI for Beginners educational video.