Weights and Biases Explained

Weights and biases, two words you will hear in every AI and machine learning conversation, but almost no one explains what they actually are. In this video, we will build the concept from scratch with real numbers, real diagrams, and zero handwaving. By the end, you will understand exactly what weights and biases are, how they are learned, and why they are the foundation of every neural network ever built. Let us start. A weight is a multiplier. It is the number that tells a neural network how much should I care about this particular input. Here is a concrete example. You are predicting house prices. You have three inputs. Size and square feet, location quality, and age of the house. Each input gets a weight. Size has a weight of 150. That means for every extra square foot, the predicted price goes up by $150. Location has a weight of 50,000. Location matters enormously, a weight of 50,000 reflects that. Age has a weight of -2,000. Negative. Because older houses are worth less, the weight encodes that inverse relationship automatically. When you multiply each input by its weight and sum everything up, you get a raw prediction. The model does not decide these weights up front. You do not write them by hand. Training discovers them. That is the whole point. Now let us talk about bias. Bias is an offset. It is a number added to the weighted sum before producing an output. Think of it as the y intercept in a linear equation. In y= mx plus b, the b is the bias. Here is why bias is essential. Without bias, if all your inputs are zero, the output is always zero. No matter what, the line must pass through the origin. That is a huge constraint. What if the correct answer when all inputs are zero is not zero. Bias fixes this. It gives the neuron a baseline output that is independent of its inputs. An analogy. Imagine your body temperature. It sits at 37° C even when you are not exercising. That baseline is your body's bias. It fires at a default level with no input required. Bias is also a learnable parameter just like weights. It gets updated during training through gradient descent. Every neuron in every layer has its own bias. It is that single number that gives the neuron the freedom to shift its decision boundary anywhere it needs to be. Let us now look at exactly what happens inside one single neuron. A neuron takes multiple inputs. Each input is multiplied by its weight. All those products are summed together. The bias is added on top. Then the result passes through an activation function that introduces nonlinearity. The equation is Z = W1 * X1 + W2 * X2 + W3 * X3 and so on plus B. Then output equals the activation function applied to Z. Let us compute a real example. Input X1 is 0.8 with weight 0.6 that gives 0.48. Input X2 is 0.5 with weight0.3 that gives0.15. Input X3 is 1.0 with weight 0.9 that gives 0.9. Add the bias of 0.2. Total Z 1.43. Apply realu max of zero and Z result 1.43. This single calculation repeated across millions of neurons thousands of times during training is how a network learns to recognize cats, translate languages, and generate text. So how do weights and biases actually get learned? The answer is gradient descent. Here is the intuition. Imagine a hilly landscape. Your current weight values determine where you are standing on that landscape. The height of your position is the loss. How wrong your predictions are. Your goal is to walk downhill to the lowest point. That lowest point is where weights and biases produce the most accurate predictions. Gradient descent works like this. First you make a prediction with the current weights and biases. This is called a forward pass. Then you compute the loss. How far off was that prediction from the truth. Then back propagation calculates exactly how much each individual weight and bias contributed to that error. Finally, you nudge each weight and bias downhill by a tiny amount proportional to its gradient. The update rule is new weight equals old weight minus the learning rate times the gradient of the loss with respect to that weight. The same equation applies to bias. This cycle repeats thousands of times on millions of examples. Each iteration the weights and biases get a little bit better. After training, they have settled at values that produce the most accurate predictions possible on your data. Now let us zoom out and see weights across an entire network. A single layer in a neural network is a matrix operation. You have an input vector X. You multiply by a weight matrix W and you add a bias vector B. Then apply the activation function. That is it. W * X + B. Repeat this for every layer. In a tiny network with three input neurons for hidden neurons and two output neurons, the first weight matrix is 3x4. That is 12 weights plus four biases. The second matrix is 4x 28 weights plus two biases. 26 parameters total in this tiny example. Now consider real networks. Lenet from 1998 has 60,000 parameters. ResNet 50 has 25 million. GPT2 has 1.5 billion. GPT4 is estimated at 1.8 trillion parameters. And every single one of those parameters is a weight or a bias found by training, not by human engineering. The math is the same at every scale. The numbers just get larger. Before training can even begin, you have to choose starting values for every weight and bias. This is called weight initialization, and it matters far more than most beginners realize. If you initialize all weights to zero, every neuron in a layer will compute identical outputs and identical gradients. They all update together. They all stay identical. The network never breaks symmetry. It effectively never learns. If you initialize weights too large, the activations become saturated. They push into the flat regions of sigmoid or tan functions where gradients are essentially zero. Learning stops. The solution is carefully calibrated random initialization. For networks using tan or sigmoid activations, the Xavier initialization draws weights from a uniform distribution scaled by one over the square root of the number of inputs. For networks using realu, the he initialization draws from a normal distribution scaled by the square root of two over the number of inputs. These formulas ensure that the variance of activation stays roughly consistent from layer to layer which keeps gradients flowing during back propagation. Modern frameworks like PyTorch and TensorFlow handle this automatically. But knowing why it matters helps you debug when things go wrong. Let us look at what weights actually look like in real trained models. In a logistic regression model for spam detection, around 1,000 weights, one per feature, values ranging from -3 to positive 5. A bias of negative0.84. Trained in seconds. High positive weight on a word like lottery. That feature strongly predicts spam. Negative weight on meeting. That feature argues against spam. Now look at GPT2 small 117 million parameters. Here something interesting happens. The individual weight values are tiny mostly between negative0.1 and positive 0.1. Why so small? Because with hundreds of millions of weights multiplied across dozens of layers even small values produce meaningful outputs. Large individual weights would cause activations to explode. The pattern holds across all deep networks. Individual weights are small, but the collective computation across millions of them is enormously expressive. This is also why transfer learning is so powerful. You can take those pre-trained weights from GPT2 and use them as a starting point for your own task instead of learning 117 million parameters from scratch. Three things commonly go wrong with weights during training and you need to know how to diagnose and fix each one. First vanishing gradients. As you back propagate error through many layers, the gradients get multiplied together repeatedly. If those multiplications keep producing numbers less than one, the gradients shrink exponentially as they travel backwards through the network. Early layers receive near zero updates and effectively stop learning. The fix use realu activation functions, he initialization and batch normalization. Second, exploding gradients. The opposite problem. Gradients grow exponentially instead of shrinking. Your loss function hits not a number and training collapses. The fix gradient clipping kept the gradient by its norm before applying the update. Third, overfitting. The weights memorize the training data instead of learning general patterns. Your training accuracy is high, but validation accuracy is poor. The fix L2 regularization adds a penalty to the loss proportional to the sum of squared weights, forcing them to stay small in general. Dropout randomly zeros out weights during training, preventing codependency. These three problems account for the majority of failed training runs. Diagnose them by watching your training curves and apply the targeted fix for each. So, here is the complete picture. A weight is a multiplier. It scales how much each input matters to a neuron's output. It is the strength of the connection. A bias is an offset. It shifts the neuron's output independent of its inputs. It gives the neuron a default firing level. Together, every neuron computes Z equals the sum of weight times input plus bias, then passes that through an activation function. A neural network is this computation repeated across hundreds of layers and millions of neurons. Training is the process of finding the weight and bias values that minimize the error on your data. Gradient descent does this step by step, computing gradients, then nudging every weight and bias slightly downhill. Initialize weights with Xavier or he. Watch for vanishing gradients, exploding gradients, and overfitting. Use the right fixes for each. That is it. Every neural network ever built from a simple logistic regression to GPT4 is doing this exact thing at different scales. If you want to go deeper on gradient descent, back propagation or how activation functions work, subscribe. Every week we go this deep.