Dot product explained (Maths Behind AI)

What is the dot product? Here is something remarkable. Every time you ask Google a question, every time Netflix recommends a show, every time you talk to chat GPT, there is one mathematical operation running millions of times per second underneath all of it. That operation is the dotproduct. It is the simplest thing in the world. Take two lists of numbers, multiply them pair by pair, and add up all the results. One number comes out but that single number carries extraordinary meaning. In this video we are going to build complete intuition for the dot product from its geometric meaning to its role in neural networks, attention mechanisms and recommendation systems. By the end you will see it everywhere in AI. Let us start. Before we write a single formula, let us understand what the dot product actually means. Imagine two arrows pointing out from the same origin. These are your two vectors. The dot productduct measures one thing. How much do these arrows point in the same direction? If both arrows point exactly the same way, perfectly parallel, the dot productduct is at its maximum positive value. The vectors are perfectly aligned. If the arrows are at a right angle to each other, perpendicular, the dot productduct is exactly zero. They share no common direction at all. In geometry, this is called orthogonal. If the arrows point in opposite directions, one forward, one backward, the dot product is negative. They are working against each other. There is also a beautiful geometric interpretation using projection. Imagine shining a light perpendicular to one vector. The shadow that the other vector casts onto it, that shadow length is deeply connected to the dotproduct. Specifically, the dotproduct equals that shadow length times the magnitude of the vector being projected onto. This is the projection intuition and it is the key to understanding why the dotproduct appears everywhere in AI. Now let us look at the two equivalent formulas for the dotproduct. The first is the component form. You take vector v and vector w both with the same number of dimensions. You multiply v_sub_1 by w1 then v2 by w2 all the way to vn * wn then you sum all those products. That is it. The second formula is the geometric form. The dotproduct equals the magnitude of v that is its length time the magnitude of w time the cosine of the angle theta between them. These two formulas give exactly the same number. They are two lenses on the same idea. The component form is what you compute with code. The geometric form is what tells you what it means. There are some important properties to know. The dotproduct is commutive. V.Wals w do.v. It is also distributive and linear in each of its arguments. And here is a special case. The dotproduct of a vector with itself equals the square of its magnitude. So the magnitude is just the square root of the self dotproduct. Let us work through three complete examples to make this concrete. First example v = 2a 3 and w = 4a 1. We multiply 2x 4 to get 8. We multiply 3 by 1 to get 3. We add 8 + 3 to get 11. The dotproduct is 11, a positive number, meaning these vectors lean in the same general direction. Second example, V= 1 comma 0 and W = 0 comma 1. These are the standard basiscomma 1. These are the standard basis vectors pointing along the x-axis andvectors pointing along the x-axis and the y-axis respectively.the y-axis respectively. 1 * 0 is 0. 0 * 1 is 0. 0 + 0 is 0. The1 * 0 is 0. 0 * 1 is 0. 0 + 0 is 0. The dotproduct is 0. These vectors aredotproduct is 0. These vectors are exactly perpendicular to each other,exactly perpendicular to each other, which makes perfect sense geometrically.which makes perfect sense geometrically. Second example, V= 1 comma 0 and W = 0 Third example v = 3a 2 and w = -2a 1. 3 * -2 is -6. 2 * 1 is 2. -6 + 2 is -4. The dotproduct is negative. These vectors are pointing somewhat away from each other. The angle between them is greater than 90°. See the pattern. Multiply pairs. Sum the results. And the sign tells you everything about the angle between them. The dotproduct has a small limitation. It depends on the lengths of the vectors, not just their direction. Two short vectors pointing the same way give a smaller dot productduct than two long vectors in the same direction, even though the direction is identical. To fix this, we normalize. We divide the dot productduct by the magnitude of V * the magnitude of W. The result is called cosine similarity and it ranges from -1 to positive 1. Pure direction, no size. A cosine similarity of positive one means the vectors point in exactly the same direction identical direction regardless of length. Zero means they are perpendicular. No relationship at all. Negative -1 means they point in exactly opposite directions. This is the foundation of almost every semantic search system you have ever used. When you type a query into Google, the system converts your query into a highdimensional vector using a language model. Every document in the database has also been converted to a vector. The system then computes the cosine similarity between your query vector and every document vector. The documents with the highest cosine similarity are the most relevant results. Let us make this very concrete by looking inside a single artificial neuron. A neuron has inputs. Let us say three input values x1, x2 and x3. Each input has a corresponding weight W1, W2 and W3. The neuron also has a bias value B. Here is what happens in the forward pass. First, the neuron computes the dotproduct of its weight vector with its input vector. That is W1 * X1 + W2 * X2 + W3 * X3. In our example, that gives us 0.6 * 0.8 8 + 0.9 * 0.5 + 0.2 * 0.3 which equals 0.99. Then it adds the bias 0.99 + 0.1 = 1.09. Finally, it applies an activation function typically realu for hidden layers. Realu just replaces negatives with zero. Since 1.09 is positive, the output is 1.09. That is it. one dotproduct, one addition, one activation. The entire learnable intelligence of that neuron is contained in its weight vector. Training a neural network means adjusting millions of these weight vectors using gradient descent, which here is the beautiful part also involves dotproducts and transposed matrix operations. The attention mechanism is the architectural innovation that powers GPT, BERT, and essentially every large language model in use today. And at its heart is the dotproduct. Here is the core formula. Attention of Q KV equals softmax of Q * Kranspose divided by the square root of D * V. Let me unpack each piece. Q is the query matrix for each token in your sequence. The query vector asks what information do I need? K is the key matrix for each token. The key vector says what information do I offer? V is the value matrix. This contains the actual content to be passed forward. To compute the attention score between two tokens, you take the dotproduct of the query vector of the first token with the key vector of the second token. A high dotproduct means those two tokens are highly relevant to each other. A low or negative dotproduct means they are unrelated. You then divide by the square root of D to prevent the dot productducts from getting too large in high dimensions, which would push the soft max into regions of very small gradients. The softmax converts the raw scores into a probability distribution. All scores sum to one. Finally, you multiply by the value matrix to produce a weighted combination of values or high attention tokens contribute more. Every single forward pass through a transformer runs this computation for every token pair in the sequence. for a sequence length of 1,000 tokens with a model dimension of 768. That is an enormous number of dot products all running in parallel on GPU. Let us go one level deeper and understand projection. The geometric operation that reveals the true meaning of the dot product. When you project vector onto vector B, you are asking if I drop a perpendicular line from the tip of a down to the line containing B, where does it land? That landing point measured from the origin is the projection of a on to b. The formula is projection of on to b equals the dotproduct of a and b divided by the magnitude of b ^ 2 * b. The scalar component just the length of the shadow is the dotproduct of a with the unit vector in the direction of b which equals the magnitude of a time cosine theta. This projection concept is central to principal component analysis. When you do PCA on a highdimensional data set, you find the igen vectors of the coariance matrix. Each data point is then projected onto these igen vectors using you guessed it, the dotproduct. The projection tells you how much of each principal component is present in that data point. There is also the selfproduct identity. The dotproduct of a vector with itself equals the sum of squares of all its components which is exactly the square of its L2 norm or uklitian length. So the dotproduct is not just about angles between different vectors. It is also the fundamental operation for computing the length of a single vector. Let us zoom out and see how the dot product connects to the AI systems you interact with every day. Every major search engine converts your text query into a dense embedding vector using a language model. The entire database of documents has also been converted to vectors. The system then computes the dotproduct or equivalently cosine similarity between your query vector and every document vector. The documents with the highest similarity scores rise to the top. This is called dense retrieval and it has largely replaced older keyword matching approaches. Recommendation systems like Netflix and Spotify work the same way. Your preferences are encoded as a user vector. Every item movie sown product has an item vector. The predicted interest score is the dotproduct of the user vector and the item vector. The items with the highest scores get recommended. Large language models like chat GPT compute dotproducts at virtually every stage of their operation. Each attention head computes dotproducts between query and key vectors. Each linear layer computes dotproducts between weight vectors and input embeddings. A single token generation through GPT4 involves billions of dot products. Even computer graphics, when calculating how bright a surface should be under a light source, the rendering engine computes the dotproduct between the surface normal vector and the direction to the light. A high dot product means the surface faces the light directly maximum brightness. A dot productduct near zero means the light grazes the surface at a shallow angle, dim. A negative dotproduct means the surface faces away from the light completely dark. The dot product is not just a formula. It is the universal language for comparing things in highdimensional space. Here is the complete picture of everything you now understand about the dotproduct. In component form, take two vectors of the same length, multiply corresponding components, sum the results, one number out. In geometric form, that same number equals the product of the two magnitudes times the cosine of the angle between them. This connects the arithmetic to the geometry. The sign of the dotproduct tells you everything about the angle. Positive, the angle is acute. The vectors lean the same way. Zero, the angle is exactly 90°. The vectors are perpendicular and completely unrelated. Negative, the angle is obtuse. The vectors oppose each other. Cosine similarity is the normalized version. Divide by both magnitudes and it gives you pure direction measurement on a scale from negative - 1 to positive 1. This is the foundation of every semantic similarity system in modern AI. The self dotproduct equals the squared magnitude giving you a direct connection between the dotproduct and vector length. projection. The shadow of one vector onto another is computed directly from the dotproduct and sits at the heart of PCA and dimensionality reduction. Every neuron in every neural network computes a weighted sum which is a dotproduct between a weight vector and an input vector plus a bias. Every attention score in every transformer is a dotproduct between a query vector and a key vector. Every recommendation score, every search relevance score, every cosign similarity, all reduced to this one operation. The dotproduct is arguably the single most important operation in all of applied machine learning. If you truly understand it, both the formula and the geometry, you have a foundation for understanding virtually any AI algorithm. In our next video, we go from the dotproduct to matrix multiplication. How chaining dotproducts across rows and columns gives you the full power of linear transformations. Subscribe to Neurom for weekly videos on the maths behind AI. See you next week.