Neural Networks Simply Explained (Theory)

Key Takeaways

what is going on guys welcome to this video in today's episode we're going to talk about the theory behind neural networks in order to get an understanding of what a neural network is how its structured and how it basically works so let us get into the explanation now let us start with the question of what a neural network actually is a neural network is basically just a mathematical structure that solves problems in the machine learning world so this could be a regression problem a classification problem a reinforcement learning problem it doesn't matter if it's unsupervised supervised whatever a neural network is used for multiple different things and they're highly used because they're so effective at problem-solving and neural networks are consisting of so called neurons so this could be a neuron here and they are consisting of layers of neurons so the first layer of neurons in a neural network is called the input layer and it's followed by a bunch of so-called hidden layers these hidden layers are basically layers in between the two outer layers so they are also consisting of a bunch of neurons here and in the end we then have an output layer and usually if you're using a an ordinary feed-forward neural network with dense layers which you have is you have all the neurons of one layer connected to all the neurons of the next layer so I'm not going to draw all the connections here right now what you get the idea every neuron of one layer is connected to every neuron of the next layer so all these neurons are interconnected I'm not going to draw all of them now just so that you get the idea basically you know something like that they're all interconnected and this goes not only for the input layer and the hidden layers it goes for all the layers so they are all interconnected here also for the output layer let's just write it down here this is the output layer and these here are the hidden layers and the hidden layers are come on the hidden layers are just layers of abstraction layers of sophistication because the only thing that you want is actually you want to put in some data and get some output for example if it's a classification problem we want to put in an image for example of some animal here and what we want to get is we want to know a cat or a dog so it could say okay it's a cat or it's a dog whatever it is and it's basically a classification problem in this case we feed in information into the input layer and we want to get a result in the output layer for example this could be of the neuron indicating that's a cat and this could be the neuron indicating it's a dog and this could be activated at 0.9 in this 0.1 which would tell us it's a 90% a cat so this is how neural networks are used for classification for example and these hidden layers in the middle are just what makes the neural network think you could say so what we're looking at here is an ordinary feed-forward neural network we're going to talk about the different types of neural networks in another video but this is an ordinary feed-forward neural network which means that we have the direction the information is always fed into one direction we're only feeding the information forward we don't have any loops for example like this year or we don't have any backward connections where we feed information back into the network we just have feed-forward from the input layer to the output layer through the hidden layers and the hidden layers are what makes this model complex because this year is just input this year is just output and everything in between are the interconnections that we can tweak the parameters that we can change in order to get the right prediction so now let us take a look at the structure of an individual neuron now an individual neuron gets a lot of inputs from a previous layer so the previous layer produces a bunch of outputs and neurons of a previous layer produce a lot of outputs and these outputs are then the input for our next layer here so i1 i2 i3 and okay it should be a three and these inputs here are just numerical values for example they could be 0.3 they could be 0.7 and 0.9 of course it could also be 25 depending on the activation function that you use we're going to talk about that in a second but you have these values here and these values are now being weighted so each connection into our neuron has a certain weight w1 w2 and w3 so these are the weights and these weights indicate how important these particular things are so we could say that weight one is five for example and this would mean that we multiply zero point three times five so we have in this case one point five as a final output that comes into the neuron then we have zero point seven let's say this is zero point 5 and this would mean it's half of its so zero point three five and then we have zero point nine and the way it could be just two and this would produce one point point eight so these are the final inputs that come into our our neuron input we now do with them is we feed them into a transformation function which is actually just summing them up so we calculate the sum of those I don't want to do the calculation right now let's just say it's roughly four and what we do then is we subtract a so called bias so we have a certain bias here and this bias is a constant value that we subtract for example it could be three three and we then subtract this bias from the sum so we say four minus three and this is one and then this is the final output of this transformation here what we do then is we feed this into one more thing called the activation function this activation function takes this input and produces a final output so activation function so we take this one here and feed it into the function so in this case the function could be a sigmoid function I don't know what the exact value of one would be zero is like zero point five so one is probably I don't know point six around zero point six let's say it's just that and distend the final output so zero point six is the final output of this neuron which is again let me pick another number here the input for another neuron in the next layer so this zero point six here is the I one for the next neuron for example so this is how it works let me repeat it one more time you get some inputs then you have the weights forty connections here so we get some numerical value multiply time the weight times the weight you do this for all the connections entering a neuron then what you do is you sum all these values up subtract a certain bias feed the result of this calculation into a certain activation function we're going to talk about activation functions in a second what kind of activation functions there are but basically it just takes some activation function you feed in the final result of this calculation then you get the final result the output of this neural network of this neuron sorry not of the whole neural network now one thing that it's important is you don't choose the weights and biases yourself you don't as a programmer you don't define the weights and biases of a neuron what we do is we let the neuron figure out using gradient ascend and back propagation we're going to talk about that in a second we're letting the neural network figure out what values are best for producing the desired results so we're training the neural network we're finding out okay how can we find the best way it's and biases so that you always produce the result that we want in the case of cats and dog images you were feeding in the pixels you're showing him okay this is a showing it this is a dog this is a cat and what you want is you want a neural network to tweak the weights and biases so that it always produces the right results this is what we're trying to do here and this is the structure of a basic neuron now let us take a look at two activation functions in order to get an understanding of what's happening when we're using an activation function and what they look like so before we do that let's just talk about how an activation could be done you could say this is the x-axis this is the y-axis as long as I get an input X which below zero I always return zero and as soon as I get an input over zero above zero I return one so this is what the basic perceptron does but the perception is very outdated and not used because this version of an activation function so to say it's pretty limited so what you do here is you basically say okay if it's zero point zero zero zero one it's a 1 but if it's minus zero point zero zero one it's a zero and it's very limited and very static so very binary very black-and-white so we have activation functions that do this in a much better way so let us take a look at the sigmoid activation function we're not getting too mathematical here but I'm going to write down the function so that you get an intuition the sigmoid function is basically Sigma of x equals I'm going to write it actually we have no space here 1 over 1 plus e to the power of minus X now we don't need to memorize this you don't need to know this when you're programming you're just typing in sigmoid activation function but this function is a little bit smoother than the basic perceptron so what we're doing here is we're basically I hope I don't do this to shitty here don't take it too seriously what I'm drawing here because it sounds perfect let me redraw this actually so basically you're having a smooth version oh that's actually quite good of the perceptron so you're having the one value here so this is the limit this is the one and this of course is the zero and whatever we input into the sigmoid function it returns a certain value between zero and one so if we get a large negative value so minus 9999 this would return almost zero it will never reach zero actually but it will return 0.001 or something I don't know the same goes for a large positive number for example 9999 this would return something that's almost 1 so 0.9999 and so on but the good thing is that you don't get the big jump from zero to one if you just pass the zero mark because if you have I don't know let's say minus 0.01 this should be around I think 0.49 or something as soon as you get to zero you get zero point five and as soon as you get to zero point one you get zero point five zero zero one or something I don't know but it's a much smoother way of giving you an output for a given input instead of just saying either zero or one we have a big jump there so this is one function that's very popular in machine learning it's called the sigmoid function it's much better than the basic perceptron but there's actually one more activation function that is even more frequently used than the sigmoid function and this is the reload function and it's used because it's so super simple a real o stands for rectified rectified linear unit and this function is super simple but super effective because what it actually does is as long as you give it a negative input it will always return zero and as soon as you give it a positive output it will just return a positive input it will just return the input as an output so basically if you pass zero it gives you a zero if you pass minus five it gives you a zero if you pass - 999 it gives you a zero if you pass five it returns five if you pass seven it returns seven and so on so you have the smooth increase but still you don't allow for negative values which is very useful because negative values might lead to a lot of confusion in a neural network so you usually want to avoid them but still you have to gradual scale off values based on the inputs so one produces one and so on so the function written down is just f of x equals max of 0 + X so if X is less than zero we return 0 and if ax is more than 0 or greater than zero we return X a very simple function very you full and we're using it a lot of times actually in a lot of projects I've already used a real Oh and in future projects we're also going to use real Oh so this function is very important so now we're going to talk about a very fundamental concept of machine learning which is called training and testing so if you know something about machine learning you probably know what training and testing is but I'm going to explain it very quickly here because it's very essential for the next topic we're going to talk about which is gradient descent and back propagation so you have a machine learning model this doesn't have to be a neural network in this case it is a neural network but it could also be a support vector classifier could be a K nearest neighbors classifier whatever you have a machine learning model and what you want to do is you want to input some data and you want this model to output the right data now this could be anything this could be a breast cancer classification this could be a handwritten digit recognition this could be a voice recognition this could be whatever you want in this case we're again going to take the example of images of cats and dogs so we feed it in we feed in images of animals and what we want to do is we want a neural network to predict if this is a cat or to recognize if this is a cat or a dog so this is the goal with this particular machine learning model and in order to make it do that we need them all to adjust its weight and biases weights and biases we're going to talk about how it does that in a second but when we're training them all what we're doing is we're providing some labeled data at least when we're dealing with supervised learning and classifications we're providing training data which could be something like fifty thousand images of cats and dogs with classification so we could say okay this image here is a cat the next one is a cat as well the next one is a doc and so on and what we do is we feed all of these images into our neural network and we tell it afterwards okay this is a cat this is a dog this is again a dog this is a cat and so on and the neural network should then adjust its weights and biases so that we are so that it learns what a cat looks like and what a dogs our dot looks like this is how we train them all now what we do afterwards once the models trained is we test it so we say I have I don't know five thousand more images of cats and dogs and I'm going to use these images to test them all now these are images that our model has never seen before it's not the training data it's completely new images and what we do is we just feed these images into the machine learning model without telling it that this is a cat or dog we just see what it says and at the same time however the data is still labeled and we know if it's a cat or dog so we just compare the output of the neural network with the actual result and we say ok yes this is true or notice it's not true and based on that we calculate the so-called error I'm going to use a red number a red color here the error is basically how much percent of the class of the training examples sorry of the testing examples were classified incorrectly so an error could be ten percent and these ten percent would indicate that 90% of the images were classified correctly in ten percent not now there is a second metric that's even more important for the computer which is the loss metric and the loss metric is calculated with so called loss function and it's a numerical value that it's not that it's not that intuitive for for us humans so this could be something like two point nine one or five point six or zero point seven now this is not a percentage of how many of how many examples were lost or how much the error is but it's just a metric that is calculated by a specific loss function now for us humans it's more intuitive to look at the error in order to realize okay how much percent of the mall was classified incorrectly or how much percent of the training testing examples were classified incorrectly by our model but the loss function is actually what we want to minimize because the error is better for us humans but the computer can better work with a loss function so our goal is when using gradient descent and back propagation is to minimize the loss function we want to get it as low as it can get because a lower loss means that the mall is more accurate so the goal is minimize the last function this is what we do with the gradient descent algorithm so now let's talk about the gradient descent algorithm now before we start here I want to mention that I'm not going too deep into the mathematics here I'm not going to explain exactly with calculus and partial derivatives how the gradient descent algorithm works the only thing that I want to do here is I want to give you an intuition of how this works maybe I'm going to do a mathematical series in the future where I tell you or where teach you how these processes and algorithms work in detail but now we're just getting an intuition here so what the gradient descent algorithm does gradient descent is it tries to minimize a given function so let's say we have a simple function here where actually the output of the function let's say we have a very very simple function here which is just this one here this could be something like x squared plus 1 for example and what the gradient descent algorithm does is it starts at a certain point so let's say it starts here this is the x-axis which is the input and the y-axis at the is the output and now what you need to imagine here this is just two dimensions so it's very very very simplified but actually what we have here is X which could be the weight and the buyers or the weights and biases and Y is the output of the last function so we give it the X which is the collection of weights and biases and the Y is the loss function so what we do is with the current weights and biases we have a certain loss we have a certain y-value this could be seven for example and what we want to do is we want to minimize it as much as possible now as you can see we'll not going we're not going to reach zero at the last function we want to get to the local minimum we want to get to the minimum of this function which is right here so in order to get there what we need to do is we need to move the ax value and in this case as you can see we need to move it to the right because when we move the ax value to the right when we increase through x value we decrease the Y value so what we do is we basically make a tiny step into the into the right direction so that the Y value gets lower so in the next step or after this step we have a value of 6.5 for example and we continue to do this over and over again until we reach the minimum and this could be one or in this case it is one so this is basically what the gradient descent algorithm does it takes a point and in then tries to to get it into the local minimum now how does it do that as I said I'm not going to get too deep and deep into the mathematics here but basically we're calculating the direction of the steepest descent we're calculating the derivative the partial derivative of x in or for X in this case and we're finding out the best way to increase the function and then we're just taking a tiny step into the opposite direction as long as this is a positive number that the gradient is positive number because when the gradient is zero we know that we've reached a valley a local minimum so to say now this might be a problem when we're using more complex functions because of course this function was very simple but what happens if we have a function that looks like this year you see I mean this is a ridiculous function right now don't take it too seriously but if I start here for example with the gradient descent algorithm I will end up here in the local minimum whereas the function minimum in this case at least what we see here the total minimum would be here so we can go much much lower than the local minimum here and this is the problem or not the problem but the limitation of the gradient descent algorithm it's not going to find the minimum of the function it's going to find a local minimum so it's going to say ok from the starting point the local minimum the local valley is this one here but yeah there are a couple of points in this function here that are lower than this particular minimum here so this is the limitation of gradient descent and we need to be aware of it now there's one more thing that needs to be taken into account here when we're working with neural networks we're not working with an x-axis and a y-axis we have one last value but we have thousands and hundreds of thousands of inputs these are weights biases we have so many neurons each of those neurons have interconnections and we have weights and biases so so many of them we have thousands and hundreds of thousands of them so in this case I just used a three dimensional graph here to show you what happens when you have two dimensions in order to give you a little understanding or intuition of what could happen it's it's not possible to imagine it but what would happen if you're dealing with five hundred thousand I mentioned for example so this would be the z axis here this is the x axis and this is the y axis and the z could be the loss and x and y are the inputs weights and biases you could say and we start at this point D here and we want to get to this local minimum which is somewhere here so in this case we're not just looking at X we're looking at x and y so we need to calculate the partial derivative again I'm not going to explain it too much don't worry but basically we're looking at the X direction in which direction do I need to move in order to decrease Z the most so we are calculating the partial derivative of X in relation to Z or to f of X whatever so in this case you know you need to go into that direction or actually you know when you calculate the partial derivative you calculate the direction of steepest ascent so in this case this one and for y it's this one and then you calculate based on these two directions you calculate are the direction of steepest descent which is the total direction of steepest descent could be something like that and then you take a tiny tiny step into the opposite direction we're not talking about a big step we're talking about a very very very tiny step and then you repeat the process over and over again until you inevitably roll into the local valley or the local minimum this happens when you have two dimensions and when you have 5000 dimensions you need to calculate for each of those 5000 dimensions the best direction to take then to calculate the best total direction to take and then take a tiny step over and over again this is what the gradient descent algorithm does and it gets quite complicated when we're dealing with neural networks so last but not least we're going to talk about back propagation and back propagation might seem a little bit confusing and we're not going to get into the mathematical formulas here as well we're just going to talk about the basic idea the basic intuition of what is happening behind the scenes when back propagation is used so basically you have a neural network let's pick the input layers here let's pick the hidden layers here very ugly drawing I know doesn't matter but these are all interconnected here and whatever and in the end you end up with a certain result so you give it an input and you get a certain output for example this could be the indicator for cat this could be the indicator for dog and the output of our neural network is 0.74 cat and 0.34 dog and what we do here is we get this percentage here so 70 percent likelihood that it's a cat and 30 percent likelihood that it's a dog and then we compare it to the actual result when we're training so the actual result could be 1 and 0 because it's 100% a cat if you know the result it's 100% a cat it's like 99% of cat it's 100 percent of cat and it's 0 percent of dog so this is what the neural network gets to see and it will then know ok I need to increase this value this value and I need to decrease this value here so this is what the neural network now knows that it has to do but of course you cannot just go ahead and change this value you cannot just change the output of a neuron the only thing that you can do is you can change the weights and the biases you cannot change anything else you cannot change the output or input in any way so what you do is you basically go back a layer and just say ok looking at this output and how it needs to change what needs to change in the previous layer so for example this neuron here if I increase it would it increase this neuron and decrease this neuron if yes ok I need to increase this neuron because this neuron increases this one and decreases this one then maybe I realize okay this neuron here radically decreases this one and slightly increases this one which is you know in the middle of what we want but since it radically decreases this one we probably want to decrease it a little bit and you proceed like that over and over again so you say ok this one also needs to be decreased for our desired result and the last one needs to be increased a lot whatever and then what you do is you go back one more layer and then you see ok in order to increase this one decrease this one decrease this one and increase this one maybe you end up with the solution that you need to decrease these 3 and increase the last one and then you do this until we reach the input layer and then basically based on these results you know ok for this particular training example I need to increase or decrease the outputs of these individual neurons and based on that you can adjust weights and biases now the problem is that all of this that we did right now is just for one training example so then after that you get a second training example and this training example might demand a totally different way of increasing or decreasing the outputs for example I might feed in another image of a cat but in this case it was a very bad output for example like this and this will demand a much higher increase in the cat neuron and a much higher decrease in the dark neuron and also the other way around maybe I have a dog and it's wrong with classified so I will need to do the exact opposite of all that so what you need to do is you need to compare output to input and see how you want to change the individual activations of the neurons and based on that you're going to adjust weights and biases and of course for example let's just look at this particular neuron here the first one wanted to increase it the second one wanted to increase it even more then the third one wanted to decrease it and so on in the end what you need to do is you need to combine all of these results all of these desired increases and decreases to end up with an average result of an average demand so you could say okay most of them we want to decrease it a couple of them we want to increase it it so you'll end up with a slight decrease whereas if all of them or almost all of them want to decrease it you'll end up with a huge decrease this is how it basically works very superficial again don't take it too seriously here but this is the intuition and the idea of how back propagation basically works so that's it for today's video I hope you enjoyed it and I hope you learned something if you like this type of theoretical videos let me know in the comment section down below and hit the like button so that I know that you like these videos also feel free to ask questions and give feedback in the comment section down below and subscribe this channel if you haven't done yet because you'll see more future videos for free so thank you very much for watching see you in the next video and bye [Music]