ADALINE: Adaptive Linear Neurons - EXPLAINED!

Key Takeaways

greetings fellow Learners now before we get into this wonderful world of adaptive linear neurons I have a thought-provoking question for you what fields are the most relevant when you think of artificial intelligence now the most obious answers for this is probably going to be something like Computer Science and Mathematics and if you consider neural networks then it would also be Neuroscience but I've also found that electrical engineering can also be pretty useful too and you can tie that very nicely with Concepts in Ai and we'll actually see that in the duration of this video but turning this question over to you what Fields do you think are most relevant when you think of artificial intelligence comment your answers down below and i' love to hear your thoughts now this video is going to be divided into three passes where we're going to first walk through the why of adaptive linear neurons then what is Adeline and then some comparisons between the per ctron model for classification and the Adeline model for classification so let's get to it in the last video we took a look at the perceptron model classifier that was created by Frank Rosen blot in 1958 and this perceptron model was inspired by the signal processing capability in human sensory systems around the same same time Bernard woodro and marcian Ted Hoff were two electrical engineers that were also interested in Signal processing but not in human biology but in electrical circuits and in order to understand now why we have these adaptive neurons we need to understand a little bit about circuits and circuit design so let's get to that so this here is a simple circuit where a circuit defines the path of electron flow and a simple circuit in the simplest form consists of a power source a switch as well as a resistor or a bulb now if we have an open switch then that means that the bulb is off which we get the response from and in a closed switch we see that electrons can flow and hence the bulb is on now for this circuit diagram if we label the switch as X and the bulb as B we can generate a truth table and also a Boolean expression with the output bulb being represented as some function of the input switch so closed switch is one open switch is zero and this would be the bullan expression now this here is a really simple circuit but circuits can get quite complex so how do we actually design more complex circuits well in 1939 Claude Shannon wrote a master's thesis in which he actually addressed exactly this and defined the steps for how you can define a circuit so it involved first constructing a true table then creating a bullan expression from this truth table then we simplify this Boolean expression using Boolean algebra and then you can reconstruct the circuit from this simplified Boolean expression and what this allowed was that the simple circuit can be constructed using this method and so circuit design now transformed because of this from an art to more of a science now the con of this however is that constructing truth tables can become INF feasible if the number of inputs and outputs is very large for n inputs there can be two to the power of n records in our truth table which is intractable so how do we solve this problem now this is where Bernard woodro and head Hoff come back and they realize that well for some circuits like traffic lights the accuracy needs to be 100% otherwise it can lead to some catastrophic results like an accident however there are many other circuits which they are interested in Signal processing specifically that don't require 100% accurate results all the time like in telecommunication now so the objective when designing a circuit should not be to minimize the number of components as Shannon had done before but instead minimize the average number of errors and so circuit designed by Bernard woodro and Marci and Ted Hoff transformed Boolean algebraic form of Designing a circuit to a statistical optimization form of Designing circuits and statistical optimization basically means involves techniques like gradient descent now just to note here at this time gradient descent did exist and it was introduced officially I guess in 1847 by a French mathematician Ki now to illustrate this statistical optimization wooder and Hoff took a fundamental element that is found in many adaptive circuits and by adaptive we mean that the circuit consists of components that can be changed and in this case these little knobs over here are voltage gains that can be changed now they also called this circuit a neuron because it is structured similar to the previous artificial neurons like the molic pits neuron that we discussed in a previous video as well as later which came on which is the John Von Newman neuron and the circuit became known as Adeline quiz time have you been paying attention let's quiz you to find out when does circuit design by minimizing the errors make sense over tro tables a when you want to simplify circuit connections B when the number of possible inputs and outputs is large C when errors do not lead to catastrophic results or D when dealing with circuits that cannot be simplified by Boolean algebra note that multiple answers may be correct here and I'll give you a few seconds to answer this question the correct answers are B and C but can you tell me why give your reasoning in the comments below and let's have a discussion now if you think I do deserve it at this point please do consider giving this video a like because it will help me out a lot now that's going to do for quiz time and pass one of this explanation but keep paying attention because I will be back to quiz you so Adeline here stands for an Adaptive linear classifier and why it's called so is going to become more apparent by the end of this pass so in this pass we're going to talk a little bit more about some details in explaining the what of Faline so it explains the mathematics and then also uses this in tandem to design an electrical circuit so let's talk about this so we know woodro and Hoff basically said that they wanted to design circuits to minimize the average number of errors this is kind of what we saw in the previous pass and so what they want to minimize is this mean squared error where this y here is going to be the reference output the actual output they want in the signal and this y hat is the output signal of the final electric circuit that they design now this output signal is going to be generated by a signal transforming device called a filter so essentially we have an input signal and the filter circuit is going to take it and generate an output signal and adaline is this filter circuit that we are designing and we want to adapt the gains those knobs that we saw before to minimize the average error now let's kind of map this a little bit to the circuit diagram so in our out ofine image that would be this prediction that we see here the output of this filter is going to be at this point here where it's going to be the sum of products of these AIS and XIs XI is going to be a switch it's an input switch then we want to kind of adjust these knobs these A's to minimize the error so that's kind of what we're saying in the math and and so what we want to do is minimize this equation right over here where we know this here is going to be the output of the adaptive filter out ofine and because we want to minimize we can take a gradient with respect to AJ so we take the gradient and then we get this big equation over here and so how much we need to tune AJ that is like one of the gains it is dependent on every single sample and from an electrical circuit standpoint this electrical circuit can become really expensive because it needs too many inputs and so in order to deal with this we draw Hof said let's approximate the gradient the total gradient with respect to that single gain to be the sum of the individual error gradients here and so what we can do is go back to the math and what we want to do is just the mean squared error for a single sample and right now we can then take the derivative and we get this equation over here so we have the gradient uh with respect to a certain knob for a single sample equal to -2 * the error time the input itself and so our actual change according to the you know if we apply this to like our gradient descent formulation is just going to be we adjust the knob by 2 e XJ and this XJ is going to be just a switch right and so if you've kind of noticed here this is like stochastic gradient descent and this is thus more practical in electrical circuit design so armed with this gradient update rule we now have an electrical circuit that's designed known as Nai adaline so it consists of 16 input switches it consists of 16 gains which can be tuned this is the bias knob over here and it consists of a reference switch which is going to give us the true signal that we desire and this voltage reading in during the training phase is going to give us the the error the magnitude of error and so if we trying to transform this knat line into a circuit diagram it kind of looks like this where we have again the input switches here or defined here as plus one and minus one the knobs are the gains here there going to be 16 of these this is the bias gain and then this over here is going to be an operational amplifier an opamp which can be designed as a summer circuit to essentially take the sum of products of every single one of these so this could be like a0 plus A1 minus A2 minus a 3 plus A4 plus A5 and so on and I say that because the X's here are just going to be plus or minus one and then we're going to get an output continuous value which is y hot which we can compare to the true reference switch value using another operational amplifier so these opamps will just take typically the difference between this plus and minus so that's Yus y hat but if you design the resistors in a certain way then the we can also make it a multiple of this difference so this Ada could just be like any small fractional value and you know this kind of probably looks very similar to like the learning rate in a gradient descent algorithm now given this architecture let's now uh iterate through the actual training procedure so for every single input and label that's in the data set we set the input via the 16 switches we then reset the reference switch to get whatever label output signal that we want want then the voltage meter is going to read the air and it's going to be determined by this formulation over here Ada is set by how the knobi Adeline has been designed specifically the opamp has been designed and then we'll adjust each knob according to this formulation that we derive with stochastic gradient descent and because we derived it with a stochastic gradient descent approach as we iterate through every single input and label these knobs will eventually converge to values that will minim the average number of errors and this overall algorithm is going to be known as the least mean squared algorithm or the LMS algorithm now during inference phase we are now given just an input and this input is set via the switches now the voltage meter is just going to read the predictions why there is no there is no reference that's set and then we are going to quantize this in the sense of like okay whatever y hat is if it's greater than zero it's going to read A+ one if it's less than zero it'll read a minus one and this would be the classification label and so in summary we have adaline it is adaptive because it has weights that can be tuned the knobs it is linear because the prediction is a linear combination of the inputs and these adjustable weights and then it's a classifier quiz time it's that time of video again have you been paying ention let's quiz you to find out how do we transform the continuous outputs to get a discrete output from Adeline a directly from the meter B using the quantizer C using the reference switch or D using the constant Source note that multiple options here may be correct and I'll give you a few seconds to answer this question the correct option is B but can you tell me why comment your reasoning down below and let's have a discussion now that's going to do it for quiz time and pass two of this explanation but keep paying attention because I will be back to quiz you in this pass let's compare perceptron and Adeline so both of these are classifiers and the perceptron was inspired by signal processing in humans while adaline was inspired by signal processing in electrical circuits now the learning procedure in perceptron is error correction so we update the weights only when an error is made whereas an analine the pro learning procedure is least mean squares algorithm and so on every sample we will have an adjustment of these gains now in the perceptron the activation function is essentially just a threshold function so the outcomes are always discret but in Adeline the activation function is linear at least before quantization quiz time oo this is going to be a fun one which of the following statements is true a the perceptron was largely influenced by human biology B the Adeline was largely influenced by electrical circuits C perceptron update is a form of stochastic gradient descent or D adaline update is a form of stochastic gradient descent note that multiple options here may be correct and I'll give you a few seconds to answer this question the correct options are all of them a b c and d but can you tell me why give your reasoning in the comments below and let's have a discussion and if you think I do deserve it at this point please do consider giving this video a like because it will help me out a lot that's going to do it for quiz time in pass three of this explanation but before we go let's generate a summary in this video we talked about Adeline Adeline is a filter circuit that we're designing to adapt gains in order to minimize the average number of Errors we walked through its training procedure which will effectively follow a stochastic gradient descent approach all in order to minimize the average number of Errors we also looked at how from the math we could construct kni Adeline and this circuit diagram where we Define out align to be adaptive because it has weights that can be tuned linear because the prediction is a linear combination of inputs and adjustable weights with a linear activation function and then is a classifier and then we ended with some comparisons between the perceptron model and the Adeline model so that we get like an idea of how they exist together and that's all that we have today thank you all so much for watching this video and if you like what you saw please do consider giving this video a like if you want to continue your AI Journey please do consider watching this video right over here you won't regret it and I will see you in the next one bye-bye