Neural Networks 101: Practical Hands-on Session | Beginner Level | DataHour by Prashant Sahu

Key Takeaways

thank you guys for joining in time and this is the only slide that I have you know seven eight hardly I want to make this more of an interactive session more on coding okay so that we are able to uh actually see how neural networks work okay so let's get started and I request the audience to put all your questions in the Q a section only I'll not be watching the chat box for your questions if there is any generic networking that you want to do uh please take the chat box uh the webinar chat but if you have technical questions related to the uh topic that I am discussing I'll be only watching the Q a section right okay so how many of you are completely new to neural networks I mean you might have just heard the name but never done anything so just give us maybe a quick hand raise I can have a count okay I see eight nine ten so it seems that a lot of people have already started working on neural networks already okay and how many of you are slightly comfortable with neural networks you might have done some project on neural networks maybe CNN models image classification image detection or object detection object tracking anything anything similar okay so we have people who just have theoretical knowledge time okay YOLO okay perfect so we we won't be going that far uh I mean uh okay do not have any knowledge second year student okay uh let me do one thing I just check the YouTube section as well okay perfect so a lot of people have some exposure so it's a mixed pack that's what I get now mostly the audience has mixed bag okay so people have already worked with tensorflow and Pi dots okay good so we would not be you know discussing about python because we have exactly one hour session from the point I start okay and it's difficult to You Know cover into or you know delve into too many things okay because I have already mentioned that this is going to be a beginner friendly session and I want to keep it that way okay so let's get started so I'll not invest too much time into the timeline just because neural networks is not something which came up in the last five years or ten years frankly so this has been there the concept of neural networks was formulated somewhere back in 1943 something okay so you can always ask chart GPT so how many of you have haven't yet used chat DPT okay let me put it this question so the whole architecture the back end of chat GPT the you know the you know the whole uh model the backend model on which the charge GPT has been trained okay is a neural network model okay so a lot of application areas are there uh IBM Watson is there Alexa alphago okay so these are already you know deployed applications yes I am watching the chat box as well okay so the thing is uh if you haven't yet tried chargpt you should be trying that out because the back end is still all the neural networks so of course in this session we would not be discussing about the architecture and we would not be able to go to that level definitely but it is good to know that so whatever we are going to learn today is very much relevant that's the point so when I ask that question uh the intention was to tell you that the concept of neural network has been there for more than 80 90 years okay it's not that uh new network came into existence in the last 10 15 years only but neural network gained popularity in the last thing 15 years I would say okay now why did it gain popularity for the simple reason training the neural networks requires lot of memory a lot of speed it's computationally extensive okay intensive I would say computationally pretty intention okay and you need that kind of you know processing power so prior to the gpus the training of neural network used to take days even on very you know High sophisticated gpus the training of very sophisticated new neural networks do take days they have been trained there are a lot of pre-trained networks that we will probably talk in the next uh session I have planned another follow-up session on pre-trained networks as well okay so they have been trained on gpus okay so the thing is prior to the Advent of gpus trading the neural network on cpuser think about the Pentium processors okay uh 386 486 okay barely you can run some windows applications and using those processes training and neural network is hell of a job so neural network when it is not trained properly cannot perform a job I mean properly that that is so popularity another big reason and that's the cloud so with the you know Cloud at hand you can scale up your computational uh Power so let's say you want a 16 GB Ram or a 160gb ram you can add as many cores as as many gpus over the cloud and you can use 160 cores or 320 codes also for your training process okay so that seamlessly possible easily Okay so with the easy which is beginning of the neural networks have become very easy and that is the reason why it has gained popularity and now we have so deep neural networks with almost 175 billion parameters are there we'll see the calculations of these parameters in this particular session what these parameters mean you might have heard this statement this chart GPT 3 or gpd3 has got 170 billion parameters and now we have the gpt4 version coming up 230 billion parameters so how do you train so many uh parameters right so that is the thing uh so let's get started I mean uh I can give you a biological analogy first and then how this biological neuron translates into an artificial neural network neural network or artificial neuron basically neuron is a building block and network is basically when you connect all these neurons together so as to process the data to learn the pattern in the data and give an output that's called as neural network so in our brain we have the brain is your main you know processing horse in your entire body right that processes all the signals which is coming from your five senses okay so all the stimulus is passed to your brain in the form of electrical signals and in the brain these nerves carry forward these electrical signals to the brain there in your brain you have these neurons so these neurons interact with each other and process those electrical signals and help us to identify the eyes what they are seeing the nose what kind of smell it is the you know tongue what kind of taste it is okay uh ears what am I hearing and if someone is pinching you what kind of touch is that okay so all that processing is done by these neurons a single neuron does not process anything uh anything you know big okay I would say so you need billions of these neurons in a proper networked way okay so as to make sense out of these signals that is the main thing right so dendrites is something which will uh these are the inputs to the neuron I would say so dendrites is something which is collecting the input from a previous layer of neuron you can say or from a pre previous neuron so dendrite will connect with another new around so that right will take the import nucleus is something where the entire information is processed okay the process information is passed through this axon which is covered with a sheath and all and these X1 terminals are there so you can see a lot of branching so the same signal is passed to multiple neurons okay so all those multiple neurons will again process the same signal and then they will again pass the processed signal to another layer of neurons that is how it goes and that's exactly the same concept we follow in our artificial neural network also so we have a input node will have a processing you know something like a nucleus and then we'll have an output layer okay so input node a hidden layer which will process all the data and output layer so at least three things we will have in our artificial neural network as well okay uh so these are the main application areas I would just go through them very fast because many of them have already known that the point is fine so computer vision so basically image detection image uh or object detection image classification uh object tracking okay a lot of these applications which deal with so just imagine you want to identify whether a given image is a cat image or a dog image or a given image uh has got a person now is that person Prashant or Donald Trump I mean so that kind of image can be classified whether this person is question month or no question that's a simple classification you can identify the objects in a particular image okay this object is a you know computer this is a mouse this is a printer so different objects can be identified so these are all separate uh you know application areas related to computer vision and then you have natural language processing so this is what the GPT is all about so you language translation means you can translate a text from one language to another language so let's say I'm speaking in English Okay and there is a real-time translation which can happen in zoom in another language like French or German okay or any other language that you want to hear so how do you translate text from one language to another language for that again you are going to use neural network sentiment analysis so this is like I actually took four sessions on sentiment analysis before in a month of November December and Jan so I don't know if any one of uh present here have attended those sessions so sentiment analysis is like given a product review or a movie review okay you can try to identify whether that person was happy with that product or okay so it's a positive sentiment or a negative sentiment or is it a neutral sentiment kind of thing speech recognition again so given a uh voice data okay we can identify what this person is saying and who that person is so there are two things you can do in speech recognition Okay so identifying the person is one application that identifying what that person is saying is further another application so all your uh speech to text converters like Google okay Alexa so you are talking to Alexa and Alexa is able to convert that spoken language into a text that text is searched in the back end so let's say you say Alexa play me a lullaby okay or play me so and so song so how does Alexa able to you know perform that action so it is able to recognize what you are saying understand what you are saying and then that text is searched over Google or maybe somewhere and it searches and fetches the relevant information so you are asking what is the weather looking today okay so it will give you the weather information and then that text is spoken back to you so that's another level of conversion okay a conversion of uh Speech back to text so all that speech to text and text to speech this is all going to use your your Networks text classification it's like uh imagine your document classification so whether this given document is a medical document legal document it's a invoice or is it a receipt okay so you can classify different pieces of documents robotics you train robots Okay motion planning control and all that so um have you seen videos of Boston Dynamics have you seen videos of Boston Dynamics okay fine and then Finance you can create a model which can predict the stock price tomorrow so that's like a stock price prediction based on the historical data based on the historical Trend okay so your feed 15 years of daily movement and then you train your neural network okay so your now neural network is able to identify the patterns basically the most usual patterns the Candlestick patterns that's one way of doing technical analysis uh otherwise if you don't want to learn technical analysis let all the technical analysis be learned by your neural network your neural network is sophisticated enough to learn all that analysis from the historical data so you need some good amount of historical data to be able to train your neural network that is one drawback of the neural networks also if you use less data the accuracy of the neural network would be actually worser than a simple machine learning model let me warn you that way okay that is how a dual network has cons as well it's not just having Pros it has commands as well so training the new network I have already mentioned is a computationally expensive intensive process that's one cons second I said the data requirement for the same application will be at least 10x so if you have 10 000 data points you can make a machine learning model very easily you will be able to get 1995 accuracy but if you use the same 10 000 data points for a neural network either you will need a very sophisticated neural network if you use a vanilla neural network you might even get just 70 accuracy right so other option is increase the data volume and the data quality right and then so on so so a lot of applications so we are done discussing this okay so how does a neural network finally look like okay so you will have basically uh at least two things some input nodes as you can see here what are the input nodes so input nodes will capture input from your uh user let's say right you have a data set you need to have a data set so everyone here is aware of let's say Iris data set is everyone aware of this Iris data set come on let me take it that way I'll try to you know give you more practical examples because if you are here directly without having a knowledge of machine learning or no exposure to data science it will be difficult because I cannot explain those terms at all okay that's much beyond the scope so everyone here who has some background of data science or machine learning would have definitely encountered Irish data set it has got four features there's got four features at this point I would strongly recommend you to catch crab two things number one a pen as a paper because my slides are pathetic number one and I would be speaking a lot of things and I would be repeating that at a pace that you are able to note it down because my slides will have just pictures there would be rarely any text okay so that it will not help you at all okay so if you start noting it will register in your mind immediately okay two things so if you are aware of the Irish data set there are four features one is supplement one is sample width one is petal limb and one is petal weight so there are four features and that's the reason I have got four nodes in my input layer so what is the first point to be noted so the first point to be noted is the input layer has got nodes I did not say neurons please listen to me very very carefully okay I am going to be technical now okay this is one very thorough mistake most people do if I simply ask you how many layers are there in this network most people would say two which is Thoroughly wrong it's a single layer neural network it's a single layer neural network so you do not ever count the input layer as a new network area for the reason that it does not have neurons the input layer has nodes that is the first point the input layer has nodes second question what is the job of the nodes the job of the node is to capture the inputs from your data set third question how do I know how many nodes will I need answer is simple the number of nodes in the input layer is exactly equal to the number of features in your data set is that very clear step by step is everyone following me and this is how I will be going through the session like it's going to be a straight away question answer sessions you should be able to answer these questions if someone is saying uh asking you okay so first and foremost once again I would repeat the input layer is never counted as a layer while counting the total number of layers in the neural network then I said input layer does not have neurons it only has nodes then I said how many nodes would be needed in your input layer that is exactly equal to the number of features in your data set so you need to have a data set you need to have a clear understanding of your data set of course how many rows and columns the number of columns is the number of features typically and the number of features will determine the number of nodes in your input layer simple the job of the nodes is to collect the values of the features perfect perfect and now I've got an output layer okay so that output layer if you see has got one neuron that output layer has got one neuron right so what is the job of the neuron here okay so the job of this neuron is to process the input so all the inputs are coming from here okay and all the inputs are coming and this guy processes all the inputs and gives an output how is that processing done that I'll tell you in the next slide okay is that very simple till now now I will show you another neural network okay now it has got one additional layer so how many layers are there in this neural network come on how many neural networks are there in this neural network two exactly don't don't make that mistake it's two layer Network perfect awesome awesome so input layer is never counted because this green things are input nodes I've got five neurons in my hidden layer so what is the job of the neuron in the output layer question answers so unless you are clear okay so what is the job of the neuron in India that job of the neurons to give it a final output the output but someone has to learn the pattern in your data right just imagine you are feeding those features what are those features simple and separate better length petal weight what is the job layer the job is to identify what is the species this Iris flower belongs to correct what is the species and there are three species and vesicle right someone has to learn the pattern the output layer does not learn the pattern output layer will only tell you whether it is what is that whatever that one of those three right the job of the output Lane neuron is to give it a final output it will tell you the final class but someone has to learn the pattern in the data set are you getting my point right so who learns that pattern the neurons of the Hidden layer that is the job of the neurons of the Hidden layer that is the reason you need the hidden layer now the next question is how do I decide how many neurons to keep in a hidden layer now this is pretty much a hyper parameter so people who are coming who have come from the data science background know that word what is the hyper parameter anyone what is the hyper parameter so hyper parameter is something which you decide as a data scientist or you know as a subject matter expert and it is pretty much dependent on your understanding of the problem and your experience so someone will take five someone will take 50 someone will take 500 and there is no correct value to it that is the meaning of a hyper parameter so hyper parameter will affect the learning process and the overall performance always right so those are hyper parameters okay for example in decision trees the hyper parameter can be set as the max depth okay the total number of leaf nodes the total number of splits the number of data points per split the number of data points per leaf the total number of features to use in a decision tree so there are so many hyper parameters if you're talking of a random Forest which is The Ensemble algorithm then the first hyper parameter is how many decision trees you want to collect and make it as a random Forest so by default that number of estimators itself is hyper parameter so I'm just giving analogy from machine learning okay the meaning of hyper parameter so here the number of neurons in the hidden layer is a hyper parameter you get to decide you have to experiment frankly is that clear now yes no quickly chat box don't use the Q a unless there is a question there okay now is it necessary that I have to have only one hidden layer is the next question answer is clearly no the number of hidden layers itself is another hyper parameter so you are the architecture you are the architect sorry you are the architect here and you are designing your own neural network for a specific application the figure that you see on this slide is called as neural network architecture you are the architect so I get to decide how many neurons to use in a hidden layer and how many hidden layers I can have hundreds of hidden layers and each hidden layer having hundreds of neurons so I will decide as per my application it is as simple as that one very very important question how many neurons to have in the output layer who decides that or what decides that if you have a pen and paper please note it down for a regression problem you will always have a single neuron in the output layer it is as simple as that no questions what is the regression problem you need one output for each sample no right house pressing data set just imagine what are the features number of rooms uh you remember Boston host passing data set okay crime rate proximity to highways tax paid percentage of blacks Charles River dummy variable and number of the 13 variables in that Boston House pressing data set and finally you have just one value of that house price just one value so why do you need hundreds of neurons in the output layer right so you just need one neuron in the output layer for a regression problem end of the story no discussions needed it is as simple as that clear next is classification problem again if you have a pen and paper immediately noted down the number of neurons in the output layer is always equal to the number of classes so if you have a binary classification you can have two neurons well actually has one neuron also but we do not prefer we always prefer to have two neurons for a binary classification and if it is a multi-class so for example for the iris data set we had three classes we had three classes correct so how many neutrons will be there in the output layer come on the chat box three perfect perfect so you guys are now clear how to design a neural network yes a basic architecture is clear now the next question is how do I train this network next question is how do I train this network okay now I'm showing you a fairly complicated uh diagram okay uh maybe I'll just take it in this presentation mode because things will appear one by one okay uh okay sorry this is how it goes okay so what are 5.1 3.5 if you guys uh remember I don't know I don't expect I don't expect okay but 5.1 is the value of the simple length of the first record in the Irish data set 3.5 is actually the value of the Supple width 1.4 is the petal length and 0.2 is the petal rate okay so basically these are your values coming from the data set okay values coming from the data set it is as simple as that time so you just take the first record maybe I can show you a simple example here so don't worry I'll share all the code with you okay I'll directly show you one data set okay this is your crime rate and so this is a Boston House pressing data set right this is a Boston house pricing data set so there are 13 features so if you start counting from here till here else tag there are 13 features and medv that is your target variable that you have to learn okay so 13 features so now what I'm saying is the values of the first record will be captured by those input layer nodes people so I would have 13 nodes in my input layer in the first pass in the first iteration you can say the first node will capture this value the second node will capture this value the third node will capture this value and so on and so forth simple yes or no come on is that making sense is that connect there perfect okay so let's go back then fine so for Simplicity I have just taken a single hidden layer for Simplicity I've taken a single hidden layer okay and I've labeled them fine so now as you see each neuron now listen to me very carefully if you have a pen and paper just note it down okay so each neuron in any layer is fully connected to all the neurons or nodes in the previous layer and all the neurons in the subsequent layer that is how I have connected Can you see each neuron here sorry each node here is connected to all the neurons of the Hidden layer and then again all the neurons of the output layer are connected to all the three neurons of the Hidden layer so such a network is called as such a network is called as fully connected Network fcm this is also called as dense Network d e n s e dense Network or a fully connected Network FC in what is fully connected fully connected means each neuron or node in any layer is fully connected to all the neurons of the previous layer and all the neurons of the subsequent layer it is that simple listen to me that word dense I'm just typing it down in the chat box because that's the exact keyword you have to remember that keyword I'll be using in my code perfect perfect okay now how does the input go okay now let me so X1 X2 X3 X4 so imagine that these are the four features and one two three are the numbering of my neurons in the first hidden layer and y1 and Y two are the two outputs simple because there are two neurons in the output layer so obviously those two neurons will give two outputs y one and Y two whatever till here it is fun next is what are these W's now W's are the weights corresponding to each of these connections please understand there is some technical stuff I am taking time because there is nothing else in the code otherwise okay so this is the only thing that people struggle to understand code is all charge GPT okay you just give a prompt give me a simple neural network which can solve the iris data set problem in just 15 seconds you will get that entire code I have just copy pasted that code in my Jupiter notebook I'll tell you very frankly so there is no fancy stuff in coding henceforth okay what is more important is to understand what is really going behind the scenes with that knowledge is what would be evaluated which will help you perfectly and will help you you know okay let me repeat that one second what are these W's W's are called as weights what is big weight is basically attached to each of the connections so how many Corrections are there from first layer to the second layer in the first layer you have four nodes the second layer you have or basically the hidden layer you have three neurons so four into three simple mathematics will tell you there are 12 connections however how have I named them w11 means its weight connecting from the first node to the first neuron W12 means weight connecting the first node to the second neuron w three one means the weight connecting the third node to the first neuron is that clear that's how so it will be a matrix this weight will be a matrix remember that that weight will be a matrix yes no it will be a matrix okay now each neuron also has so connections have weights please understand these weights are not attached to the neurons the yes once again I repeat this okay these weights are not attached to the neurons weights are attached to the connections these bias terms are attached to the neurons so you have weights and you have biases so if you have three neurons you'll have three bias terms it is as simple as that okay I have got a very good question shubham so is asking who decides the initial weights random initialization that's what you do in gradient you say no it's random initialized correct always so you can initialize it with zeros you can randomly initialize with random numbers which follow a uniform distribution or random initialization which follow a normal distribution and now we have some sophisticated algorithms uh which is glow Rod normal Lee Kun so some of these uh kernel initializes are there I'll talk about them in my code it is there in my code so just wait till I Venture into the into the code there I'll specifically tell you how to specify these weights initially by default if you do not specify they will be randomly initialized is that clear everyone yes no not only the weights but also the biases will be randomly initialized clear so how many biases you will get three biases for the hidden layer there is no weight and bias for the input layer why why there is no weight and bias for the input layer because the input layer does not have neurons perfect now the next layer that is basically from the hidden layer to your output layer those connections will also have weights so how do I use these weights I'll tell you that okay just wait okay so those will also have uh you know some weights here V1 v11 is basically the weight connecting the neuron one to the output layer neuron one v21 is basically the weight connecting the second layer second neuron of the Hidden layer so the first neuron of the output layer and so on basically right and then we have the BIOS terms for the neurons of the output layer correct so again you will get a matrix so here it is bias terms are three cross one and uh 2 cross 1 and you have the weight Matrix like 4 cross 3 and 3 cross 2 so one you can take a snapshot of this because this is very very important and that will help you in the further understanding okay why is the bias required so basically what happens is when you train the neural network these randomly initialized weights and bias they will get updated via an algorithm called as back propagation okay but before I Venture into the back propagation we'll have to understand how does a forward pass look like okay how does the forward pass work okay and then uh so basically okay yeah interesting okay so this is again summarizing the same thing in terms of matrices this time in terms of matrices so w Matrix represents all the connections from your input layer to the first hidden layer and V Matrix represents all the connections from the hidden layer to the output layer correct and then you have the predictions fine now you can see those uh matrices on your left side okay now what is the net input so can you read this net input to the first neuron of the first hidden layer so I am talking about this okay let me highlight that okay I'm talking about this guy so this guy gets a weighted sum of imports so these weights are multiplied to the actual numbers so what were the actual numbers five point three three point one one point four two point something right so these were the values of x 1 x 2 x 3 and Export so these were coming from your data set these were coming from the data set so these numbers are now multiplied by their corresponding weights so what you have is a weighted sum of the inputs not directly the inputs so the first operation that you have to understand is it is a weighted sum of the inputs and you are adding the bias term as well so the bias term will take care of the I would not say non-linearity but basically any gain or shift is there in the overall after the overall calculation that's the bias thing okay so this is your net input to the first neuron of the output sorry header layer okay now next is this clear this okay now this can be summarized in a simple Matrix operation like this W transpose X plus b okay I will not get into the total mathematics but I can help you this in terms of code as well okay uh okay so whatever is the net why is this important to understand let me just slightly highlight that why is this important okay so imagine that you have the method that you have 100 neurons in the hidden layer and then you have um again 50 60 neurons in your output layer then if you try to do this scalar operation okay this is a scalar operation if you try to do this scalar operation W1 X1 plus W two x two plus W two X Plus and so on till w 100 x 100 that will be pretty tedious and this is for one neuron then you repeat the same set of operations for the next neuron and then you repeat the same calculations for the third neuron for the fourth neuron for the fifth neuron that you keep repeating for all the 100 neurons that manual calculation is extremely tedious and not at all scalable what is the scalable solution this Matrix operation is a scalable solution right so when you do this you get a net uh so all these calculations happen in just one Matrix operation see this is the power of linear algebra that's the reason people say you should be really good in linear algebra right so this is the Matrix operation and you can verify why this operation works okay you can go through the recording later and why this works I've already mentioned in terms of dimensions so now the net output of the Hidden layer okay so this should have been off not two this is the net input okay and the net output so this is the hidden layer which is getting this as the input fine what is the output of that the output of that is this calculation the output of that is calculation so this calculation will be the output this output so what is that calculation if you see I have used one function called as relu here can you see here relu function linear unit so basically what I am doing is I am transforming my net input what is my net input w x transpose plus b that is my net input so I am manipulating that input by using some function and that function is called as activation function why do you need to use that activation function because your neural networks will otherwise will not be able to learn the non-linearities in your data set so all these activation functions are non-linear functions I have a list of all these I can show that quickly before we get back to this slide once again so um there so here you can see there are some other activation functions are there you can see this is a relu activation function and then you have sigmoid activation function which goes from 0 to 1 you have soft Max activation function and then you have something like tanh as the activation function which goes from -1 to plus one and these are non-linear operations so basically here the x is your total weighted sum of the inputs X is the total weighted sum of the inputs okay so you will apply a non-linear activation function so that your neural network is able to learn the knowledge if you do not apply a non-linear activation function then your entire neural network will be will be as simple as a Premier regression model it is as simple as a linear regression model right because Matrix operation is like what multiplying and adding multiplying and adding and that's what you do in your linear regression also like what is the linear regression model W and X1 plus W 2 x 2 plus 0 3 x 3 and so on right that's a linear regression so if you do not apply a non-linear function on that the net neural network will always remain linear so you have to use a activation function so that your hidden layers are able to learn the non-linearities the non-linear patterns in your data set so this is the simplest way I can put because I really do not want to get into all the calculations and stuff but this is the simplest way of explaining okay and then that output goes as input to the output layer once again I repeat this output will go as this output will go as sorry so this output will go as input okay that output will go as input to the output layer input to the output layer so this is the net input okay and that input is again transformed by another activation function did you notice I have changed the activation form here so that brings to the question how do I decide which activation function to use in which scenario I'll give you a clear answer to that okay but again you have to have your pen and paper ready okay because there is no slide noting that down okay so I tell you very clearly in black and white which activation function to use for which kind of problem and for which layer so it is dependent on the problem that you are solving number one and it is further dependent on the layer that you are talking about so I'll give you a very clear answer to that right now I have just used a randomly relevant softmax just for the sake of demonstration okay fun so this final output that you get here for the last statement before we go to the code okay one last statement before we go to the code this last output that I get is what we call as a forward path is one forward pass you speed the data from here you see things from here it goes to the hidden layer and then it goes to the output layer and finally you get some predicted value that's what your predicted value is right so the moment you get a predicted value can you see your data is Flowing linearly this is why we call it as a forward pass I will now show you how to do one forward pass using simple numpy ready I'll uh exceed the session and this is typically the case with me okay so maybe another 15-20 minutes or maybe even 8 30 okay I hope you guys are fine with that okay uh okay so I am clearing all my annotations here and directly jumping to the code part because I don't want to get into the too much of the mathematics that will not be very helpful so now I am going to tell you okay the activation functions okay I'm going to tell you all the activation functions so if you go to keras.io can ask for it tensorflow and you can use tensorflow you can use piano you can use pytorch sorry you can use Cafe you can use fast API so there are five most commonly used Frameworks to create neural report once again I repeat there are five most commonly used Frameworks to create neural networks okay one is tensorflow one is Tiano one is pie Dodge one is Cafe and one is fasted green okay so these are the five most commonly used Frameworks you need a framework to create a neural network I did not mention the name Keras is basically a wrapper at the back end it can connect with tensorflow or it can connect with piano either of them okay so it's just a wrapper what does the wrapper mean so one line of Keras is basically 15 lines of tensorflow so I'm not going to teach you tensorflow but remember that 15 lines of tensorflow can be wrapped up into a single line of Keras code so that Keras makes creating the neural networks training the neural network and visualizing the output Ultra simple okay I'll straight away head to the API Docs uh yeah uh that was just okay so if you go on the left side here you can see activation functions okay not there so I'll just search here okay I'll just search activation and you'll get all the list of the activation functions which are available okay now you don't have to use all of them I'll give you a simple example or a simple you know yeah so activation layers no no not this there is a web page which gives you all the list of the yeah these are the most commonly used activation list but no not this I'm looking for the page which has the list of all the activation functions so we have it here which is not optimizes license it is there maybe I'm just in the same okay so this has a lot of activation for example yes these are the available activations correct so relu is one of them and then you have sigmoid you have soft Max next one and you have soft plus and so on so all the equations see there are a number of the equations are also given for each of them the equations are also given if you see this documentation page the equations are also given for each of them so I have used these equations and I just wanted to create a visualization how do they look like okay so the first thing is I'm I just want to visualize so I have a linear activation I have reload so linear means what goes in comes out as it is no processing them that's a linear activation once again what goes in comes out as it is see no processing them rectified linear unit where the output will be 0 if the input is negative once again I repeat the output would be 0 if the input is negative what is the input input will be your weighted sum of uh inputs right W 1 x 1 plus W 2 x 2 plus and so on that is the net input on which I am running my activation function correct and it is X basically if it is positive so if the value of x is 10 the output is thin if the value of x is minus 2 then the output will be 0. right sigmoid this is the equation and for Tan H this is the equation okay so fine I have created some values of X for x axis and these are remaining you know code for the plotting so I'm not going to discuss all that so just have a look at the linear Activation so what goes in comes out it is as simple as that relu for all the negative values can you see for all the negative values the output is zero and if the value of input is positive the output is exactly the same so for 2 it is 2 for 3 it is 3 for 4 it is 4 and so on coming to the sigmoid whatever be the input the output is always capped cap between 0 to 1. so if the x value is minus infinity then the output would be exactly 0 it is approaching 0 but it will become 0 only at minus infinity similarly at plus infinity this will approach to become one so all the intermediate inputs the output will be tapped at zero to one right in the case of tan H the plot is exactly same only that this is not from 0 to 1 it is from minus one to plus one that's the only difference okay so that's your tan X so these are the different activation functions then the next thing is I would need a loss function so what is the loss function okay so this is how you would visualize the loss function okay so what is the loss function uh I would strongly suggest everyone to go through this article I will give you the link of this article if you go to Analytics Vidya blog okay um I'll give you one article you should read that okay so I'm just going to that blur and I would want you guys to read one article so there should be a search button somewhere search button okay yeah it is there I'm searching for optimization so what is optimization what are optimizes and what are loss functions all this is very clearly mentioned in this article which I have written and I have already taken one data r on this optimization and the role of optimization in machine learning and deep learning so we do not have enough scope to you know discuss about all the optimization here so that's the reason I'm giving the link of this article so what are optimizers what are loss functions which loss functions are available which loss functions are supposed to be used to solve which kind of problems every single thing is discussed in this article I'm just giving you the link of this article in the chat box this will help you so please note this down okay so here uh you will see a table so these are the different algorithms and these are the loss functions which is so what is the loss function I'll just tell you in one line what is the last concept ion loss function is a function which you are trying to minimize while finding the optimal values of the weights and bias once again I repeat loss function is a function which you want to minimize while finding the optimal values of the weights and the bias why do you need to find out the optimal values of the weights and bias because random values of the weights and bias will give you random outputs please understand that a neural network where the weights and the bias terms okay these are W's and V's if they are random numbers the output will also be a random number so you need these bits and buyers to be learned properly so during the training process okay we are trying to find out the optimal values please listen to me carefully we are trying to find out the optimal values the right values of these weights and bias okay and how do we do that we do that by minimizing the loss function that exact statement you will find in this article okay now which loss function is used in which machine learning algorithm this is the table that is a there is another table so loss functions are used everywhere for regression algorithms for classification algorithms and you will find one statement here please read this in deep learning you can create your own neural networks and choose your own loss functions that is the beauty you can choose your own last function as per the problem and all the available loss functions are here so I'll just click on that link and you can see all the loss functions available here right so these are all the possible loss functions but you don't have to know all of them you don't have to know all of them because I'll give you a clear idea which loss function to use in what kind of scenario so I've already committed I'll tell you which activation function to use when and now I am also going to help you which loss function to use them so two things that I'm going to help you okay okay so but please help yourself in terms of the optimization why we are doing and all that okay so everything is mentioned here very clearly what is an optimization problem what is now the third thing is Optimizer there is a third word called as Optimizer what is that optimizer now the optimizer is the actual algorithm which is trying to find out the optimal values so here there is the section called as how do we solve this optimization problem or there is something called as Optimizer okay so that statement will also be there somewhere right so I'm just clicking on this link so these are the available optimizers in detail it's very clearly mentioned here okay there are some advanced optimizers in deep learning and what is this Optimizer Optimizer is the algorithm which will find the optimal values of the weights and bias by minimizing the loss function that's the complete statement once again I repeat Optimizer is the actual algorithm which will find the optimal values of the weights and bias during your training process by minimizing the loss function right so what all optimizers are there now you have all the list okay um you can see here Adam I think yeah actually these are the available optimizers you have SGD that is a stochastic gradient descent RMS prop atom is adaptive momentum and then Max native is your nestro accelerated adaptive momentum okay and so on so there is this I'll not get into all the technical details but you can experiment with these optimizers so which Optimizer to use for which kind of problem now that's pretty much a trial and error there is no single answer to that there is no signal you will have to experiment okay which Optimizer to use for which kind of problem but there is a clear answer to choice of uh loss function and activation function right that is what I am going to help you with next okay so coming back to my code um so this is how you have your mean square error loss function normally we are going to use the mean square error loss function for a regression problem and for a if you want I can just note that down quickly for you and I have purposefully not put that in the slide so that I want you guys to note it okay so how do I decide which loss function to use and which Activation so we'll start with the activation functions okay so activation function so normally normally for hidden layers go for relu don't waste time in experimenting with a number of things that is what people have already done in the last 90 years don't waste your time unless and until you have a very strong reason not to go for relu go for relu it is as simple as that end of the story no discussion needed no questions to be asked people have figured this out the hard way this was the one of the your biggest breakthrough 2015 by Jeffrey hinton's group when they came up with this railroad activation function so all the hidden function hidden layers and all the very much all the sophisticated hidden layers of every single neural network you'll find really by default end of the story now coming back to the output layer now output layer will depend if it is a regression problem go for uh you can go for radio also or linear now why do you want to go for relu suppose your predicted value is a stock price now you know that the stock price cannot go negative so value will never give you a negative output but linear can give you a negative output so you have to be clear on that okay so if you can accept negative outputs then go for linear and if you cannot accept negative outputs go for Value end of the store if you have a classification problem see it is that simple okay it is that simple you just have to remember for classification problem okay go for soft matters directly don't waste time in sigmoid design directly go for software then then now coming to [Music] um so I've already mentioned the number of nodes in a hidden layer sorry number of number of nodes in the input layer will be equal to the number of features that that's very clear similarly number of number of neurons because I am going to create a network based on these inputs okay neurons in the output layer player is equal to is equal to 1 for regression problem straight away I have already mentioned and that will be again equal to the number of equal to the number of classes application it is as simple as that number of neurons in the hidden layer is a hyper parameter so you can take anything as you want it's fine now uh activation function is the loss function so loss function for regression problem directly go for MSE mean square error loss function or mean absolute error loss function or there is mean Square logarithmic error also you can experiment with all these but this is the most common this is the most common is the most common but there are others also so go with MSC by default until an unless you have a strong reason I'll share this text file don't worry mine okay I'll just copy and paste it in the chat box at the end for a classification problem so for a classification problem okay the loss function can be taken as a binary cross entropy uh but I would normally so if you are taking binary cross entropy BCE okay then binary cross entropy then you must make sure make sure that you have only one neuron in the output layer and and so you have to make sure two things if if you are taking binary cross entropy okay for a binary classification for binary certification okay so there are things that you have to make sure so