Implementing a Neural Network using Pytorch | DataHour by Shanup Peer

Key Takeaways

okay hello everyone so first of all let me also welcome everyone to today's data session on implementing a neural network using pytorch so the the meeting come here has already given a brief feature about me so I'll just quickly uh get it out of the way so I'm shaluk here I'm a principal data scientist at open loop based out of Dubai in India I'm primarily involved in building custom analytics solutions for various sectors and verticals of business again the primary focus being in Predictive Analytics and time series forecasting um I'm also very actively involved in the AI at Tech space as a data analytics Mentor with springboard um very closely associated with the Deep learning.ai group in multiple roles for testing of new courses and also as a mentor for the new courses and specializations and I'm also an Adjunct professor of data science at contact based out of Taylor Auto California so that's about me um so without further Ado let's get into the um business side of things so I'll start it out with an intro to neural network um so for today's session um I have given the prerequisites um one being an overall understanding of the neural network architecture and its elements and two having a working knowledge or familiarity with python so that we don't need to spend more time on it so the way I proceed is first we look at the neural network anyways because it is very relevant not very deep but just enough to set the stage so we look at the architecture and the workflow for the neural network after that we'll open up about Pi torch and discuss in brief about tenses which is one of the most important things Concepts one of the most important concept when we get into the deep learning framework and I in a moment I give you an idea as to why that is so relevant and important and finally we'll go Hands-On with the implementation details in The jupyter Notebook okay so let's get started this is a general architecture for a neural network it's a typical one it has the input layer the output layer and any number of hidden layers with again with any number of neurons in these hidden layers the choice of the number of layers as well as the number of neurons in the hidden layers has to be arrived at through experimentation while the input layer and the output layer is not in our hands because it is determined by the number of features in the input and the number of classes or labels that we need to provide for the output so that is something that is defined by the problem but the hidden layers that's in our apps how many hidden layers should we have and how many neurons should be there in each of those hidden layers the important thing being to decide on just enough number of layers and neurons so it's not like you can just go decide I have 100 hidden layers and maybe 1000 neurons in each layer no the idea is that we need to have just enough number of layers and neurons that can serve our purpose which at the end of the day is to be able to predict reliably in technical terms we call it accuracy of the model when it comes to real life problems time and money to train the model also becomes a decision Factor so we cannot go overboard on the number of layers and neurons uh thinking that the sky is the limit coming to the hidden layers we have the fully connected layers we call them fully connected layers because if you look at it this is the first layer or the input layer every every input from the previous layer is connected to each of the neurons so all four inputs go to this neuron all four inputs go to this neuron all four inputs go to this zero likewise the output of this neuron this neuron and this neuron all three of those go to the next layer neurons as well so that's why we call it a fully connected layer okay of course the waves and the bias of each neuron will be different from the remaining neurons otherwise there would be redundancy so each of these neurons fits in as individual pieces of a larger person each of the neurons will learn a small part of the final output helping the neural network to finally piece them all together to arrive at the final output the key thing about the hidden layers that differentiates a neural network from a linear regression model let's say is the activation function which we have represented here with f okay so let's uh take a quick closer look at each neuron so this is if I zoom into each neuron this is what I see so from the previous layer we'll have all of the inputs coming into this neuron and then there is an activation function so essentially what happens is the this is the equation that gets calculated at each neuron W X plus b where X is the input any number of inputs that many number of waves will be there and there will be a single bias for each neuron but after all of that there is a function that completely wraps this the output of this this equation the WX plus b and this is where the power of the neural network comes in okay and this particular activation function again for a neural network to become powerful enough to be called as a universal function approximator this function has to be a non-linear activation function if it is a linear activation function this entire neural network will get reduced to a linear regression unit a single linear regression unit so this is why it's essential for this function to remain as a non-linear activation function until let's say a few years ago the sigmoid activation function was a very popular choice but we have come to understand that excuse me that it has its limitations Vanishing gradients Etc today the more popular activation function is the value and its radians like the Leaky Revenue so this is this is the fundamental building block of the neural network now let's look at the neural network workflow so essentially we start out with the input data so whatever data prep that needs to be done we do it and then we provide the data for the first and the most important phase which is called as the training phase so when it comes to the training phase I'm assuming that's why I mentioned this as a prerequisite I'm hoping that everybody is familiar with uh these step-by-step activities that happen so we have uh epochs or uh the number of training Cycles where the entire data gets seen by the modeling ones so we we call them as the epochs then we Define a loss function it can be uh it's up to us to Define what the loss function is we have um the root mean Square the cross-entropy depending on whether it's a regression versus a classification problem so any number of last functions we can decide and then there is a learning or an optimization algorithm the most basic one and the most important one being the stochastic gradient um again within that so the whole process is about gradient descent but within that we have many variants of the optimization algorithm a very famous one being the um Adam optimization algorithm um you know one of the limitations that a stochastic gradient descent has in the general sense is that we looked at all of these neurons that did in layers and all of the each of these neurons has weights and biases and at the end of the day what we're trying to achieve is we are trying to find out the um the final value of the weights and the biases such that our model is able to predict accurately right so it is through this optimization algorithm in this case the gradient descent that this weights these weights and the biases get updated in each epoch okay so if you look at the stochastic plate in the center algorithm um the way it's done is all of the parameters from you know all of the neurons in all of the Hidden layers they're all packaged with a single learning rate learning rate is the speed at which uh the the updates happen to the weights and the biases in each iteration right when it comes to Adam though um it is structured in such a way that each parameter each weight and bias can have different learning rates also as the learning progresses um it keeps watch of what's happening to each parameter update and looking at how quickly or how slowly the update is happening it's able to modify the learning rate for each of these parameters this so it's quite dynamic in that sense which is why it has now become one of the most popular optimization algorithms okay now um back propagation I hope everybody's uh has heard about this term because this is that single thing that uh of course the activation function as I mentioned earlier the non-linear activation function is what gives it the power but the the overall algorithm is pieced together by something called as back propagation okay so to understand back propagation we need to also understand that there was a loss function right so we look at the predicting power of the neural network by giving it an input and looking at what is predicted by the neural network and we also have the equivalent actual value that we are expecting so the last function as I said the mean square error or the cross entropy or whatever it is is a measure of the difference between the actual and the predicted values okay now this is only defined at the output layer because we only know the target value we have the output layer in none of the other layers we we have no clue as to what the target is so everything has to go through a forward pass and it has to come to the output layer where we have a Target value waiting for us and now we predict uh based on the whatever architecture that we have put up and whatever dates and biases that we had in each of the layers of the neurons we are able to make a prediction and then we make a comparison of it right the back propagation is we take the class and we take it all the way back right so um we have so I mentioned we have n number of layers with K number of neurons in each layer which varies from layer to layer right so instead of just modifying the final weights the the the the key to the neural network architecture is that every single weight and bias are right from the output layer all the way to the first layer is modified and this is done using the concept called as part propagation so we back propagate the loss from the output all the way till the first layer and then we update the weights and biases right across right from the first layer all the way to the output layer and this is what separates the neural network algorithm from many of the other machine learning environments so this happens this keeps going on and Epoch after Epoch learning cycle after learning cycle and finally we arrive at the final weight and bias for each neuron of each layer so this is where we say that the training is completed we have certain criteria that we set we say that okay we can call it um as the final set of weight and biases if the the loss is this much or if the loss between one Epoch to the next Epoch is very minimally changing so there are different criterias that we can set to decide when is it that we should stop this training process but once we do that we have the final sets of weights and biases for neuron for each layer and then we say that we have a model this model is now ready for prediction or testing or evaluation different terminologies that we use and again in the evaluation phase we have several evaluation metrics that we can use so with that I've set the stage for what a neural network is now it's time to look at pie torch what is my touch it's it's a very popular research deep learning framework and I'll tell you why I've put the research in inverted commas originally it was designed by Facebook meta but now it's open source so now the prominence that we give for by torch okay to understand that let me open something so yeah so this is a this is a very uh popular um website papers with code um it's a website that tracks excuse me the latest and the most famous machine learning papers okay and let's take a look at what it indicates about the popularity of various Frameworks okay so let's look at December 2018 we see that um so this is a nice it gives you a good uh statistic right so when you look at December 2018 uh and these are research papers okay so we haven't come to the industry side of things this is research where papers are getting published so we look at December 2018 we see that tensorflow had about 35 percentage of the papers being published okay so it's a research paper is being published but the paper was based on a tensorflow framework and the pytorch had 31 percentage and these are some of the other Frameworks that were available now let's move fast forward all the way to December 2022 and you're going to see something very interesting happening okay from 31 percentage of tensorflow papers in 2018 today it is four percentage and from 35 percentage uh was it 35 uh sorry from 31 percentage by torch papers in 2018 today it is 63 percentage so I will let you be the judge of where this is going and in terms of popularity okay so it is high time that we we can no longer ignore uh pie torch um it is uh 63 percentage is not a small thing uh that's a huge domination but research papers I I wouldn't um uh take the liberty of talking uh how things stand in the industry because that the jury is still out on that one okay but very clearly the writing is on the wall the um there is a huge popularity of pie torch in academy and research okay and most importantly the comparison is mind-boggling look at where it stands 63 versus 4 percentage of tensorflow so our time is well invested if we decide to adopt pythons for our deep learning development and research activities okay now going back slide so uh there's no question about why we should be knowing by touch it's uh the statistics clearly show it and obviously the a lot of the research papers and research in machine learning and deep learning especially happening on fighter on on pytorch framework means that many of the deep learning models pre-built right is already made available for you in pie torch framework so you can go to the torch Hub or the torch Mission dot models and many of the latest and the best and the greatest models are all sitting there for you and you can just invoke them um you know we are all aware of what transfer learning can do for us right so we are all able to use the most latest models out there within our development code so um so many good things if we're going to be sitting with pie torch okay moving on so now when we talk about Frameworks especially deep learning Frameworks we have all heard about gpus and the power of the GPU and you know we need gpus to run the Deep learning Fame Frameworks and all of that so what is the tensor it's a data structure that is used throughout deep learning Frameworks what are they very simply put they're just numbers or arrays um it's basically a data structure so even a number is can be a tensor an array can be a tensor so a scalar is a tensor a vector as an uh in an array a one-dimensional array that is also a tensor a matrix as in a multi-dimensional array that is also a tensor so in that sense it is no different but the data structure overall is different from what we have seen let's say in numpy um right but to the naked eye it all seems the same but then what is that big difference and why do we hear about tensors and always we need to operate with tensors when we come to the Deep learning Frameworks so the answer to that is as great as it is and as prevalent as it is cannot utilize the power of the GPU to accelerate the numeric computations obviously it goes without saying I mean it's it's uh it's a very open fact that gpus can give us 50x maybe 100x faster speed than your normal CPUs right depending on the state of the art if you go to some of those GPU architectures and chips right so those gpus they need the tensor uh tensors are able to utilize the GPU um again in tensor that is that is an option called as two device so we can we can specify uh depending on the machine that you're using for the Run if it is a CPU machine use a CPU but if it's a GPU machine definitely use the GPU so it allows us to look to check the device where it's running to find out if it's a CPU or a GPU if it has a GPU capability then immediately switch on to the GPU mode and then the power uh our acceleration is just in minutes so um there's no question about it numpy has its limitations when we're looking for that accelerated power and tensor gives it to us which is why you're going to see whether it's tensorflow or whether it's by torch you're going to see that whatever data that we have before we start using it we convert it to tensors and there are very simple functions and methods that are available whatever format the data is you can easily convert it to tensors so now for the real thing we need to create a neural network in pythons how do we do it there are two methods or flows that are available in in pytarch okay one is the sequential one and the other is the functional one the sequential one which we'll see in a moment is it's more or less simple pythonic syntax which people from us people from like us who are not from the pure software fraternity can easily understand but also there is another flow which is the functional flow on the functional architecture right this is where you're going to see quite a bit of oop constructs the object-oriented programming okay and one thing that I have noticed and excuse me heard about regarding pie torch is its Affinity to Pure software constructs such as book Concepts not very palatable for the non-software background people within the data science community and hence it deters even some of us from getting our hands dirty with it um so one of the things that I would like to do today is dispel some of those apprehensions it's not as bad as we think it is it's quite standard you don't need to know too much about local cons oops constructs just you need to know what you need to place where and the rest is pretty much run-of-the-mill stuff okay and with that uh stage being set I think without any further delay let's get into the Jupiter notebook so firstly we'll start with a regression problem okay so um the pytorch workflow not very different from the typical neural network workflow so we need to get the data ready so there we have a small difference we need to in in addition to whatever we do to get the data ready we will need to turn it into tensors That's essential then after that we pick a pre-flame model or build a new one um so we've already discussed this is just the normal neural network workflow we need to pick a loss function and we optimize it if you're going to build it from scratch if you're going to use a pre-plane model that's up to us but if it's a new one or even if you're training fine tuning a pre-trained model you need to build a training Loop obviously we need to fit the model to the data and finally make a prediction on the training data itself we look at how things are going we are able to calculate the loss so we are able to evaluate the model in that sense by looking at the loss and then we improve it whatever we need to change so the hyper parameter tuning happens uh you know which which starts with changing the architecture itself the number of neurons in each layer the number of layers the number of epochs the learning rate all of those things these we will need to do we will train it on the training data but we will check all of this on the validation Data before we finally look at the performance on the testing data okay so we start out with your simple libraries that we need um so we import torch start nnsnn is going to be that so NM is the neural network one so this is going to be one of the key libraries that we will be invoking so as I said uh you take any number you can convert it to a tensor so if it's a number just say torch dot tensor and a simple number gets converted to a tensor whether it's an array that gets converted all you have to use is the same command the torch dot tensor and pass the array as the input if it's a multi-dimensional one a matrix same thing no difference so it's that simple to convert a non-tensor to a tensor so we will use the California Housing data set so we're going to pull in the data so first of all we will do the train and the test splitting of the data but after we do that here you can see we have called the tars.tensor so each of these data we we the data that we have created the train data the test data the the train labels the test labels everything gets converted to a tensor just a sanity check what's the shape of the input we have 20 640 samples and each sample comprises of eight features and we have equally the same number of labels that we would like to predict so this is how the input data looks like just said that your normal sanity checks this is what the output lab Target looks like obviously we're doing a regression model so this is going to be a floating number which is the Target that we are trying to predict so we will start out with uh the simple thing the simple model the the simple architecture that we have so I mentioned earlier that you have sequential as well as functional and sequential is very much pythonic in its syntax while the so before we share the bad news let's see the good news the good news is very simple you have a fully connected layer right you start with eight inputs your first layer comprises of 24 neurons your second layer comprises of 12 neurons so the way we call this is NM dot linear is basically the fully connected layer okay so if you want to connect your inputs to the first set of hidden layers you have to the first hidden layer you are basically saying that I have eight inputs and I need 24 neurons in my first hidden layer okay and then specifically separately you will have to call the the activation function so don't think that if you call NN dot linear the the the the function is going to be automatically uh taking care of the activation function there are Provisions for it but uh if you want to be specific about it we call the value function after we call the linear function right the output of this is then passed on is it is then becomes the input the next hidden layer so basically what you're saying is there are 24 neurons uh in the previous layer now I want it to be connected to 12 neurons in the next hidden layer so this is my second unit layer okay further so again we have the activation function which is the value that we uh explicitly uh mentioned otherwise it will take the linear function right by default which is not what we want as I mentioned earlier if you don't have a non-linear activation function and you can have any number of layers and neurons but if you give it a linear activation function it all equates down to a single linear regression equation that entire Mighty neural network all boils down to a single linear regression equation model so which is why we need the specific non-linear activation function so now we go to the third hidden layer where from the 12 neurons in the previous hidden layer we need to connect it to six neurons in the uh in the new hidden layer and finally you're coming to the output layer where six neurons in the previous hidden layer will all connect to a single neuron which will give you the output which will give me the output or the overall neural network so um this is how we have defined the overall structure of my neural network now for a few important things so we need a loss function right so this model is now able to predict but before it can predict it needs to learn and if it needs to learn it needs a loss function so that it can find out how good is its learning so in this case being a a regression model we we will select the mean square error which which is the last function right and then I mentioned that we need an Optimizer the the basic Optimizer is a stochastic gradient descent but if you want a lot more power during the learning process Adam is an excellent choice and obviously you have a learning rate to begin with but Adam built-in has capability to keep changing this for different parameters for different neurons and at different stages of the learning also it is able to change this learning rate that is what makes Adam so powerful okay now we've looked at we discussed about the Learning Cycle right so we have we have to define the number of epochs that we want to run it and if you want to run it in batches we can do that so we create a batch of data so 10 samples in a batch right now what happens is here is where we need to understand some of the important constructs these are standard whatever model you create that those changes will happen here right but the training part the evaluation part it's quite standard it does not change right as you will see now I'll go from the regression model to the classification model you'll see that only the the overall structure has changed of the neural network but the Learning Cycle the testing cycle everything the constructs are same in fact it's a copy paste you don't need to change anything there right except the model name whatever it is so a few things that we need to understand okay um so if we go by step by step first we have a forward pass where you have your weights and biases and the model starts calculating based on the input it will go through each neuron the new each neuron will give its output then those neurons get pass on those outputs to the next hidden layer and to the next hidden layer and so on and so forth till it reaches the output at the output you have a prediction and that that is where we calculate the loss we calculate the loss saying that the model is predicted so and so but this is what we were expecting so we we are able to compare we predicted value with the training uh the data that we had right so from there what we do is now in by touch um a few things that are slightly different from other Frameworks which we need to take care of is um you know that when you do the gradient descent um the equation is you sum sum all the losses right so the problem with our more than a problem a feature with pie torch is that all of these losses get added up right but when we when we go to a new Epoch or a new learning cycle we want those losses to be reset and then again we will look at for each input because during any entire uh samples in a batch or the entire samples in the training data you seen once so the the total loss is calculated by looking at the loss of each sample right so it's getting added up now pie charts what it does is unless you explicitly say that you know uh flush out whatever you did in the previous cycle it will hold on to it and so whatever loss you add in the previous Epoch in the previous learning cycle it will keep that within the model and the next next cycle when you are again calculating the loss because now now what happened in the previous cycle is no longer relevant because you have actually modified the weights and biases in each Epoch right but pytots will hold on to it and and it will keep it there so we need to ask it to flush it out this is what this this method does the optimizer.0 grad is something that you will need to invoke in every work to tell by dots flush it out what happened in this Epoch we don't need it anymore finally we I mentioned back propagation which is where the loss from the output layer is propagated all the way back to the input so we need to explicitly say backward so that the back propagation can happen take the loss back propagate it and that is done by invoking this method the last start backward finally all the losses have been propagated backwards we have calculated the gradient in each of those neurons any with respect to each of the parameters of each of the neurons in each of the Hidden layers right now what do we do the final thing to do is for that paper we need to update the width and the bias of each neuron of each hidden layer depending on the gradient that was calculated and that is done when we call Optimizer dot step so these are the standard steps involved whatever model that you're using to run the training so you can you can just memorize this in fact whatever you can see in this notebook you can copy them take it wherever you want that's how it is okay so in a moment I will share with you how that happens so as you can see here we are starting the epoch okay now from there what we're doing is we we are creating for in this case we are taking it batch by batch So within ET block we would like to do this batch by batch that has its advantages also overall on the performance so we won't get into the details of it even if it's not a batch if you're going to do it uh you know the entire training set in one shot that's also fine so what we do is we call model we already created the model and we have given it a handle the handle is model big number at model one model two whatever right but we need to invoke that model correctly whatever name you have given it so we're going to say model and I'm going to give it the input data right based on which we have created a loss function also so the loss function we have created here we said is the mean square error loss so then we as per the step by step that I just discussed here we first invoke the model which will do the forward pass and bring the first output out then calculate the loss okay once the loss is calculated we we are going to say we're going to say reset okay reset whatever you have about the previous now if it's the first Loop that you're running the first Epoch this is not required but for all the epochs together we need this so as a standard uh practice we will first call zero Grant so whatever you have evaluated and kept if it is there in your memory flush it out you want to start with a clean State and this we need to we need you to do it in every Epoch once that is done we'll call the back propagation propagate it all the way back no once that is done this as I mentioned you have propagated everything back there's only one more thing to do because the gradients have all been propagated all the way back to all the neurons now we we say that I want you to update all the parameters and we do that by saying Optimizer dot step so these are the importance steps involved first the forward pass second calculate the loss for that forward pass reset whatever you have in your memory do the back propagation based on the back propagation whatever you have calculated on the gradient I want you to update all the parameters done this is your training loop it's over right now if you want to do some kind of evaluation when you say evaluation it's not on the training you are you're going to evaluate whatever the model has done um we we call something called as model dot the back now I will get into this in more detail the important thing that I would like to take away from this linear regression model is that you call these four steps and this is standard this is candidate as you will see in a moment I'm going to go into a totally different problem classification problem and you'll see that the exact same steps show up there as well okay so with that said this is as simple as this you create the model right in in as if this was the sequential architecture very simple we get to the bad news later um but after that you define the loss function you specify what Optimizer you want and then bank it's step by step just invoke it call the loss function where you set the optimizer call me backward ask the model to do the back propagation update the parameters you're set you're done right so I will go into the evaluation and the prediction aspect of the neural network in the next notebook where we will discuss what this is okay so with that let me open up the next notebook is this is this visible the the new the classification problem notebook that I just opened so now we're going to look at a classification problem okay a little more uh a few more things will show up there in addition to what we saw in the linear regression model again standard to a classification model versus uh what typically happens in the regression Mark so the workflow remains exactly the same nothing changes so first of all in this case instead of using a data set I thought we create some data so basically we create some circles okay so you we want to create two circles so that we are able to create a boundary um to differentiate between these two circles it's a classification so I either want the prediction of a blue dot or a Red Dot that's what I'm trying to achieve okay so uh first of all whatever we created right this was this this was created using the simple python method which is create circles right but whatever we do since it's it's since we created it as a numpy array there is a method there is a function that we can invoke to convert the numpy array to a torch which is a tensor right as I mentioned uh before you you get into any kind of activity with a deep learning framework such as tensorflow by torch Etc make sure that the data has been converted into the tensor format again it's nothing different to the naked eye and when you look at the data but behind the scenes there is plenty different and that the GPU is able to leverage for Superior performance okay again standard stuff we will do the train and test split um so in this particular case what we did was we first converted it to the torch and to the tensor and then we're doing the split in the previous notebook we did the split then we converted to the tensor okay now comes the building up of the model so here is where the bad news starts right for some of us so today we are going to show that it is not as bad as we think so in this classification model I'm going to use the functional model earlier the sequential model was quite simple we just called nn.linear NN dot value done problem solved right but here you're going to see a few new things okay so for this is this is this is the part that people with the null software background kept kindly slightly intimidated by okay so you get to see all of this oops constructs so let me try to break it down frame number one um yes we are we are going to deal with classes and certain methods of those classes so number one we will need to create a class for every neural network model that we are creating this class needs to inherit from the base class in pie torch which is NN dot module this is already there it is defined it is kept there all neural network models will have to this will be the parent class and we will need to reference this okay number one when we create the class we have to create the init method for it okay and in the init method we call a specific instance by Yeah by specifying self okay now the next thing that you need to do is called super dot give it what does Super dot init mean Super means I am calling the parent class and basically what this does is It's recording the parent class and initializing the init method of the parent class okay why do you need to do that because the question would be see if I'm already referenced referencing The Base Class the parent class then all methods in that parent class are available for me in this particular class right then why do I need to do this again you need to do this again yes granted all methods available uh in in the parent class is available for you in this subclass that you are creating now but there are certain attributes in the parent class right attributes that get initialized only only when you call the init method of the parent class now some of those attributes are required by the methods that you're now going to reference here so uh it's it's a vicious cycle there so we don't need to think too much about it again as I mentioned yes you need to First overcome the the understanding that yes that is going to be a bit of class and hook constructs being there but this is again standard stuff so you create a new model in which you will again just like your sequential you're going to call the NN dot linear nothing much has changed there except that you will say self self dot whatever the name of the layer is right now why do we use the functional methodology the functional architecture it is much more capable and Powerful in terms of its flexibility compared to a sequential architecture sequential architecture as the name suggests you see questions so only when you want to do things in a sequential manner are you able to use the sequential architecture if you want to jump around right if you want to connect the output of one layer by skipping and going to two layers later or things like that any kind of flexibility that you want the sequential method cannot be the sequential architecture cannot be used it is purely sequential so if you have simple basic stuff basic neural network architectures by all means use it but if you want something more so you have heard about Skip connections right in in your NLP models or even in your computer machine models right so uh for those kind of situations you cannot use the sequential architecture which is where the functional architecture comes in functional architecture comes with a bit of book constructs but very standard stuff so you first Define the init method you call the super dot init method every time every time you do this standard and then you have to use the self dot layer so this is one extra pick that gets added because with with the hook constructs and the class methodology you will need to reference it with respect to each instance which is done with the self so again standard self dot layer one and this name is whatever you want to give it self dot layer1 it's a fully connected layer it has two inputs and 10 outputs again with very no different from what we invoked in the sequential architecture also so any number of hidden layers so each one of this is a hidden layer so here in the init method you're not connecting it so that's why after each fully connected layer you don't see the value right this is where you only initialize whatever you want to use this is not where you are connecting the model okay the connecting the model happens in another method called as a forward method the forward method again is defined is already defined in your base class but whenever you invoke The Base Class you have to mandatorily you have to mandatorily overwrite the forward method and it's in the forward method that you connect all the layers that you have defined how do you connect it so when you call the forward method again you have to call the self because you're referring to a particular instance and then this is your input your input data right so you say that give my input to the first fully connected layer right so again we have named it the fix we can name it as anything then what you're saying is the output of this first fully collected layer I want to pass it to a value okay and I'm going to name it also as X you can change it you can say A1 A2 A3 A4 so then in the same way you need to change these names this way so then you're saying that the output of the value I want to give it to my next fully connected layer then that the output of the fully connected layer I want to pass it to already and then finally I have another layer okay here you're going to see something different so in a classification model you typically have a sigmoid right if it's a if it's a binary uh classification you'll have a sigmoid with which you can find out if it is zero or a one right we are not putting that Sigma in here so it has been proven it has been proven that it's it's better it's better not to put the sigmoid in in the overall flow of the the layers instead what we will do is it is better to invoke it in the loss function when we calculate the loss function what we will do is internally if you if you invoke the BCE loss so you have the BC loss function and you have the BC with logits loss right if you invoke the PC e with logit's loss what it does is whatever is the output of the forward pass it will fit a sigmoid layer on the output side of it and after that only it will calculate the loss so you could question what difference does it make well it seems like the research has shown that it is better that the the the the sigmoid function remains as part of the loss while calculating the loss instead of it being mentioned as part of the overall layers itself so this is something that has just been proven so it's better to follow that so we have defined the model now as you can see nothing else has changed you define the loss function as you can see here since we did not use the the sigmoid layer in the last layer because eventually it's a it's a binary classification problem so you need to convert whatever output came out you need to convert it within to zero or one but in our model that provision has not been kept so we need to use the pce with logit's loss what it does is it does the BCE but before it does the BC and calculates a loss it will fit a sigmoid layer to the output of our model and then so we we Define the loss function again just in as in the previous case we have specified which Optimizer I want to use I have specified here that I will go with the sigmoid the stochastic gradient descent right now look at this you call the model you calculate the loss you reset whatever the pie charts had with it from the previous epoch can do the back propagation based on the back propagation you update on the parameters done done now this is your training did anything change nothing well the only thing that changed was you add you had a few uh oops constructs but very standard nothing that keeps changing right it's if you buy standard uh methodology if you use the init and the super and you define the forward method you're setting both right now comes once you have done the training part now comes the evaluation part of the testing part right now unlike for people who have been exposed to tensorflow in tensorflow you have your model.fit model dot predict model dot evaluate model dot compile so for each activity there is a separate function for it so when you say model dot compile it will take all the loss function and the optimizers and it will keep that it's only when we call the model.fit does it do the training but in this case you see that the training happens when you call model itself right and the loss function is calculated and the optimizer is done yeah how much time it will take uh maybe another 10 minutes is that okay yeah actually uh next data is some skills so that just I'm just asking you can take your time okay but please try to uh wind up early yeah sure so um in fact I'm almost done I just need to mention a couple of important things when we're looking at testing or the in the evaluation mode where after you train the model you would like to see how it performs on the new data so when you do that you have to it is good practice to First say model dot event why in your model you might have let's say drop out layers you might have batch normalization these are things that are essentially required only during the training phase and since like your tensorflow you don't have a model dot compile model dot fit model dot predict pythons doesn't know in what mode are you running this it's only model you call model it does everything right so but when you say model dot eval it's basically telling by torch in a whatever extra things that you're doing for the training phase which is uh using the batch normalization the Dropout layers remove that don't keep it there I I I'm not asking you to invoke those because I'm just using the I just want to predict using the model that has just been created so this is One Direction that you need to give to Fighters that I am in testing mode secondly torch.inference mode now what happens with the normal flow is that the gradient gets calculated and it gets updated so now when you say the torch.inference mode this will improve your performance of the model because the you're again letting it know that I don't want you to do anything to the gradients calculate the gradients update that gradients none of those things right I am because I'm totally in the inference mode so all I want you to do is just tell me what is the output of the model when I give this input the reason being that see look this is we are in the testing mode now is it any different from the training mode no it's the same thing just the data has changed this is the reason why we specify model dot eval and with torch dot inference mode these two things if you specify you're telling pytorch that now I'm giving you test data just make me a prediction don't bring in your Dropout layers batch normalization don't try to change your gradients don't calculate the gradients um modify it I I want you to use what you currently have that is what these two methods we uh the the direction given to by touch um to make it work in that mode so um that is pretty much as you can see uh the even though we have used the functional mode um where it is oops constructs you can very clearly see that everything is standard the only thing was that you needed to create a class invoke the parent class which is NN dot module which is standard for any neural network you need to Define an init method where you will call all the layers the activation functions don't connect it you and then you connect it in the forward method this is a method that you will have to use you will have to overwrite mandatory and this is where you connect the model after that the training process the evaluation process is pretty much standard as I mentioned you can just copy paste the code and put it anywhere and it will run so with that I think we have slightly gone extra on the time but yeah that we will stop the uh my talk so maybe a few questions I can take