Image Generation using GANs | Deep Learning with PyTorch: Zero to GANs | Part 6 of 6

Key Takeaways

hello and welcome to deep learning with pyo zero to Gans this is an online certification course being organized in collaboration by free code camp and Jovian today we are on lesson six deep learning with pyo 02 Gans and the topic is generative adversarial networks or Gans my name is Akash and I'm your instructor so over 5 weeks we have been learning various Concepts in deep learning and today we are going to put them all together to train a generative adversarial Network or Gan which can go from producing random images like a random noise like this to producing high quality images of handwritten digits and Anime faces and what's more interesting is that the exact same kind of model will be used to produce both of these images and the only thing that will differ is the data set the models are trained on now if you have been following along then by completing the three weekly assignments and a course project on the course page and submitting them on the course page you will be able to earn a certificate of accomplishment for this course so please visit the course page and let's get started so the course page is zero.com and on this page you can enroll for the course and you can also view the previous lessons lessons 1 through 5 and the previous assignments assignment 1 2 3 and the course project but today we looking at generative adversarial networks and we are also going to cover an additional topic called transfer learning towards the end so let's get started now on the lesson page you will be able to watch a recording and you can catch a Hindi version here as well and the first notebook we are going to use today is called generating anime faces using Gans so let's open it up this is a Jupiter notebook hosted on the Jovian doai platform and you can see here this notebook has some explanations and below here it also has some code and to run this code you have two options you can run it using free online resources which we have been doing so far or you can run it on your computer locally and you can follow the instructions here we will be using Google collab so I'm going to scroll up here click on the Run button and click run on collab now I click authorize and run on collab this may ask you to connect your Google Drive so that this notebook which was in a read only mode on Jovian can be put onto your Google Drive and executed on Google collab an online platform for running jupyter notebooks and now we can see here we have the same Jupiter notebook except this time it's on Google collab and the code when we execute it will run on Google servers where they have given you some free gpus with a limited access so the first thing we will you will do is click on the runtime button and select change runtime type and then from the change runtime type you want to definitely select GPU now if you have subscribed to collab Pro you can also pick a high Ram machine and when you subscribe to collab Pro you also get Priority Access to higher more powerful gpus so let's do that so now we have selected GPU as our runtime and we can start executing our code so the first thing we need to do is run the first cell which is going to install the Jovian library and connect this collab notebook with the Jovian notebook that we just executed this is so that when we want to capture a snapshot of this jupyter notebook we can do that easily using the Jovian Library so always run this first cell whenever you run a notebook from collab from Jovian onto collab now when you do run the first cell it initializes the runtime which means it connects to the server on Google's infrastructure where all of this code will be executed so today's topic is training generative adversarial networks or Gans in pyo and we've been building towards this step by step and you can see the previous notebooks in the series here we've already seen how to run the code now we we've also seen how to use a GPU which was to go into runtime and change runtime type and select GPU on Google collab but if you're not using Google collab on other platforms you have the option of gpus as well so on kaggle notebooks you have a sidebar from where you can select an enable a GPU now if you have a Linux or a Windows laptop with an Nvidia graphics card installed then you can install the Nvidia Cuda drivers so Cuda is the language used to communicate with GPU so you need to install the Nvidia Cuda drivers and then you need to install the version of pytorch compatible with gpus and then you will be able to use a GPU on Linux and windows on binder and Mac OS you will not be able to use gpus so the models that we train here will take a lot longer to train without gpus but that said you can still run all of this code without a GPU now let's talk about generative modeling so far we have been using deep neural networks for something called supervised learning which is image classification or classification in general or regression which so class ification involves telling which category a particular sample and input belongs to and regression involves predicting a continuous number from an input however generative adversarial networks or Gans as we call them use neural networks for a very different purpose and that is called generative modeling which is an unsupervised learning task so there are no labels or targets and it involves automatically discovering and learning the regularities or patterns in input data in such a way that the model can be used to generate or output new examples that could have been drawn from the original data set so what that means is if you're given a data set of human faces you can then train a model you can train a gan which can learn to generate new human faces which look like they may have been picked from that data set so for instance there are many different there are many different data sets containing human faces online and one of the gas that has been trained on human faces produces images like this high quality images of human faces and in fact you can visit this site this person does not exist.com and this site is connected to a generative adversarial Network running on the cloud and every time you reload this page it makes a request with a random input to that model and that model generates this high quality image of a human face and and this person does not exist so this image has been generated right here right now this very second each time you're getting a new image out of this and you can see that there are so many details it's a very detailed image of a face you can see there is teeth hair and even accessories like glasses and you can even see that there are backgrounds and there are people around as well in a lot of cases now if you wish you can go and check the source code of this so this is a generative adversary Network so you can check out how it works there are some some explainer videos here as well but we will today be learning something very similar to this so we will not be generating human faces that requires training a model for probably a few days to maybe a few weeks on a really large data set we will do something slightly easier which is generating anime faces so if you watched any anime or cartoons we will generate anime faces using the same technique that has been used here now there are many approaches for generative modeling in general but what is called a gan a generative aders network is something that takes this particular approach towards generative modeling so we have two models here normally so far we have seen that there has only been one model and we give an input to the model and it gives a prediction We compare the predictions with the targets and then we perform gradient descent and that trains the model and then we can use the model for making inferences that is what we have learned so far in the last five weeks that is typically how a classification or regession model works now a generative adversarial network has two different models so two different neural networks there is a generator Network and there is a discriminator network now the generator Network the input to it is some random input some random Vector so you feed in a random Vector into the generator Network and it generates an image or it at least tries to generate an image so you give it an input and it generates an image if you give it a different input it generates a different image and this is the model that we really want to train we want to make this model better and better at generating images which look like they may have been drawn from a particular data set and to do that we use a second Network called a discriminator so the discriminator is a model its only job is to be able to differentiate between real images which is images drawn from the actual data set and generated images which is images generated by the generator so the discriminator is uh responsible for taking real images and uh generated images and discriminating between them telling which one is which so now we have these two models and the way the training works and this is very interesting it's a very elegant trick that is used to train Gans because there is no real targets here so how do you really make the generator better and the the way to do that is first come up with a good discriminator so what we do is we take the discriminator we put in some a batch of real images let's say 128 real images into the discriminator and then we put in a batch of fake images or 128 generated images from the discriminator and then we train the discriminator so we use gradient descent because we know that for real examples the discriminator should predict real and for fake examples the discriminator should predict fake so we use that fact to train the discriminator so that the the discriminator after training gets good at differentiating between real and generated images then once we have trained the discriminator just a little bit we then use the discriminator as a part of the loss function of the generator so what we do is we take a generated image from U we take some generated image from the generator we pass them through the discriminator and we try to fool the discriminator so we try to then update the model update the generator model train the generator model so that the generated images get better so that the generator can now fool the discriminator and we'll see this in a lot more detail but I just want you to capture the intuition here that first we train the discriminator to be able to differentiate between really bad generated images and actual images from the data set then keeping the discriminator fixed we train the generator to produce slightly better images that can fool the discriminator and so the generator now gets better then we train the discriminator once again it can get better at differentiating the slightly better fake images versus real images and then we train the generator again so the generator now generates even better images and that is a cycle that we repeat over and over and over batch by batch epok by Epoch and at the end of that process you end up with a generator model which can generate images like this so This that is why this approach is called a generative adversarial Network because the generat Ator and discriminator are like adversaries the generator tries to fool the discriminator the discriminator tries to catch the generator's bluff so that's how we train again and that's what we are going to talk about today but Gans can be notoriously difficult to train because you can see that there is a subtle interplay between the discriminator and the generator if one gets too good then it is no longer an even match between the two so that's where it is training Gans is hard and they are extremely sensitive to hyperparameters activation functions and regularization so in this tutorial we will train a again to generate images of Anime characters faces so by the end of the tutorial we will be able to generate images which look somewhat like this you can see that these are not high quality images of anime faces these are slightly lower in in resolution and you can also see that they are a little bit noisy they're not perfect but they're getting pretty close and we'll see how to do that and since Gans are sensitive to all these parameters and activations and regularization we will make some very interesting choices which I will talk about uh which you may not have seen so far so the data set we are going to use is called the anime faces data set it consist of about 63,000 cropped images of anime faces and since generative modeling is an unsupervised learning task these images will not have any labels so we we're just going to work with images of anime faces so let's just uh run through a couple of things here we are going to set a project name in a project name variable we will use this later for capturing snapshots and sending our notebook to Jovian we'll see and if you are running this on a platform where you do not have access to a GPU then you may want to where you do not have all the libraries installed then you may want to install the required libraries here if you do not have access to a GPU you can use this CPU versions of pytorch if you have access to a GPU you can use the GPU versions of pytorch on Google collab however none of these are required because all the libraries are already installed so whichever one you need to run just uncomment the the exact line and run it and you will be able to install the libraries now the first thing we need to do is get the data set and we're going to get the data set from kaggle so if we click this link you can see here that on kaggle this particular user has created this anime faces data set um it has been scraped from this website get you.com and you can see here on on the data tab you will be able to see that these are all the images that are present here you can browse them as well there seem to be about 63,000 over 63,000 faces how do we get this into Google collab now now to get a kaggle data set not just this one but any kaggle data set into Google collab you can use a library called open data sets so you can see this here it's a open source Library developed by Jovian and the open data sets Library provides a helpful function called download so you can just use this function open data sets or OD it's imported often as OD od. download and pass the URL to a kaggle data set so for instance we can take this data set and then pass it to open data sets. download and it will be able to download the data set into Google collab or wherever you're running your jupyter notebook so this is a pretty useful Library definitely use it in your projects whenever you're working with kaggle and do give it a star as well now there's one other step involved here because open data sets uses the kaggle official API so you need to provide access to your kaggle account you need to capture you need to provide some API credentials so that open data sets can get this data from your kaggle account and the way to do that is to go to kaggle.com now on kaggle.com you can then click on your profile and go to the account page so this is on the sidebar you click on account and then on the account page you will see a option here called API there's a section API and you click on create new API token so when you click the button create new API token it is going to download a file called kaggle Json and this contains this contains an API token which will be used to access your kaggle account from anywhere so the kagle Json file will look something like this I'm not going to open my file here because I don't want to show you the secret key on video so here we have the kagle Json file it will have a username key where the this which will contain your kagle username and it will have a key or the token and that key value will be here it will be a long string now when you run open data sets. download you will be asked to enter your username and enter your kaggle key and you can get them from this kagle Json file and you can paste them so let's see how that works so here we are installing the open data sets Library using pip and then we are importing open data sets as OD and using the data set URL that we have set here passing it to od. download so here it's going to ask me for my kagle username and I can enter my kagle username and it is also going to ask me for my kagle key once again I can get that from the kagle Json file now you can enter your kagle key here here but there is one other way to use open data sets which is a little bit easier so I'm just going to stop this cancel this operation here and open the files tab and on the files tab I'm going to click the upload button and I have already downloaded my kagle Json file so I'll simply upload the k. Json file into the files tab so now the kagle Json file is available to collab and open data sets can automatically pick pick it up so if we run open data sets. download with the data set URL this time it's going to read the kaggle credentials and you can see that it is now downloading the data so the data will get downloaded in a folder which has the same name as the URL portion of the data set indicating that specific data set so let's see here it is not only downloaded but it is also unzipped for you as well so now you can see this folder anime data set anime face data set and in inside this anime phas data set there is a single folder called images which contains all 63,000 images in jpeg format so if you open this you can see here that there is a big list of images here so there you go this is what the images look like each one is a JPEG image and there are no labels there are no classes so that's why there is a single folder called image so now let's load this data set using the image folder class from torch vision and we will also resize and crop the images to 64 pixels x 64 pixels that will make it a slightly easier for us to work with generating very large images using Gans will often require a lot of training and a much bigger model than we can create here so we will work with 64 pixels x 64 pixels another thing that we're going to do is we're going to normalize the pixel values and if you remember our discussion from C 10 what we did there was we calculated the mean and standard deviation for each of the channels red green and blue and then we used those means to normalize the pixel values this time though we are not going to calculate a mean we're going to do something much simpler we're going to pick a a mean of 05 and a standard deviation of 0.5 as well and here the purpose is not really to normalize pixels across channels but the purpose is to take the pixels which are which have values pixel intensities in the range 0 to 1 and translate them into a range minus one to 1 now this is one of those eccentric things that we do while training Gans it is more convenient for training the discriminator when the pixel values are in the range of minus 1 to 1 which is if there if they are symmetrically distributed around zero rather than being in the range 0 to 1 and we will see this come up again when we generate images from the generator okay so think about that this normalization is not really to change the effects of different channels this is simply to move the pixel intensities from the 01 range to the minus one one range so we import data loader image folder and transforms then we create this we set the image size to 64 we going to use a batch size of 128 for creating our data loader and then we're going to use the means of 0.5 and standard deviations of 0.5 for normalizing the data so here we are creating the image folder with the data directory and then we passing the transform since we are going to have more than one transform we need t. compose and T refers to torch Vision transforms so we're going to compose a bunch of transforms which means we're going to apply them one after the other the first thing we do is apply t. resize and that is going to resize every face image into 64 pixels by 64 pixels then we are going to take a center crop because it's possible some of these images may be rectangular a center crop is going to pick the Central Square crop out of it the 64x 64 crop then we're going to convert them into tensors so now we end up with image tensors and finally we're going to normalize them using these stats so that the values within the tensors go from the 01 range to the minus one one range so that's our train data set and then here is our training data loader so we have the training data set and then we pass the batch size we also pass shuffle through then this is because each time we want to generate batches from the training data set in each Epoch we want to use the images in a different order and that just helps the randomization helps train and generalize the model faster now we're also going to set num workers to three and pin memory to true this is simply going to ensure that we use multiple course from our machine to read the images all right we have data loaders and we have a data set as well so let's create some helper functions to denormalize the image tensors and display some sample images from a training batch now here we are going to import P torch and we're going to use the torch Vision utility function make grid and we also going to use PLT uh matplot li. P plot to plot the images so first we're going to define a function dor now remember for training a model we are going to use these image tensors which have been normalized with pixel values in the minus one to one range but to view these images we will have to bring them back into the 01 range so that is what this dor function does it takes the image tensor multiplies it with the standard deviation of 0.5 and adds the mean and that's just going to denormalize it bring it back into the 01 range and now we can define a couple of helper functions show images and show batch so show images simply takes a bunch of image tensors and then it a maximum number of images that it should show and it simply plots them so it plots them in a grid like this and then the show batch function takes a data loader and gets a batch of images from the data loader M simply shows the images so let's look at a batch of data so here is a batch from the training data loader when we call show batch we fetch a batch from the data loader and then we break out but before breaking out we show the images and inside show images we are going to denormalize the images so we are going to take the images and change the pixel values back to 0 to 1 and then we are also going to convert that into a grid here so using the make grid function in torch vision and finally to show it using matte plot lip we need to just flip the dimensions around a little bit and that is why we have this permute function remember in pytorch images have the color channels in the First Dimension whereas matte plot lib requires the color channels to be in the last Dimension and that's why we have this Peru now if the show images function does not make sense then that would be it would be a good idea to go back and revise chapters 4 or five we've covered this in a lot more detail there but the idea here is to take a bunch of images and display them in a grid like this so this is our training data you can see that we have many different kinds of faces here you have faces looking forward you have faces looking to the side you have many different hairstyles as well many different hair colors sometimes you have hair accessories you have different colors of eyes you have male you have female you have both kinds of data you have different Expressions you also have accessories so this is a fairly a Vari data set it has many different variations even though it is just Anime faces and it's still a challenging data set to work with so let's see how we can train a generator to produce images and let's see how close we can get to these real images in the time that we have today so before we continue I'm just going to install the Jovian library and then create a snapshot now remember that this notebook is running on Google collab and that can shut down anytime so I'm just going to my Jovian profile grabbing my API Key by clicking on the API key button and when I run Jovian docit and provide my API key the Jovian Library captures a snapshot of this notebook and puts it on my Jovian profile so that I can run it later even after this Google collab notebook has shut down okay so always remember to record snapshots of your notebooks so moving right ahead before we Define our models let's just create some utilities for working with gpus once again we've covered these in a lot of detail in the previous lessons so we won't get into the code here but we'll just see what these functions are and how we'll use them so the first thing is the get default device function now it's always a a good idea to write your code in such a way that it can work with a GPU if one is available otherwise it should use the CPU and that's where we're going to use this get default device function it checks whether a GPU is available using torch. ca. is available and for this to return true three things should hold true the server or the compute or the computer where you're running this code the execution environment should be connected to the hardware which is an Nvidia GPU or graphics card then you should have the Cuda drivers so Cuda is the language used to communicate with GPU so you should have the Cuda drivers installed and third you should have the py to version compatible with gpus so if you have all these three things which are already ensured in collab but you may have to ensure them manually if you're running this on your computer if you have all these three things then you will get back then we are returning TS short device Cuda which is going to be a pointer to a GPU similarly otherwise if you do not have these if you do not have GPU available then we're going to return to short device CPU and get default device is automatically going to pick a GPU or a CPU based on whichever is available now that's one the second is a two device function so the two device function can take some data or some or a model and then move it onto a given Target device and finally we have this device data loader class so we have data loaders for training and there's no validation data loader here there's just a training data loader so the training data loader is used with within a for Loop now the device data loader can be used to wrap the training data loader so that whenever a batch is pulled out in a for Loop that batch is immediately moved to the device just in time just before being used so these are the three GPU related utilities that we have so first let's get the default device now if you have everything set up properly and you have a GPU you should see device type Cuda here if you do not see device type Cuda that means you're using a CPU on collab you may want to go and change the Run time type if you see CPU here not just right now but whenever you are working with a deep learning project and we can now move our training data to the data we can now wrap our training data using device data loader so that it automatically transfers batches of data to the GPU when we access them so here we have converted the training data loader into a device data loader by passing the original data loader and the target device which is a GPU so now let's start defining our model the first model that we will Define is the discriminator because it is somewhat easier to understand the discriminator takes an image as an input and it tries to classify that image as a real image or a generated image in this sense it's like any other neural network we've trained so far and we'll use a convolutional neural network which outputs a single number for each image so we give it an image which is a three Channel by 64x 64 image and for that image the discriminator returns a single number as an output and then we will interpret this number as the probability that the discriminator thinks that this this image was drawn from the actual data set right so if the discriminator outputs zero then it is predicting that the image was generated from the generator and if the discriminator output something close to one like 0.9 then it is 90% sure that the image was picked from the actual data set so that's a pretty straightforward uh classification problem and this kind of classification is often called a binary classification and this is slightly different from the multiclass classification which we've done so far where we had maybe let's say 10 categories or 10 digits or 10 categories of everyday objects this is much simpler where there is just one output which is whether the image is real or not so let's define the model and as we've seen uh since this is going to be a straightforward convolutional neural network we do not need to extend any classes here we simply going to use the nn. sequential class so we use nn. sequential and into nn. sequential we pass a set of layers and then we can use this discriminator model and when we pass in some input it can go through the layers one by one and come out from the other side so let's look at the layers here so here first of all we're going to start out with an input image of size 3x 64x 64 and of course it's going to be a batch of images not just a single image but we have not added the batch Dimension here because we never affect the the model never really affects the batch dimension for each image in the batch it gives an output of the shape that we'll see so first of all we have a convolutional layer so here we have a con 2D layer and this goes from channels three input channels to 64 output channels and it uses a kernel size of four and a stride of two so this is something interesting that we had not seen earlier we have seen kernel sizes of three and we have seen stride of two or we've seen stride of one so far and normally after a convolutional layer we also have a Max pooling layer so the convolutional layer retains the size of the input so if you have a 64x 64 input being passed in then the output is also 64x 64 however when you use a stride of two then the convolution is going to [Music] skip ahead so it's going to use every second pixel to place the kernel so for example the kernel is first placed here and then it is placed here and then it is placed here you can see that it is skipping one pixel both in the horizontal and the vertical Direction and that is why the image size reduces to half the output size is half and then the kernel size of four is simply to make sure that we are covering all the pixels properly so this is the part that I will leave for you as an exercise you can see this image here where you have this 6x6 image and it has a padding of one pixel around it and then we passing a kernel of size 3x3 now instead of a 3X3 suppose this was a 4x4 kernel just try and draw it out on paper and see just make sure that it will still be a 3X3 output when you have a stride of two okay so that's what we do here kernel size of four stride of two and we will do that throughout and the output from this convolutional layer and then of course after that we have the batch normalization layer and then we have an activation function so the output is going to be 64 channels we go from three channels to 64 channels and then the size of the image the height and width they become half because of the stride of two so that becomes a 32 pixel x 32 pixel feature map and then we repeat that process so we go from 3x 64x 64 to 64x 32x 32 and then then we go from 128 by 16 by 16 so we double the channels but we half the height and width using the stride of two once again we go 256 by 8X 8 512x 4X 4 and then the final convolutional layer has zero padding so since we have a 4x4 image and on the 4x4 image if you do not have any padding and you using a kernel of size 4 that is going to fit exactly onto the image into exactly one position so the 4x4 image is going to get reduced to a one by one feature map and then another thing that we're doing is we're going from 512 channels to one channel so ultimately we take the entire output we take the image which started out as a 3X 64x 64 feature map and then we increase it to a 512x 4x4 feature map But ultimately we reduce it down to a single number using a smart convolutional layer which has a kernel size of four and zero padding okay so the convolution math is something that you should try and spend some time on just convince yourself that these will be the output sizes take pen and paper try to create each layer separately and then pass try to pass the input of one layer try to pass the image as an input to the first layer take the output and then pass it as an input to the second layer that's a good thing about jupyter notebook you can just add more cells you can create these layers layer one layer two layer three and so on and then you can pass an sample image or maybe a batch of images through each of the layers one by one and observe the shapes okay so do spend some time here and it's really it's really important to cultivate the intuition for being able to tell what the output is going to be from a convolutional layer but the important thing for us here is that the image has been reduced to a single number A 1 by one by one tensor and then we we going to flatten it out into a single ve into a single Vector a vector of size one instead of a 1 by 1 by one tensor and finally we're going to apply the sigmoid function so the sigmoid function let's see the graph of the sigmoid function here so this is a sigmoid function this is its formula its formula is 1 + e to the- X so what it does is it takes numbers and it reduces them it squishes them down into a 0 to one range so if you have a large positive number then that is going to come close to one and any positive number is going to then end up in the range point5 to 1 and if you have a large negative number or any negative number that will end up in the range 0 to.5 so that's a sigmoid function and we apply the sigmoid function because we want to interpret the output of the discriminator which is a single number per image we want to interpret it as a probability and the probability that the image passed was a image drawn from the actual data set the probability that it is a real image and not a generated image and that is why we use sigmoid here and this is how binary classification is different from multiclass classification so when we had 10 classes we used soft Max but when you have a single class you simply use sigmoid now there is one other important detail here and this is about the activation function now so far we have been using the rectified linear unit this is what you may have seen seen in many different models that we have trained or elsewhere we use a rectified linear unit function but here in the discriminator we use something called The Leaky Rel or leaky rectified linear unit activation function so first let's understand what it is so the Leaky reu is a very slight variation on the relu now you may know that the relu is a function which simply ignores negative values so if the input to the r function is -1 -2 -.1 the output is zero but the input if the input to the r function is something positive like 1 2 or three the output is the same number so at more than at greater than z r is the function yal X at less than z r is the function 0er leaky R is slightly different in the sense that it allows a small negative portion to pass through as well so greater at greater than zero it has the same function it has the same formula as Ru which is yal X but if it is less than zero the input then it has a function y ax and typically this a is a small number less than one and that is what is called a leaky value so we are using leaky value with the factor a of0 2 so whenever there are negative values they are going to get multiplied by 0 2 and why do we use it so this is once again one of those eccentric choices that we have to make for our model to train properly so this is the justification this is given in the in the paper that explains this uh model so what happens is because we use the discriminator as a part of the loss function to train the generator Ru would lead to a lot of outputs from the generator being lost so that's where leaky Ru allows the pass of a small gradient signal even for negative values so not just the positive values but also the negative values and that makes the gradients from the discriminator flow stronger into the generator so the idea here is the image is generated from the generator we want to make sure that once an image has been generated from the generator none of that information is lost when we are passing it through the discriminator and using a loss function to train the generator okay so instead of passing a zero slope it passes a small negative gradient so think about that and one good way to just come up with the right justification here is to try replacing leaky relu with reu here and try to train the model with the exact same parameters and see if you get a better result or a worse result now if you get a worse result then think about it think about why leaky relu may be helping in the process so next we can move our model to the device so we simply going to call the two device function which is going to take the discriminator and the device and it is going to move all the weights within the discriminator onto the GPU let's just run this so now that brings us to our generator Network the input to the generator is typically a vector or a matrix of random numbers and this is called a latent tensor so because neural networks are deterministic you give them an input and comes out an output now from the generator we want to generate different random images of anime faces and to do that what we need to do is we need to give it random inputs so the generator will take a latent tensor which which will be of shape 128x 1 by 1 which is basically 128 random numbers so we will give the generator 128 random numbers or a batch of 128 random numbers each and each of these 128 random numbers a 128 by 1X one tensor is then going to get converted into an image tensor of size 3x 28 by 28 okay so it takes the random tensor converts that into an image and when you give a different random tensor it will give you a different image and then the idea would be to change the weights of the generator so that each time you give a random tensor the image that comes out looks like an anime face drawn from the actual data set and that is what we will train the generator to do later but to do this to go from 128x 1X 1 to the shape 3x 28x 28 we need to do something which is the reverse of a convolution right so typically in a convolution you would start out with an image and end up with a certain number of output outputs for instance for the cifar 10 data set we ended up with 10 outputs for the discriminator network here we ended up with one output and here it seems like we starting with 128 numbers and we want to end up with an image so to achieve this we will use something called the con transpose 2D layer from pyo which performs what is called a transposed convolution now some sometimes a transposed convolution is also called a deconvolution although there is a subtle difference between them and you can check out this blog post that I've linked to here to learn more about it so we're at a point now where you have all the basic concepts in place and any new Concepts like different layers or different activation functions or different regularization techniques are things that you can learn about online through blog post by even reading papers reading tutorials reading books to fill in the gaps in your knowledge Because deep learning is such a fast evolving FEI that field that it is not possible to cover all of these Concepts in a single course and in fact even if even if one did all of them would become outdated pretty quickly because there are new things coming out but what we do have right now is we have a strong grasp of the understanding of the basics things like gradient descent things like um loss functions activation functions layers so at this point I will encourage you to pause the video at some point and go through and read the convolutional transpose 2D layer and see how it works but here is a brief intuition what we do is suppose you have a 3X3 image so you can see the blue the blue squares here represent the pixels of the image we take the 3X3 image and then we pad it so we pad it not just around the image but also between the pixels so you can see that there is some padding here between the pixels and there is some padding around the pixels so we do this padding and then we take a 3X3 kernel and then we run it over this padded feature map and now you can see as we run the kernel we from a 3X3 image we go and create a 6x6 a 5x5 image in this case okay and now you can start doing a little bit of the math what happens if you use a kernel of size four what happens if you use a stride of two and so on but this is the basic idea behind a deconvolution or a con transpose dud layer and one other way to look back at this is imagine you were doing a convolution on the output then you would end up with the shape of the input okay so when you apply this reverse technique what kind of a convolution will it take to end up with a 3X3 feature map well you would probably have to start with something like a 5x5 output okay so keep that in mind uh it's the exact in terms of the feature map sizes it is the exact opposite of what a convolution does so let's define our generator now so now we have the latent size of 128 that we're going to use let's we've put that in here now the generator is also going to be be a sequential model and we're going to start out with latent size * 1 by 1 so you can try changing the latent size and see if that gives you slightly different results but the basic idea behind Laten size is that if you had just one vector then you would have just one dimension of control for the generator so you can think of the different latent vectors as controlling different aspects of the generated image maybe the first latent Vector may end up controlling the color of the hair and the second latent Vector may end up controlling the color of the eyes the Third latent Vector may end up controlling the third latent the third element of the latent Vector may end up controlling the shape of the mouth or the expression the fourth latent Vector may end up controlling which way the face is turning so the more I mean it's not exactly this but it's roughly this where some combination of latent vectors controls a specific attribute of the generated image once the generator has been trained so the more latent vectors you have the more variety you can get in your generated images typically something like 64 128 is good enough for simpler images for more detailed images like human faces you may want to use a much bigger latent size so now the generator takes a latent Vector which is of size 128x 128 128 x 1X 1 and of course this is going to be a batch of latent vectors not a batch of latent tensors not just a single latent tensor so it takes this latent tensor and then it uses a con 2D transpose so here you have a con 2D transpose we go from from the latent size number of channels so 128 channels to 512 channels and then we going using a kernel size of four and stride of one so together what these do is these take an image from 1ex one shape to a 4x4 shape and I'll let you work this out by looking into con transpose 2D and of course we have batch norm and then we have the normal relo activation function so for generator we do not need the Leaky Rue only for the discriminator then once again we have another con transpose 2D layer and this time we go from 512 channels to 256 channels and once again the size of the feature map doubles with a kernel size of four and stride of two and padding of one so now we have 256x 8X 8 and then once again we repeat it and we end up with 128x 16x 16 we repeat that once again and then we end up with 64x 32x 32 and finally we end with 3x 64x 64 so you can see we've first increased the channels to 512 and uh increase the feature Map size to 4x4 and then we double the feature Map size each time but we reduce the number of channels into half each time so at the end we end up with 64x 32x 32 and finally the final layer is going to take these 64 channels and output three channels so the end result of the convolution of the generator function is to take a latent tensor of shape 128x 1X 1 and convert it into a 3X 64x 64 image okay now there's one other interesting detail here you will notice that we have here An Tan H or a hyperbolic tangent activation function now what is that doing what that does is it the hyperbolic tangent function reduces values into the range minus1 to 1 so when a 3X 64x 64 feature map is generated from this convolution it can have values anywhere in the range from minus infinity to Infinity the tan the hyperbolic tangent function reduce them reduces them into the range minus 1 to 1 so the outputs of the generator are going to be images which have pixel values in the range minus 1 to 1 now if you recall when we created our data set you may remember that we applied a normalization which converts the real images from a from having a 0 to 1 range of pixel intensities to having them in the range minus 1 to 1 so now you see how it all fits together that images picked from the data set after normalization have pixel intensities in the range minus1 to 1 and they are of shape 3x 64x 64 and images generated from the generator after applying the hyperbolic tangent activation at the last layer will also have the shape 3x 64x 64 and have pixel values in the range minus1 to 1 so we have set up the generator in such a way way that it is going to generate images which at least have the same ranges of pixel intensities as the discriminator and then the next step will be to train the generator so that it's not only the ranges that match but even the pixels have such such values that when you pick put all the pixels together it looks like the image of an anime face okay but this is the first step that the generator is able to generate images which fall whose values whose pixel in I ities fall within the same range minus 1 to 1 so that's so that's our generator function now we can try out our generator function we can see what kind of images it is generating simply by first generating some random latent tensors so we create we use torch. randn which is going to create random tensors with values in the range minus one to 1 picked from a gsan distribution but random is the important part here and we're going to create a batch of Random tensors so we have a batch size of 128 so we have 128 and then we're going to create tensors of size latent size by 1 by one so here is the output so the fake so our XB or a batch of inputs to the tensor has a shape 128x 3x 64x 64 sorry it has a shape 128 by 128 by 128 by 1 1 by one so batch size is 128 and then the latent size is also 128 then we pass them so let's print that out here xb. shape then we pass this batch of data into the generator and let's run the generator here first so we pass this batch of data into the generator so that the generator converts each latent tensor into an image of shape 3x 64x 64 and then to show these images we can use the same show images function that we had earlier so what we're going to do inside show images is we are going to take these fake images and take the pixel values and denormalize them from a minus one one range to a 0 to one range so now you can see here that after denormalizing when we put them into a grid this is what the images look like which is basically random noise and this is something that you might expect because we have not really trained the generator so far we have simply set it up so that it can generate these images so it's just generating random pixel values which looks like noise and we will now train both the generator and the discriminator but before that let's also move the generator to the device using the two device function so now we have our discriminator model and then we have a generator model as well and let's train them and first we will train the discriminator and then we will use the discriminator to train the generator now since the discriminator is a binary classification model which is it outputs a single number indicating whether the image is is real or not the probability we can use something called the binary cross entropy loss function the binary cross entropy is very similar to the Cross entropy that is used for classification problems where there are multiple classes except here it's used for a single class so let's see how to train the model here is a function train discriminator it is going to use um it is going to we are going to give it a batch of real images so this is one batch of training that that is going to be performed by this function so it's going to get a batch of real images as input and it is also going to get an Optimizer and this Optimizer is has been set up to modify the parameters of the discriminator okay so first the first thing that we do is we clear all the gradients within the discriminator so we called optd do0 grad then we pass real images through the discriminator so here we take a batch of real images so this this will be drawn from the data set we'll see how this is used later on but we take the batch of real images and we pass them through the discriminator and then we get a set of predictions now obviously for each image you're going to get a single number so if you have a batch of 128 images you're going to end up with 128 numbers each of them are in the range of 0 to 1 but what we really want the discriminator to predict the target for the real images is one so it should be predicting one for each real images so so the probability that the image is real is one and therefore the targets are one and now we have predictions and we have targets so now we know how to calculate the loss so now we can call the F do binary cross entropy function and we pass in the real predictions which are the predictions which are made by the discriminator on a set of real images and then we pass in the targets the targets are all one so that gives is the loss which we'll use for gradient descent apart from the loss as an evaluation metric just so that we can visualize we can tell how well the discriminator is doing we are also going to calculate the average prediction that the discriminator made for the batch of images so we're going to take a torch. mean of the real PRS and that is simply going to give us the average of all the predictions for the real images and we going to call that the real score so we want the discriminator real score to get as close to one as possible for it to become good at detecting real images so that's our one part of training the discriminator passing in a back B of real images but then we also need to pass in a batch of fake images to tell the discriminator which images should be classified as fake so then we once again create a batch of latent tensors so here we use a random batch we use tor. randn and we pass in the batch size the latent size in 1 by one and that's going to create latent tensors a batch of latent tensors of size 128 by 1 by one then we're going to put the batch of latent tensors into the generator and end up with a batch of fake images remember as we did uh just here above we're going to end up with a batch of fake images 128x 3x 64x 64 so this has the exact same shape as a batch of real images but initially it is going to be random noise so we take all of these fake images and we know that for these fake images the targets should be zero because the discriminator should be predicting that the probability of these images being real is zero so the targets are zero and then the predictions for the fake images can be gotten by simply passing the fake images into the discriminator so we pass the fake images into the discriminator get some predictions and the targets we set them all to zero and that is done using torch. zos here so you can see that the real targets were torch. ons and here we were using torch. zos and now we can calculate the fake loss or the loss for fake images which is once again F do binary cross entropy pass in fake predictions and fake targets and we end up with the fake loss then we call torch. mean and then we call torch. mean on fake predictions that that also gives us the fake score so we want the score for fake images the average score to be as close to zero as possible so once again this is another evaluation metric that we will keep in mind and now the overall loss for the discriminator is the real loss obtained by passing a batch of real images and the fake loss obtained by passing a batch of fake images and then we call do backward Ward on the loss so remember when we call Dot backward on the loss that computes the gradient of the loss with respect to every single weight of the discriminator so now the gradients are computed and then we called optd do step so this is the discriminator optimizer we are calling do step to perform gradient descent here what this will do is this will modify the weights of the discriminator and remember just the discriminator it's not going to touch the generator yet so it is going to modify the weights of the discriminator make them slightly better so that the discriminator gets slightly better at telling fake images from real images in the next batch okay so that's our gradient descent we have trained our discriminator using one batch of images from the training set and one batch of fake images generated inside this function and now finally we simply going to return the loss and the real score and the fake score for the purpose of tracking so that we can track the discriminator training progress so that that's the discriminator now let's come to training the generator now the discriminator was straightforward because at the ultimately it was a classification model and the only real subtle part there was using the generator to generate the input to the discriminator now training the generator is not that obvious because the outputs of the generator are images and this is where we employ a rather elegant trick which is to use the discriminator as a part of the loss function and this is how it works we generate a batch of images using the generator just as we have done so we create some random latent tensors put them into the generator and we get back a batch of image tensors 3x 64x 64 now we pass those images into the discriminator and outome some predictions for each of those images the discriminator is going to make a prediction now the generator's purpose is to fool the discriminator so if we had the perfect generator and we were able to fool the discriminator then the all the predictions would be one so what we do is we calculate the loss by setting the target labels to one so we do this because the generator's objective is to fool the discriminator then we use that loss to perform gradient descent that is to change the weights of the generator okay so we do not affect the discriminator we change the weights of the generator so that it gets better at generating real life images so what happens is you pass a batch of data get some images pass them through the discriminator set the targets to one and calculate the cross entropy loss then you perform gradient descent and change the weights of the generator the next time the generator is going to generate some images for which the discriminator which has not been affected so for which the discriminator is going to predict something closer to to one so we train the generator to fool the discriminator by using the discriminator as a part of the loss okay so this is the rather elegant trick and think about this a few times why we are setting the target labels to one for generated images and if you can understand this you understand the basic idea behind Gans and this is how it looks like in code so in code it's actually really simple the train generator function simply takes the optimizer for the generator and first it clears the gradients so opt g.0 grad is going to clear all the gradients so that we can all the gradients are wiped out and we can start fresh then it's going to create a batch of latent tensors so here we generate a batch of latent tensors using torch. randn you can see they have latent size by 1 by one is the shape they have and it's a whole batch then we take that latent tensors pass them into the generator and get back some fake images so now we have a batch of fake images now we pass those fake images into the discriminator and get some predictions and then we set the target to one so Target is set to torch. On's and then we calculate the loss so this is the loss for the generator overall loss so the calculate the loss and this loss is once again F do binary cross entropy where we pass in predictions and targets so here you can see that if we were to put all this logic inside this single line of code the discriminator is actually part of the loss function so we are using the discriminator as a function not as a neural network but as a function to convert images into numbers and we want those numbers to be close as close to one as possible to make the generator better now we call do backward on the loss function now that is going to calculate the derivatives of the gradients of the loss with respect to the parameters of the generator the weights and bias and then we call opt G so the generators Optimizer we call the do step function and that is going to update State the weights of the generator so that in the next batch or in the next time we try to train train the generator it's going to generate images which are able to better fool the discriminator and keep in mind that we keep the discriminator constant here so the training of the generator and the discriminator is done in tandem first we train the discriminator to get good at separating U fake images from real images and then we train the generator to get good at fa at fooling the discriminator using the train generator method and finally we simply return the loss of the generator so that's the training functions and before we train the model now one thing we have to remember here is there is no validation set so sometimes it can be hard to interpret what's happening is the generator getting good is the discriminator getting good is something going wrong and that is where we have to do some visual inspection so what we're going to do is we going to save intermediate outputs after every single Epoch we going to generate a batch of images and we're going to save that batch of images as a file as a PNG file so let's see how to do that we're going to use a save image function from torch vision and torch vision. utils contains a lot of very useful functions to work with images which is loading images saving images converting a grid creating a grid of images and many more so do check them out it also contains many utilities for downloading data from different sources now we're going to put these images into a folder called sample directory samp uh into folder called generator or generated which we are putting into the variable sample directory and we're going to create this directory here so if we run the code you can see that this directory gets created here it is then we have this function called save samples so the save samples function is going to first generate uh a bunch of fake images using some latent tensors so we're going to give it a batch of latent tensors and it's going to generate images from the latent tensors then it it is also going to construct a file name so it's going to construct a file name generated Hyphen Images hyphen and this is going to be a number.png so it's going to be something like generated images 0000 .png 001.png 002.png and so on and the index is what is going to control that so in some sense using the index we are specifying the file name of the generated image and then we are simply going to call save image on the denormalized fake images so we are going to denormalize the batch of fake images and we're going to call save image and all save image does is converts the batch of images into a grid and then saves that as a file a PNG file so it's going to save it at that at that particular file name in that particular directory which is a generated directory and finally if we have set show equals true then it is also going to show or display those images so that's our save samples function once again if this does not make sense then what all you have to do is just create new code cells and run each line one by one in fact that is how we created the function in the first place first we used four or five cells to step by step create each of these things and then we put the code into a function okay so now for let's create a set of latent tensors that we can keep using Epoch after Epoch because we want to look at the outputs of our model so it would be good to look at the outputs on the same set of latent tensors otherwise if you also change the latent tensors you may not be able to compare the performance of the model Epoch after Epoch so let's just create a fixed set of latent tensors using torch. randn so we're going to create 64 tensors here of latent size of of latent size comma 1 comma 1 so we're not going to use a full batch of 128 we're just going to have 64 so that we can display it like this in an 8X 8 grid so let's create those fixed latent tensors which we will use after each Epoch and let's first save a set of samples before we start training so now we're calling save samples and we're calling it with the index zero and it is saving the file generated images 00.png and you can see here in generated if we open this you can see that this is the image that has been saved here and now we ready to start training the model so before we do that let's just record one more snapshot so here is a training Loop here and this training Loop this fit function is going to be very different from the fit functions we've seen so far so we are going to train the generator and discriminator in Tandem and once again to very quickly revise what that is going to look like is first we're going to train the discriminator for a batch so we take some real examples some generated examples put that into the discriminator and then get the results calculate the loss so for the real examples the prediction should be one and the generated examples of target the target for real examples is one the target for generated examples is zero We compare the targets with the predictions calculate loss using binary cross entropy and update the discriminator so that it gets better at making this differentiation then we take some random inputs put them into the generator and then we take the generated images run them through the discriminator and try to then train the generator so that the discriminator predicts one or that is the generator fools the discriminator using the fake images okay so we train the discriminator then we train the generator and then we repeat and let's see what that looks like in code so here is a fit function it simply takes the number of epoch and the learning rate and then the start index so this is the index that is used for naming the files that are saved at the end of each Epoch so first we empty the Cuda cache so this is going to remove any remaining unused data from the GPU so that we have a fresh memory of the GPU to use then we also have these lists where we can track some of the losses and scores so we have the generator losses the disc minor losses the real scores and the fake scores and then we have these optimizers so we are going to use the Adam Optimizer here and once again this is one of those things that has that is a little bit eccentric it has been found that the adom optimizer with certain configurations so you see these betas here so the Adam Optimizer with all these with these betas so these values of 0.5 and 999 works well for training this specific model and you can look through the documentation of item to understand what these betas are this is something that we leaving as an exercise for you so we creating two optimizers one for the discriminator and one for the generator so the discriminators optimizer is going to change the weights of the discriminator and the generators Optimizer is going to change the weights of the generator now we start with the epoch we start with the training and in each Epoch we get batches of data from the training data loader and you can see there is this additional wrapper called tqdm here this tqdm is a helpful widget a Jupiter widget it's a helper Library which will show a progress bar for each for the for each Epoch so as each batch comes in you will see a progress bar and let me just kick off the training here because it's going to take some time okay I think we just need to make sure that we have executed everything okay I see the train generator function has been executed so I'm just going to kick off the training here because it's going to take a while in fact let's do fewer epochs let's do only 10 epochs right now all right so now we have our uh we go through each batch of data and then for each batch we first train the discriminator so we pass in the batch of real images and note that we are not really interested in the targets that we get from the original image data set in fact there is only one class there is only one folder given to the image folder data set which is the images folder so we not there are no real labels for us to use from the data so we simply take the real images and put them into the discriminator and then we also pass the optimizer and then we call train discriminator and that's going to train the discriminator using some real images and some fake images and then return the loss the real score and the fake score then we going to train the generator so to train the generator we simply call pass in the optimizer for the generator it is then going to generate some fake images using the generator pass them through the discriminator set the target to one and calculate the loss and then update the weights of the generator okay so the discriminator gets a little better and then the generator gets a little better and then we repeat that process so then we and that's pretty much it this is the training Loop essentially then here we are simply recording the losses so we append the last Batch's loss we append the discriminator loss real score and fake score so that we can study them later and then here we are logging the losses so we are simply printing out the output at the end of the epoch so we printing out the epoch now number we printing out the total number of epo the generator loss discriminator loss real score and fake score from the last batch and finally here we are saving the sample files so we are saving Epoch plus start idx this is the index that we are passing so this is going to as the epox increase the file name is also going to increase from 0000 1 02 03 04 and so on and then we pass in the same fixed latent tensor so that at the end of each OC we generate an a set of images using the same latent tensors and then we set show to false because we do not want to display them we just want to save them to file and finally we return the entire set of losses real scores and fake score so although this fit function looks big there's really only this much code here which is the actual meat of the training Loop which is to run over the epox get real images and then train the discriminator and the generator okay so then we are going to train this model with a learning rate of 0.2 for 10 Epoch and you can see here that we've called the fit function with these 10ox and this learning rate and we've also logged these hyper parameters here using jan. log hyper params as you try different experiments make sure to log your hyper parameters for each experiment and also log the metrics that you get at the end so this is going to take a while but we have already trained this before so let me just show you what that looks like you can see that the reals the discriminator loss starts to decrease here and the generator loss is pretty high but over time you will see that the discriminator loss actually increases because the generator gets better so that's where it becomes difficult to interpret the losses where sometimes the generator loss is high sometimes the discriminator loss is high you can also see the real score of the discriminator and the fake score so the real score if that gets closer to one that means the discriminator is getting really good at differentiating between real and fake images and U if the fake score gets close to zero as well but really what is going to tell us how the training is proceeding is by looking at these generated images so if we open up this folder here and look at you can see generated images zero that was pretty bad this was just random noise but then when we look at generated images 01 things start to get interesting in just one Epoch our discriminator has been trained to differentiate between real and fake images and then the generator has been trained using the discriminator trying to fool the discriminator so you can see that the generator has started generating images which look somewhat like aneme faces you can see that there is hair here and as the latent vectors vary the color of the hair varies similarly you can see that there is a structure of the face coming together and then as the latent vectors vary the structure of the face varies as well you can see the basic formations of eyes you can see small hints of the nose and you can see small hints of the faces so our trick of generating of uh training the generator and discriminator in tandem is actually working and already just from the first Epoch you can you can start to see that our model has started to generate anime like faces although they're not very well defined right now and we let the strain for a while but if you scroll down here you will be able to see that there are some of these older um images so after the fifth Epoch you get to this point where it's starting to look much more well defined it's starting to look much richer you can see that the hair now has not only the color but it also has the shine it also has these strands uh it's still struggling with the eyes the faces are a little bit messy but it has started to generate if you look closely enough you'll also find that there are both male and female uh faces here you are seeing different colors of hair you are seeing different colors of eyes then by the 10th Epoch it gets much better so by the 10th Epoch it's getting pretty good the hair is getting really good uh you can see that there are different variations here as well so this is a lot of hair this is this is very little hair this probably looks more like a male face and you can see that you there are variations in terms of where the face is turned so this face is turned left and this face is turned right and so on still a bit messy of course and by the 20th Epoch it's getting pretty good wouldn't say perfect yet but at least some images look pretty good if you come get to the 25th Epoch so by the 25th eoch it's it's looking almost indistinguishable uh indistinguishable from actual images for instance if you pick this image you could pass off this image as something that was actually a screenshot from an anime or if you look at this one for example you can see that the there's hair there's eyes well defined and there's a nose and the face is turned in a certain direction there's even some detailing on the skin and as you continue to train your model over multiple epochs try training it for 40 EPO 50 EPO 100 Epoch and see how good the generator gets okay now it would be nice to generalize to visualize the training process by combining the sample images generated after each Epoch into a video and this is what we can do using the open CV Library so I'm not going to get into the code here but the basic idea here is that it is going to take each of the images and it is going to convert them into a training video and when we train it this is what it looks like so it starts out with random noise and then as the generator as the discriminator gets better and the generator gets better at fooling the discriminator you can see that slowly it's starting to improve Epoch by Epoch by Epoch it's getting better and better and better and it goes back and forth sometimes you will see a degradation in the quality of images that's probably because the generator still exploring the space of images that are being generated there is always this subtle inter subtle competition going on between the generator and the discriminator so keep that in mind and we can also visualize how the loss changes over time but this is one of those cases where the graph is not really going to be very helpful for us in general what we do want however is that the generator's loss should get as low as possible as the generator's loss gets lower and lower that means the generator is going to fool the discriminator better and better but at the same time we do not want the discriminator loss to get too high at the same time because we do not want to have a bad discriminator because if a bad if you have a bad discriminator then the generator does not have a lot to work with then here we can plot the real and fake scores as as well so you can see that we actually have a pretty good discriminator probably the discriminator could get a little bit worse sometimes you see here it the real score gets to around. 5 7 and the fake score gets to around. 2 or so but in general the discriminator is going doing a decent job at this point when you see something like this you may want to maybe train the generator not once but twice per batch and you can try these things you can try to see how you can uh slightly affect each of these things that the training of the generator and the training of the discriminator to get a better result okay so there's a lot here to experiment with in a generative adversarial Network so I will let you experiment here you experiment with the activation functions experiment with the learning rate experiment with the optimizer experiment with the training process itself maybe first train the discriminator for a few batches separately and then train the generator for a few batches separately see if there is some kind of a strategy that you can figure out to train the Gan faster try training for a longer time try training for 50 Epoch 100 Epoch and see if you get well defined faces and one thing you may have noted here is that it doesn't matter what data you put into this because so far we've not used any information about the actual data set here so you could replace the anime faces with a data set of human faces you could replace it with a data set of handwritten digits you could replace it with a data set of paintings and art is another very interesting thing that you can generate with Gans and because art is subjective and Abstract sometimes the result are actually quite impressive you can replace it with images of landscape you can replace it with images of dogs and cats you can replace it with images of everyday objects so do experiment with this this is a very interesting field and there's a lot of creativity that you can apply here in fact a lot of artists have started using Gans to generate images and they've seen they've identified different ways of controlling the different um controlling the different different aspects of the generated images one other thing to experiment with is the latent tensors so because here we've used only 64 latent tensors but you can try to change each latent tensor you can try to vary increase and decrease each specific dimension of a latent tensor and see how that affects the image so use this notebook as a playground try to experiment try to let your curiosity run wild with Gans and you will find some really interesting results and if you find something interesting do write a blog post about it do let us know about it let people know on the Forum this is a open area of research right now so something even if you spend maybe a couple of days working on this you're probably going to discover something new which nobody else has so do spend some time with Gans you can see here our training has completed and we have been able to generate images for about 10 epochs so in 10 epochs we are about halfway there it's not that great but in about epox it will get pretty good so that's generative adversarial networks and if you go back to the lesson page you will be able to see here that there is another another notebook here which is to generate handwritten digits using Gans this is almost exactly the same thing in fact you can use the exact same notebook and simply change the data set instead of the instead of the anime faces data set you can use the amness data set uh but here there is A variation on the network as well if you want to see you don't have to use convolutional layers for the mnist data set you can actually just use normal linear layer so if we go down here you can see the discriminator here the discriminator is a lot simpler it is nn. sequential of nn. linear and Ley Ru and sigmoid and similarly the generator is also a lot simpler it simply contains three linear layers so you do not need to use convolutions and deconvolutions to create Gans you can actually just do it with simp simple linear layers as well you can see here here the generator starts out generating random noise but over time as we train the generator the generator gets better and better and over after 10 Epoch it looks something like this after 50 Epoch it looks like this and after about 300 Epoch and this takes it trains much faster so about 300 Epoch in it generates pretty good images of HR digits okay so do try this out as well and you can watch this video as well you can see you can study how the losses change here and you can compare how the generator discriminator performance varies between this model and the other model that we looked at so that's Gans and then there is one other topic closing topic that we will cover today as we wrap up the course and that is called transfer learning so transfer learning is a technique used for computer vision used specifically very commonly with convolutional neural networks so we'll just touch upon this briefly and as we go through this also use this um last part as sort of a tutorial for the course project on how to structure a course project so let's open up this notebook on Google collab once again what we're going to do is click on runtime and select change runtime type and make sure that we have the GPU selected here and then we're going to run the first cell so now runtime is initialized so we've learned about convolutional neural networks and if you dig further if you look up online and try to figure out what exactly a convolutional neural network learns you will learn that each kernel so in each layer you have many kernels and each kernel actually learns something specific about the image and then as you have layers the the kernels the kernels understanding also becomes layered so what do I mean by that when suppose you have an image like this so this is an image of a sign which says 60 then the first layer of kernels is going to identify things like edges so maybe one kernel May identify a edge going towards the top right another kernel May identify a vertical Edge another kernel May identify a horizontal Edge another kernel May identify slightly circular edges and so on and then as we pass them through a second layer and through the second layer once again these kernels learn slightly more higher level features so what we doing is as we increase the number of channels as we increase the number of channels the number of kernels in each layer also increases and we go from identifying lowlevel features think like edges to identifying mid-level features things like shapes so for example the circular shape the square shape um patterns and then we go to identifying more higher level features like actual let's say faces of dogs and then wheels and then cars and things like that so your convolutional neural network builds a layered understanding of your data and as you pass so as you pass your data through the neural network and as you train the model over and over uh this is what is called layer visualization if you actually look at each layer and try to see which kind of images activate a particular layer the highest you will find that in a particular layer different kernels get activated by different types of images so when it is a lower level layer which is something closer to the input the kernels are very simple they get activated by shapes and then as you go to higher and higher layers they get activated by specific patterns and IM and eventually at the topmost layers or or the outermost layers which are closest to the output each layer each kernel is responsible for identifying a specific object or a specific part of an object okay and this is a far deeper field than we can get into right now but I have provided some sources here on how a convolutional neural network learns and how to visualize what each layer of a convolutional neural network learns so do use this information here but keeping this in mind that a convolutional neural network creates a creates a layered understanding of data or and these this data is ultimately images so in some sense it is creating a layered understanding of the world so keeping this in mind there is a technique called transfer learning that we can apply and the idea behind transfer learning is very simple we take a we take a network a convolutional neural network that has been already trained on a very large data set so there is a data set called the image net data set which contains 1 million images across thousand categories and several companies and several research Labs have trained dozens of models in fact hundreds of models on the image net data set so the idea behind transfer learning is that we take one of these models which is already trained and these are called pre-trained models and we use some of the layers of these models to then train models train custom models for a custom data set that we are working with with so the idea here is that the the features learned in the lower layers the lowlevel features the mid-level features the high level features they are going to be useful for for solving any image classification problem or any computer vision problem in general the only thing that we may need to change is the classifier at the end so remember we have a linear layer at the end of each convolutional uh Network which that linear layer is simply taking the output of the last convolutional layer and then giving it converting that into a vector based on the number of classes that are a that we need to learn okay so this is what we're going to see here and for that we're going to use the Oxford trip it pets data set so this is a data set available from the fast AI course by the way which is another course that you should definitely check out course. fast.ai and this contains 37 categories so remember C4 10 we worked with 10 categories but this contains 37 categories so that's a pretty large number of categories but it only contains about 200 images per class so in total it only contains about 7,000 images and that makes it a pretty hard problem and these images also have large variations in scale pose and lighting so these are breeds of pets cats and dogs so that can already be quite difficult to differentiate from because a lot of breeds of dogs look quite similar and many breeds of cats look similar and we have only about 200 images per class compare that with C 10 where we had 60,000 images and we had only 10 classes so it seems like this should be a much harder problem than C4 10 so let's see how far we can get with this by using what is called a pre-train model so first we're going to download the data set so to download the data set here on the on this page course. fast. a/d sets you can get the URL now this is a direct URL this is not a kaggle data set so this is a direct download URL to the raw file oxer it. tgz and that's why using the download URL function from pytorch torch Vision but if this was a data set hosted on kaggle you could use open data sets and that is the idea here for your course project whether you're picking a data set from kakle or you're picking a data set from FTI you have very simple ways of downloading that data set on kaggle use the open data sets library for uh fast AI data sets use download URL or for any data set that is hosted somewhere where you have a direct link for it something that you can open in a browser and that will download the file the same URL can be used with download URL as well then once the file is downloaded we can convert it into let's download it first once the file is downloaded we can extract it out and convert it into a folder containing all the files this might take a while depending on the size of your data set so while that downloads let's see okay it looks like it's downloaded here and if you refresh the files tab you will see that there is a data folder here and in the data folder we have Oxford Tri it pet and here we have images now unlike the data sets that we have seen so far far in this there is just a single folder called images and then the actual so the actual uh class of the image is indicated here so for instance Abyssinian _ one indicates that this image belongs to the class Abyssinian and this particular image has the ID one so you can see that these are all the Abyssinian images similarly if we scroll down you can see Bengal is another breed so each of these is a breed either of cat or dog and then we have Burman and then we have Bombay and then we have a British short hair and we have a bunch of different we have 37 different classes actually so this is something very different from the image folder data set that we have seen so far you can see here that here the class is actually part of the file name so what we're going to do is we're going to create a helper function called par breed which which can pick up a file name and all that does is it is going to take the file name and it is going to split the file name at the underscores and then it's going to get the parts of the file name then it is going to ignore the last part of the file name so for instance from Main cone 52 um 52 is going to be ignored and then we are going to join it back using spaces so here for example you can see that we go from Japanese _ chinor 23 to Japanese chain which is the class of this particular file Japanese chain and now we can also view images to to to view images we can use the pil Library so from pil we import the image class and then to open uh we Define this function open image which takes a path and it is going to open up an image and then convert it into RGB so let's view an image here let's call open image and um let's call open image with Files 4 once again and let's try to show it so we import matplot lib do pipot as PLT and we can do PLT do IM show okay we we will also need to join the data directory here so we see we just say os. path. jooin data di so there you go now we have the full path of the image so this image represents a Japanese Chin and you can look through few more images here it's always a good idea to look at some images before you do any processing or before you try to train a model so now we know how to load an image and now we also know how to get the class of the image which is the breed of the dog or the cat so now the next step here is for us to create a custom data set we can no longer use the image folder data set because all the images are in the same folder there are no different folders per class but what we can do is we can create our own custom class by extending the data set class from torch so from torch. .data you get the data set class and the data set class here we should probably import it so let's import the data set class here so the data set class requires three functions to be implemented an init function a len function and a get item function so the Len function simply Returns the length of the data set and the get item function is supposed to return given an index the I element from the data set so here what we're going to do is uh to the init function or the Constructor we are going to pass in the root which is the root directory where the files are contained and then the transforms that need to be applied so we're going to store the root directory but we're also going to get the list of files in the root directory and then if the file name ends with a JPG because sometimes depending on which operating system you're on sometimes there might be some other files there as well so we only need to work with images so if the file name ends with JPG then we keep that file name so we get a list of files ending with JPG from the root directory then what we also do is we call pass breed on each file name and that gives us a list of classes for each of the files and then we convert that into a set so the set is going to remove duplicates and then we convert it back into a list so this entire line of code is going to give us a list of the unique classes that are present in the data set and then finally we also store the transform so this is the Constructor where we are getting a list of files from the root directory then in the get item function if we want to get the I item first we get the file name of the I file and then we join the root directory with the file name using os. path. jooin to get the full path of the file then we open the image using the full path of the file and whatever transforms have been passed into our into our data set uh into our Constructor here we apply those transformed on the open image and finally we also get the index of the class so from self. classes we get the index of the class name obtained from the file using par breed okay so try going through this step by step and try to understand here the idea here what we're trying to show you here is how to create a custom data set you may not always have an image folder you may have sometimes you may have all the all the files in a single folder as you have in this case and then the class may be a part of the file name sometimes you may have a CSV file so if you have a CSV file then you will need to load up the CSV file using a library like pandas and inside the get item function and inside the unit function you may want to then pass the CSV file the data frame and then from the get item function you may need to use the CSV file to actually get the class of the data now what it is is not important if you are able to write python code you should be able to figure it out and for a particular data set if you're not able to figure out how to load it in in pytorch then simply post on the Forum now the place to post is you go back on the lesson page whichever lesson you have a doubt in and click on the course discussion forum and then on the course discussion forum you can ask a question here by clicking new topic or you can also open up a particular topic and ask a question there if you see something related so please do that now you're at a point where you should be able to approach different data sets and spend some time with it it may seem difficult at first but you can see here in total to work with a custom data set it took us only about 15 lines of code you simply defined a Constructor which stores all the important information and then you have this Len function and get item function so definitely use something like this for your project if you need to so now that we have our pets data set class our custom data set class we are going to now pass it in the data directory and we're going to pass in a list of transforms that we need so here we have the transforms the first transform that we're going to do is we going to resize the images into size 224 because you have all these images in very different shapes and sizes some are tall some are broad some are square some are very large some are very small so we're going to resize them into 224 pixels x 224 pixels then we are going to add a padding of 8 pixels so we're going to add a reflection padding and then we also going to do a random crop so first we resize the image into 225x 224 and when we do that when we do resize only the shorter Edge becomes 224 the longer Edge can sometimes be longer than 224 so then we do a random crop to ensure that we get a square crop out of the image so no we no longer have a square we no longer have rectangular images now we have square images and then we pass those convert those images into tensors using torch. two tensor and finally we normalize them using stats from the image net data said so this is one important but subtle thing here we are going to use a model which has been trained using the image net data set the imaginary data set contains 1 million images from uh belonging to about a thousand classes so we're going to use a model that has been trained there and as we did for CA 10 the images the input tensors were normalized before training that model so the model understands not generic RGB images but the model only understands images that have been normalized using this mean and the standard deviation so whenever you use a pre-trained model and we'll see what that is you need to know what are the normalization statistics that were used to normalize the data when training the model because if you pass unnormalized images that will not make sense to the model because it's possible that the model has been trained with RGB values in the range minus one to 1 but if you're passing in images with which have RGB values in the range 0 to 1 the model it may not make sense to the model and hence the pre-training may not be very useful so we apply this normalization using image net stats whenever we are doing transfer learning so we do the normalization and that creates our data set for us so that creates our data set for us and you can see that our data set contains 7,390 images we can also check data set. classes here so if we just do data set. classes you can see these are all the different classes that are present in the data set let's check the length so there are about 37 classes here we can also visualize a batch from the data we can also visualize a single image so here we have defined a function called denormalize which can take an image or a batch of images and denormalize it so it takes the normalized tensors which have been normalized using the image net statistics and converts them back into normal image tensors where pixel values go in the range 0 to 1 so in the show image function we take an image tensor and we take a label and then we display the labels for instance here this is the main and its label is 25 the number and the text is Main for that specific class and this is what it looks like you can try a few more this is uh looks like this is a Kon dog looks like this is a Persian cat looks like this is a basset hound and you can see that there is some cropping happening it was possible that this was more of a rectangular image which has now been cropped into 224x 224 and you can try changing the crop size but in general you do want to keep the images Square because ultimately all of these images from a different from all of these images are going to be stacked together to create a single batch of image tensors and for the batch to be a single tensor each image needs to have the same shape and the easiest way to ensure that is to make each image a square so if you convert each image into a square of a size 224 then when you have a batch of images they can be combined into a tensor very easily so that's our data set and then we can create training and validation sets out of it so we set the validation percentage to 0.1 or 10% and that gives us the validation set size so we can now use the random split function from toss. u. data to create a training set and a validation data set and you can check their length here so you have about 6,600 images for the training set and 739 for the validation set a very small number compared to C 10 1110 is the size but at the same time we have almost 40 classes and some of these classes are hard to differentiate too and next we convert uh we create data loaders so we're going to use a batch size of 256 because on the Google collab GPU you can use a larger batch size if you get a Cuda out of memory error you may want to convert it into 128 and then restart restart the notebook once again so sometimes if you use too large a batch size that is going to slow down your training or it is going to cause an outof memory error and when you get an outof memory error you have to restart the kernel so let's now create data loaders training data loader and validation data loader and then let's create this function show batch and all of these functions from this notebook you can use these utility functions feel free to copy paste them into your notebook the idea here is not that you should be able to write all of this by yourself from scratch that is absolutely not the idea you can use whichever code you need the idea is for you to find an interesting data set go to kaggle go to fast a go to Google data set search and a bunch of other places find an interesting data set and then use this code make the necessary modifications to work with that data set so here we have the show batch function something that you can reuse and let's look at a batch of data now once again this might take some time because it is going to process a whole bunch of images and these are pretty large images so this is a batch of data you can see you have cats and dogs and different breeds and some some breeds can be very different to difficult to differentiate but I'm sure these three dogs although they look somewhat similar are from different breeds and then sometimes the cropping is a bit odd like in this case you can see that the face of the dog has gotten cropped out in this case you can see the face of the cat has gotten cropped out in this case you can hardly see it's only half of the body and that is what makes this even more difficult because these are in different shapes and we are probably Flying Blind to a large extent here when we taking Square crops one option to avoid this is to pad the images so that they become Square now this is something that you can try out I'll leave this as an exercise for you take the rectangular images and then pad them in the right fashion so that they become square and maybe you can use a zero padding maybe you can use a reflection padding that's up to you so that way you can retain the entire image inside the square but there's a lot of variety here so now we're ready to train our model and first to define a model we need to learn about transfer learning now you do not have to use transfer learning on your course project you you can train a model from scratch this is not a requirement but if you wish you can also use transfer learning it's totally up to you um so this is the part where depending on what exactly you're trying to do you may or may not use this part but the idea behind transfer learning is we take this network which has been trained on the image net data set and it has a bunch of convolutional layers then Rel layers and pooling layers then convolution then pooling convolution pooling and then finally it has some fully connected layers and it is trained to make predictions it is trained to give outputs which correspond to the classes of the image net data set now what we typically do is we take one or more layers from the very end so we take the layers at the very end sometimes there's just one fully connected layer sometimes there are two three fully connected layers so we can make that cut at any point so we make a cut at least somewhere before the final layer and we replace it with a new classifier okay so this is what it looks like in code so here we have once again our image classification based class now if you have not seen this before you can go back and review lessons 3 4 or five and you will see the exact same class being used there what this defines is this defines the training step which calculates the loss for a batch of training data it defines the validation step which calculates the loss and accuracy for a batch of validation data and it defines the validation Epoch end which calculates the overall loss for the validation set and finally defines the EPO end which uh prints out the losses and accuracies at the end of an Epoch so this is common for all image classification problems and that is we use the cross entropy laws for instance that is common we use the accuracy evaluation metric that is common and that is why we have created this Base Class and you can use this in your project so please feel free to use this and because we are working with an image classification problem all we need to do is we need to extend the image classification based class so here we're defining a model called the pets model and the pets model is going to use the image classification Base Class here so let's extend this class so we extend the image classification base class and then we need to Define just two functions the init or the Constructor function and the forward function now in the Constructor function we are going to pass in the number of classes so that's one thing and then we're also going to pass in an additional parameter called pre-train equals true and we'll see what that does now the first thing this we've seen earlier is to invoke the initializer of invoke the init function of the image classification base which itself extends nn. module from pytorch so this is a necessary line but here is where it becomes different instead of creating our own model where we would do something like nn. sequential and inside it put in a bunch of convolutional layers and maybe create residual blocks and maybe create U have activation functions have fully connected layers have flatten and a whole bunch of layers instead of doing that we are going from torch Vision we are importing models and we are using models. resnet34 so let's see what resnet 34 is so if you go to models. resnet34 and here let's just call this function this returns a model you can see that that this returns a model that is already defined for us so it looks like it has a convolutional layer then it has a batch normalization then it has a relue and then it has a max pool 2D and then it has these sequential layers so something called a basic block and anything that is called a basic block here in resnet 34 is actually a residual block so this is one residual layer which has two convolutional layers and activations and then it is going to add the input back to the output similarly you have another residual block you have another residual block then you have more residual blocks and so on so this is a model that is already implemented for us we do not need to write it and one other thing you can do is you can set pre-trained equals true so when you said pre-train equals true what that does is it also downloads you can see here that there was a short download here so it is downloading weights for this specific model called the resnet 34 model it is downloading weights from this model when it was trained on the imet data set so somebody very kindly for us has taken this model and trained it on the image net data set which contains a million images and um belonging to a thousand classes and they've trained it possibly for days or weeks and they've trained it on multiple gpus possibly on a cloud cluster somewhere and they've spent maybe thousands of dollars on it but at the end the combined the the condensed knowledge of all the images in that data set is captured within the weights of the model so the weights of the model the resnet 34 you can see this pth file this weights of the model have then been uploaded to pyo's website from at this URL and when you pass pre-train equals true pyos not only instantiates this rest net model but it also then loads the weights from this pre-trained model into resnet 34 okay so now at this point when you when you have models. resnet 34 with pre-trained set to true so this is your resnet that is trained on the image net data set so now if you pass it if you pass in an image into this resnet it can come up with a prediction and the prediction will be a number or the prediction will actually be probabilities for thousand classes so it can come up with probabilities for thousand classes and then you can interpret those probabilities and you can pick the highest probability as a prediction of the model and it is making prediction for the classes from image net now that is not very useful for us because we are working with a different data set here and we have only 37 classes not thousand so what we want to do is we want to remove the last layer of this reset 34 model and instead insert our own last output layer our own linear layer or fully connected layer as it as it is called and that will have only 37 classes so that's what we going to do here so first we create a network uh first we create model models. resnet34 and then if if pre-train was set to true in the Constructor we create the pre-train model and that we set to self. network then inside the network you can see here there are there is layer one layer two and so on and finally there are there is this FC or what is a fully connected layer it's also called a l linear layer or a feed forward layer there are different ways of calling it but it's the same thing it's nn. linear layer it takes the last 512 features that we get out of the previous layer and it converts them into thousand instead of doing this we are going to replace it so we are going to take nend we going to create a new nn. linear layer the number of input features is going to be the same as the old fully connected layers features so instead of we simply take self. network. fc. in features which is going to five which is going to be 512 and here we're going to now have the number of classes or the number of outputs set to 37 which will be passed in through the Constructor so our pets model takes the reset 34 with pre-trained weights and removes the last layer and replaces it with a fully connected layer or a linear layer with 37 classes that's it so now we have all the weights for all the layers except the last layer and we have a new layer with new last layer with randomized weights now that is a lot lot better than starting out with a model that doesn't know anything and this is because of the hierarchical learning of convolutional neural networks we know that the first convolutional layer will learn something like um something like lines and directions and shapes and the second convention layer will combine those to identify patterns and shapes and so on and by the time you get to the thir the 32nd layer 33rd layer and by the way resnet 34 is called resnet 34 because it has 34 trainable layers so that would be 33 convolutional layers and one linear layer so by the time you get to the 32nd 33rd layer it there is a lot of condensed Knowledge from a million images that you can reuse okay so that's our convolutional neural network with a pre-trained model which we have modified then we have these utilities for training on a GPU we have get default device which is going to uh give us a pointer to a device based on what is available if a GPU is available it gives us a pointer to a GPU otherwise it gives us a pointer to a CPU we have the two device function to move tensors and models to devices and then we have the device data loader helper class what that is going to do is that is going to move batches of training data into the GPU when we try to access them and then here we have our training Loop now this is something that we've covered in a lot of depth in lessons 3 4 and 5 so I will defer to that you can go back and watch those videos uh but the idea here is you can take this cell and you can copy paste it into your uh into your project as well so here we have a evaluate function which calculates for the the model its overall metrics on the validation set overall validation loss and accuracy then we here we have the fit function which can take a number of epoch and then train the model using a given learning rate on the training loader and evaluate it on the validation loader and use the given optimization function by default which is optim do SGD then we had also defined something called the fit one cycle function which applies a certain strategy for weing the learning rate so it slowly starts with a low learning rate and slowly increases it and then decreases it over time so this is something that we learned the one cycle policy is of increasing the learning rate and then decreasing it helps us train the model much faster because it helps the model first it points the model in the right direction and then it trains the model it the model makes big steps in the right direction as the learning rate increases and then the learning rate decreases as the model Narrows in on a specific optimal minimum and there towards the end we train with a very small learning rate for a longer time so then the model can really zero in to the best results so that's our fit fit one cycle function so now let's train our model so we get the device the default device and then we train our then we create our training data loader and validation data loader by wrapping the existing training and validation data loaders so the benefit of wrapping them in device data loader um which also takes the device is that every time a batch is fetched the batch is automatically moved to the GPU as well okay okay so let's start training the model here and I'm going to kick off both of these trainings for you to see so now we create the model and we create it in the exact same way that we had done so we are going to pass to it you can see pets model accepts two arguments it accepts the first argument which is the number of classes then the number of output classes we can get using Len data set. classes and then the second argument it accepts is pre-trained so first we will use pre-rain weights so if we set pre-trained equals true then the weights of resnet 34 will be downloaded and used so that all the layers except the last one have the weights from the image net training and the last one is randomly initialized and then before we train we evaluate the model on the validation set so we are evaluating it on the validation data loader and you can see that the loss is 3.8 and the validation accuracy is only about 4.55% and this is something that you would expect because there are 40 classes almost 30 37 classes so a random guess on 37 classes would only be about 2 to 3% accurate so even though we are we are using a pre-trained model since the last layer is still randomly initialized so the predictions are still random but when we train the model and here we are going to train the model using fit one cycle when we train the model all the information that is already there in the previous layers is going to be quite useful and only the last layer is going to train and change very rapidly so we train the model model for six Epoch with a maximum learning rate of 0.1 remember we using the one cycle policy of increasing and then decreasing the learning rate over the duration of the epox and then we are also going to apply some like regularization techniques like gradient clipping weight Decay and we going to use the Adam Optimizer all of these are covered once again in a lot of detail in lesson five so do go back and review it and for your project you can use the one cycle policy if you wish and if you wish you can use the normal training loop as well which has a fixed learning rate it's totally up to you so we call fit one cycle with the given epox maximum learning rate model training data loader validation data loader and the optimization functions and remember in lesson five it took us about 5 minutes to train a model from scratch on the cart and data set to get to about 90% accuracy this time we are working with far fewer images so there we had 60,000 this time here we only have about 7,000 and we are working with more classes there we had 10 classes here we have only uh we here we have 37 classes and we are working with images of different shapes rectangular images and we are working with data that is hard to differentiate we have breeds of dogs which look very similar yet within just about 2 and a half minutes you will see right now that we get to an accuracy of about 80% so you can see that we got to an accuracy of uh 79.9% so Approximately 80% in just under 3 minutes and that's pretty that's pretty good that is probably the best we have done so far in terms of uh fast training and all this has happened because of transfer learning because we are using an a model that has been trained on imet because all of the convolutional layers already know how to identify shapes and patterns and and faces and features and things like eyes and noses and faces of dogs and so on and faces of cats and so on right so only the last layer has been really trained and the rest of the layers also have been modified but not as much and you can see the difference here is when you try to train the same model from scratch so now this time we going to create a second model and once again this is going to use pets model this is going to use resnet 34 and we are going to pass in the same number of classes we are going to replace the last layer but this time we are not going to use the pre-trained weights from the image net training so if you do not have the pre-trained weights and then we evaluate the model initially it looks the same it's about 1.8% accurate once again it is as good as random initially because the last layer is randomly initialized and in fact all the layers are randomly initialized this time and this time when we train you can see that the accuracy does not really improve to the same extent so you can see after the first Epoch we get to about 0.02 and after the second Epoch we get to about um 0.05 which is 5% accuracy and then we are only about getting to 10% accuracy here you can see by the third Epoch we were already up to 20% accuracy and then it picked up really fast it went from 20% to 61% to 76% to 79% so let's wait for a couple of minutes and see what the see what happens here so there you go it is still only at about 13% and as you let this train you will notice that while the pre-trained model reached an accuracy of 80% in less than 3 minutes the model without the pre-trained weights could only reach an accuracy of 24% and this is what makes transfer learning probably the single most powerful technique in computer vision so in general in the real world whenever you're working on a project you always always always want to use a pre-trained model now for this particular course project um you may not you you can try training a model from scratch too because that's a good way to learn but when you're out there working on a real world project something that you want to deploy into the world you will you're going to get a good Head Start If you use a pre-train model so always use a pre-train model whenever you can and this training is just completed and you can see that it is about 20 3.9% accurate so now you can you know what the next steps are you if you had a test set then you can evaluate the model on the test set and Report the test set accuracy now at this point you should also look at examples you should pick examples from the test set or from the validation set and you should then make predictions on those examples now say our model is only about 80% accurate here so if it is 80% accurate that means 20% of the images from the validation set so the validation set is of size about 700 So 20% of that is about about uh 140 images so around 140 images have been misclassified so at this point you should look at each of those 140 images that have been misclassified and check why they have been misclassified is it because maybe the image got cropped in a funny way is it because that the two classes are quite confusing using to begin with is it some other reason and then you can use that information to then improve your data set or to improve your model so you can maybe start try using a different model try using res net 50 try using dense net 101 try using efficient net there are a lot of different families of pre-trained models that are available all you need to do is search for image net pre-trained models pytorch and you will end up with a whole bunch of them in fact if you just look inside models you will see a whole bunch of bunch of them so if we just do dir models you can see there are a lot of models here and there are from they are from a certain set of families you have vgg squeez net Shuffle net rest net mobile net a denet and then there are a few more which you can simply copy over from elsewhere they are parts of different libraries so do try that out do experiment with it and see how far you can get and in fact it would be a good idea to try and get to about 95% accuracy on this and in fact you can try to get to 95% accuracy in less than 5 minutes so that's a challenge for you to try out on this transfer learning notebook and also use this notebook as an inspiration for your course project if you wish you can train a model from scratch or if you wish you can train a model using transfer learning that's totally up to you but the process is exactly the same so if we come back here onto the course project so for the course project you will pick a data set of your choice and you will apply the concepts that we've learned in this course to train deep learning models end to end with py toch experimenting with different hyperparameters and metrics so you can find data sets online and we'll talk about where to find data sets then you need to understand and describe the modeling objective so first you need to explain what type of data is it are you working with images or you if you want you can also work with tabular data you can also work with text and audio although there is some figuring out you will have to do here but you have the foundation to figure out these things on how to work with text and how to work with audio and then you need to figure out what type of problem it is so it could be a classification problem where we trying to categorize objects or in inputs into different classes it could be a regression problem where you're trying to predict a single continuous number or it could be a generative modeling problem where you're trying to generate new images so you can use the Gans notebook as a starting point then you may need to clean the data if necessary you may need to just make sure that all the files are in the right formats you may need to Define your custom data set as we did today for the transer learning uh tutorial you may want to plot some graphs you may want to ask some questions on how many how many elements are there in each class which classes are small which classes are large do you see any issues with the images themselves are there any corrupted images these are all issues that will come up with real world data then you need to perform your modeling so that means you need to First Define a model you can use a pre-trained model you can use a pre-trained you can use an existing architecture like resnet 34 without the pre-trained weights or you can define a network architecture on your own all three of them will give you different learnings and what I would recommend is on the same data set try all three ideas use a pre-train model use a model without the pre-train weights and Define your own custom model with your own regularization you can then pick some hyper parameters and train the model you can use any training strategy you can you can use a a simple training strategy the fit function that we've already defined or you can use learning rate analing or scheduling something like the one cyle policy and you can even compare the two and see how how one is different from the other you can then make predictions on samples and then you can evaluate it on a test data set if you have a test data set you also need to create a train test validation split wherever possible and then you can save the models weights because you've trained you've spent a lot of time training model you can record the metrics to Jian and then you can try different hyperparameters and regularization techniques to change your model so to get a pass grade in the course project you should have trained a model and the model should be giving some good result now it doesn't have to be perfect but your model should be getting to let's say if it is 10 classes then you should be getting to at least 60% accuracy on 10 classes if you at 10% accuracy that's pretty bad that's as good as random or if you're doing a generative adversarial Network then you should be getting some something reasonable it doesn't have to be perfect once again if it is a regression problem again your mean squared error should be reasonable if you're predicting values in the range of uh let's say 100 to 200 then the error should not be so large that your prediction is so far away from the values that it doesn't make sense so try to pick a data set where you can get a good result and how do you pick data sets here are are some sources from where you can get data sets so you can go to kaggle.com sets and here you can get some public data set so you just search for image classification if that's what you want to do and you have tons of image classification data sets another good source is course. fast.ai so this is one place where you have a whole bunch of data sets now some of these are not image classification problems so some of these are more advanced problems like image segmentation or object detection now that will require to learn something new if you have the time if you have the enthusiasm do learn about object detection do learn about image segmentation and try to create a model on one of these more advanced problems then there is also this list of awesome deep learning data sets that you can use some of these you know already some of these you some of these are new some of these are very large but try not to pick too large a data set because that may take a long time for you to complete uh it normally what you want is you should your training Loops your Epoch should not last more than a second or two so even if you're working with a few thousand images you can use transfer learning or if you working with 10,000 images try to reduce the image sizes keep them small uh make sure that you are able to maybe at least train one model every day so that over a week or two you can get to a pretty good point where you have tried at least 10 15 different ideas 10 15 different sets of hyper parameters so look through these sources and then there are also a lot of these other data sets not all of them are image data set some of them are tabular data and you can apply the same techniques instead of convolutional layers you will use feed forward or linear layers uh so do do use these you can use any of the notebooks from the from this course as a reference so as a starter notebook you can click the new notebook button and create a blank notebook and then run that on collab and type the code yourself step by step but you can use any of these notebooks as a reference you can use any of the code from any of these notebooks directly you do not have to worry about plagiarism there because all of this code is reusable code but what you should not do is you should not take somebody else's project and submit it definitely pick pick a data set that you find interesting and then write your own code make your own modifications to this code to make it work with that data set do not copy paste from somebody so there's a fine distinction there but we think you can figure it out so please do that and then you can make a submission here by pasting The Notebook link and if you have the time and we know that not everybody might but if you have the time if you can put in another maybe four five 6 hours try to summarize your learnings in a blog post you can write a blog post about your course project about the data set about the model about the approaches you've tried about about the hyper parameters about the learning rate scheduling about uh transfer learning you can write a blog post about anything that has been covered in this course writing a blog post is a great way to consolidate your understanding and it is a great way to share your knowledge with others and more importantly your notebook and your blog post will be really helpful for you when you look for jobs and internships because you will be able to share your work and it will not just be a line on your resume without any evidence it will be actual project hosted on Jovian or hosted on medium or wherever you write your blog and uh whoever is viewing your profile can then look at that project and understand that you not only uh you're not just claiming to know deep learning you actually know it and you have done work on a real world project and that is going to go a long way in helping you move forward in your data science career so please do that and that is the course project you you have time to do the course project so please work on it and now a very quick recap of the course so we started out with the very basics of pyos we talked about tensors matrices vectors how to represent them we saw how to convert numpy arrays into pyos tensors and back and forth we saw how to compute we saw how to perform operations on pyo tensors and we saw how to use the autograd functionality which is to automatically compute derivatives of um results with respect to the input tensors and then we use that functionality to implement gradient descent so we used gradient we implemented gr gradient descent using a linear regression problem we tried to predict crop yields of apples and oranges in a bunch of different regions by looking at the temperature rainfall and humidity in that region first we did it from scratch using Matrix operations and then we saw how to use py to built-ins things like nn. linear things like the optimizer class loss functions and then we also we also defined our first training loop our first fit function to train a deep learning model and and you can go back and revise any of those those basic concepts whenever you need to then building on top of that we saw how to work with images so we saw how to go from working with tabular data like crop yields using temperature rainfall and humidity to working with image data and ultimately what we said was images are simply pixels and each pixel is a number so we can flatten it out into a vector of 784 pixels in this case from the amness data set 28x 28 pixel images and then we trained we uh we used the same approach we had in in the previous lecture which was the linear regression uh approach but this time we used it for classification and that is when it it's now gets called logistic regression we passed it through a linear layer by multiplying with the weights Matrix adding the bias and then for loss we used the cross entropy loss instead of the mean square error loss because this time we wanted to interpret the outputs as probabilities and we wanted to maximize the probability of the correct class so we had 10 outputs for 10 classes and then we used the cross entropy laws and once again we used gradient descent to train our model we also learned about the concept of training set and validation set so the training set is used to train the model and the validation set is used to evaluate the model because in the real world the valid the model is going to be used on data that it has not already been trained on so that's that's one thing that we did we also saw how to finally evaluate on the test set we saw how to make predictions on example images from the test set and why that is important then we moved on to feed forward neural networks we saw how you can add an activation function after a linear layer and then add another linear layer so that way we created our first hidden layer and we understood why that activation function why that nonlinearity is important because it makes our model more powerful it gives our model the power to learn nonlinear relationships which was not possible in uh linear regression or logistic regression model another capability it gives us is to increase the number of hidden layers so that we can have a layered learning which ultimately came together in convolutional neural networks but each layer can learn basic features and then the next layer can learn features based on top of the previous layer and so on and you can increase the size of each layer so the hidden size can be increased arbitrarily large so the number of parameters in the model can go from uh something that is is fixed and determined by the by the data to something that is completely flexible and you can in fact by the time we came to reset 34 the the number of parameters goes into millions so you can create models with millions of parameters even when you're working with small data sets and the way to do that is to create layer by layer and you need to have an nonlinear activation between the layers otherwise the whole thing simply collapses to a linear transformation so nonlinearity along with this power of adding multiple lay ERS and increasing the size of each layer makes neural networks really powerful for learning any kind of um any kind of function any kind of relationship between inputs and targets and that was the universal approximation theorem that we learned about we also then understood why we need to use gpus because we are doing a lot of Matrix operations and gpus or Graphics processing units or gaming cards are optimized to do Matrix operations orders of magnitude faster than CPUs and we saw how to move our data to a GPU we saw how to detect a GPU and we saw how to move our model to a GPU and train on the GPU then we learned about the convolution operation so we saw some of the limitations of our feed forward neural networks where we teaching we're treating each pixel as an independent entity although that is not the case pixels come together in a patch to create patterns and then patterns come together to create higher level patterns and that is how an image contains information which we can perceive so the convolution ution operation the kernel is something that can capture that it is a single patch that passes over the entire image and each kernel in a convolution starts to learn something about the image and as we layer convolutional layers we as we add more and more convolutional layers each layer uh builds a higher level understanding the lower level the lower layers learn edges and shapes and curves the higher layers learn patterns the even higher layer learns things like eyes and noses and objects and finally as you come to the final ler that learns to recognize specific objects from specific classes of the training data set then we looked at how to use some techniques to improve convolutional layers because one problem that we run into often is overfitting that the model gets really good on the training data set but is unable to give the same level of accuracy on the validation data set because it starts to memorize examples from the training data set and the way to counter that is regularization so we saw how to use techniques like data augmentation data normalization to not only counter overfitting but also train the model faster uh then we uh looked at regularization techniques like batch normalization and weight Decay and gradient clipping and Dropout and we also used residual networks so residual networks add the input of a convolution layer back into the output and what that does is it allows the layer to Simply learn the residual or the change between the input and the output rather other than an entire transformation which makes it more powerful and using residual networks with regularization techniques like batch normalization we can create really deep neural networks which can sometimes go into hundreds of convolutional layers and then using the right data augmentation and regularization we can TR we can train these models really really quickly so we got to about a 90% accuracy on the C4 10 data set in less than 5 minutes which is pretty close to the stateof the-art and finally today we looked at generative aders IAL networks where we took all the things that we had learned over the course of the 6 weeks five weeks five and a so weeks and applied them to this problem of generating generative modeling and that tells you how versatile neural networks are that you just need to change a few things here and there and you can point it into towards any data set and solve many different kinds of problems we saw how to generate images of anime faces and they got pretty good in just about 10 to 15 Epoch and then we saw how the same approach can be used to generate images of handwritten digits all we need to do is change the data set and then we also looked at transfer learning a really powerful approach that makes deep learning so powerful where models that have been trained on a really large data set like image net with millions of images and thousands of classes over days and weeks by spending thousands of dollars the pre-trained weights of those models can be used in models that we create all we need to do is we need to customize the final layer to point uh to Simply give the kind of outputs that we want so we took a reset 34 um model and then we removed its final layer and replaced it with a new fully connected or linear layer which could which had 37 outputs for the data set that we were working with the Oxford triple it pets data set if you had a different data set you would have a different number of outputs um but you can change that and that is where we are right now so in 6 weeks we've covered a lot of material and I hope you've been working through the assignments if not you can still do them definitely do the assignments it's not enough to just watch the lectures uh so definitely do the assignments definitely work on a course project while you have all this information in the in your head it is the best time for you to consolidate and improve your understanding by doing a course project so definitely do it within the next few weeks because after that you may not really have the same level of understanding and then you may never find time to get back and review all the lectures or the um videos you have the lectures you have the videos you have the Jupiter notebooks you can use all the code that you need from any of these notebooks as a reference you have sources for data sets you have you can go to kaggle you can go to fast aai wherever you need to go so review the lecture videos run the jupyter notebooks on collab complete the course Project Share your work and exchange feedback back sharing is a very important part because we are learning this all around the world it's important that we have a community that we interact with so please participate in the Forum discussions it's not only to ask questions but also to help other people and also to just feel motivated that there are so many people around the world who are taking this step which is a a very big step of trying to learn something as complex as deep learning on your own on the side while you are already holding a job or already holding already work uh studying something else so sometimes it it's natural to feel down or demotivated sometimes and that is where the community can really help you so do try and participate in the Forum discussions do try help each other help other people out the more you help people the more your understanding improves we have a great community and I just want to give a shout out to everybody who has been answering questions on the Forum um and some of the some of the people from the previous runs of the course have also been answering so please do help each other out you have the free code Camp Forum you have the Jovian Forum you have um you have the YouTube comments use any format of discussion that you need you can always hit us up on Twitter you can reach us on email we are always available to help you out and what to do after this course the one single best thing that you can do is to create more projects on real world data sets and on different problems that is the best way to learn learning by doing is the right way so do not try to jump to a different topic immediately do not try to do get into the into the habit of doing more courses try to create projects that is the only way you learn so do create projects one other thing you can do is you can go to kaggle and try to participate in a kaggle competition in fact we have a kaggle competition that you can try out and we'll share the link in email it is a competition that we conducted 6 months ago and but you can still make a submission to it you can still see how you're doing there do share your knowledge write a blog post write a tutorial or just answer questions on the Forum or we are also inviting people to create videos on their course projects you know you've put in a lot of work a lot of effort into your course project so we will be inviting people to create videos on the course project which we will then be sharing on social media and YouTube and on newsletters and so on so definitely if you're interested just let us know on the Forum or just write to us uh tweet at us and we will uh we will reach out to you and help you set it up and then once you have done enough work in this domain once you have also created some public repository of your projects and you can put them up on Jovian you can share on the free code Camp Forum you can share them on GitHub you can write blogs only then should you go and U do more courses U but if you want to dig deeper into specific Concepts that we've covered then few courses that I will recommend are deeplearning.ai which is a more theoretical course but it it definitely does a great job of covering all the theoretical aspects and all the mathematics behind deep learning if you want to go there although it's not entirely necessary another great course is fast.ai now fast.ai is a course built around a library fast AI which gives you a much higher level interface where you do not have to write a lot of code with just about five six lines of code you can train a deep learning model very easily and uh it is a very versatile library and it's a great course as well so do check out fast. A2 so with that I will see you on the forums and zero.com is the course website where you can find all the lessons all the videos all the material all the assignments and you can also find the link to the Forum there you can find us on Twitter on @ freecode Camp Jovian ML and I am akashian is on Twitter and I have one final an announcement to make so our next course is on data structures and algorithms in Python this is a really useful course if you are looking to improve your coding skills if you're looking to understand more about computer science theory if you've picked up some python but you don't really know how maps and lists and different algorithms within python work it is also really useful from the perspective of interviews so no matter which role you're applying for you are most likely to end up in an interview where you will be asked to code something like binary search or uh b or breadth first search or depth first traversal of a graph or sorting algorithms and things like that so this is a great course for to learn those things as well and we will have a lot of exercises which will ultimately be quite similar to the kind of questions that you will get asked in coding tests and interviews so definitely do go to py algorithms. comom and enroll for the course the course will be starting sometime in January so with that this is Akash signing off on behalf of free code camp and Jovian I hope to see you soon in the meantime continue learning continue building projects continue being active on the Forum all the best good luck and goodbye