How to build custom Datasets for Images in Pytorch

Key Takeaways

[Music] in this video we will implement a custom data set to load our data in all the videos i've done it's just been about creating a model or saving a model uh you're using transfer learning etc but a really big problem is loading the actual data and there are a few data sets that you can load very nicely from pytorch but let's say you have your own images or maybe some data set that you downloaded online how do you actually load the data efficiently so that's the question we're going to try to answer in this video first i have some standard code here that's working except we have no way to load the data the data that we have is a data set that i've downloaded of cats and dogs and what i've stored is in a csv file i've stored the image name that's in this specific folder and also the second column here is the target so one for dog and zero for cat what we're going to start with is we're going to try to def create a class which is going to be used to to load our data set so we're going to do cats and dogs data set and we're going to inherit from data set then we're going to define an init and what we're going to send in is a csv file so the csv file i just showed a root directory which is going to be the root directory to the to the images and then we're going to have transform which is optional if we want to have it then we're going to use self.annotations to be panda read csv csv file so we're going to use panda to help us read the csv file we're not going to do anything complicated then self.groupdirectory will just be the directory and self.transform will be transformed so the really the functions that we're going to have are len so the length and then we're going to have get item and what we want get item to do is return is a get item is going to return a specific example i a specific image and corresponding target to that image so we're going to do first the length is pretty easy we're going to return len of self.annotation in our case we have 25 25 000 images uh 50 of cats 50 of logs then so what we're going to start with is we're going to find image path we're going to do os.path.join self.root directory and then we're going to do self.annotations.i look and then index comma 0. so remember we send in a particular index that's we don't choose the index that's python does for us but we're gonna do self.annotation which is the csv file we're gonna uh this the row i and the column zero right the first column was the name of the image then we're going to do image is io.imread image path then we're going to do y label is so the y label will just be at that same except the first column and all that we're going to do is we're going to do int convert to an integer and then we're going to do torch dot tensor and in between we're going to do if.self.transform we're going to do image is self.transform of image and then that's it so we've loaded the image we've done some transform which is optional uh if we send in transform it's going to do the transformation and then what we want to do is we want to return the image and the y label okay so this class we've created is for a specific it just loads one image and the corresponding target to that image now we can go back to our load of the data and we're gonna first say see we're gonna do from custom data set so that's the let's see that's the the python file let's actually make this full screen we're going to import cats and dogs data set then we're going to say data set is copy that and then we're going to send in the csv file which we called cats dogs.csv root directory will be cats dogs resized and transform the transform we will use is since we're going to load it as an image we need to convert it to a tensor so we're going to do transforms dot 2 tensor and now one thing we can do here see valid syntax be probably one less yeah then one thing we can use also is we can use train set comma test set so we can use uh for this we can we can use torch.utils.data.random split from the data set and let's do 20 000 to the training set 5000 to the test set and then we're going to define our train loader to be data loader so this is the standard way all that really we've done that's unique to our specific data set is we're going to use that class that we created now it's just pretty much the standard that we always do so we do data set equals train set batch size equals batch size shuffle equals true and then we just copy this for the test loader and test set yeah and that should be it so that's how we load data if we have images in a folder and we have a cc file that describes the image name and also the target corresponding to that specific image so let's uh so for fun what i've done here is i've imported the google net [Music] google net architecture with pre-training and yeah i didn't cover this code but it's pretty standard code we have the loss we have the optimizer we have some training here of the network and then in the end we have check accuracy on the training set and the test set yeah so let's try to train on this just for curiosity on the 5000 images in the test set we got about 96 97 accuracy which is pretty cool so hopefully you were able to follow the video on how to load data if you've gathered your own data set and if you got any question then leave them in the comment below thank you so much for watching the video and i hope to see you in another video