Build a validation set with Keras

Key Takeaways

[Music] in this video we're going to be demonstrating how to use care us to create a validation set in our previous video we showed how to train the model that we've been working with in the past several videos of this playlist and we're going to continue working with the same model so as a prerequisite to this video I would recommend you watching all of the videos in my getting started with Karis playlist first so that you can fully understand where we are and what we're dealing with in this video the videos that will get you up to speed to where we are now we'll cover the prerequisites need to start working with Karos how to pre-process data for training building a Karass model and training a Karos model so assuming that these prerequisites have been met we're now going to show how to create a validation set in this video so before we show how to do that we need to discuss exactly what a validation set is and before directly answering that let's start with a bit of background so we've built our training set that contains all of our samples and during each epoch our model is going to be getting trained over and over and over again on that data and continuing to learn the features of that data so the hope is then later that you can take this model and apply it to new data and the model will be able to accurately predict on data that it's not seen before based solely on the data that it was trained on so with the validation set you're essentially taking some percentage of your existing samples in your training set and saying no you're not going to train on these samples instead you're going to validate on them so you want to train on most of the data that you have and your training set and then you want to take data that you've stored elsewhere and a validation set and have the model predict on that data during training and look and see how well it did so it's learning over and over again the features of the training set and then during each epoch when it's being trained it's going to be predicting on the validation set so moving forward you're not going to just be seeing the loss in accuracy on your training set you're also going to see loss and accuracy on your validation set as well the thing that's cool about this is that your models not seeing this validation data but or since it wasn't included in the training set so essentially it's brand-new data so the models taking what it's learned from that training and then trying to generalize and make a prediction on data that it's not seen before in the validation set and this allows you to see how well your model generalizes it also helps you make sure that you're not overfitting and over fitting is when your model is only learning the specificities to the training data and it's not able to generalize well on data outside of the training set all right so now that we've got down what a validation set is how do we actually create one so there's two ways one is that you can just have a structure that holds your validation set you would tell the fit function which is the function that we call when we say model dot fit you would tell that about your validation set just like you would the training set in labels so let's just see how that would look so here's what we're calling fit like we did in previous videos so now if I created a set called valid set I would set it equal to a structure that looks like this it would be an array and each element in this array would be a tuple that would have a sample in its corresponding label and the next element would have a sample in a corresponding label and these samples and labels are not the ones that are included in your existing training set and labels this is going to be separate data so if we did that then whenever we called model dot fit we would pass in validation data equals and then we would pass in our variable here that we called a valid set and then whenever we ran this function the model would be training on everything in this scaled trained samples here along with its corresponding train labels and it would be validating on our valid set there's another way within Kerris that you can create a validation set and it's a bit simpler to me so let's get this out of here we're not going to use that anymore and rather than specifying validation data equals valid set we are going to specify validation split equals 0.1 so what is this doing caris is going to look at the scaled trained samples along with its course labels and it's going to split out 10% in this case because I'm supposed to find point 1 but you can specify any fraction between 0 & 1 it's going to split out whatever you specify here into a validation set that's going to basically look like that validation set that we just created as an example so it's going to take this validation data the 10% that it's splitting off from the training samples it's going to hold it out and whatever it trains it's not going to be training on this 10% here it's going to be training on our scale train samples with their corresponding labels and then it's going to be validating on the 10% that you've split out so what we're going to do now is just run this cell again so before this is what we saw we saw we have loss and accuracy being displayed for each epoch and we ran 20 epochs here we had lost going steadily down and accuracy going steadily up now let's see what it looks like whenever we run the same exact set function the only thing that we've changed is that we added this validation split equals zero point 1 parameter all right so now we see we have similar output but we have two new attributes that are being shown so along with the loss and accuracy as we were seeing last time we now have Val loss and Val accuracy which is the loss in accuracy that we're getting only on this 10% of our data which is in our validation data set and if we look here we have this output that says training on 1890 samples validating on 210 samples so I have 20 100 samples total and I specified the validation split to be 10% so that's where this 210 is being calculated from so if we look now solely at the Lawson accuracy what we had last time we see it we're starting at 0.7 three on the loss and we are steadily going down until we reach about 0.3 0 on the accuracy we're starting at about 50% and we are steadily going up to again reaching about 93% all right so that is similar to what we saw last time when we didn't have a validation set now if we look at our validation loss so our loss calculated only on these 210 samples that are being validated on on each run through the data on each epoch we're going from point seven one loss and we are steadily declining until we reach about point one nine and then with the accuracy we're starting at about 50% steadily climbing until we're actually reaching about 100% accuracy on our validation set so like I said this is really good for you to be able to have to see how well your model is generalizing on data that it's not been trained on and which is ultimately able to tell you if your model is overfitting to your training set or not in this case we see that our models not overfitting and the reason why is because we have similar results both on our training loss and accuracy as well as our validation loss and accuracy so an indication of overfitting would be if our model was continuously going up in accuracy and down in loss for our training loss in accuracy but our validation loss and accuracy we're not doing so well so maybe they're stalling out or maybe the accuracy can't get past 50% that would be a good indication that our model is learning only the specificities of our training data and it's not generalizing well on data that it's not seen before now one last thing that I'd like to point out before we wrap up here is whenever we specify this validation split equals 10% recall how I mentioned last time this shuffle equals true parameter means that all of the data that your model is training on it's going to be shuffled over each epoch that's not going to be true for your validation data so the validation split is actually taking whatever you specify here in my case 10% is going to take literally the last 10% of the data that are in your training samples so it's not going to be shuffled it's going to be the same data every time for over every run and it's going to be the last 10% that is here so because of that I did need to make a quick change in how I was generating the sample data that we used here so if you were following along from the beginning and actually writing the same code and using the same data as I was and used the data that I generated in the pre-processing data video then I'm just going to scroll back up to the top of this notebook and show you the change that I made so that you can pause the video rearrange your code in a similar fashion as well and like I said the only reason I did it is so that the data at the end of my training samples was at all uniform because then our validation split wouldn't be a very accurate depiction it would all be uniform data and that our model wouldn't be able to validate really well on that so if we scroll back up here this is where I generated the sample data so I'll just leave this on the screen here so that you can maybe pause the video and compare what I have in this cell to what you wrote previously so that you can rearrange the only thing that I did actually is change the order in which these four loops run so I'll let you take a look at that let me know if you have any questions about that step in the comments below in future videos we're going to continue learning new techniques that we can use with Kara's we're going to start building more complex models and convolutional neural networks so I hope you stick around for those and I hope you found this video helpful if you did please like the video subscribe suggest and comment and thanks for watching [Music]