TensorFlow 2.0 Crash Course

Key Takeaways

hey guys and welcome to a brand new tutorial series on neural networks with python and tensorflow 2.0 now tensorflow 2.0 is the brand new version of tensorflow still actually in the alpha stages right now but it should be released within the next few weeks but because it's an alpha tensorflow has been kind enough to release us that alpha version so that's what we're gonna be working with in this tutorial series and this will work for all future versions of tensorflow 2.0 so don't be worried about that now before i get too far into this first video i just want to quickly give you an overview of exactly what i'm going to be doing throughout this series so you guys have an idea of what to expect and what you're going to learn now the beginning videos and especially this one are going to be dedicated to understanding how a neural network works and i think this is absolutely fundamental and that you have to have some kind of basis on the math behind a neural network before you're really able to actually properly implement one now tensorflow does a really nice job of making it super easy to implement neural networks and use them but to actually have a successful and complex neural network you have to understand how they work on the lower level so that's what we're going to be doing for the first few videos after that what we'll do is we'll start designing our own neural networks that can solve the very basic uh mnist data sets that tensorflow provides to us now these are pretty straightforward and pretty simple but they give us a really good building block on understanding how the architecture of a neural network works what are some of the different activation functions how you can connect layers and all of that which will transition us nicely into creating our own neural networks using our own data for something like playing a game now personally i'm really interested with neural networks playing games and i'm sure a lot of you are as well and that's what i'm going to be aiming to do near the end of the series on kind of our larger project i'll be designing a neural network and tweaking it so it can play a very basic game that i've personally designed in python with pygame now with that being said that's kind of it for what we're going to be doing in this series i may continue this on future uh in later videos and do like very specific neural network series maybe a chat bot or something like that but i need you guys to let me know on what you'd like to see in the comments down below with that being said if you're excited about the series make sure you drop a like on this video and subscribe to the channel to be notified when i post the new videos and with that being said let's get into this first video on how a neural network works and what a neural network is so let's start talking about what a neural network is and how they work now when you hear neural network you usually think of neurons now neurons are what compose our brain and i believe don't quote me on this we have billions of them in our body or in our brain now the way that neurons work on a very simple and high level is you have a bunch of them that are connected in some kind of way so let's say these are four neurons and they're connected in some kind of pattern now in this case our pattern is completely like uh like random we're just arbitrary we're just picking a connection but this is the way that they're connected okay now neurons can either fire or not fire so you need to be on or off just like a one or zero okay so let's say that for some reason this neuron decides to fire maybe you touch something maybe you um smelt something something fires in your brain and this neuron decides to fire now it's connected to in this case all of the other neurons so what it will do is it will look at its other neurons and the connection and it will possibly cause its connected neurons to fire or to not fire so in this case let's say maybe what this one firing causes this connected neuron to fire this one to fire and maybe this one was already firing and now it's decided it turned it off or something like that okay so that's what happened now when this neuron fires well it's connected to this neuron and it's connected to this neuron well it's already got that connection but let's say that maybe when this one fires it causes this one to unfire because it was just fired something like that right and then this one now that it's off it causes this one to fire back up and then it goes it's just a chain of firing and unfiring and that's just kind of how it works right firing and unfiring now that's as far as i'm going to go into explaining neurons but this kind of gives us a little bit of a basis for a neural network now a neural network essentially is a connected layer of neurons or connected layers so multiple of neurons so in this case let's say that we have a first layer we're going to call this our input layer that has four neurons and we have one more layer that only contains one neuron now these neurons are connected now in our neural network we can have our connections happening in different ways we can have each uh what he called neuron connected to each other neuron so from layer to layer or we can have like some connected to others some not connected some connected multiple times it really depends on the type type of neural network we're doing now in most cases what we do is we have what's called a fully connected neural network which means that each neuron in one layer is connected to each neuron in the next layer exactly one time so if i were to add another neuron here then what would happen is each of these neurons would also connect to this neuron one time so we would have a total of eight connections because four times two is eight right and that's how that would work now for simplicity stake we're just going to use um one neuron in the next layer just to make things a little bit easier to understand now all of these connections have what is known as a weight now this is in a neural network specifically okay so we're going to say this is known as weight one this is known as weight two this is weight three and this is weight four and again just to re-emphasize this is known as our input layer because it is the first layer in our connected layers of neurons okay and going with that the last layer in our connected layer of neurons is known as our output layer now these are the only two layers that we really concern ourselves with when we look and use a neural network now obviously when we create them we have to determine what layers we're going to have and the connection type but when we're actually using the neural network to make predictions or to train it we're only concerning ourselves with the input layer and the output layer now what does this do and how do these neural networks work well essentially given some kind of input we want to do something with it and get some kind of output right in most instances that's what you want input results in an output in this case we have four inputs and we have one output but we could have a case where we have four inputs and we have 25 outputs right it really depends on the kind of problem we're trying to solve so this is a very simple example but what i'm going to do is show you how we would or how a neural network would work to train a very basic snake game so let's look at a very basic snake game so let's say this is our snake okay and this is his head um actually yeah let's say this is his head but like this is what the position of the snake looks like where this is the tail okay we'll circle the tail now what i want to do is i want to train a neural network that will allow this snake to stay alive so essentially its output will be what direction to go in or like to follow a certain direction or not okay essentially just keep this snake alive that's what i want it to do now how am i going to do this well the first step is to decide what our input is going to be and then to decide what our output is going to be so in this case i think a clever input is going to be do we have something in front of the snake do we have something to the left of the snake and do we have something to the right of the snake because in this case all that's here is just the snake and he just needs to be able to survive so what we'll do is we'll say okay is there something to the left yes no something in front yes no so zero one something to the right yes no and then our last input will be a recommended direction for the snake to go in so the recommended direction could be anything so in this case maybe we'll say the recommended direction is left and what our output will be is whether or not to follow that recommended direction or not or to try to do a different uh recommendation essentially or go to a different direction so let's do one case on how we would expect this neural network to perform without train like once it's trained right based on some given input so let's say there's not something to the left so we're going to put a 0 here because this one will represent if there's anything to the left the next one will be front so we'll say well there's nothing in front the next one will be to the right so we'll say right and we'll say yes there is something to the right of the snake and our recommended direction can be anything we'd like so in this case we say the recommended direction is left and the way we'll do the recommended direction is negative 1 0 1 where negative 1 is left 0 is in front and 1 is to the right okay so we'll say in this case our recommended direction is negative 1 and we'll just denote this by direction now our output in this instance should either be a zero or a one representing do we follow the recommended direction or do we not so let's see in this case following the recommended direction we keep our snake alive so we'll say one yes we will follow the recommended direction that is acceptable that is fine we're going to stay alive when we do that now let's see what happens when we change the recommended direction to be right so let's say that we say one as our recommended direction again this is dern here then what should our output be well if we decide to go right we're gonna crash into our tail which means that we should not follow that direction so our output should be zero so i hope you're understanding how we would expect this neural network to perform all right so now how do we actually design this neural network how do we get this work how do we train this right well that is a very good question and that is what i'm going to talk about now so let me actually just erase some of this stuff so we have a little bit more room to work with some math stuff right here but right now what we start by doing is we start by designing what's known as the architecture of our neural network so we've already done this we have the input and we have the output now each of our inputs is connected to our outputs and each of these connections has what's known as a weight now another thing that we have is each of our input neurons has a value right we had in this case we either had 0 or we had 1. now these values can be different right these values can either be decimal values or they can be like between 0 and 100 they don't have to be just between 0 and 1 but the point is that we have some kind of value right so what we're going to do in this output layer to determine what way we should go is essentially we are going to take the weighted sum of the values multiplied by the weights i'm going to talk about how this works more in depth in a second but just just follow me now so what this symbol means is take the sum and what we do is i'm going to say in this case i which is going to be our variable and i'll talk about how this kind of thing works in a second we'll say i equals 1 and i'm going to say we'll take the weighted sum of in this case value i multiplied by weight i so what this means essentially is we're going to start at i equals 1 we're going to use i as our variable for looping and we're going to say in this case we're going to do v1 times vi or sorry vi times wi and then we're going to add all those so what this will return to us actually will be v1 w1 plus v2 w2 plus b3 w3 plus v4 w4 and this will be uh our output that's that's what our output layer is going to have as a value now this doesn't really make um much sense right now right like why why are we doing this weights what is this multiplication well just follow with me for one second so this is what our output layer is going to do now there's one thing that we have to add to this as well and this is what is known as our biases okay so what we're going to do is we're going to take this weighted sum but we're also going to have some kind of bias on each of these weights okay and what this bias is known as it's denoted by c typically but essentially it is some value that we just automatically add or subtract it's a constant value for each of these weights so we're going to say all of these these connections have a weight but they also have a bias so we're going to have b1 b2 b3 and b4 uh well we'll call it b instead of c so what i'll do here is what i'm also going to do is i'm also going to add these biases in when i do these weights so we're going to say bi as well so now what we'll have is we'll have at the end here plus bi or plus b1 plus b2 plus b3 plus b4 now again i know you guys are like what the heck am i doing this makes no sense it's about to make sense in one second so now what we need to do is we need to train the network so we've understood now this is essentially what this output layer is doing we're taking all of these um weights and these values we're multiplying them together and we're adding them and we're taking what's known as the weighted sum okay but how do we like what are these values how do we get these values and how is this going to give us a valid output well what we're going to do is we're going to train the network on a ton of different information so let's say we play 1 000 games of snake and we get all of the different inputs and all the different outputs so what we'll do is we'll randomly decide like a recommended direction and we'll just take the state of the snake which will be either is there something to left to the right or in front of it and then we'll take the output which will be um like did the snake survive or did the snake not survive so uh what we'll do is we'll we'll train the network using that information so we'll generate all of this different information and then train the network and what the network will do is it will look at all of this information and it will start adjusting these biases and these weights to properly get a correct output because what we'll do is we'll give it all this input right so let's say we give it the input again of zero one zero and maybe one like this is a random input and let's say the output for this case is um what do you call it so one is go to the right the output is one which is correct well what the network will do is say okay i got that correct so what i'm going to do is i'm not going to bother adjusting the network because uh this is fine so i don't have to change any of these biases i don't have to change any of these weights everything is working fine but let's say that we get the answer wrong so maybe the output was zero but the answer should have been one because we know the answer obviously because we've generated all the input and the output so now what the network will do is it will start adjusting these weights and adjusting these biases it'll say alright so i got this one wrong and i've gotten like five or six wrong before and this is what was familiar when i got something wrong so let's add one to this bias or let's multiply this weight by two and what it will do is it'll start adjusting these weights in these biases so that it gets more things correct so obviously that's why neural networks typically take a massive amount of information to train because what you do is you pass it all of this information and then it keeps going through the network and at the beginning it sucks right because it has this network just starts with random weights and random biases but as it goes through and it learns it says okay well i got this one correct so let's leave the weights and the biases the same but let's remember that this is what the way in the bias was when this was correct and then maybe he gets something wrong and it says okay so let's adjust bias one a little bit let's adjust weight one uh let's mess with these and then let's try another example and then it says okay i got this example right maybe we're moving in the right direction maybe we'll adjust another weight maybe we'll adjust another bias and eventually your goal is that you get to a point where your network is very accurate because you've given it a ton of data and it's adjusted the weights and the biases correctly so that this kind of formula here of this weighted average will just always give you the correct answer or has a very high accuracy or high chance of giving you the correct answer so i hope that kind of makes sense i'm definitely over simplifying things in how the adjustment of these weights and these biases work but it's not crazy important and we're not going to be doing any of the adjustment ourselves we're we are just going to be kind of tweaking a few things with the network so as long as you understand that when you feed information what happens is it checks whether the network got it correct or got it incorrect and then it adjusts the network accordingly and that is how the learning process works for a neural network all right so now it's time to discuss a little bit about activation functions so right now what i've actually just described to you is a very advanced technique of linear regression so essentially i was saying we're adjusting weights we're adjusting biases and essentially we're creating a function that given the inputs of like y z w or like left front right we are giving some kind of output but all we've been doing to do that essentially is just adjusting a linear function because our degree is only one right we have weights of degree one multiplying by values of degree 1 and we're adding some kind of bias and that kind of reminds you of the form mx plus b we're literally just adding a bunch of mx plus b's together which gives us like a fairly complex linear function but this is really not a great way to do things because it limits the degree of complexity that our network can actually have to be linear and that's not what we want so now we have to talk about activation functions so if you understand everything that i've talked about so far you're doing amazing this is great you understand that essentially the way that the network works is you feed information in and it adjusts these weights and biases there's a specific way it does that which we'll talk about later and then you get some kind of output and based on that output you're trying to adjust the weights and biases and and all that right so now what we need to do is talk about activation functions and what an activation function does is it's essentially a non-linear function that will allow you to add a degree of complexity to your network so that you can have more of a function that's like this as opposed to a function that is a straight line so an example of an activation function is something like a sigmoid function now a sigmoid function what it does is it'll map any value you give it in between the value of negative one and one so for example when we create this network our output might be like the number seven now this number seven well it is closer to one than it is to zero so we might deem that a correct answer or we might say that this is actually way off because it's way above one right but what we want to do essentially is in our output layer we only want our values to be within a certain range we want them to be in this case between zero and one or maybe we want them to be between negative one and one i'm saying like how close we are to zero making that decision how close we are to one something like that right so what this sigmoid activation function does it's a non-linear function and it takes any value and essentially the closer that value is to infinity the closer the output is to one and the closer that value is to negative infinity the closer that output is to negative one so what it does is it adds a degree of complexity to our network now if you don't if you're not a high level like math student or you only know like very basic high school math this might not really make sense to you but essentially the degree of something right is honestly how complex it can get if you have like a degree 9 function then what you could do is you can have some crazy kind of curve and stuff going on especially in multiple dimensions that will just make things like much more complex so for example if you have like a degree nine function you can have curves that are going like like this uh like all around here that are mapping your different values and if you only have a linear function well you can only have a straight line which limits your degree of complexity by a significant amount now what these activation functions also do is they shrink down your data so that it is not as large so for example right like say we're working with data that is like hundreds of thousands of like characters long or digits we'd want to shrink that into like normalize that data so that it's easier to actually work with so let me give you a more practical example of how to use the activation function i talked about what sigmoid does what we would do is we would take this weighted sum so we did the sum of w i v i plus b i right and we would apply an activation function to this so we would say maybe our activation function is f x and we would say f of this and this gives us some value which is now going to be our output neuron and the reason we do that again is so that when we are adjusting our weights and biases and we add that activation function and now we can have a way more complex function as opposed to just having the kind of linear regression straight line which is what we've i've talked about in my other machine learning courses so if this is kind of going a little bit over your head it may be my lack of explaining it i'd love to hear in the comments below how you think of this explanation but essentially that's what the activation function does now another activation function that is very popular and is actually used way more than sigmoid nowadays is known as rectified linear unit and what this does is it let me draw it in red actually so we can see it better is it takes all the values that are negative and automatically puts them to zero and takes all of the values that are positive and just makes them more positive essentially or like to some level positive right and what this again is going to do is it's a non-linear function so it's going to enhance the complexity of our model and just make our data points in between the range 0 and positive infinity which is better than having between negative infinity and positive infinity for when we're calculating uh error all right last thing to talk about for neural networks in this video i'm trying to kind of get everything like briefly into one long video is a loss function so this is again going to help us understand how these weights and these biases are actually adjusted so we know that they're adjusted and we know that what we do is we look at the output and we compare it to what the output should be from our test data and then we say okay let's adjust the weights and the biases accordingly but how do we adjust that and how do we know how far off we are how much to tune by if an adjustment even needs to be made well we use what's known as a loss function so a loss function essentially is a way of calculating error now there's a ton of different loss functions some of them are like mean squared error that's the name of one of them i think one is like um i can't even remember the name of this one but there's a bunch of very popular ones if you know some leave them in the comments love to hear all the different ones but anyways what the loss function will do is tell you how wrong your answer is because like let's think about this right if you get an answer of let's say maybe our output is like 0.79 and the actual answer was one well that's pretty close like that's pretty close to one but right now all we're going to get is the fact that we were 0.21 off okay so 0.21 off so adjust the weight to certain degree based on 0.21 but the thing is what if we get like 0.85 [Music] well is this like this is significantly better than 0.79 but this is only going to say that we were better by what is this 0.15 so we're still going to do a significant amount adjusting to the weights and the biases so what we need to do is we need to apply a loss function to this that will give us a better kind of degree of like how wrong or how right we were now these loss functions are again not linear loss functions which means that we're going to add a higher degree of complexity to our model which will allow us to create way more complex models and neural networks that can solve better problems i don't really want to talk about loss functions too much because i'm definitely no expert on how they work but essentially what you do is you're comparing the output to the what the output should be so like whatever the model generated based what it should be and then you're going to get some value and based on that value you are going to adjust the biases and the weights accordingly the reason we use the loss function again is because we want a higher degree of complexity they're non-linear and you know if you get 0 if you're 99 like say you're 0.1 away from the correct answer we probably want to adjust the weights very very little but if you're like way off the answer your two whole points maybe our answer is negative one we want it to be one well we want to adjust the model like crazy right because that model was horribly wrong it wasn't even close so we would adjust it way more than just like two points of adjustment right we'd adjust it based on whatever that loss function gave to us so anyways this has kind of been my explanation of a neural network i want a very i want to state right here for everyone that i am no pro on neural networks this is my understanding there might be some stuff that's a little bit flawed or some areas that i skipped over and quickly actually because i know some people probably gonna say this when you're creating neural networks as well you have another thing that is called hidden layers so right now we've only been using two layers but in most neural networks what you have is a ton of different input neurons that connect to what's known as a hidden layer or multiple hidden layers of neurons so let's say we have like an architecture maybe that looks something like this so all these connections and then these ones connect to this and what this allows you to do is have way more complex models that can solve way more difficult problems because you can generate different combinations of inputs and hidden what is known as hidden layered neurons to solve your problem and have more weights and more biases to adjust which means you can on average be more accurate um to produce certain models so you can have crazy neural networks that look something like this but with way more neurons and way more layers and all this kind of stuff i just wanted to show a very basic network today because i didn't want to go in and talk about like a ton of stuff especially because i know a lot of people that watch my videos are not pro math guys are just trying to get a basic understanding and be able to implement some of this stuff now in today's video what we're going to be doing is actually getting our hands dirty and working with a bit of code and loading in our first data set so we're not actually going to do anything with the model right now we're going to do that in the next video this video is going to be dedicated to understanding data the importance of data how we can scale that data look at it and understand how that's going to affect our model when training the most important part of machine learning at least in my opinion is the data and it's also one of the hardest things to actually get done correctly training the model and testing the model and using it is actually very easy and you guys will see that as we go through but getting the right information to our model and having it in the correct form is something that is way more challenging than it may seem with these initial data sets that we're going to work with things are going to be very easy because the data sets are going to be given to us but when we move on into future videos to using our own data we're going to have to pre-process it we're going to put it in its correct form we're going to have to send it into an array we're going to have to make sure that the data makes sense so we're not adding things that shouldn't be there or we're not omitting things that need to be there so anyways i'm just going to quickly say here that i am kind of working off of this tensorflow 2.0 tutorial that is on tensorflow's website now i'm kind of gonna stray from it quite a bit to be honest but i'm just using the data sets that they have and a little bit of the code that they have here because it's a very nice introduction to machine learning and neural networks but there's a lot of stuff in here that they don't talk about and it's not very in-depth so that's what i'm kind of going to be adding and the reason why maybe you'd want to watch my version of this as opposed to just reading this off the website because if you have no experience with neural networks it is kind of confusing some of the stuff they do here and they don't really talk about why they use certain things or whatnot so anyways the data set we're going to be working with today is it's known as the fashion mnist dataset so you may have heard of the old dataset which is image image classification but it was like digits so like you had digits from zero to nine and the neural network classified digits this one's a very similar principle except we're going to be doing it with like t-shirts and pants and um what do you call like sandals and all that so these are kind of some examples of what the images look like and we'll be showing them as well in uh in the code so that's enough of that i just felt like i should tell you guys that the first thing that we're going to be doing before we can actually start working with tensorflow is we obviously need to install it now actually maybe i'll grab the install command here so i don't have to copy it but this is the install command for tensorflow 2.0 so i'm just going to copy it here link will be in the description as well as on my website and you can see pink pip install hyphen q tensorflow equals equals 2.0 0.0 hyphen alpha 0. now i already have this installed but i'm going to go ahead and hit enter anyways and the hyphen q i believe just means don't give any output when you're installing so if this runs and you don't see any output whatsoever then you have successfully installed tensorflow 2.0 now i ran into an issue where i couldn't install it because i had a previous version of numpy installed in my system so if for some reason this doesn't work and there's something with numpy i would just pip uninstall numpy and reinstall so do pip uninstall numpy like that i'm obviously not going to run that but if you did that and then you tried to reinstall tensorflow 2.0 that should work for you and it should actually install its own version of the most updated version of numpy now another thing we're going to install here is going to be matplotlib now matplotlib is a nice library for just graphing and showing images and different information that we'll use a lot through this series so let's install that i already have it installed but go ahead and do that and then finally we will install pandas which we may be using in later videos uh in the series so i figured we might as well install it now so pip install pandas and once you've done that you should be ready to actually go here and start getting our data loaded in and looking at the data so i'm just going to be working in subline text and executing my python files from the command line just because this is something that will work for everyone no matter what but feel free to work in idle feel for you to work in pycharm as long as you understand how to set up your environment so that you have the necessary packages like tensorflow and all that uh then you should be good to go so let's start by importing tensorflow so import tensorflow as tf like that i don't know why it always short forms when i try to do this but anyways we're going to import uh or actually sorry from tensorflow we'll import keras now keras is an api for tensorflow which essentially just allows us to write less code uh it does a lot of stuff for us like you'll see when we set up the model we use keras and it'll be really nice and simple and just like a high level api and that's the way that they describe it that makes things a lot easier for people like us that aren't going to be defining our own tensors and writing our own code from scratch essentially now another thing we need to import is numpy so we're going to say import if i could get this here import numpy as np and finally we will import uh matplotlib so mat plot lib in this case dot pi plot as plt and this again is just going to allow us to graph some things here all right so now what we're going to do is we're actually going to get our data set loaded in so the way that we can load in our data set is using keras so to do this i'm just going to say data equals in this case care as dot datasets dot fashion underscore mnist uh and this is just the name of the data set there's a bunch of other data sets inside of keras that we will be using in the future now whenever we have data it's very important that we split our data into testing and training data now you may have heard this me talk about this in the previous machine learning tutorials i did but essentially what you want to do with any kind of machine learning algorithm especially a neural network is you don't want to pass all of your data into the network when you train it you want to pass about 90 80 percent of your data to the network to train it and then you want to test the network for accuracy and making sure that it works properly on the rest of your data that it hasn't seen yet now the reason you'd want to do this and a lot of people would say why don't i just give all my data to the network it'll make it better not necessarily and that's because if you test your data on if you test your network on data it's already seen then you can't be sure that it's not just simply memorizing the data it's seen right for example if you show me five images um and then like you tell me the classes of all of them and then you show me that the same image again you say what's the class and i get it right well did i get it right because i figured out how to analyze the images properly or because i'd already seen it and i knew what it was right i just memorized what it was so that's something we want to try to avoid with our models so whenever we have our data we're going to split it up into testing and training data and that's what we're going to do right here so to do this i'm going to say train in this case train underscore images and train underscore labels comba in this case test underscore images comma test underscore labels and then we're going to say this is equal to data dot get underscore data so not get a load underscore down now the reason we can do this is just because this load data method is going to return information in a way where we can kind of split it up like this in most cases when you're writing your own models for your own data you're going to have to write your own arrays and for loops and load in data and do all this fancy stuff but keras makes it nice and easy for us just by allowing us to write this line here which will get us our training and testing data in the four kind of variables that we need so quickly let me talk about what labels are now so for this specific data set there are 10 labels and that means each image that we have will have a specific label assigned to it now if i actually i'll show you by just printing it out if i print for example train underscore labels and let's just print like the zero with uh i guess the first training label so let me just run this file so python tutorial one you can see that we simply get the number nine now this is just what is represent like the label representation so obviously it's not giving us a string but let's say if i pick for example 6 and i hit enter here you can see that the label is 7. so the labels are between 0 and 9. so 10 labels in total now the thing is that's not very useful to us because we don't really know what label 0 is what label 9 is so what i'm going to do is create a list that will actually define what those labels are so i'm going to have to copy it from here because i actually don't remember the labels but you can see it says here what they are so for example label 0 is a t-shirt label 1 is a trouser 9 is an ankle boot and you can see what they all are so we just need to define exactly this list here so class names so that we can simply take whatever value is returned to us from the model of what label it thinks it is and then just throw that as an index to this list so we can get what label it is all right sweet so that is um how we're getting the data now so now i want to show you what some of these images look like and talk about the architecture of the neural network we might use uh in the next video so i'm going to use pi plot just to show you some of these images and explain kind of the input and the output and all of that so if if you want to show an image using matplotlib you can do this by just doing plt dot im show and then in here simply putting the image so for example if i do train not labels images and let's say we do the seventh image and then i do plt.show if i run this now you guys will see what this image is so let's run this and you can see that we get uh this is actually i believe like a pullover or a hoodie now i know it looks weird and you've got all this like green and purple that's just because of the way that kind of matplotlib shows these images if you want to see it properly what you do is i believe you do cmap equals in this case uh plt.c i think it's like cm.binary or something i gotta have a look here because i forget uh yeah cm.binary so if we do this and now we decide to display the image it should look a little bit better let's see here uh and there you go we can see now we're actually getting this like black and white kind of image now this is great and all but let me show you actually what our image looks like so like how was i just able to show like how was i just able to do this image well the reason i'm able to do that is because all of our images are actually arrays of 28 by 28 pixels so let me print one out for you here so if i do train underscore images let's do seven the same example here and print that to the screen i'll show you what the data actually looks like give it a second and there we go so you can see this is obviously what our data looks like it's just a bunch of lists so one list for each row and it just has pixel values and these pixel values are simply representative of i believe like how much i don't actually know the scale that they're on but uh i think it's like an rgb value but in grayscale right so for example we have like 0 to 255 where 255 is black and 0 is white and i'm pretty sure that's how getting the information in someone can correct me if i'm wrong but i'm almost certain that that's how this actually works so this is gray null but this is these are large numbers and remember i was saying before in the previous video that it's typically a good idea to shrink our data down so that it's with within a certain range that is a bit smaller so in this case what i'm actually going to do is i'm going to modify this information a little bit so that we only have each value out of 1. so we instead of having a 255 we have it out of 1. so the way to do that is to divide every single pixel value by 255. now because these train images are actually stored in what's known as a numpy array we can simply just divide it by 255 to achieve that so we'll say train images equals train images divided by 255 and we'll do the same thing here with our test images as well now obviously we don't have to modify the labels as well also because they're just between zero and nine and that's how the labels work but for images we're going to divide those values so that it's a bit nicer so now let me show you what it looks like so if i go python tutorial 1.5 pi and now you can see that we're getting these decimal values and that our shirt looks well the same but exactly like we've just shrunk down our data so it's going to be easier to work with in the future with our model now that's about it i think that i'm going to show you guys in terms of this data now we have our data loaded in and we're pretty much ready to go in terms of making a model now if you have any questions about the data please don't hesitate to leave a comment down below but essentially again the way it works is we're going to have 28 by 28 pixel images and they're going to come in as an array just as i've showed you here so these are all the values that we're going to have we're going to pass that to our model and then our model is going to spit out what class it thinks it is and those classes are going to be between 0 and 9. obviously 0 is going to represent t-shirt where 9 is going to represent ankle boot and we will deal with that all in the next video now in today's video we're actually going to be working with the neural network so we're going to be setting up a model we're going to be training that model we're going to be testing that model to see how well it performed we will also use it to predict on individual images and all of that fun stuff so without further ado let's get started now the first thing that i want to do before we really get into actually writing any code is talk about the architecture of the neural network we're going to create now i always found in tutorials that i watched they never really explained exactly what the layers were doing what they looked like and why we chose such layers and that's what i'm hoping to give to you guys right now so if you remember before we know now that our images they come in essentially as like 28 by 28 pixels and the way that we have them is we have an array and we have another array inside it's like a two-dimensional array and it has pixel values so maybe it's like 0.1 0.3 which is the grayscale value and this goes and there's times 28 in each row of these these pixels now there's 28 rows obviously because well 28 by 28 pixels so in here again we have the same thing more pixel values and we go down 28 times right and that's what we have and that's what our array looks like now that's what our input data is that's fine but this isn't really going to work well for our neural network what are we going to do we're going to have one neuron and we're just going to pass this whole thing to it i don't think so that's not going to work very well so what we need to actually do before we can even like start talking about the neural network is figure out a way that we can change this information into a way that we can give it to the neural network so what i'm actually going to do and what i mean most people do is they they do what's called flatten the data so actually maybe we'll go i can't even go back once i clear it but flattening the data essentially is taking any like interior list so let's say we have like lists like this and just like squishing them all together so rather than so let's say this is like one two three if we were to flatten this what we would do is well we would remove all of these interior arrays or list or whatever it is so we would just end up getting data it looks like one two three and this actually turns out to work just fine for us so in this instance we only had like one element in each array but when we're dealing with 28 elements in each sorry list listed array they're interchangeable just in case i keep saying those uh what we'll essentially have is we'll flatten the data so we get a list of length 784 and i believe that is because well i mean i know this is because 28 times 28 equals 784 so when we flatten that data so 28 rows of 28 pixels then we end up getting 784 pixels just one after each other and that's what we're going to feed in as the input to our neural network so that means that our initial input layer is going to look something like this we're going to have a bunch of neurons and they're going to go all the way down so we're going to have 784 neurons so let's say this is 784 i know you could probably hardly read that but you get the point and this is our input layer now before we even talk about any kind of hidden layers let's talk about our output layer so what is our output well our output is going to be a number between 0 and 9. ideally that's what we want so what we're actually going to do for our output layer is rather than just having one neuron that we used kind of in the last the two videos ago as an example is we're actually going to have 10 neurons each one representing one of these different classes right so we have 0 to 9 so obviously 10 neurons or 10 classes so let's have 10 neurons so 1 2 3 4 5 6 7 8 9 10. now what's going to happen with these neurons is each one of them is going to have a value and that value is going to represent how much the network thinks that it is each neuron so for example say we're classifying the image that looks like a t-shirt uh or maybe like a pair of pants those are pretty easy to draw so let's say this is the image we're given a little pair of pants what's gonna happen is let's say pants is like this one like this is the one it actually should be all of these will be lit up a certain amount so essentially maybe we'll say like we think it's 0.05 percent this we have like a degree of certainty that it's 10 percent this one and then it is like we think it's 75 percent pants so what we'll do when we are looking at this output layer is essentially we'll just find whatever one is the greatest so whatever probability is the greatest and then say that's the one that the network predicts is the class of the given object right so when we're training the network what we'll do essentially is we'll say okay well we're giving the pants so we know that this one should be one right this should be a hundred percent it should be one uh that's what it should be and all these other ones should be zero right because it should be a zero percent chance it's anything else because we know that it is pants and then the network will look at all this and adjust all the weights and biases accordingly so that we get it so that it lights this one up directly as one at least that's our goal right so uh once we do that so now we've talked about the input layer and the output layer now it's time to talk about our hidden layers so we could technically train a network that would just be two layers right and we just have all these inputs that go to some kind of outputs but that wouldn't really do much for us because essentially that would just mean we're just going to look at all the pixels and based on that configuration of pixels we'll point to you know these output layers and that means we're only going to have which i know it sounds only 784 times 10 weights and biases so 784 times 10 which means that we're only going to have 7840 weights right weights and biases things to adjust so what we're actually going to do is we're going to add a hidden layer inside of here now you can kind of arbitrarily arbitrarily pick how many neurons you're going to have in your hidden layer it's a good idea to kind of go off based on percentages from your input layer but what we're going to have is we're going to have a hidden layer and in this case this hidden layer is going to have 128 neurons so we'll say this is 128 and this is known as our hidden layer so what will happen now is we're going to have our inputs connecting to the hidden layer so fully connected and then the hidden layer will be connected to all of our output neurons which will allow for much more complexity of our network because we're going to have a ton more biases and a ton more weights connecting to this middle layer which maybe we'll be able to figure out some patterns like maybe it'll look for like a straight line that looks like a pant sleeve or looks like an arm sleeve maybe it'll look for concentration of a certain area in the picture right and that's what we're hoping that our hidden layer will maybe be able to do for us maybe pick on pick up on some kind of patterns and then maybe with these combination of patterns we can pick out what specific image it actually is now we don't really know what the hidden network or hidden layer is going to do we just kind of have some hopes for it and by picking 128 neurons we're saying okay we're going to allow this hidden layer to kind of figure its own way out and figure out some way of analyzing this image and then that's essentially what we're going to do so if you have any questions about that please do not hesitate to ask but the hidden layers are pretty arbitrary sorry i just dropped my pen which means that you know you can kind of experiment with them kind of tweak with them there's some that are known to be to do well but typically when you're picking a hidden layer you pick one and you typically go at like maybe 15 20 of the input size but again it really depends on the application that you're you're using so let's now actually just start working with our data and creating a model so if we want to create a model the first thing we that we need to do is define the architecture or the layers for our model and that's what we've just done so i'm going to type it out fairly quickly here and again you guys will see how this works so i'm going to say model equals in this case care keras.sequential i believe that's how you spell it and then what we're going to do is inside here put a list and we're going to start defining our different layers so we're going to say keras dot layers and our first layer is going to be an input layer but it's going to be a flattened input layer and the input underscore shape is going to be equal to 28 by 28 so remember i talked about that initially what we need to do is well we need to flatten our data so that it is uh passable to all those different neurons right so essentially oh i gotta spell shape correctly shape correctly so essentially whenever you're passing in information that's in like a 2d or 3d array you need to flatten that information so that you're going to be able to pass it to an individual neuron as opposed to like sending a whole list into one neuron right now the next layer that we're going to have is going to be what's known as a dense layer now a dense layer essentially just means a fully connected layer which means that what we've showed so far which is only fully connected neural networks that's what we're going to have so each node or each neuron is connected to every other neuron in the next network so i'm going to say layers.dense and in this case we're going to give it 128 neurons that's what we've talked about and we're going to set the activation function which we've talked about before as well to be rectify linear unit now again this activation function is somewhat arbitrary in the fact that you can pick different ones but rectifier linear unit is a very fast activation function and it works well for a variety of applications and that is why we are picking that now the next layer is going to be another dense layer which means essentially another fully connected layer sorry and we're going to have 10 neurons this is going to be our output layer and we're going to have an activation of softmax now what softmax does is exactly what i explained when showing you that kind of architecture picture it will pick values for each neuron so that all of those values add up to one so essentially it is like the probability of the network uh thinking it's a certain value so it's like i believe that it's eighty percent this uh two percent this five percent this but all of the neurons there those values will add up to one and that's what the soft max soft max function does so that actually means that we can look at the last layer and we can see the uh probability or what the network thinks for each given class and say maybe those are two classes that are like 45 each we can maybe tweak the output of the network to say like i am not sure rather than predicting a specific uh value right all right so now what we're going to do is we're going to just set up some parameters for our model so we're going to say model.compile and in this case we're going to use an optimizer of atom now i'm not really going to talk about the optimizer uh atom is typically like pretty standard especially for something like this we're going to use the loss function of sparse and in this case underscore categorical i believe i spelt that correctly and then cross entropy now if you're interested in what these do and how they work in terms like the math kind of side of them just look them up there's they're very famous and popular and they're again are somewhat arbitrary in terms of how you pick them now when i do metrics i'm going to say metrics equals accuracy and again this is just going to define what uh we're looking at when we're testing the model in this case we care about the accuracy or how low we can get this loss function to be so yeah you guys can look these up there's tons of different loss functions some of them have different applications and typically when you're making a neural network you're you'll mess around with different loss functions different optimizers and in some cases different metrics so now it is actually time to train our model so to train our model what we're going to do is model.fit and when we fit it all we're going to do is give it our train images and our train labels now we're gonna set the amount of epochs so now it's time to talk about epochs now epochs are actually fairly straightforward you've probably heard the word epoch before but essentially it means how many times the model is going to see this information so what an epoch is going to do is it's going to kind of randomly pick images and labels obviously corresponding to each other and it's going to feed that through the neural network so how many epochs you decide is how many times you're going to see the same image so the reason we do this is because the order in which images come in will influence how parameters and things are tweaked with the network maybe seeing like 10 images that are pants is going to tweak it differently than if it sees like a few better pants and a few that are a shirt and some that are sandals so this is a very simple explanation of how the epochs work but essentially it just is giving um the same images in a different order and then maybe if it got one image wrong it's going to see it again and be able to tweak and it's just a way to increase hopefully the accuracy of our model that being said giving more epochs does not always necessarily increase the accuracy of your model it's something that you kind of have to play with and anyone that does any machine learning or neural networks will tell you that they can't really like they don't know the exact number epochs they have to play with it and tweak it and see what gives them the best accuracy so anyways now it is time to actually well we can run this but let's first get some kind of output here so i'm going to actually evaluate this model directly after we run it so that we can see how it works on our test data so right now what this is doing is actually just training the model on our training data which means we're tweaking all the weights and biases um we're applying all those activation functions and we're defining like a main function for the model but if we actually want to see how this works we can't really just test it on the training images and labels for the same reason i talked about before so we have to test it on the test images and the test labels and essentially see how many it gets correct so the way we do this is we're going to say test underscore loss test underscore ac which stands for accuracy equals model dot evaluate is that how you spell it maybe and then we're going to do test images test underscore labels and i believe that is the last parameter yes it is so now if we want to see the accuracy of our model we can simply print out test underscore acc and we'll just say like tested acc just so we know because there is going to be some other metrics that are going to be printing out to us when we run this all right so now that we've done that let's actually run our file and see how this works so this is it this whole part here is all we actually need to do to create a neural network and do a model now actually let me just quickly say that this keras.sequential what this does is it means a like a sequence of layers so you're just defining them in order where you say the first layer obviously is going to be your input layer we're flattening the data then we're adding two dense layers which are fully connected to the input layer as well and that's what our model looks like and this is typically how you go about creating a neural network all right so let's run this now and see what we get so this will take a second or two to run um just because obviously there is well we have 60 000 images in this data set so you know it's got to run through them it's doing all the epochs and you can see that we're getting metrics here on our accuracy and our loss now our test accuracy was 87 so you can see that it's actually slightly lower than um what do you call it like the accuracy here oh it's the exact same oh it actually auto tested on some data sets but anyways so essentially that is um how this works you can see that the first five epochs which are these ones here uh ran and they increased typically with each epoch now again we could try like 10 epochs 20 epochs and see what it does but there is a point where the more epochs you do the actual like the less reliable your model becomes uh and you can see that our accuracy was started at 88.9 essentially and that was on like that's what it said our model accuracy was when we were training the model but then once we actually tested it which are these two lines here uh it was lower than the the tested or like the trained accuracy which shows you that you obviously have to be testing on different images because when we tested it here it said well it was 89 but then here we only got 87 right so let's do a quick uh tweak here and just see what we get maybe if we add like 10 epochs i don't think this will take a crazy long amount of time so we'll run this and see maybe if it makes a massive difference or if it starts leveling out or it starts going lower or whatnot uh so let me let this run here for a second and obviously you can see the tweaked accuracy as we continue to go i'm interested to see here if we're going to increase by much or if it's just kind of going to stay at the same level all right so we're hitting about 90 percent and let's see here 91. okay so uh we got up to 91 but you can see that it was kind of diminishing returns as soon as we ended up getting to about seven epochs even yeah even like eight epochs after this we only increased by marginal amount and our accuracy on the testing data was slightly better but again for the amount of epochs five extra epochs it did not give us a five times better result right so it's something you gotta play with and see now in today's video what we're going to be doing is just simply using our model to actually predict information on specific images and see how you can actually use the model i find a lot of tutorial series don't show you how to actually practically use the model but what's the point of creating a model if you can't use it now quickly before i get too far into the video i would just like to show you guys something that i'm super excited to announce because i've been waiting for them to come for a long time and it is the official tech with tim mugs so you guys can see them here i just want to quickly show them to you guys if you'd like to support the channel and get an awesome looking mug i actually really like them then you guys can purchase them just by i believe underneath the video it shows like the teespring link um but yeah they're awesome they look really good and the reason i've been holding out on showing them to you guys is because i wanted to wait till i received mine to make sure that it was up to quality and that it looked good enough uh to sell to you guys essentially so if you'd like to support the channel um you can get one of those if not that's fine but if you do decide to buy one please send me like a dm on twitter instagram or something and let me know so i can say thank you to you guys so anyways let's get into the video um so what i'm gonna do actually is i'm gonna uh we need to continually train the model every time we run the program which i know seems like a pain but unless we want to save the model which i guess i could actually show in this video later as well we just have to train it and then we can use it directly after so after we've you know tested this we don't need to do this evaluate anymore we are trained the model we can use it to use it we actually just need to use a method called predict but i'm going to talk about kind of how this works because it is a little finicky we're not even just finicky but just not intuitive so essentially when you want to make a prediction using the model i'm going to set up just a variable prediction here you simply use model.predict and then you pass it a list now what you would think you would do is just pass it like the input right so in this case we just pass it some input that's in the form 2828 and it would predict but that's not actually how it works when you want to make a prediction what you need to do is put whatever your input shape is inside of a list or actually well you can do it inside of the list but you can also do it inside an mp array as well like a numpy array and the reason you need to do that is because what predict does is it gives you a group of predictions so it's expecting you to pass in a bunch of different things and it predicts all of them using the model so for example if i want to do the predictions on all of my test images to see what they are i can do prediction equals model.predict test images and if i print out like prediction uh you guys will see what this looks like so let's run this here and see what we get so obviously we have to train the model each time which is a little bit annoying but we can save it later on and obviously this one runs pretty quickly so it's not a huge deal all right so there we go so now you can see this is actually what our predictions look like now this is a really weird kind of like looking prediction thing we're getting a bunch of different lists now that's because right our output layer is 10 neurons so we're actually getting an output of 10 different values and these different values are representing how much the model thinks that each picture is a certain class right so you can see we're getting like 2.6 to the e to the negative zero six which means that obviously a very small number so it doesn't think whatsoever that it's that and then i'm trying to find if we can see ones that aren't like to the e but apparently it's we didn't really get lucky enough with it showing because i just cut some of them off here but if i print out let's say like prediction zero and i guess we're gonna have to run this again i probably should have thought of that then you guys will see exactly what the prediction list looks like and i'm gonna show you how we can actually interpret this to determine what class it is because this means nothing to us we want to know is it a sandal is it a shoe is it a shirt like what is it right so there you go so this is what the list looks like so if we look through the list here we can see these are all the different probabilities that our our network is predicting so what we're actually going to do essentially is we're going to take whatever the highest number is there we're going to say that is the predicted value so to do that what we do is we say np dot arg max okay and we just put it around this list now what this does is it just gets the largest value and finds like the index of that so in this case since we have 10 neurons the first one is representing obviously t-shirt the last one is representing ankle boot it'll find whatever neuron is the largest value and give us the index of that neuron so if it's like the third neuron then it's going to give us a pullover right and and that's how that works so if we want to see the actual like name though rather than just the index then what we need to do is just take this value and pass it into class names so we'll say class underscore names and then we'll index whatever the value is that this np dot arg max prediction zero gives us right so let's run this and see what we get now all right so there we go so we can see that now we're actually getting ankle boot as our prediction which makes a lot more sense for us right rather than just giving us like that prediction array or whatever it was okay so that's great but the thing is how do we how can we validate this is actually working well what we need to do now or not what we need to do but what we should do now is show the input and then show what the predicted value is and that way we as the humans which know obviously which is which can validate that so what i'm going to do actually just set up a very basic for loop and what this for loop is going to do is loop through a few different images in our test images and show them on the screen and then also show the prediction so show what they actually are and then show the prediction as well so to do this i'm just going to say 4 i guess in this case i in range 5 and what we'll do is i'm going to say plt dot grid i'm just going to set up a very basic like plot to show the image i'm going to image show our test underscore images i right i'm going to do this cmap thing so i'm going to say cmap equals in this case plt.cm.binary which is just going to give us like the gray scale and then i'm going to say plt dot x label which just means underneath and i'm going to say is equal to actual and in this case i'm going to say plus and what do we want to do we need to get the actual label of our test image which would be in test underscore labels i and then what i'm going to do is add a header and say this is what the model predicted so to do this i'm going to say plt dot i believe it's oh sorry not header so that title and the title will simply be prediction plus in this case we're going to say prediction and then i now the reason we can do this or sorry we're going to have to literally copy this this whole arg max thing and we'll put that here except instead of zero we're going to put i and just that way it will show all of the different images right so now what i'm going to do is for each loop here i'm going to plt.show which means i'm going to show those images so we can see exactly what they look like so quick recap in case i kind of skimmed over some stuff all we're doing is setting up a way to see the image as well as what it actually is versus what the model predicted so we as the humans can kind of validate this is actually working and we see okay this is what the image and the input is and this is what the output was from the model so let's run this and wait for it to train i'll fast forward through this and then we will show all the images okay so quick fix here um i just ran this and i got an error we need to do class names and then test labels i and that's obviously because the test labels are going to have like the index of all of these so i can't just put like the number value i have to put the class names so that we get the correct thing anyways i hope that makes sense you guys let's run this now you can see that was the error i ran into again fast forward and then i will be back all right so i am back now this is a little bit butchered in how i'm actually showing it but you can see that it's saying the prediction for this was the ankle boot and it actually is an ankle boot now if i close this it'll just show four more because that's the way i've set it up so now you can see that prediction pullover it actually was a pullover all right we see we get prediction trouser it actually was a trouser and prediction trouser actual trouser uh prediction shirt actual shirt and obviously if you want to see more you could keep looping through all of these and doing that now say you just want to predict on one image well what you could do for example is uh and this is kind of a weird way what i'm about to do but you'll see let's say we wanted to just predict like what the seventh image was well then what i would do is just say test images 7 which is going to give us that 28 by 28 array and then i would just put it inside of a list so that that way it gets it's given the way that it's supposed to look but that also means that our prediction list right we're going to get is equal to this it's going to look like prediction and then it's going to have this and then inside it's going to have all those different values so it's going to have like 0.001 0.9 but it's going to be a list inside of a list so that's just something to keep in mind when you're working with these predictions because that is really the only way to do it and that this is exactly what tensorflow recommends on their website as well if you're just predicting for one item just put it inside of a list so that it's going to work fine so anyways that has kind of been it on using the model to predict stuff in future videos we'll get into a little bit more advanced stuff this was a very easy classification problem just really meant to give you an introduction and personally i think if you never worked with any machine learning stuff this is pretty cool that in a few minutes of just kind of writing a little bit of code whether you understand it or not you can create a simple model that can classify fashion items like a shirt a t-shirt and i don't know that's pretty cool to me and in future videos obviously we're going to be doing a lot cooler stuff it's going to be a little bit more advanced but hopefully you guys can stick with it i'd love to know what you guys think of the series so far so please leave a comment down below it helps me to kind of tweak my lessons and all that as we go forward if you guys enjoyed the video please leave a like and subscribe and i will see you again [Music] now in today's video what we're going to be doing is talking about text classification uh with tensorflow 2.0 now what i'm going to be doing just to be fully transparent with you guys here is following along with the actual official tutorials on the tensorflow 2.0 tutorial uh now i find that these are actually the best in terms of like kind of a structure to start with to understand very basic neural networks for some pretty simple tasks i would say and then we're gonna stray away from those we're gonna start using our own data our own networks our own architecture and we'll start talking about kind of some of the issues you have when you actually start applying these to real data so so far you guys have noticed and i've seen some comments on it already that the data is really easy to load in and even pre-processing it like in the last one we just divided everything by 255 like that's really simple in the real world your data is definitely not that nice and there's a lot of stuff that you need to play with and modify to make it actually usable so anyways we'll follow along with this one for today and essentially the way that it works is we're going to have movie reviews and we're just going to classify them either as either positive or negative now what we'll do is we'll just look at some of the movie reviews and then we'll talk about the data we'll talk about the architecture using stuff to predict some issues we might run into and all of that now i don't know how many video parts this is going to be i'm going to try to record it all at once and just split it up based on how long it takes but with that being said enough talking let's get started so what we're going to do obviously is start in our file here and again this is going to be really nice because we can just steal kind of the data from keras what we'll start by doing is just importing tensorflow as tf we're going to say from tensorflow import keras and then we're going to say import numpy as np now before i start i ran into a quick uh issue when i was actually trying to do this just following along with the official tutorial and that was that the data that i want to grab here actually doesn't work with the current version of numpy that comes with tensorflow it's on their github as an issue but anyways to fix this what we need to do is install the previous version of numpy so to do this uh what i'm actually going to do is just say pip um i think i do like pip numpy version or something because i want to see what version it is uh incorrect command let's say pip version number i want to find what version it is and then just go down uh to that version okay so i found the version of numpy uh what we're going to do now is actually just install the correct version of numpy to make it work for this tutorial now this should be fine for everything going forward and if you want to install the most recent version of numpy after doing this go feel free but to do this all i'm going to do is just say pip install and then numpy equals in this case 1.16.1 i believe the version we're using right now is 0.3 at least at the time recording this but just change it to this version and hopefully in the future they'll fix that issue so that we don't have to do this but anyways i'm going to install that um yeah you're going to have to add two equal signs and i already have this installed so that should just not do anything but you guys just make sure you do that i'll leave the command in the description now after we do that what i'm going to do is just load in the data i'm going to say data equals in this case care as dot data sets dot i m what is it i am db now i believe this stands for like some something movie database i don't really know but anyways that's what the database is and we're going to do the same thing we did in the previous tutorial which is just split this into training and testing data so to do that i'm going to say train underscore data train underscore labels comma and then in this case we'll say test underscore data and then test underscore labels equals in this case data.load underscore data now we're just going to add one thing in here which is num underscore words equals in this case 10 000. now the reason i'm doing this is because this data set contains like a ton of different words and what we're going to actually do by saying num words equals ten thousand is only take the words that are the ten thousand most frequent which means we're going to leave out words that usually are only occurring like once or twice throughout the entire data set because we don't want to throw those into our model and have things like be more difficult than they have to be and just have data that's kind of irrelevant because we're going to be comparing obviously uh movie reviews and there's some words that are only in like one review we should probably just omit them because there's nothing to really compare them to in other data sets anyways i hope that kind of makes sense but that's not super important we're just going to do num words equals uh 10 000. it also shrinks our data a little bit which makes it a bit nicer now what we're going to do next is we're actually going to show how we can display this data now if i start by actually just showing you like the train underscore data and let's pick like the zero with one so i guess the first one and i print this out to the screen so i'll just go if i could get to pyth python and i guess in this case we'll have to do i probably should just type this to start uh tutorial 2 when this actually prints out probably going to take a second here just to download the data set you can see that what we have is actually just a bunch of numbers now this doesn't really look like a movie review to me does it well what this actually is is integer encoded uh words so essentially each of these integers point to a certain word and what we've done just to make it way easier for our model to actually classify these and work with these is we've given each word one integer so in this case maybe like the word the integer one stands for something the integer 14 stands for something and all we've done is just added those integers into a list that represents where these words are located in the movie review now this is nice for the computer but it's not very nice for us if we actually want to read these words so what we have to do is find the mappings for these words and then find some way to actually display this so that you know we can have a look at it now i'll be honest here i'm just going to take this from what they have on the tensorflow website on how to do this typically you would create your own mappings for words with your own dictionary and you just already have that information but fortunately for us tensorflow already does that so to do that i'm going to say word underscore index equals in this case imdb dot get underscore word underscore index like this now what this does is actually going to give us a dictionary that has those keys and those mappings so that what we can do is well figure out you know what these integers actually mean so when we want to print it out later we can have a look at them so what i'm going to say now is word underscore index equals in this case k colon and then we're going to say what do you call it v plus 3 for k v in word underscore index dot items so i might have been incorrect here this doesn't actually give us the dictionary this just gives us like tuples that have the string and the word in them i believe and then what we're doing here is we're going to say instead of c sorry this should be v my apologies is we're going to get just break that tuple up into k and v which stands for key and value and the key will be the word the value will be obviously the integer yes that's what it will be and we're going to say four word items in index we'll break that up and then we're just going to add a bunch of different keys into our data set now the reason we're gonna start at plus three is because we're gonna have actually one key or three keys that are gonna be like special characters for our word mapping and you guys will see how those work in a second so i'm gonna start by just saying word index and in this case i'm going to put in here uh pad we're going to talk about this in a second so don't worry if you guys are kind of like what are you doing right now i'm going to say word index and in this case start equals 1 and say word underscore index in this case i believe it's like u n k yeah that's correct we're gonna say u n k equals two now u n k just stands for unknown and i'm gonna explain all this in a second but it's easier just to type it out first and we're gonna say word index in this case inside this tag unused and we're going to say equals three so what i'm doing essentially is all of the words in our training and testing data set have um like keys and values associated with them starting at one so what i'm doing is i'm just going to add three to all of those values so that what i can actually do is assign my own kind of values that are gonna stand for padding start unknown and unused so that if we get values that are not valid we can just assign them to this essentially in the dictionary now what i'm going to use for padding you guys will see in just a second essentially it's just so we can make our all our movie sets the same length so we'll add this what's known as pad tag and we'll do that by adding zero into our actual movie review list so that we're gonna make each movie review the same length and the way we do that essentially is if they're not the same length so maybe one's 100 maybe one's 200 we want all them to be 200 the 100 length movie list will for what we'll do is we'll just add a bunch of padding to the end of it to make it length 200 and then obviously our model will hopefully be able to differentiate the fact that that is padding and then we don't care about the padding and we shouldn't even bother really like looking at that right all right so now what i'm going to do is add this kind of complicated line here uh just to i don't even know why they have this to be quite honest this is the way the tensorflow has decided to do their like word mappings but apparently you need to add this reverse underscore underscore word underscore index which is equal to dictionary and then here we're going to say value comma key for key comma value uh in word underscore index i believe that's correct and what this is going to do actually sorry not word index word index.items what this is gonna do is okay i understand now now that i've typed it out you just swap all the values in the keys so that right now we actually have a dictionary that has all of the like the keys first which is going to be the word and then the values where we actually want it the other way around so we have like the integer pointing to the word because we're going to have our data set that is going to contain just integers like we've seen here and we want these integers to be able to point to a word as opposed to the other way around so what we're doing is just reversing this with a reverse word index list just our dictionary sorry essentially that's what this is doing here all right now that we've done that the last step is just to add a function and what this function will do is actually decode essentially all of this training and testing data into human readable words so there's different ways to do this again i'm just going to take this right from the tensorflow website because this part's not super important and i'd rather just you know do it quickly than spend too much time on it so we're just going to say return blank string dot join in this case we're going to say reverse word index dot get in this case we're going to say i comma question mark now what this does essentially if you don't know how the get works is we're going to try to get index i which we're going to define in a second if we can't find a value for that then what we'll do is just put question mark and that's which is a default value which means we won't crash if we're having like a key uh error in our dictionary and we're going to say for in this case i in text i don't know why where i have text typed i think i might have messed something up here so one second here oh text array is the parameter my apologies so anyways that's what this is going to do is just going to return to us essentially all of the the keys that we want or the human readable words my apologies so now what we'll do is we'll simply just print out decode review and i'm just going to give it some test data so let's say test for example 0 and i guess we're going to do test underscore data it doesn't really matter if you do train or test data but let's just have a look at test data zero and see what that actually looks like so let's run that assuming i didn't make any mistakes we should actually get some valid output in just a second this usually takes a minute to run up imdb is not defined what did i type here i typed that as data my apologies so where we say imdb which is right here we just need to replace that with data in my other file i called it imdb so that's why i made a mistake there but let's run that again and hopefully now we will get some better looking output so let's wait for this and see dict object has no attribute items this needs to be items classic typos by tim one more time third time is a charm hopefully let's see and there we go so now we can see that we're actually getting all of this decoded into well this text now i'll allow you guys to read through it but you can see that we have these kind of keys that we've added so start which is one which will automatically be added at the beginning of all of our text and then we have these unks which stand for unknown character essentially and then we don't have any other keys in here but say for example we had like some padding we had added to this we would see those pad tags as well in here so that is essentially how that works if you'd like to look at some other reviews just mess around with kind of the values in the index here throw them into decode review and then we can actually see what they look like now something to note quickly is that our reviews are different lengths now i've talked about this already but let's just compare two reviews to really test that i am not just making this up so i'm going to say test underscore data why would i have a capital here test underscore data zero so the length of test and record data is zero and the length of let's try test underscore data one just to prove to you guys that these are actually different lengths which means there's something kind of fancy we're going to have to do with that padding tag which i was talking about there so let's go into text classification let's go cmd and then python in this case tutorial2.pi now i guess we're going to get that output again which is probably what's causing this to just take a second to run you can see that we have length 68 and we have length 260. now this is not going to work for our model and the reason this doesn't work is because we need to know what our inputs shape sorry and size is going to be just like i talked about before we define the input nodes or the input neurons and the output neurons so we have to determine how many input neurons there's going to be and how many output neurons there's going to be now if we're like we don't know how large our data is going to be and it's different for each what do you call its entry then that's an issue so we need to do something to fix that so what we're gonna do is we're gonna use this padding tag to essentially set a definite length for all of our data now we could go ahead and pick the longest review and say that we'll make all of the reviews that length but what i'm going to do is just pick an arbitrary number in this case we'll just do like 250 and say that that's the maximum amount of words we're going to allow in one review which means that if you have more than 250 words in your review we're just going to get rid of all those and if you don't have 256 words or 250 words or whatever it is we're just going to add these padding tags to the end of it until eventually we reach that value so the way to do this is again using these fancy tensorflow functions now if you don't like these um functions and like what these do for you and how they just kind of save you some time go ahead and try to write them yourself and if you want help on how to do that feel free to reach out to me on discord or in the comments or whatever but i personally just use them because it saves me quite quite a bit of time in terms of like typing out the functions and i already know how to do a lot of what these functions do so for me it doesn't really make sense to just retype them out when i can just use these kind of fancy tools so what we're going to say is we're going to redefine our training and testing data and what we're going to do is just trim that data so that it's only at or kind of normalize that data so it's at 250 words so to do that i'm going to say train underscore data equals in this case keras dot pre-processing um no idea if that's how you spell it we'll have to check that in a second dot sequence dot pad underscore sequence uh so pre-processing i think that's correct i guess we'll see and then in here we have to define a few different parameters so what we'll first do is we'll give that train underscore data we're going to say value equals which will be the pad value so what we add to the end of in this case our numpy array to pad it per se and in this case we'll just use this pad tag so we'll say literally word index pad so let's copy that and put that there we're going to say our padding equals in this case post which just means we're going to pad after as opposed to before we also could pad before but that doesn't really make too much sense for this and then what we'll say is max in this case len equals and then you pick your number that you want to make all of the values equal to now tensorflow did like 256. i'm just going to do 250 and see if this makes a difference in terms of our accuracy for the model and i'm literally just going to copy this and change these values now to test underscore data instead of train underscore data and this will do the same thing on our other data set oops didn't mean to do that so test underscore data like that so quick recap here because we are at 17 minutes now essentially what we've done is we've loaded in our data we've looked at our data we've created the word mappings essentially for our data so that we can actually figure out what all these integers mean we've created a little function here that will decode the mappings for us so we just pass it a word review that's integer encoded it decodes it and then it we can print that information out to the screen to have a look at it what we've just done now is we've done what's called pre-processing our data which means just making it into a form that our model can actually accept and that's consistent and that's what you're always going to want to do with any data that you have typically it's going to take you a bit more work than what we have because it's only two lines to pre-process our data because keras kind of does it for us but for the purpose of this example that's fine all right so now that we've done that it's actually time to define our model um now i'll show you quickly just to make sure that you know you guys believe me here that this is working in terms of pre-processing pre-processing our data so it's actually going to make things the same length so we'll say train underscore data test underscore data let me just print this out to the screen so python tutorial 2. uh again we're going to get these integer mappings but we'll get the length at the end as well and another error of course we need to add an s to these sequences again my apologies guys on that um classic typos here so anyways i had pre-process processing sequence we need sequences and now if i run this you can see that we have a length of 250 and 250 so we've kept that consistent now for some oh i'm printing i don't know why this is printing two oh it's because i'm printing it here and then i'm printing it here um but you guys get the idea in that we've now made them actually the same size so let me remove these print statements um all of them so we can stop printing train data zero up here as well and now let's start defining our model so i'll just say model down here is a little comment just to help us out so what i'm going to do now is similar to what i've done before except in the last one you might have noticed that the way i defined my model was uh here i'll show you in a second once i finish typing this so we did keras.sequential and then what we actually did was just had a list in here that had all the layers that's fine you can do that but in this case we're going to have a few more layers so what we're going to do actually is add these layers just by doing model dot add it's precisely the same thing as before except instead of adding them in this list we're just gonna do it using this method so now we're gonna say keras dot layers dot in this case embedding and i'll talk about what these layers do in a second so we're gonna do ten thousand sixteen and then we're just going to actually copy this um four times and just change these layers and the kind of uh parameters as well so now we're gonna say global average pooling 1d and then do that and then we're going to add a dense layer here and another dense layer and change these parameters so we'll say dense and we'll say in this case 16 and we'll say activation equals relu our rectify linear unit whatever you guys want to call it and then we'll do down here one and activation equals rectifier linear unit as well uh actually sorry not real really we're gonna do sigmoid my apologies so now we'll actually talk about the architecture of this model and how i came up with picking these layers and well what these layers are well what we want essentially is we want the final output to be whether the review is good or whether the review is bad i think i mentioned that um at the beginning of the video so what we're actually going to do is just have either that like we'll have one output neuron and that neuron should be either zero or one we're somewhere in between there to give us kind of a probability of like we think it's like twenty percent one eighty percent zero something along those lines now we can accomplish that by using sigmoid because what it will do again we've talked about the sigmoid function is it'll squish everything so whatever our value is in between zero and one which will give us a nice way to test if our model is actually working properly and to get it the value that we want hey guys so now it's time to talk about word embeddings and this embedding layer and then what the global average pooling 1d layer is doing now we already have an idea of what these dense layers are with these activation functions like relu and sigmoid but what we're actually going to do today or i guess just in this video is talk about the architecture of this network kind of how it works on a high level understanding and then in the next video what we'll do is actually get into training and using the network so what i'm going to do first is just start by talking about these first two layers and specifically what this embedding layer is because it's very important and then we will draw the whole network or the whole i guess network is the right word way to put it the whole architecture and talk about how it fits together and what it's actually doing so let's get started now the easiest way to kind of explain this is to use an example of two very similar sentences so i'm just going to say the first sentence is have a great day and the next sentence will be have a good day now i know my handwriting is horrible so just give me a break on that um it's also hard to kind of write with this tablet so that's my excuse but anyways these two sentences looking at them as human beings we can tell pretty quickly that they're very similar now yes great and good maybe one has more emphasis on having an amazing day whatever it is but they're very similar and they pretty well have the same meaning right maybe we know when we would use the sentence and kind of the context in which like these words great and good are used in day and day and all this right it just we understand what they are now the computer doesn't have that same understanding at least right off the bat when looking at these two sentences now in our case we've actually integer encoded all of our different values so what we end up having are all of our different words sorry is our sentences end up looking something like this so we're going to have this first word will represent a 0. a will be 1 great will be 2 and a will be 3 so then down here we'll have 0 1 in this case we're going to say good is 4 and day is 3 as well so this means if we integer and code these sentences we have some lists that look something like this now this one clearly is the first sentence and this one down here will be the second sentence now if we just look at this and we pretend that you know we don't even know what these words actually are all we can really tell is the fact that two is different from four now notice what i just said there two is different from four when in reality if we look at these two words we know that they're pretty similar yes they're different words yes they're different lengths whatever it is but we know that they have a similar meaning and the context in which they're used in this sentence is the same now our computer obviously doesn't know that because all it gets to see is this so what we want to do is try to get it to have an understanding of words that have similar meanings and to kind of group those together in a similar form or in a similar way because obviously in our application here of classifying movie reviews the types of words that are used and the context in which they are used really makes a massive difference to trying to classify that as either a positive or a negative review and if we look at great and good and we say that these are two completely different words well that's going to be a bit of an issue when we're trying to do some classification so this is where our embedding layer comes in now again uh just to say here one more time like we know these are different but we also would know for example say if we replace this four with a three well all our computer again would know is that two is different from three just like four is different from two it doesn't know how different they are and that's what i'm trying to get at here is our embedding layer is going to try to group words in a similar kind of way so that we know which ones are similar to each other so let me now talk about specifically the embedding layer so let me just draw a little grid here now what are embedding layer actually does kind of like i don't want to say the formal definition but the more mathy definition is it finds word vectors for each word that we pass it or it generates word vectors and uses those word vectors uh to pass to the future layers now a word vector can be in any kind of dimensional space now in this case we've picked 16 dimensions for each word vector which means that we're going to have vectors maybe something like this and a vector again is just a straight line with a bunch of different coefficients in some kind of space that is in this case 16 dimensions so let's pretend that this is a 16 dimensional vector and this is the word vector for the word have now in our computer it wouldn't actually be have it would be zero because again we have integer encoded stuff but you kind of you get the points we'll say this is the word vector for half now what we're going to do immediately when we create this embedding layer is let me actually get out of this quickly for one second is we initially create 10 000 word vectors for every single word and in this case every single number that represents a word so what we're going to do is when we start creating this embedding layer we see that we've have an embedding layer is we're going to draw 10 000 word vectors in just kind of some random way that are just there and each one represents one word and what happens when we call the embedding layer is it's going to grab all of those word vectors for whatever input we have and use that as the data that we pass on to the next layer now how do we create these word vectors and how do we group words well this is where it gets into a bit complicated math i'm not really going to go through any equations or anything like that but i'll kind of give you an idea of how we do it now we want to so let me get rid of this word have because this is not the best word vector example and let's say that this word vector is great now upon creating our word vector our embedding layer we have two vectors we have great and we have good and we can see that these vectors are kind of far apart from each other and we determine that by looking at the angle between them and we say that this angle maybe it's like i don't know 70 degrees or something like that and we can kind of determine that great and good are not that close to each other but in reality we want them to be pretty close to each other we want the computer to look at great and good and be like these are similar words let's treat them similarly in our neural network so what we want to do hopefully is have these words and these vectors kind of move closer together whether it's good going all the way to great or great going all the way to good or vice versa right we just want them to get close together and kind of be in some form of a group so what we do is we try to look at the context in which these words are used rather than just the content of the words which would just be what this looks like we want to figure out how they how they're used so we'll look at the words around it and determine that you know when we have a and a and a and a maybe that means that these are like related in some way and then we'll try to group these words now it's way more complicated than that don't get me wrong um but it's kind of like a very basic way of how they group together we look at the words that surround it and just different properties of the sentence involving that word and then we can kind of get an idea of where these words go when which ones are close to each other so maybe after we've done some training uh what happens is our word embeddings are what is known as learned just like we're learning and teaching our neural network and we get we end up getting great and good very close together and these are what their word vector representations are we can tell that they're close again by looking at the angle in between here maybe it's like 0.2 degrees and what that means is these two vectors which are just a bunch of numbers essentially are very close together so when we feed them into our neural network they should hopefully give us a similar output at least for that specific neuron that we give it to now i know this might be a little bit confusing but i'm going to go we're going to talk about this a bit more with another drawing of the whole network but i hope you're getting the idea the whole point of this embedding layer is to make word vectors that and then group those word vectors or kind of like make them close together based on words that are similar and that are different so again just like we would have grading good here we would hope that a word vector like bad would be down here where it has a big difference from great and good so that we can tell that these words are not related whatsoever all right so that's how the embedding layer works now what ends up happening when we have this embedding layer is we get an output dimension of what's known as 16 dimensions and that's just how many coefficients essentially we have for our vector so just like if you have a 2d line so like if this is our grid in 2d and we say that this is x and this is y we can represent any line by just having like uh some values like ax plus b y equals c now this is the exact same thing that we can do in in n dimensions which means like any amount of dimensions so for a 16 dimensional line i'm not going to draw them all but we would start with like ax plus b y plus c z plus d w and so on and we would just have again 16 of these coefficients and then some kind of constant value uh maybe we call it lambda that is like what it's what it equals to what the equation equals to and that's how we define a line i'm pretty sure i'm doing this correctly in uh in n dimensions so anyways once we create that line what we actually want to do is we want to scale the dimension down a little bit now that's just because 16 dimensions is a lot of data especially when we have like a ton of different words coming into our network we want to scale it down to make it a little bit easier to actually compute and to train our network so that's where this global average pooling 1d layer comes in now i'm not going to talk about this in two depth in too much depth but essentially the way to think of the global average pooling 1d is that it just takes whatever dimension our data's in and just puts it in a lower dimension now there's a specific way that it does that but again i'm not going to talk about that and it's not super important if you care about that a lot just look it up and it's not like crazy hard but i just i don't feel the need to go into it in this video so anyways let's now start drawing what our network actually looks like after understanding how this embedding layer works so we're going to initially feed in a sequence and we'll just say that this is like our sequence of encoded words okay so say this is our input and maybe it's something like zero seven nine like a thousand two hundred a thousand twenty uh we have like nine again maybe we have eight it's just a bunch of different essentially numbers right so we're gonna pass this into our embedding layer and all this is gonna do is it's gonna find the uh representation of these words in our embedding layer so maybe our embedding layer well it's going to have the same amount of words in our vocabulary so it'll look up say zero it'll say maybe zero means zero's vector is like zero point two zero point and it goes to 16 dimensions but i'm just gonna do like two for this example here maybe seven its vector is like seven and nine point zero and it just keeps going like this and it looks up all these vectors so it takes all of our input data and it just turns them into a bunch of vectors and just spits those out into our next layer now our next layer what this does is it just takes these vectors and just averages them out and it just means it kind of shrinks them their data down so we'll do like a little smaller thing here and we'll just say like average okay so i'll call this one embedding and that one is average now this average layer now is where we go into the actual neural network well obviously this is a neural network but we go into the dense layers which will actually perform our classification so what we're going to do is we're going to start with 16 neurons and this is just again an arbitrary number that we've picked for our network you can mess around with different values for this and i encourage you to do that but 16 is what tensorflow decided to use and what i'm just following along with so we're going to have 16 neurons and we're going to pass all of our now 16 dimensional data or whatever dimensional data it is into these neurons like this now this is where we start um doing the dense layer so we have this dense layer and this is connected to one output neuron like this so what we end up having is this embedding layer which is going to have all these word vectors that represent different words we average them out we pass them into this 16 neuron layer that then goes into an output layer which will spit out a value between 0 and 1 using the sigmoid function which i believe i have to correct myself because in other videos i said it did between negative one and one it just takes any value we have and puts it in between zero and one like that all right so that is kind of how our network works so let me talk about what this dense layer is doing just a little bit before we move on to the next video so what this dense layer is going to attempt to do essentially is look for patterns of words and try to classify them using the same methods we talked about before into either a positive review or a negative review i'm going to take all these word vectors which again are going to be like similarly grouped words like great good are going to be similar input to this dense layer right because we've averaged them out and embedded them in all this and then what we're going to do is we're going to try to determine based on what words we have and what order they come in what our text is and we hope that this layer of 16 neurons is able to pick up on patterns of certain words and where they occur in the sentence and give us a accurate classification again it's going to do that by tweaking and modifying these weights and all of the biases that are on you know all of these different what do you call it layers or all of these connections or whatever they are and then it's going to give us some output and some level of accuracy for our network all right so now it's time to compile and train our model now the first thing we have to do is just define the model give it an optimizer give it a loss function and then i think we have to define uh the metrics as well so we're going to do is going to say model equals in this case or sorry not model equals model dot compile if i spell compile it correctly and then here we're gonna say uh optimizer we're gonna use the atom optimizer again i'm not really gonna talk about what these are that much if you're interested in the optimizer just look them up and then for the loss function where you're going to use the binary underscore cross entropy now what this one essentially is is well binary means like two options right and in our case we want to have two options for the output neuron which is zero or one so what's actually happening here is we have the sigmoid function which means our number is going to be between zero and one but what the loss function will do is pretty we'll calculate the difference between for example say our output neuron is like 0.2 and the actual answer was zero what will give us a certain function that can calculate the loss so how much of a difference 0.2 is from zero um and that's kind of how that works again i'm not going to talk about them too much and they're not like i mean they are important but not to really like memorize per se like you kind of just mess with different ones but in this case binary cross entropy works well because we have two possible values zero one so rather than using the other one that we used before which i don't even remember what was called something cross entropy we're using binary cross entropy okay so now what we're gonna do is we're actually gonna split our training data into two sets and the first set of our training data is going to be called validation data or really i guess you can think of it as a second the order doesn't really matter but what we're going to do is just get some validation data and what validation data is is essentially we can check how well our model is performing based on the tunes and tweaks we're doing on the training data on new data now the reason we do that is so that we can get a more accurate sense of how well our model is because we're going to be testing new data to get the accuracy each time rather than testing it on data that we've already seen before which again means that the model can't simply just memorize each review and give us either a zero or one for that it has to actually have some degree of i don't know like thinking or operation so that it can work on new data so what we're going to do is we're going to say x underscore val equals and all we're going to do is just grab the train data and we're just going to cut it to a thousand or 10 000 entries so there's actually 25 000 entries or i guess reviews in our training data so we're just going to take 10 000 of it and say we're going to use that as validation data now in terms of the size of validation data it doesn't really matter that much this is what tensorflow is using so i'm just kind of going with that but again mess with these numbers and see what happens to your model everything with our neural networks and machine learning really is going to come down to very fine what's known as hyper parameters or like hypertuning which means just changing individual parameters each time until we get a model that is well just better and more accurate so we're going to say that x val equals that but then we're also going to have to modify our x train data to be train underscore data and in this case we're just going to do the other way around so 10 000 colon now i'll just copy this and we're just going to replace this again with instead of test uh actually oh we have to do this with labels sorry what am i thinking so we're just going to train change this to be labels and then instead of x val it's just going to be y value and then y train so yeah we're not touching the test data because we're going to use all that test data to test our model and then we're just going to use the uh the training stuff or the validation data to validate the model all right so now that we've done that it is actually time to fit the model so i'm just going to say uh like fit model and you'll see why i'd name this something different in a second it's going to be equal to model.fit and in this case what we're going to do is we're going to say x underscore train y underscored train we're going to say epochs uh is equal to i that's how you spell it 40 and again you can mess with this number and see what we get based on that and we're going to say batch underscore size equals 512 which i'll talk about in a second and then finally we're going to say validation underscore data equals and in here we're going to say x underscore val y underscore val and i think that's it let me just check here quickly oh one last thing that i forgot to do we're gonna say verbose equals one verbose equals one now i'm not gonna lie i honestly don't know what verbose is i probably should have looked it up before the video but i have no idea what that is so if someone knows please let me know but the batch size is essentially how many what do you call it um movie reviews we're gonna do each time or how many we're gonna load in at once because the thing is it's kind of i mean we're loading all of our reviews into memory but in some cases we won't be able to do that and we won't be able to like feed the model all of our reviews on each single cycle so we just set up a batch size that's going to define us essentially how many at once we're going to give and i know i'm kind of horribly explaining what a batch size is but we'll get into more on batch sizes and how we can kind of do like buffering through our data and like going taking some from a text file and reading into memory in later videos when we have like hundreds of gigabytes of data that we're gonna be working with okay so finally we're gonna say results equals and in this case i believe it is model dot evaluate and then we're going to evaluate this obviously on our test data so we're going to give it test data and test labels so test underscore data test underscore labels like that and then finally what i'm going to do is just actually print out the results so we can see what our accuracy is so say print results and then get that value so let me run this quickly neural networks text classification let's go cmd and then python text or that's not even the one we're using we're using tutorial 2 sorry and let's see what we get with this this will take a second to run through the epoch so i'll fast forward through that so you guys don't have to wait all right so we just finished doing the epochs now and essentially our accuracy was 87 and this first number i believe is the loss which is 0.33 and then you can see that actually here we get the accuracy values and notice that the accuracy from our last epoch was actually greater than the accuracy on the test data which again shows you that sometimes you know when you test it on new data you're going to be getting a less accurate model or in some cases you might even get a more accurate model it really just you can't strictly go based off what you're getting on your training data you really do need to have some test and validation data to make sure that the model's correctly working so that's essentially what we've done there um and yeah i mean that that's the model we've we tested and it's 87 accurate so now let's actually have let's interpret some of these results a little bit better and let's show some reviews let's do a prediction on some of the reviews and then see like if this our model kind of makes sense for what's going on here so what i'm going to do is i'm just going to actually just copy some output that i have here um just save us a bit of time because i am going to wrap up the video in a minute here but essentially what this does it just takes the first review from test data gets the model to predict that because we obviously we didn't train on the test status we can do that fine we're going to say review and then we print out the decoded review we're going to print out what the model predicted and then we're going to print out what the actual label of that was so if i run this now i'll fast forward through the kind of training process and we'll see the all right so this is what essentially our review looks like so at least the one that we were testing it on and you can see that we have this little start tag and it says please give this one a miss for and then br stands for like break line or go to the next line so we could have actually added another tag for br uh if we noticed that this was used a lot in the review uh but we didn't do that so you see br unless this is actually part of the review but i feel like that should be like break line in terms of html anyways and then we have some unknown characters which could be anything that we just didn't know what it was and it says and the rest of the cast rendered terribly performed says the show is flat flat flat brbr i don't know how uh michael madison could have allowed this one on his plate he almost seemed he'd what does it seem to know this wasn't going to work out and his performance was quite unknown so all yeah so anyways you can see that this probably had like some emojis in it or something and that's why we have all these unknowns and then obviously we made this review which was pretty short to be the full length of 250 so we see all these pads that did that for us and then we have a prediction and an actual value of zero so we did end up getting this one correct now i think it'd be interesting actually to write your own review and test it on this so in the next video what i'm going to do is show you how we can save the model to avoid doing like all of this every time we want to run the code because realistically we don't want to wait like a minute or two before we can predict a movie review every time we just want it to happen instantly and we definitely can do that i just haven't showed that yet in the series because that's kind of in like later what you do after you learn machine learning um and obviously like this this model trained pretty quickly like we only had about uh what was it like 50 000 test data set which it seems like a large number but it's really not especially when you're talking about string data so in future videos we're gonna be training uh models that take like maybe a few days to train at least that's the goal or maybe a few hours or something like that so in that case you're probably not going to want to train it every time before you predict some information so that'll be useful to know how to save that so in today's video we're going to be doing is talking about saving and loading our models and then we're going to be doing a prediction on some data that doesn't come from this actual data set now i know this might seem kind of trivial we already know how to do predictions but trust me when i tell you this is a lot harder than it looks because if we're just taking in string data that means we have to actually do the encoding of the pre-processing removing certain characters making sure that that data looks the same as the data that our neural network is expecting which in this case is a list of encoded numbers right or of encoded words that is essentially just numbers so what we're going to do to start is just save our model so let's talk about that now so up until this point every time we've wanted to make a prediction we've had to retrain the model now on small models like this that's fine you have to wait a minute two minutes but it's not very convenient when you have models that maybe take you days weeks months years to train right so what you want to do is when you're done training the model you want to save it or sometimes you even want to save it like halfway through a training process this is known as checkpointing the model so that you can go back and continue to train it later now in this video we're just going to talk about saving the model once it's completely finished but in future videos when we have larger networks we will talk about checkpointing and how you how to load your or train your model in like batches with different size data and all that so what i'm going to start by doing is just actually bumping the vocabulary size of this model up to 88 000. now the reason i'm doing that is just because for our next exercise which is going to be making predictions on outside data we want to have as many words in our model as possible so that when it gets kind of some weirder words that aren't that common it knows what to do with them uh so i've done a few tests and i noticed that with the what he called with the vocabulary size bumped up it performs a little bit better so we're going to do that so anyways we bumped the vocabulary size and now after we train the model we need to save it now to save the model all we have to do is literally type the name of our model in this case model dot save and then we give it a name so in this case let's call it model dot h5 now h5 is just like an extension that uh means i don't know it's like i honestly don't know why they use h5 but it's the extension for a saved model and keras and tensorflow so we're just going to work with that and that's as easy as this is it's just going to save our model in binary data which means we'll be able to read it in really quickly and use the model when we want to actually make predictions so let's go ahead and run this now and then we're going to have the model saved and then from now on we won't have to continually train the model when we want to make predictions i'm going to say python tutorial 2 and i'll be right back once this finish finishes running all right so the model has finished training notice that our accuracy is slightly lower than it was in the previous video really kind of a negligible difference here but anyways just notice that because we did bump the vocabulary size so anyways now that we've saved the model we actually don't have to go through this tedious process every time we run the code of creating and training and fitting the model and in fact we don't actually need to save it as well either here to load our model in now that it's saved and you can see the file right here with all this uh this big massive binary blob here all we have to do to load this in is just type one line now the line is whatever the name of your model is it doesn't matter i'm just going to call it model is equal to in this case keras dot models dot load underscore model and then here you just put the name of that file so in this case model.h5 now what's really nice about this as well is you can actually train a bunch of different models and tweak like hyper parameters of them and only save the best one what i mean by that is like maybe you mess with for example the amount of neurons in the second activation layer uh or something like that or in the second hidden layer and then you train a bunch of models you figure out which one has the highest accuracy and then you only save that one that's nice as well and that's something you could do like overnight you could run like your script for a few hours train a bunch of models figure out which one is the best only save and then use that one so anyways we're going to load in this model notice that i've actually just commented out this aspect down here because we're not going to use this anymore and now what we're going to start doing is actually training or testing the model on some outside data so i've gone ahead and picked a movie review for one of my favorite movies some of you guys can read this if you want uh but it's the lion king absolutely love that movie so i've decided to go with this this review was a 10 out of 10 review so a positive review and we're gonna test our model on this one now i actually did take this off like the imdb website or whatever that's called um but the data set that they use is different so this is you guys will see it why this works a little bit differently and what we have to do with this so this is in a text file so what i'm going to do is load in the text file here in code and then get that big blob that string and convert it into a form that our model can actually use so the first step to do this obviously is to get that string so we're going to say with open and in this case i've called my file test.txt and then i'm just going to set the encoding because i was running into some issues here you guys probably don't have to do this i'm just going to say utf hyphen 8 which is just kind of a standard text encoding and we're going to say as f now again the reason i use with is just because that means i don't have to close the file afterwards better practice if you want to use that and now i'm going to say for line in f dot read lines which essentially just means we're going to get each line in this case we only have one line but if we wanted to throw in a few more uh reviews in here and do some predictions on those that would be very easy to do by just keeping this code structure just throw another line in there and now i'm just going to say we're going to grab this line and we're going to start pre-processing it so that we can actually feed it to our model now notice that this when we read this in all we're going to get is a large string but that's no good to us we actually need to convert this into an encoded uh list of numbers right and essentially we need to say okay so of that's a word what number represents that put that in a list same with all same with uh same with animation right and we keep going and keep going pretty well for all of the words in here and we also have to make sure that the size of our text is only at max 250 words because that's what we were using when we were training the data so it's expecting a size of that and if you give it something larger that's not going to work or it might but you're going to get a few errors with that so anyways the first step here is i'm going to say n line is equal to line dot and i'm going to remove a bunch of characters that i don't want so i'm just going to say dot replace i think this is the best way to do it but maybe not um and i'm going to replace all the commas all of the periods all of the brackets and all of the colons and i'll talk about more why we want to do that in just one second so we'll do dollar place i guess this dollar place should probably be outside the bracket uh and then we'll replace with a bracket with nothing and i know this is there probably is a better way to do this but for our purposes it's not really that important and finally we will replace all our colons with nothing as well now again the reason i'm doing this is because let's go here if you have a look for example when we split this because we're just going to split this data by um spaces and to get all the words what will end up happening is we're going to get words like company comma we're going to get words like i'm trying to find something that has a period like art dot and then a quotation mark right and we don't want those to be words in our list because there's no mapping for art period there's only a mapping for art which means that i need to remove all of these kind of symbols so that when we split our data we get the correct words now there'll be a few times where the split doesn't work correctly but that's okay as long as the majority of them are working well same thing with brackets right i can't have irons and then a closing bracket is one of my words so i need to get rid of that now this reminds me i need to remove quotation marks as well because they use quite a few of those in there i don't know why i closed that document uh so let's do that as well with one last replace so say dollar place in this case we'll actually just do backslash quotation mark and then again with nothing now i'm adding a dot strip here to get rid of that backslash n and now we're going to say dot split and in this case we'll split out a space now i know this is a long line but that's all we need to do to remove everything and now we actually need to encode and trim our data down to 250 words so to encode our data i'm going to say encode equals in this case uh and we're just literally we'll make a function called like review underscore in code and we'll pass in our endline now what review and code will do is look up the mappings for all of the words and return to us an encoded list and then finally what we're going to do and we'll create this function in just a second don't worry it doesn't already exist is we're actually going to use what we've done up here with this test data train data keras pre-processing stuff and we're just going to apply this to in this case our encoded data so we add those pad tags or we trim it down to what it needs to be so in this case we'll say encode equals keras.preprocessing instead of train data we'll just pass in this case actually a list and then encode inside it because that's what it's expecting to get a list of lists all right so now that we've done that our final step would be to use the model to actually make a prediction so we're going to say model.predict and then in this case we'll pass it simply this encode right here which will be in the correct form now we'll save that under predict and then what we'll do is just simply print out the model so we'll say print or not the model sorry we'll print the original text which will be the review so in this case we'll print line and then we will print out the encoded review just so we can have a look at what that is and then finally we'll print the prediction so what whether the model thinks it's positive or negative so we'll just say predict and in this case we'll just put zero because we're only going to be doing um like one at a time right okay sweet so now the last thing that we need to do is just simply write this review in code function and it will be good to go and start actually using our model so i'm just going to say define review underscore in code this is going to take a string we'll just call that s lowercase s and what we're going to do in here is set up a new list that we're going to append some stuff into so i'm just going to say like return let's just say like encoded equals and then i'm going to start this with 1. now the reason i start 1 in here is because all of our data here uh where it starts has a one so we're just going to start with one uh because we won't have added that in from uh the other way i hope you guys understand that just we're setting like a starting tag to be consistent with the rest of them and now what we're going to do is we're going to loop through every single word that's in our s here which will be passed in as a list of words we'll look up the numbers associated with those words and add them into this encoded list we're going to say for word and in this case we're going to say word in s now we'll say if word in this case word underscore index and again we're going to use word underscore index as opposed to reverse word index because word index stores all of the words corresponding to the letters or not the letters the numbers which means that we can literally just throw our data into word index and it'll give us the number associated with each of those words so we're going to say if word in word index then we'll say encoded dot append and in this case we'll simply append in this case word index word now otherwise what we'll do is we'll say encoded dot append to now what will happen is we're going to check here if word if the word is actually in our vocabulary which is represented by word index which is just a dictionary of all the words corresponding to all the numbers that represent those words now if it's not what we'll do is we'll add in that unknown tag so that the program knows that this is an unknown word otherwise we'll simply add the number associated with that word now one last thing to do is actually just do word.lower here just to make sure that if we get any words that have some weird capitalization they are still found in our vocabulary so like words at the beginning of a sentence and stuff like that uh and now with that being done i believe we're actually finished and ready to run this code so what's nice about this is now that we've saved the model we don't have to train it again so i can literally just run this and it should happen fairly quickly fingers crossed let's see all right must be a list of integrals found non-iterable object so what error is that here um in code encoding coding code all right so print review and code ah well it would be helpful if i returned the encoded list and that would have been our issue there so let's run that one more time and see what we're getting there and there we go sweet so this is actually the review i know it's very really hard to read here but if you guys want to go ahead and read it feel free since it's on the lion king it's obviously a positive review and then you can see this is what we've ended up with so our review has been translated into this which means we've actually trimmed quite a bit of the review and you can see that wherever it says two that is actually a word that we didn't know or that wasn't in our vocabulary four represents the that's why there's a lot of fours and then all the other words have their correspondence right now fortunately for us we picked a 88 000 vocabulary which means that we can get indexes like 20 000 whereas before it would have all been under 10 000. and you can see that our prediction here is now 96 um positive which means that obviously like we were going between zero where zero is a negative review and one is a positive review so this classified correctly as very positive review and we could try this on all other kinds of reviews and see what we get but that is how you go about kind of transforming your data into the form that the network expects and that's where i'm trying to get you guys at right now is to understand that yes it's really easy when we're doing it with this kind of data that just comes in like imdb like keras load data but as soon as you actually have to start using your own data there's quite a bit of manipulation that you have to do and things that you might not think about when you're actually feeding it to the network and in most cases you can probably be sure that your network is not actually the thing that's happening incorrectly but it's the data that you're feeding it is not in the correct form um and it can be tricky to figure out what's wrong with that data so with that being said that has been it for this video i hope you guys enjoyed that's going to wrap up the text classification aspect here of neural networks hey guys so in today's video i'm going to be showing you how to install tensorflow 2.0 gpu version on an ubuntu linux machine now this should work for any version of linux or any linux operating system although the one i am going to be showing you on is ubuntu 18.0.4 now you may notice that i'm actually on a windows machine right now and that this is actually just an ubuntu terminal that's open now i'm actually just ssh into a server that i have that contains two 1080 graphics cards so gtx 1080s and that's how i'm going to be showing you how to do this now quickly if you don't understand the difference between the cpu and the gpu version the cpu version is essentially just way slower and you would only really use the cpu version if you don't have a graphics card in your computer that is capable of running tensorflow 2.0 gpu so quickly before we go forward and you guys get frustrated with not being able to install this make sure that you have a graphics card that actually works for this programmer for this module that means you have to have a graphics card that is a gtx 1050 ti or higher those are the ones that are listed on tensorflow's website as compatible with tensorflow 2.0 gpu if you want to have a quick thing without having to go to the website to see if yours works if it has four gigs of video ram and is a gtx generation card or higher it most likely works with tensorflow 2.0 now i don't know about all the different cards if you have any questions leave them below i'll try to answer that for you but any 1060 1070 1080 or rtx cards that have cuda cores on them will work for this essentially you just need a cuda enabled gpu so you can check if yours meets that requirement before moving forward now to do this i'm just going to be following the steps listed on the tensorflow website now you may run into some issues while doing this but for ubuntu this is pretty straightforward and i'm essentially just going to be copying these commands and pasting them in my terminal now if you'd like to just try to do this without following along with video go ahead but i will be kind of showing you some fixes that i ran into while i was doing this um so let's go ahead and get started so actually let me just split the screen up so we can have a look at both of them at once i'm in my linux machine right now you just have to get to the terminal you notice that i don't even have a desktop and i'm literally just going to start copying and pasting these commands now the first thing that we need to install is actually cuda now cuda is what allows us to use the cuda cores on our gpu to actually run the code so just go ahead and keep copying these commands it will take a second and i actually already have this installed on my machine so i'm going to go through the steps with you guys but again if anything is different on my machine that's probably because it's already installed so if you don't know how to copy it into a window like this you just right click on your mouse and it'll copy if you're using a server like i am um but anyways we'll just go through all of these and keep going now i will have all these commands listed in my description as well and that should show you guys you know if the website goes down at any point you can just copy it from there as well so yeah literally just keep going all we're doing here is adding nvidia packages we're gonna make sure we have the nvidia drivers for our graphics card that are correct and then we're gonna go ahead and install tensorflow 2.0 uh so yep go through these commands there's not really much for me to say as i copy these in and eventually we will get through them all all right so now we're going to install the nvidia driver you can see that's all commented out on this tensorflow website here copy that and again just continue to go i don't really have any commentary for you guys here so we'll copy this this is going to install obviously the development and runtime libraries which we need and it says minimum four gigs or approximately four gigabytes which will mean that's how long it how many gigabytes it's going to take up on our machine so this will take a second and i'll fast forward through these stuff if it does take a while finally we're going to install tensor rt i don't even know what this is but apparently it's required and then after we're done this we should actually be finished installing everything that we need for tensorflow 2.0 to work again if you guys want to go through this just go to the website copy all of these commands in order paste them into here and they should work properly now finally what we have to do is actually install tensorflow 2.0 so we've got all the dependency dependencies installed and now to install tensorflow 2.0 we're just going to say pip3 install tensorflow and i believe we're going to say hyphen gpu and then equals equals 2.0 point i got to find it up here to make sure that we do it correctly 2.0.0 hyphen alpha zero like that so then we'll do that and that should install tensorflow 2.0 for us now i already have this installed but this will actually take a few minutes to install because there is quite a bit of stuff that it needs to download on your computer so anyways that has been it for installing tensorflow 2.0 on your computer using the gpu version again throughout the rest of the neural network series i'm going to be going forward doing this on an ubuntu machine so running all of the code i'll do the development of windows throw the files on my server train the model train the models excuse me and then take the models off and use them on my windows machine so if you want to validate if this is working you can really quickly just do python 3 in linux and then you can cite do import tensorflow and doing that you shouldn't get any errors and if you don't get any errors then you have successfully installed tensorflow 2.0 now a few errors here if you guys are still listening and stuff wasn't working if for some reason when you install tensorflow and you notice that it's not using your gpu go ahead and uninstall the cpu version of tensorflow so just pip3 on uninstall and then tensorflow and i guess you'd have to just do just tensorflow like that and that will install the cpu version if it is installed in your machine so anyways that has been it for how to install tensorflow 2.0 gpu version on ubuntu pretty straightforward just go through copy these commands and if you guys have any questions or errors please just leave them in the comments below and i will try my best to help you out