Prepare your data for ML | Text Classification Tutorial Pt. 1 (Coding TensorFlow)

Key Takeaways

[Music] [Applause] hi everybody I'm Laurence Moroney from the tensorflow team at Google and today we're going to talk about text classification it's part one of a two-part series will will focus on the data and getting it ready to train a neural network you will do this hands-on using a workbook that you can find at the link in the description below and I'll step you through it text classification has some unique challenges so before you get coding let me step you through some of these first of all neural networks typically deal with numbers are not text when learning patterns that can be used for prediction or classification so in this case we're looking at learning from movie reviews to see if those reviews are positive or negative and the first step of course is to change the words into numbers that represent them there'll be a little bit more processing of these words into vectors determining their sentiments and we'll cover that in the next video so let's get coding first first things first I'll have to check the licenses before I begin and now I'll import tensorflow and numpy I'll also use care Us and print out the version of tensorflow that I'm using okay now it's time to get the data set the IMDB set is included with care us so let's download it and let's take a look at what's in there note that in this case the nice folks that care us have done the work for us of converting the words into integers they've also sorted them into a dictionary so that lower numbers are the most common words and higher numbers are the least common words so when we loaded the specified 10,000 words this will then give us the top 10,000 words that are used across all of the reviews okay now we've loaded the data and we have our training data and labels as well as our test data and labels it's also nicely sorted into integers for us which is a great first step for learning let's see what the data looks like next first we'll look at our training data you'll see that we have a total of 25,000 items of data and 25,000 labels describing them the labels are very simple it's zero for a negative view and one for a positive one a reviews look like this it's just a long set of numbers and these are the indexes into the array of words the review will start with a 1 indicating the start of the review so the first word in the review is word number 14 which translates to the word this followed by the value 22 which translates to the word film the next bit of code is then a handy-dandy way of decoding the review note that the values zero through three are reserved with one being the start of the review as we mentioned a moment ago and zero is for padding now this is important and you'll see that in a moment I can now decode the review and see that one 14:22 other start character and this and then film it's pretty cool right now earlier I skipped over this piece of code showing me the length of the review so for example the first movie was 218 words long and the second was 189 words long now that's really awkward and it's confusing to train a neural network if all of the training data is of different lengths so let's pick a standard length for every review and if it's longer we'll trim it to that length and if it's shorter we'll pad it to that length the Charis pre-processing api's make this really easy here you can see I'm taking the training and test data and making sure it's 256 words long if I need to pad it then I'll pad it with the pad character which is the 0 that we saw earlier a quick look will now show that it worked they're all 256 words long and if I now look at my first set of training data you'll see that it's padded by zeros remember it had been 218 words long so the extras get patted out to make it 256 great our training and test data is now ready so in the next episode you'll take a look at how to design a neural network to accept this data and to train a model to determine the sentiment of movie reviews I'll see you there [Music] you [Music]