Data Preprocessing | Introduction to Text Analytics with R | Part 11

Key Takeaways

hi welcome to video number 11 series on in text analytics with our I am your host Dave Langer so we have here is in our studio environment where all of the code through the end of video 10 has been executed and at the end of video 10 we built a very powerful model a model that achieved more than 97% accuracy in terms of detecting ham versus spam and our text messages as measured by 10-fold cross-validation repeated three times we use the mighty random forest that code took a long time to run so what we did is we loaded up some binaries that I had pre ran and cached in the github to actually see what the performance was now one of the things that we looked at last time which bears repeating is that we look at the variable importance plot here we noticed that when we added spam similarity became very important it was a way out here it essentially dwarfed all the other features that we had and what we also saw was there was some indications here that maybe spam similarity while quite an effective feature as measured by our cross-validation runs may potentially be a problem for us in terms of overfitting for two reasons one because it so much dominates everything else and two because when we looked at the sensitivity and specificity we saw that this this particular feature actually helped us detect ham correctly but it didn't actually it actually decreased our ability to detect spam so we thought it didn't raise both of those metrics simultaneously that could be indicative of a potential for this to overfit but there's only one way to be absolutely sure about that and that's what we're going to start working on in this video which is let's start prepping our test data the 30% that we held out for testing that needs to be set up now what we all right what we have is since our test data our test data from from way back from video 1 video 2 video 3 but it hasn't actually been pre processed so we've done a lot of work in this video series on our training data we've tokenized we've lowercased we remove stop words we've performed stemming we've added by grams we've done things like tf-idf and we've done SVD all of those sorts of things now have to be done to the test data as well because we have a trained model we have a trained model but we've pre-processed the training data so much that the model can only work with data in that similar format so everything that we've done to the training data we now have to do to the test data as well to get it in the right format to get it in the right semantic space to get in the right geometry so that it will work with our trained bottle so that's what we're going to do so looking at the code go ahead and expand this a little bit not surprising the first thing we're going to do is we're going to tokenize just like we saw before we're going tokenize we're in a lower case we're going to remove stop words we're then going to stem we're going to add by grams and then we're going to convert this to our document term frequency matrix so we go ahead and run all this code right here real quick and here's the first thing that you notice first thing you notice right so we have our training data has 3901 documents we're totally familiar with that we've seen that all along in this video series and our training data has twenty nine thousand two hundred ninety seven features and it's 99.9% sparse now what's super interesting about the test data is that not surprisingly because there's as much smaller number of documents write less than half the number of documents notice that we have less than half the features now that's going to be super interesting because not surprisingly our machine learning model that we've trained expects the same features that it's always seen was working on test data so for example assuming that we didn't do any SVD let's say let's say we didn't get down to 300 singular singular vectors which is just just you know just just working with the by grams the grams and by grams in tf-idf we would expect that these two numbers to be exactly the same because our model is expecting in the case of not using SVD 29,000 to our 97 features to sum total of the uni grams and by grams and that's what it expects it says look I need a data frame that's twenty-nine thousand two hundred ninety seven features wide I don't understand anything else you can send me more data than that that's fine but you have to send me at least the twenty nine thousand two hundred ninety seven features bad I've been trained with better data I expect that's the contract that you and I have now you'll notice here that we don't have that so we're gonna have to fix this up later so this is super important to realize in the beginning which is this when you do text analytics in particular you're going to train on a historical set of data that historical set of data is going to have a collection of words a collection of diagrams and collection of trigrams that reflect the data that you had at that point in time it's fully expected that in production after you move a text analytics model in production that you're going to see documents with new words in them and in text and tweets for example this is extremely common that people will actually morph the types of characters and words and phrases over time and you would fully expect that there's going to be this gradual diversion between the nature of the textual data that you're going to get in production with your new data versus what you train your model on and this is a prime example of that right here we can see for a fact that even with our 70/30 split of our data we're getting features we're getting different numbers of features and that's totally expected so it's really expected so we're going to be doing things with our test data to essentially morph make transform new test data so that it looks like our older training data and this is what happens all the time in text analytic systems in production and one of the things that you have to realize is that your text analytics models once they get put in production eventually become stale they eventually need to be retrained because eventually you'll get an ever-increasing amount of new types of words new types of diagrams and trigrams in your test data that are not reflected in your model and can't actually be used essentially we're going to have to strip them out and we'll see how that's done later well to strip them out at a certain point that that that sheer amount of data that you've never seen before becomes so valuable that you want operate into your model which essentially means your model gets stale and it needs to be retrained from scratch and redeployed so that's one thing you need to be very clear on when you're when you're engaging these types of projects with your stakeholders is that look once we build a text analytics model and put it in production you're not done that model will need constant care and feeding you'll need refreshing overtime airglow is going to need budget it's going to need it's going to need staff to actually do that sort of work ok enough about that ok so we've got our document term frequency matrices no problem so here's here's the step that I mentioned so we need to make sure we need to make sure that our tests document term frequency matrices tables data frames have to look exactly like the DFM s that we use for training right because if they don't when we do tf-idf when we do SVD it won't be valid the trolls transformations on the test date will not be valid ergo our predictions will not be valid so we have to make sure we do all the things all the pre-processing exactly the same way between training and test and then we have to ensure that the base document term frequency matrix the DFM is exactly the same in terms of the number of columns not only the number of columns but also each column has to be in exactly the same order and it also has to have exactly the same meaning in terms of the order so for example if column 37 was the word fluke I'm just making this up if column 37 in training was the word fluke it needs to calm 37 in the test data frame also needs to be the terms see what so they have to be exactly aligned so this code actually does this so you can see here the DFM select and as always don't hesitate to use the help system DFM select select features from ADF m or fcm okay now if we do this and run these three lines of code just see exactly what I'm talking about look at that notice that I've got now I've got twenty nine thousand two hundred ninety seven features this is great right this means that now my 2d FM's are aligned the training DFM had a particular structure we have now enforced that the test DFM has that exact structure which is great because now we can apply our additional transformations so next up we're going to go ahead and of course use the mighty tf-idf on the test data now as i mentioned earlier in the series more than once and it bears repeating is so important notice this is why we've had to cache the IDF values remember earlier I made a made a big speech about how we have to cache the IDF values no matter if you write the code yourself or if you use another package of somebody else Rodan are the original IDF values have to be attained from the training examples because we know now that the DFM s are going to look the same we're going to force them to look the same between train and test but in fact they are not exactly the same ergo to prickly project the new test data into the same vector space same tf-idf vector space that we created for training data we need to reuse the same IDF values from the training data okay so I won't drain this code because essentially we've seen it all before first thing that we can do is we can apply the TF calculation to our test matrix and that will come up here shortly and we can see the structure of the resulting calculation okay no sir we've got now we've got our terms have been up in the rows and our documents around the columns just as we saw before and then we'll go ahead and apply the tf-idf calculation now notice notice here this is important we using the IDF values that we calculated from the Train there this is super important this is super important because in production you're going to get new rows one at a time right new texts are going to come through one at a time let's say in a hypothetical production system and we're going to need to project each one of those texts one at a time into the same vector space so we have to have the IDF values to do that so we'll go ahead and go ahead and run all the rest of these on all the rest of these lines of code now one thing to mention it like the law that code is running is that when you're looking at doing tf-idf in a production system be very very sure that if you decide to use a package that the package is api of the collection of functions that it provides will actually allow you to calculate an IDF all the IDF vector values for the training data and then maintain them over time because some of them some of the libraries just kind of assume that all of your data is just in one big matrix and you don't have access to the actual underlying IDF values and of course that won't work for you in a production system okay so you can see here everything is has been ran just fine we're going to complete the we're going to complete the in computing we're going to fix the incomplete cases and you see here that based on our summary we in fact have no missing values so everything's good to go so now just to recap we've taken our text data we've done our text pre-processing tokenize we lowercased remove stop words we stand removed punctuation all that good stuff all that stuff is gone we added you know grams and buy grams we created DFM we've now tf-idf DFM after we made sure the DFM had the same structure as our training DFM so now that's surprisingly the last thing we need to do is we need to actually apply our SVD projection right because that was the last step we need to actually get down from our 29,000 297 columns down to the 300 singular vectors from our SVD that are the most predictive and this is the calculation that does that and we saw this in video of eight when we looked at the mathematics the slide there's a slide to show this equation so essentially what we need to do is we need to take the inverse the Sigma inverse right this is the inverse of the seer values we multiply that by the trans those of the you matrix which essentially is the the eigenvectors of the term correlations and then lastly we need to then multiply that by the transpose of our actual tf-idf data frame so if I run this line of code here and it'll take a second here to run and what we should get back is a nice data frame that will be 1600 or so rows because that's the number of documents that we have a number of texts in our test set and then also should have 300 columns so if we do this video ah there you go sixteen hundred seventy-one rows each one a row each row is one of our test tweets and then our 300 columns notice that these dimensions are extremely important especially this 300 columns right we have now projected our test data into the same semantic space latent semantic space that our training data resides so this is really super cool okay now now we can build ourselves a test data frame you see here I'm going to add them we're going to have our test data we've got our labels from our test data we got our SVD raw features and then we're going to add text length all right this mimics the same features that we had in our training data and then next up we're going to go we're going to add similarities because remember we said we're kind of concerned about this spam similarity calculation but we should certainly test out just to make sure we have a hypothesis but this data scientist we want to verify our hypotheses with data so we're going to go ahead and test out how accurate our predictions are on the test set with the the similarities thrown in okay so this code probably bears a little bit of explanation so first thing that we're going to do is we've got our raw data or our SVD data right we have 1671 rows of our test tech our test text messages sorry test text messages 671 rows and then we have our 300 columns are 300 columns of SV deed features for the test data that's going to be the basis and then what we're going to do is we're going to add rows we're going to add rows at the bottom at the bottom here in what we're going to add is we're just going to go back to the original training data the SVD training data and say okay look you know we really need to do here is we need to actually pull out those vectors that pull it to the spam the core late to the spam right so if I run spam indexes here you can see we calculate those before these are all the indexes in the training data of the text messages that are spam and we have you know a little over 500 of them so that says look go back to the training data grab only the vectors only the vectors that correspond to spam and in particular grab the document vectors so these are these are these are the projection of the spam SMS messages into the latent semantic space so we say okay look take our test data and then graft on to the end of that the rows that that give us the spam so that essentially gives us essentially a data frame with our the documents that we care about right which are the test text messages but we're also adding in the data for the spam the representation of the spam SMS messages because we'll need that I've actually calculate the average similarity between each individual test text message and spam okay now what we do here is we say okay look we've got this data frame let's go ahead and calculate the cosines between them I'm just going to run this code and explain it while it's running so we're going to say okay look you know what we've got the first 1670 Rose of our test similarities data frame is our test text messages the next 500 or so Rose is the spam SMS messages that we got from the training dataset and then we'll say okay look we know that cosine similarity will calculate cosines between all column vectors in a table so what we do is we say okay fine look if we transpose this our documents now become the columns right because they're rows in the test that similarities data frame so transposing it will actually make the documents into the make the wrote the document rows into the document columns and then cosine will calculate all the similarities between these documents which will be cool because they'll make our calculations a lot easier as we'll see in the next line of code okay so you can see here I'm going to go ahead and pre populate a spam similarity feature on our test SVD data frame this will be the placeholder that will hold all of these similarity values and then we're going to go and say okay look the spam columns the spam columns in our new test similarities data frame essentially all of the columns that start after the original test data and go to the end so let me let me explain what that means a little bit more detail right so if we do the dimension on test dot similarities it should be 20 about 20 200 rows 20 hundred rows and 2200 columns right it's a square matrix matrices right but we know that not every every column in every row is actually a spam message we know that only the only the end rows and the end columns actually correspond to the spam messages because remember we added to the end up here the spam messages okay great so this actually tells us what actually is the spam columns in terms of new numbers and you can see here I've just run this real quick you can see yep it's everything starting at 16 72 those that's one more than we actually have in terms of test rows all the way up through 21 94 okay cool and that allows us to a nice simple calculation right here go through every row in the test set every row in the test this is Phoebe every row and calculate the spam similarity and all the spam similarity is is the average the mean of the Train similarities for that row in question and the spam column the spam columns that actually calculates them on individual basis okay this mimics the code that we saw before all right so we run all that and you can see now that we've got our our spam similarity feature is now totally calculated so now we can create some predictions all right so we say okay look you know what take our third random forest that we created from a cross validation right which we saw was over 97% accurate as measured by cross-validation and then use that view that mighty random force that we built to create some predictions for the test data and then we can do we can do confusion do a confusion matrix on our predictions versus the actual labels right here so let's go and run that code and this output is super interesting super interesting okay look at this a commonly used definition of what is known as overfitting in machine learning is doing much worse on test data than you did on your training data now in our training data as measured by 10-fold cross-validation repeated three times our mighty random forests scored over 97% accuracy and notice here that I'm test data it only scores eighty six point five nine percent accuracy and more importantly notice this who is this our sensitivity is 100% and our specificity is zero because our model predicts that everything is ham our model predicts that everything is ham it predicts it doesn't even try to save anything spam it says look you know it everything's here it's all good all these text messages are good it's not a problem now we know that to be patently false we know that because we created stratified samples both training and test so we know the proportion of the spam messages in the test dataset mimics that of the training dataset so we know for a fact there's spam in there but our model currently given the features that we provided to it automatically assumes that everything is ham now we had a hypothesis that said look this is this could be the direct result of the fact that our spam similarity our spam similarity feature is going to overfit because essentially what it does is the the spam similarity feature is very very good at differentiating ham from spam in our training data only it does not generalize to data it's never seen before it's too tightly bound to the training data ergo it's probably you know it's probably a risky feature to have and this kind of bears it out this is look you know what okay um yeah I I have almost an 11 percent drop in accuracy and I'm predicting everything to be ham just blankets so that I mean this isn't this isn't a this is not this is not a good situation so this is highly indicative of lower fitting okay so this is a great place to end video number eleven we've accomplished a lot in this video most importantly we've discussed arguably one of the the most important things in rolling out a production text analytics solution and that is being very deliberate being very careful to ensure that when we have brand new data in this case SMS text messages that come into the system that we perform all the transformations all the tokenization the stemming the stop word removal the bigram trigram whatever tf-idf calculations and lastly the latent semantic analysis projections to make sure that that test data lands in a format lands in a geometry that our training our trained model first much first first of all understands and then to can actually work with productively because here's the worst case scenario worst-case scenario is is that you do something funky in your data pipeline maybe of a bug maybe you missed a step what have you and what happens is that the model recognizes the data it doesn't say there's any formatter but yet the projection the final representation of that test data is not actually in the exact format which the Train model can work with most effectively and what you end up getting essentially is no error information from the from the code but what you get are essentially suboptimal predictions that that's the worst case scenario another scenario which is which is bad but not not nearly not nearly as bad is that you make a mistake in your pipeline you submit the new text information for a prediction and the code just throws an error saying look I don't know what to do with this record it's not in the format I expect or maybe there's some other sort of errors that's actually it's bad but it's it's it's far less insidious than the previous problem which is you get no error signal at all you just get a bad customer experience so we covered this in depth hopefully you you hopefully you understand how important this is okay the next video in this series video number 12 will be the conclusion we will wrap up we will show how we can fix this current situation where we are obviously overfitting you know here our model is around 11 percent lower in accuracy on the test holdout than it is in our cross-validation runs and we're going to see how by eliminating the spam similarity feature will actually increase our accuracy test holdout which shows us that removing spam similarity is indicative of producing a model that is more generalizable that is one that will work better on unseen data in production okay so if you like what we're doing here on the data science dojo YouTube channel please subscribe we'll be producing new content weekly and if you subscribe that's the best way to keep abreast of our latest tutorials also if you have any questions at all about this video series feel free use the comments section of this video we had data science design so Joe we monitor a YouTube channel frequently and we try to answer any and all questions promptly next up if you like what we do a data science dojo more generally we're on social media so if you follow us on Facebook or Twitter or LinkedIn you can tap in and tap into a curated stream of data science goodness that you may find useful in your daily work and lastly I hope to see you in an upcoming data science dojo boot camp this is Dave Langer signing off and until next time I wish you very happy data sleuthing you