Machine Learning with Text in scikit-learn (PyData DC 2016)

Key Takeaways

Hello, my name is Kevin Markham and I'm the founder of data school. Last month I was at the PI data conference in Washington DC and I presented a 90-minute tutorial called machine learning with text in scikitlearn. They recorded it and I got permission to post it on my channel for you to enjoy. Now before we get started, I've got two important notes. Number one, there's a link in the description below to a GitHub repository containing all the code you see in the video in case you want to follow along. Number two, if you like this content, but want to go deeper into scikitlearn and natural language processing, I offer an online course called machine learning with text in Python that you should consider taking. There's a link to learn more in the description below. With that said, I hope you enjoy the video and I'll see you afterwards. All right, we're going to go ahead and get started. Uh, so welcome everyone. Uh, thank you so much for being here. My name is Kevin Markham and this is machine learning with text in scikitlearn. Uh, so really happy to be here. It's my privilege to be here. Um before I do a little introduction of myself and the tutorial, um let me go ahead and tell you what you need to do to set up so that while I'm introducing myself, you can get uh situated and once we jump into code, you'll be uh right along there with me. So, as I mentioned as folks are walking in, uh there's a GitHub repository for the tutorial. It's at bit.ly/patc. Uh I would show you I I don't have any internet access right now. Um but uh the the repository is there if you can access it. Um there's only a couple files there really. You just need two things. One is a data set and it's just SMS.tsv small data set. And the second thing you need is either the tutorial notebook or a tutorial script, regular Python script. So if you're able to get those fi those files, that would be great. Uh if you're not, uh I have some flash drives with the uh just two flash drives with both Anaconda installers if you need Anaconda. Um as well as the tutorial files. Um so I've just got two, but if anyone needs them, don't be shy. Just come up and get them. Uh and you know, feel free to return them at the end. Uh but if you need the tutorial files, they're on there. Um so um in terms of what software you need to um uh for today uh you'll need scikitlearn and you'll need pandas and their dependencies. If you've installed Anaconda then you're all set. Um you don't need the latest versions. Um and both Python 2 and three are fine for this tutorial. Um and that's that. So while you're getting set up I'll just introduce myself. Uh, and I'm very used to standing up while I teach, so this is a little strange to be sitting down here, but I'll try to sit up as tall as I can. Um, so, uh, my name is Kevin Markham. Uh, I have been living in DC since 2005. Uh, live on Capitol Hill. I'm the founder of Data School. Uh, I teach data science in Python online. Um, I have tons of free material out there. I have one paid course and some more to come. Um just curious if anyone here has heard of data school or seen any of my videos or blog posts or Okay, few folks throughout. Um thanks for watching. Thanks for your interest. Um so my background is computer engineering. Uh I was previously the lead instructor for the part-time data science course at General Assembly. I taught that course uh six times in a row and uh really enjoyed it. And then I was an instructor fellow there. I helped to train instructors. Uh really enjoyed my experience uh there, but now I focus full-time on data school. Uh so online uh data science education. So really a pleasure to be here. Uh fun fact about myself if you are a tech geek is that I have been using the D'vorak keyboard since 1999. Anyone else using D'vorak? Oh yeah, there two people. Wow. I can't even believe it. Um, all right. So, uh, I thought it was like one out of every I don't know 500,000 people. Um, but, uh, yeah. So, uh, in terms of how the tutorial will work, uh, I'm going to be using the Jupyter notebook. Um, you can use the Python script if you like. It's really up to you. Uh, I am going to introduce the notebook briefly for those who would like to use it but are not super familiar with how to use the notebook. Um so we'll just spend a few minutes on that. If you have downloaded the the tutorial files, um uh just go ahead and uh navigate to the repository you've cloned or downloaded or wherever those repositories files repository files are stored and go ahead and uh go to the command line in that folder and just type jupitter notebook and that is juy t and when you hit enter uh that will hopefully launch a browser window and it should look like this. And you'll just click the thing that says tutorial.ipynb and it will open up something that hopefully looks like this. Okay. Uh flash drive for anyone. Um so um I have written most of the code in advance though uh I would encourage you to kind of run it along with me. Um the main reason I've written it in advance is so that you can catch up if you miss something. um while you're kind of exploring the code on your own, but feel free to write the code yourself if you prefer. If you um uh there is not like a ton of code in here. So even if you um prefer to write it yourself, you should be able to keep up with it. Okay. Um if you have really major problems running the code, there is a file in the repository called tutorial with output and that has all the output. You can browse it to it on GitHub. Just view the static version. You won't be able to run it, but at least you can look at it and refer back to it. Okay, so uh that's the story of how this is all going to work. Um I would uh pull up the repository and read the description from there, but I can't do that since I can't get the internet. However, I have the markdown document, so I'm just going to read the uh description of the tutorial real quick in case you uh haven't seen it. Um, although numeric data is easy to work with in Python, most knowledge created by humans is actually raw unstructured text. By learning how to transform text into data that is usable by machine learning models, you drastically increase the amount of data your models can learn from. So in this tutorial, we will build and evaluate predictive models from real world text using scikitlearn. Okay. So, if you've been to all of the other text related tutorials today, this is different. Um, it does not overlap almost in any way. So, hopefully you'll get something new out of it. Let me close that. All right. Um in terms of what I expect you already know uh ideally you know some Python you know some basic machine learning principles and you have basic experience with both scikitlearn and pandas. So the question is how many of you fit into that category? Oh okay a lot of you. Um okay even if you did not raise your hand that is okay. um some of the stuff that I think of as review you will see for the first time that's fine too. I think you'll still walk out with a basic understanding of pandas and scikitlearn and machine learning. So uh once again I'm going to uh introduce the notebook for just a few minutes. We will jump then right into the lesson and um you can ask questions at any time. I'll try to scan, but it's a pretty wide room, so feel free to make some sort of not awkward noise, just like, you know, cough or whatever, and I'll I'll look over. Um, so if you have any questions right now, let me know. Any questions? It doesn't matter. Three, three or two is fine. Okay. So, as I said, feel free to uh interrupt with at any time with questions, but uh just jumping right into the notebook. Um so, real quick, what is the Jupyter notebook? It is essentially a IPython interpreter plus a browserbased notebook. Um so, again, if you haven't already, you usually start it from the command line or you can start it from a program called Launcher. If you use Anaconda, just search for Anaconda launcher, then click on, you know, um, Jupyter Notebook. Um, it used to be called the IPython Notebook in case you're wondering. Uh, but if you type Jupyter Notebook again, you'll be taken to this dashboard, what I call a dashboard. And then if you click on a notebook file, you'll be taken into that file. Okay. So, real quick, couple things uh key to know about the notebook if you're going to be using it today. Um number one is that notebooks contain a combination of markdown code uh Python code and plots uh and they can have other things interactive things. Um they are divided into cells and uh the cell is basically you navigate up and down with your keyboard and you're navigating from one cell to another. Okay. So the cell is the unit of stuff in the notebook. If you want to run a cell you just hit control enter or command enter. Uh, so I'm just navigating with my keyboard to this first cell and hitting control enter. And it ran. And you know it ran because it now has that number in the little box to the left. It's telling you it's the first thing that ran in the notebook. I could run it again. Control enter, command enter, and it's the second thing that ran in the notebook. Okay. So, um, there's a lot of keyboard shortcuts. You don't need to really know them for today, especially if you're just running my code. Um, but the main thing you need to know uh to avoid uh working well with notebooks in general is that there's something called command mode and something called edit mode. Right now when we're in command mode, now we're in edit mode. Comm uh Whoops. Command mode. Edit mode. So you notice it's changing from green to blue. I'm just hitting escape or enter. So in edit mode, when you type on the keyboard, it types. In command mode, if you hit keys like an A, it will create a cell above. B, it will create a cell below. X, it will cut a cell. So, you have to be careful about when you type in a notebook because if you are in command mode again where there's no like flashing cursor, typing letters will run keyboard shortcuts. Those keyboard shortcuts will change things in your notebook. But if you are in edit mode and your your cursor is flashing, you're just going to type in the cell. Okay. So, um anyway, uh I hope that wasn't uh too boring or needlessly uh well needless if you've I guess already know the notebook. Um final thing I'm going to say before I show you how to close it, reopen it. Um is just that uh it only cares about the run order on the page, not the order of stuff on the page. So you could say a equals 2 and then create a cell above it and say print a and that will work. Uh all that matters is I ran cell three, what's now cell three before cell 4. Okay, so it's all about run order, not order on the page. Um I'm going to go ahead and uh if you want to save this uh you just save and checkpoint um and I will do that. I'm going to just close and halt and it will take you back to this uh dashboard. And if you reopen it, this is the final thing if you're like ready for machine learning. Uh the final thing I will say is that when you open a notebook and you see like this, so I've see that it looks like there's an environment populated with a variable called a that's a uh an illusion. So the environment is actually empty right now. So when you open a notebook, it doesn't matter what it looks like. The environment is empty. You need to rerun cells. Um, so just kind of a tip for you. Okay, I'm going to just delete a couple cells. And uh what I'm going to do, and you may want to do this if you were playing around with me, uh is just sell um actually you can just say um either way sell uh all output clear or um kernel resto restart and clear output. Uh slightly different, but doesn't really matter too much. Um and now my notebook is clear and I'm kind of ready to get going. Okay. All right. Any quick questions about the notebook before I actually get into the content? All right. Um, so, uh, the agenda, what are we talking about? And I'm going to clear out my screen a little bit. Uh, turn off the header and toolbar so that there's a little more space. Um, but you can use, um, uh, those, uh, to like run cells and whatnot. Okay. Agenda. Um, part one, uh, is model building in scikitlearn. If you are experienced with scikitlearn, this is a refresher. If you are not, this is new content. Then we're going to talk about how to represent text as numerical data. Uh and then in three and four, we're essentially going to apply what we've just learned onto an actual uh real data set rather than the play data set I've created for for parts one and two. Um part five, we're going to build and evaluate a model. And then part six, we are going to compare some models. So, uh, I've timed it that I think this will, um, probably run over, so I may not get to all of part six. Um, but we'll see. Um, but I'm just going to go right into part one, and I'm going to be asking you a lot of questions so that, uh, you stay alive and awake. Um, uh, all right. So I am going to go ahead and load an example data set. So how many people here have heard of the Iris data set? All right, it's kind of machine learning 101. Uh so it's a famous data set because it's an easy one to work with. Okay, we're just going to load it. So this code isn't super important um in terms of understanding how that code works. You're just loading a built-in data set from scikitlearn. Okay. And storing it in an object called iris. All right. Um then we are going to create our x and y. Okay. Uh the this is our feature matrix and our response vector. And features you might call them predictors or inputs or attributes. Uh you might call the response the target or the label or the output. Um the convention is to call them X and Y. Um and they happen to be stored as attributes of the iris object. Doesn't really matter that part. The point is we're creating our X and Y, our feature matrix and response vector. Uh so I'm curious as to who can tell me uh why the um X is capital and the Y is lowercase. Yeah, right here. X is a vector. X is a matrix 2D twodimensional Y is a single dimensional object. The convention is to make your capital X uh because it's two dimensional Y is one dimensional uh not hugely important just a convention to understand might help you read some code every now and then. And if we check the shapes, these are numpy attributes. Um uh sorry, these are numpy arrays and uh they have shape attributes. And the shape of x is 150x4. Uh the shape of y is 150 by nothing because it's one-dimensional. It's not 150 by 1, it's 150 by nothing. Okay. Um so that is important. It's like the difference between a pandas series and a pandas data frame with one column. They're actually different things. Um so x two-dimensional, yed one-dimensional. These things are actually important for being able to use scikitlearn because if you don't get object shapes, you can't actually build models without getting errors. Okay. Um okay. So uh we know from the 150 number that uh there are 150 observations or samples or records in our data set in our Iris data. Okay. Um I'm going to use I'm going to go ahead and import pandas. And this is kind of just for display purposes at this point. We'll get into actual pandas usage later. But uh what I'm doing here is I'm coercing X which uh again is a numpy array. I'm coercing it into a data frame and I'm adding column headers and then I'm showing you the head which is a a dataf frame method to show five rows. Okay. So this is the top of our uh feature matrix. For those of you who don't know the iris data, um there are 150 rows. Uh and the four features are sele length, sele width, pedal length, and pedal width. Okay? And I promise there's a reason I'm going through this even if you already know like basic machine learning. Uh you'll see the two takeaways and we'll apply them to text. Okay, so that's just the top of the feature matrix. And now here's the response vector. It's a bunch of zeros, ones, and twos. Those are integers that represent classes. The Iris um data set is generally used as an example of classification. So these are the classes. This is the response. This is what we're predicting. Okay. All right. First takeaway that's going to seem obvious, but it will be immediately applicable in the next section. In order to build a model, the features must be numeric and every observation must have the same features in the same order. So the first question, why do the features have to be numeric? Why do the features have to be numeric? I know you know this. Don't be shy. Be part of the matrix. One more time. To be part of the matrix. Uh yeah, but why does it have to be numbers? Why can't it be strings? Okay, models do math. Math requires numbers. Thus, your inputs must be numbers. Okay, bottom line. Um you can transform text to numbers, which is what we'll do. But uh you must have numbers. Okay. Uh second thing, why does every observation have to have the same features in the same order? In other words, in the first row of iris, it's seal length, sele width, pedal length, pedal width. Why does it have to be the same order in the next row? Why do you have to put sele length first in the next row rather than second, third, or fourth? any well it depends what the model is because every model does a different operation but you're getting at the point which is like these aren't labeled I mean I've added labels but they aren't labeled so scikitlearn assumes that if like a given row or sorry a given column of data is all the same feature I mean it's kind of obvious obviously you put your feature all in one row. You don't split it in sorry in one column. You don't split it into two columns. But it's super important that that is the case. Okay? And we'll see why that's relevant with text. Okay. Um all right. We're going to have one more takeaway in a minute, but first we're going to uh refresh ourselves on the um the basic model building process. I have a video series about scikitlearn. Call this the four-step model building process. It's import, instantiate, fit, and predict. So that's one, two, three, and four. So import, import the class. What is the model? Instantiate, instantiate the model, create an instance of the model. Uh, in this case, I'm importing the K neighbors classifier. Doesn't really matter uh what I import. That's what I chose. Um, instantiate the model. I'm creating an instance of the K neighbors classifier with the default parameters. That's a model that like that's an object that knows how to do KN&N classification. And then we're fitting the model with data which is where the model learns the relationship between the features and the response. So I'm doing KN&N.fit X, Y. Okay. And uh again I'm using the default model parameters. All right. Uh if you're new to this, then that's fine. That is a pattern that's useful to learn for model building. Uh if it's review, I want to give you kind of the other takeaway. Okay? Um which is we're going to make a prediction. Okay? We've trained a model. We're going to make a prediction. If you want to make a prediction, the new observation must have the same features as the training observations both in number and in meaning. Okay. So I'm going to pass 3542 okay to the predict method meaning I have this out of sample observation please predict a class for uh this observation it predicts class one okay the classes were 0 1 and two it predicts class one so the question here there are two questions okay number one what if I had passed in three features or five features, what would have happened? An error. An error. And why would there be an error? A feature that doesn't exist or an external feature. Yeah. So um there as he was saying that you know it's a feature that either doesn't exist or there's too many features like scikitlearn has learned the relationship using a KN&N model between four features and a response. So you have to give it four features to get a predicted response. You can't give it three, you can't give it five, you have to give it four, right? Kind of makes sense. Um the other thing is they have to mean the same thing. So I didn't like when I did predict I didn't say sele length equals 3 width equals 5 etc. I just put them as 3542 and I have to know that I put them in that order because those are the same orders that were in the training data. That was the order of features in the training data. Okay. So that kind of makes sense. Uh kind of that that takeaway. All right, any questions up to now? Okay, so let's get into something that I assume is probably new for almost all of you, which is representing text as numerical data. Okay, so um all right. So um what we're going to do is use a tiny bit of text that I have made up so that we can build a pretend model. We can vectorize it which is what we'll do in a minute and um then we'll move on to a real data set. But you have to understand really small data be before you can understand smallalish data. Um so we're I'm going to create a Python list of three strings. It's called simple train. Um, and it's trained cuz it's training data. It's simple cuz it's simple. Um, and they are pretend SMS messages. One is call you tonight. The second is call me a cab. And the third is please call me please. I just made these up. I did not actually send these texts. Um, and uh we are going to come back to the response vector, but we are going to predict we're going to learn uh desperation. Okay. So that's our response value. Okay. So in the training data, I've got a a list of three integers 00 and one. The it's called is desperate. So the first text I would call not desperate. The second text not desperate. The third text, the third SMS message desperate. So 001 this is going to be we're going to learn the relationship between simple train and is desperate. And then we're going to make a prediction for some other text. Okay. All right. So, go ahead and run those if you are uh following along in that way. Okay. And you might have noticed, you might be thinking to yourself, how are we going to run a model on simple train because Kevin just said that uh models have to be trained on numbers and these are not numbers. The response are numbers, but these are not numbers. Okay. So the scikitlearn documentation I'm just going to read a tiny bit out of it. Uh text analysis is a major application field for machine learning. The raw data a sequence of symbols cannot be fed directly to the algorithms themselves as they expect numerical feature vectors with a fixed size rather than raw text documents with variable length. Okay. The Iris data as you might have noticed had numerical feature vectors of fixed size. What we've got is raw text documents of variable length. Okay. So we need to turn them into numbers so that we can train a model on them. Okay. Uh we're going to use something called count vectorzer. And count vectorzer uh its description is it converts text into a matrix of token counts. And you don't have to really understand what a matrix of token counts is. You'll see what it is in a minute. Um the thing you uh probably want to know about count vectorizer is that um it has the same pattern as the four-step model building pattern except different fourth step. Instead of import instantiate fit predict, it's important fit transform. Okay. Um, we're going to import it from sklearnfeature extraction.ext because this is how you extract features from text. Okay. So, import count vectorizer and then instantiate it into an object called vect. Um, and uh, so now we've got an object vect that knows how to do vectorization. Okay. Then we're going to learn the vocabulary of the training data. Um, and that's what happens during the fit method. Okay. So when we say vect.fit simple train, it learns the vocabulary of the training data. That is all it does. Learns the vocabulary. And you'll see in a minute why uh that's useful. Okay. So I'm going to run that fetch.fit simple train. Uh these are all the default parameters for account vectorzer. And then once you have fitted a vectorzer with um uh with text, you can run this method called get feature names. Okay. And it will output your vocabulary as a list. All right. Uh that's our vocabulary. Um there are five things. Is it ignoring punctuation? Oh, great question. Uh, so there are I was about to say there are five things you might be surprised about or you might be thinking, hm, what is it doing? And one of those five things is it is uh ignoring punctuation. Okay. What are four other things that it's doing that you may or may not have expected? What did it do to go from the training data, call you tonight, call me a cab, please call me, please? And I'll move this. Well, I'll leave it there. Call you tonight. Call me a cab. Please call me please. Scroll down and we get cab. Call me please tonight. You. Yep. It removed the A. It's not because it's a stop word. It's because it's uh less than two letters long. But um it doesn't automatically stop. It does not. So there's a a parameter called stop words that you can tell it to remove stop words. So it removed this is a regular expression pattern token pattern. It's an escaped regular expression pattern. Um and that's saying only allow uh two word characters in a row. That's the minimum for a token. Okay. So we've got uh the A is missing. We've got that punctuation is missing. What are three other things that happened? Capitalization. Capitalization was removed. So it's all lowercase and there's actually a parameter called lowercase equals true. Any others? It is uh I mean yes um yes it did uh take the text converted to unic code. Uh whether or not you expected that or not I guess that might be unexpected. If you're in Python 3 you won't see the U because strings are unicode by default there. But uh what are a couple other things? Did not there is no duplication. If there were two of a word, there is now one of it. Okay, it's like the dictionary. There's only one per word. You don't need it there twice. Um and one other thing I'll mention, maybe it's kind of obvious uh and not worth noting, but it put them in alphabetical order. Okay, so it's lowercase alphabetical order. Uh no, um words less than two characters. There's no punctuation, no duplicates. Okay, so we have learned our vocabulary. This is called the fitted vocabulary. Okay, and now we're going to talk about this step for a little while after I run it. But what we're going to do is we are now going to say vect.transform simple train. Okay, vect.t transform simple train. So we are running the transform method on the fitted vectorzer and passing it the same text and it's outputting something I'm calling simple train DTM. DTM stands for document term matrix. Same thing as a term document matrix except maybe a different orientation. But let me go ahead and well I'll run it. it's going to so um just like model fitting there there's no output it just happens in place uh with vectorizer fitting it also occurred in place whereas vectorzer transformation sorry vectorizer transtransform method does output an object our document term matrix and I'm just going to print it out if you will and it says it's a 3x6 sparse matrix with nine star stored elements. Okay, we'll come back to a sparse matrix in a minute. But who can tell me why it is 3x 6? And this you probably only know this if you already know what a document term matrix is. There's three entries and six features. Yes, three entries, six features. 3x6 document term matrix is what it's called. Documents are the rows. Terms are the columns. Documents are the pieces of text. Terms are the tokens. These six tokens or features. He su he kind of jumped ahead and said like these are the features. Well, this is going to be our feature matrix. Okay, we just extracted features from text. Okay. So, it's a 3x6 matrix because there are three documents and six terms. Okay. All right. Let's take a look at it uh with some visuals so that we can kind of well visualize. And it turns out that document term matrices have a two array method that transform it from a sparse matrix to a dense matrix and uh like a normal matrix if you will. And here is our text as numbers. Okay, so the title of this section was representing text as numerical data. This is our text as numerical data. Okay. So, if you're trying to figure out what that is, I'm going to put it in a data frame and I'm going to put those feature names as the column headers. Okay. So, what have we done? Uh, call you tonight has become 0 1 0 0 1 1. Call me a cab has become 1 1 0 0. and please call me please has become 0 1 1 2 0 0. Okay. So what we've done is created a document term matrix. Each document is a row and it's simply counting the number of times a term appeared uh in that document. Okay. So this is known as the bag of words strategy. And I'll just read this uh this briefly. Each individual token occurrence frequency is a feature. So the columns are features. The vector of all token frequencies for a document is a sample. Okay, sample observation. Thus a corpus of documents corpus means collection of documents. It can be represented as a matrix with one row per document, one column per token. Okay, what we just did is called vectorization. That's why it's called count vectorizer because this is vectorization. This is called the bag of word strategy. How many people have heard of bag of words in the past? Uh, keep your hand up if you weren't really sure what that meant when you heard it. Maybe most of you. Okay, that's fine. Bag of words. It sounds all fancy, but this is just ignoring word order and counting things. Technically, you could normalize it and whatever, but the simple bag of words strategy is count them, forget about word order. Okay, it's called bag of words. It's like throwing the words in a bag. You can no longer tell the order. Okay, all you can do is pull them out and count them. That's it. So, that was vectorization. All right. Before uh we come back to using this uh document term matrix, we're going to talk about sparity very briefly in case you are interested in why it's a sparse matrix. But uh we'll just check the type of the document term matrix, this 3x6 matrix. And I'm going to go ahead and print the comments, sorry, the contents. Okay, who can tell me what we are looking at? Nonzero elements. One more time. The nonzero elements. The nonzero elements. What is on the left and what's on the right? The position in the in the table and the value. Great. So on the left is the position in the table. On the right is the value. Okay. So why did it do this? Well, uh the psych learn documentation on this is is quite good. I'll just read this out quickly. As most documents will typically use a small subset of words in the corpus, the resulting matrix will have many feature values that are zeros. For instance, a collection of 10,000 documents will use a vocabulary of 100,000 words, but each document will only have 100 to a,000 words. Okay? So you can imagine with a lot of documents you create a really wide document term matrix, right? It's wide because there's one column for every word. There's a lot of words. There's a column for every word that appears at least once in any document. And you can imagine you have a lot of documents. It has a lot of zeros going across because most documents don't have most words. Okay? So in order to be able to store such a matrix in memory but also speed up operations, implementations will typically use a sparse representation. So a sparse representation is you don't store the zeros. You only store the locations and contents of the nonzeros. Okay? If you didn't know, if you have a numpy array that's filled with a million zeros and a single one, it will still take as much space as a million and one random numbers. Okay, it it like zeros are you have to store them. But in a sparse representation, you only store the non-zeros. Okay, um that is just so you know why uh count vectorzer outputs a sparse matrix. It's for storage purposes um and for computational purposes. Um and I will mention that uh you probably didn't check it out, but on the readme page of the repo, I have about I don't know 20 different resources that relate to today's content. Um so that if you're interested in this content and you want to learn more, you can read through those or check out videos, whatever. One of those links is about sparse representations in scikitlearn and implications on performance. Yeah. So, uh you're asking so she was asking about whether I'll be talking about waiting important words. So the strategy I'm going to present uh today will not use any waiting. You could use instead of count vectorizer, you could use TF vectorzer, which is TF transformer. Wait, count vectorzer followed by TS IDF transformer, I think. Um, but the bottom line is that will normalize around rarity um or word importance, if you will. Um, I won't be getting into that today. Um, partially because you you got to learn to walk before it can run and partially because you don't always need TF for machine learning. Um, turns out that non-normalized can sometimes work better than normalized. Say you want something like you want to add a feature like maybe like the average word length. Can you can that fit here? Do you have to use a different model? Great great question. Okay, so his question was I want to add another column to this. Okay? Because if you didn't pick, if you didn't catch this part, I know I'm speaking fast because I've got a decent amount of content, but if you didn't catch this part, what we're looking at right here is our feature matrix. We are going to train our model on this. And he's asking, what if I want to throw in another feature like average word length? Okay, so there are threeish good ways to do that. One is um you can use uh you can build your feature matrices separate and and hstack them together using scypi. H stack means is like vstack is like stack rows. H stack is stack columns. So you can do that um but you have to convert them both to sparse because you can't add a sparse and a dense matrix together. You have to make them either both sparse or both dense. So you add the matrices. The another option is you use something called feature union which is harder to learn but but integrates better into the scikitlearn workflow. It's called feature union and that allows you to make unions of features. And the third option is a way to avoid that by training two separate models one on the text one on the non-ext features and ensembling them. Okay. So those are your three options. Um, I could it would take a w like an hour to talk about all of those or even a lot longer. So, I'll just leave it at that. But those are are good options. Okay. All right. Uh, other questions? All right. Um All right. Let's uh let's see. Let's um finish off this section. So, we're almost done with this section, but there's something really key I haven't shown you that is going to be critical to using this material effectively. Okay, so we're going to pretend to build our model to predict aspiration. Okay, so we're going to say uh KN&N equals K neighbors classifier. And I'm going to say N neighbors equals 1. You'll know why I'm doing that if you know KN&N. And if you don't, it's fine. I'm just modifying a model parameter. And then I'm saying KN&N.fit fit simple train DTM is desperate. Okay. So instead of fitting X and Y, I'm fitting my X which is simple train DTM a 3x6 matrix with is desperate instead of Y and is desperate is a list of length three. Okay, so it's 3x 6 is my X and three is my Y. Okay, so I'm learning the relationship between the words in the document term matrix and is desperate. And then if I want now I' I've got my train model. Okay, so the challenge is how do I now make a prediction for a new piece of text? Okay, so let's pretend you built a classifier for your phone where you only you wanted like a little red icon by all desperate text messages and uh so we've trained it on past ones that you've hand flagged as desperate and now it's got to make a prediction for every new text message that you receive. So that is the uh that is what we're doing. Okay, so here's our example text for model testing. Please don't call me. Okay, it is a list. This is pretend once again. Um, this is a list of one string. Okay, and I want to remind you of that uh that takeaway I talked about 20 minutes ago. In order to make a prediction, the new observation must have the same features as the training observations in both number and meaning. In other words, I can only pass this to KN&N.predict if it's numbers, if there are six features, and if those six features mean the same thing as they did before. Okay, those are our restrictions. Okay, so you're thinking, how do I do that? Well, here's what we do. We say vect.t transform simple test. We skip the fit. We only do the transform because fit is where a document term sorry fit is where vocabulary is learned. transform is where the document term matrix gets created. So we don't want to relearn a vocabulary. We just want to use the existing vocabulary and build a document term matrix. So I'm going to say vec.transform simple test store it as simple test dtm and uh convert it to a dense array dense matrix and I get 0 1111 0 0. Okay. And let's look at that in pandas. And what happened? Lost the don't. All we've got is please call me. Okay. So, it did create a feature matrix with the same six features, which is super important. Same six features. They meant the same thing. It counted correctly in please don't call me. That please was once, call was once, me was once. Don't is missing. Here is the toughest question I think I've asked you, which is why is it mostly okay that it dropped don't? Why is that mostly okay? One more time. Uh, sorry, one more time. The model does not have it in the formula. The model doesn't have it in the formula. Uh, I'm going to restate. Uh, let's see. Yep. Since it's in the bag, you won't know what to do or what? Uh it's not about that's a little that's kind of correct but it's a little deeper than we need to go here. It's not about like don't is negating and it's um it's closer to what the first gentleman says that it's not in the formula and I will restate that as the model does not know the relationship between the word don't and desperation. Okay, think about that for a second. Okay, let's pretend we're talking about the iris data and we have sele length, sele width, pedal length, pedal width. We train a model on those four features and then we make a prediction when someone like measures a flower and says, "Here are the measurements. What type is it?" And we put it in our model. Well, let's pretend you were doing that and someone said to you, "Hey, I think numb petals would be a useful feature for this model." you as the person putting it in in code would say, "Well, that's great, but my training data doesn't know anything about the relationship between num pedals and the class." Right? The model has never learned any relationship between the feature we didn't put in and the class. So, it's cool that we could create a new feature called don't. And maybe don't is or is not a good predictor of desperation, but we'll never know because it wasn't in our training data. And I think that's what he was getting at. So it wasn't in our formula. The model didn't learn the relationship between don't and the response, so we just drop it. Okay? And you might say, well, there's predictive value in like what words you drop and how many words you drop. And you might be right. I don't know if that's empirically true that it matters how many words you drop. Um, with a big enough training set, you're you're not going to drop that many words anyway. Um, okay. Does that make sense? Any questions about dropping that word? Okay. Um, finally we will make our prediction. We will predict whether simple test is desperate. KN&N.predict predict simple test DTM. Okay, I'm passing it that document term matrix with one document and we get a prediction of one. We get a prediction of desperate. It probably learned I mean the problem here is inadequate training data. It probably learned I mean we know what it learned. It learned that uh saying please twice is a good predictor of desperate and it learned that call is a better predictor of not desperate. So when it saw please um anyway that's just uh you know a quick interpretation. Um the point is it made a wrong prediction but that's just because uh or I would judge it as a wrong prediction and that's just because it didn't have enough training data. The point is um well really we've gone through the entire life cycle here. Okay. So let me quickly summarize what we just learned in this section. Vect.fit on train learns the vocabulary of the training data. Vec.t transansform on train uses the fitted vocabulary to build a document term matrix. And ve.transform transform on test uses the fitted vocabulary to to build another document term matrix ignoring tokens it hasn't seen before. That's just how it works. Okay. All right. Any questions on that? Yeah, please. When you use cross validation, you're going to change the number of features based on which sample you're Yes. How does that? So, uh, so um I think I know what you're asking. uh he well he's asking about uh how cross validation would play with this because um each time you will have a different set of training data in your 10 five or 10 or whatever different folds and what happens when you do proper cross validation with text data is you actually do the the feature extraction within each fold of cross validation. So you do you actually do the split into 9/10 and 1/10th. You train on the 9/10s and you uh fit and transform on the 9/10 and you predict using transform only on the 110th and you redo that vectorization process 10 times for 10fold cross validation. So to actually do that you have to use a pipeline and you do a pipeline of vectorization and um model building but um or sorry um yeah of of cross validation of your model. Um if you've never heard of that pipeline cross validation etc don't don't worry about it. Um uh the point is there is some added complexity using cross validation when you have text data because you are doing feature extraction. Okay, other questions. We're going to apply this very soon or rather right now. So if you're tired of really toy data, we'll have a slightly larger data set. Okay, reading a textbased data set into pandas. Um, if you're following along, uh, this what you'll want to do is make sure that SMS.tsv is in the same working directory as your notebook. Okay. So, uh we're going to use pdread table. Uh it's a tab separated value file and I'm going to set the column names. Okay. And we're going to store that as a data frame called SMS. If you get an error on that line and you're not sure what to do, but you have internet access, go ahead and uncomment this and run that instead. Okay? and it'll just read it from a from the GitHub repository because pandas can read from a URL. Okay. All right. So, we've got our SMS data frame. I'm going to look at the shape attribute and we've got two columns, 5,000 rows, and we'll take a look at the first 10 rows. And this is a data set of SMS messages that have been marked as either HAM or spam. Ham is non-spam if you're not familiar with the term. So you've got spam email and you've got ham email. Okay, I think it's brilliant. Whoever came up with that ham is non- spam. I think that's hilarious, but you may not. That's fine, too. Um, so uh yeah, these I don't know what language all of this is in. They're kind of Englishesque. Um, so I don't know I can't remember the source of the training data. um but they are real SMS messages that have been uh um handmarked as ham or spam. So we're going to take a look at the class distribution. Um I'm going to use the value counts method on the label series. So if you've used pandas, I'm using dot notation. SMS is the data frame. Label is the series. Value counts as a series method. And it's telling us that it's about 80% ham and 20% spam. And that's relevant to check out because you uh it's hard to build a model when you have uh severely imbalanced classes. So um it's useful to know your class balance. All right. The other data preparation we're going to do is we're going to convert the label to a numerical variable. Okay. So I'm going to use the map method and map ham to zero and spam to one and store it as a new series called label num. And uh we'll check that it worked. And you can see that each ham is now a zero. Each spam is now a one. There are some reasons uh that it's handy to encode your response as a numerical variable. Um and it used to be required for some classification models but I think today in the latest version of scikitlearn you can actually use classes that are non- numeric okay your feature matrix has to be numeric your classes your your response value technically I don't think has to be numeric for any model okay but in any case we're going to use label num instead of label all right so Um, now we've got to do some vectorization. But before that, we're going to do a train test split for model evaluation. But before that, I have to define my X and Y. Okay, so I just want to remind you of how we defined our X and Y when we were working with the iris data. Okay, X was um well, whoops, sorry. X is 150 by 4. Y is 150 by nothing. Okay. When I'm defining X and Y for eventual use with count vectorzer, I'm actually making two objects both of which are onedimensional. It's not 5572 by something. It's 5572 period. Okay. So why why when I was building a model with the iris data did I uh have a two-dimensional object for my feature matrix and why do I have an an a a a uh x that is onedimensional in this case you haven't vectorized I have not vectorized and what is going to happen when I vectorize Yes, it will turn from a one-dimensional object into a two-dimensional object. Count vectorzer will create the second dimension. So, you need to start it as one dimension because it will expand out from kind of like you can picture it as like well a pandas a panda series which is just like one big column and you can picture it expanding where the rows are still the messages but the columns are the features, the tokens, the counts. Okay. So, does that make sense? So, one-dimensional object will become a two-dimensional object when we vectorize. Okay. All right. Before I vectorize, I'm going to do train test split. Train test split I can't talk about in depth if you're not familiar, but it's a technique for model evaluation. And model evaluation is a way you evaluate your model uh by pretending that the past is the future and then predicting the future and then you can evaluate it. So you want to choose between models or you want to choose model tuning parameters. Okay. So we're going to use train test split for model evaluation. And this is kind of the uh standard way I do train test split. Uh well I mean uh this is this is going to change by the way in the scikitlearn that just came out like two weeks ago. They broke the API and now there's a different module called model evaluation. Whatever for now this is how you do it but you will have to update this code at some point. Um train test split X and Y. We're going to break it into X train X test Y train Y test. Okay objects. Um whoops sorry wrong cell. And uh the x's are uh well the x the training data is now 75% of the data. The testing is 25% of the data. Okay. So we've broken it into four pieces. All right. Uh that is that is the default. Yes. He was asking what's that is that the default? And yes the default of train test split is 7525. You can change that though. All right. Another tough question, but I think you've got it. Um, why are we splitting the data before vectorzing? We are going to vectorize next but we have split into train test split before vectorzation. Why is that? I did that on purpose. But what is that purpose? Uh, yeah, I have it right here. You don't want to train with your testing data. You don't want to train with your testing data. Yes. Um, there's a little more to it. Uh, you don't want to train what? Train isn't the word I'm looking for, but you don't want to train what with your testing data. This data is unknown. Should be unknown to your model. It should be un It's still going to be unknown. There's a very subtle point here. The number of features should match only the training. The number of features should only match the training data. Okay, I'm going to rephrase that. That's right on, but I'm going to say it in a different way, which is you the whole goal of model evaluation is uh to simulate the future. Okay, that's how model evaluation works. You pretend the past is the future and then you predict the quote future but you can evaluate because you know the future because it's the past. Okay. So what happens in the future? Well in the future you will see words you have never seen before. Thus you don't want to run count vectorzer on all of your data and then split your document term matrix. You want to split your data and then vectorize it so that you have an accurate as possible accurate as possible simulation of the real world in which you will see data in your test set. You will see words in your test set that should be ignored, right? Because if you vectorize then you split. You are not going to ignore any tokens when you do your testing. And ignoring tokens is a good thing for good model evaluation because in the future you will see words you have never seen before. Okay. If that that's like a really subtle point if you're new to this uh if it was me it would blow my mind and I'd figure it out in in like a month. But um hopefully that's helpful to those of you that are kind of uh following along. Any questions about that? Sorry, can I say it again? Why why you train be sorry why you train test split before uh vectorization. Okay, if you vectorized first and then split, which is what we don't want to do, but if we did that, you would split a document term matrix into two pieces. And uh when you did the fit and transform on the first piece, sorry, you've already done that. Um, you want your um your testing document term matrix to have features that have never been seen before because that is a simulation of the real world. In the real world, you will have words in your out of sample data that you have never seen during training. And the only way to simulate that is to split before vectorization. But even that split there still chance that the test where some entry doesn't have the word show up showed up in the train. Uh yes, and that's a good thing because that is an accurate simulation of the real world. If we I'm happy to talk more about this. I'll um I'm gonna go ahead and move on, but I'm I'm happy to talk about any of this in depth uh afterwards. Um okay, so we've done our train test split and now we're going to vectorize. And this is going to be stuff you've seen already. Um we're going to instantiate the vectorzer once again. We're going to learn we're going to do ve.fit on x-ra and vec.transform transform on X-ray because remember you do the fit on X-rain you do the transform on X-rain and it outputs a document term matrix okay now you could if you're thinking boy that seems so efficient to run use the same object twice in a row why don't they combine that and they did and it's called a fit underscore transform so a fit underscore transform is a fit followed by a transform. Okay, so that's more computationally efficient. So in the real world, people mostly just do fit transform, not fit then transform. Okay, but I do it separately when I'm teaching because I want you to understand the difference between the fit and the transform. Fit vocabulary learning transform is building the document term matrix. Okay, so we've built X-train DTM. We're going to examine it and if you are like following along and you're getting everything you can tell me what is 4179 and what is 7456 that is the size of the sparse matrix but why is it that size and how about someone I have not heard from right there Okay. Great. Great. So he said 4179 is the number of examples also known as observations also known as records and uh documents and 7456 is the number of terms also known as tokens also known as features. Okay. So 4179 7456. Okay. Yes. Shouldn't you shoot a vector on the X and then apply the transformation using that vector app separate? So you get the same columns. Yes. So you get the same test. So what we've done so far is all on X-Rain to create my training document term matrix and actually the next step is to create my testing document term matrix and it is going to have the same number of columns. Who can tell me what the what the size of X test DTM is going to be without running that line of code? What is X test DTM the size? 133 1393 by 7456. All right. So the number of documents is the number of documents in the testing data. The number of terms is the same because we did not relearn the vocabulary for the testing set. We reuse the vocabulary that we learned during model training. Okay? So that's why they're identical. And not only is that like a good thing, but if you had a different size, it's like you are going to have failure during like model prediction. Okay? So you have to do this. This isn't just like a good idea. Uh this is, you know, you have to do this. Okay. Let's see. We have got 20 minutes left. So I will um maybe not get to section six, but we'll see. uh we're going to build and evaluate a model uh in specifically we're going to uh build a multinnomial naive bay model uh we're going to treat it as a blackbox uh if I was teaching you about naive bays it would take me an hour or two perhaps and we don't have that kind of time so we're just going to treat it as a black box however it is popular for working with uh text okay uh it it turns out that with text data there are lots of nice uh attributes of knife bays, which is why knife bays is is um commonly used with text data. Okay, so what are we going to do? Uh we're going to use the same pattern we've used before with models. Import, instantiate, fit, and predict. And we're going to add on a step of evaluation. Okay, so from sklearn.nbay import multinnomial knife bays. There are multiple different knife bases bases I guess um knife by no I don't think that's right um knife bases in um scikitlearn Gaussian knife base and um we're using multinnomial knife base and uh we're going to oh sorry yes please neighbor as well uh you could you could um so k nearest neighbors is not going to work well with wide data. So, um this is another like super long topic of how do you choose a model? But K nearest neighbors is a good model. Well, some people never use it, but it could be a good model when you have a very narrow tall data set. Small number of features, large number of examples. This is large number of features, smallish number of examples, not going to do very well. Okay. Um, so we wouldn't in this case. Um, but uh you could always try. Um, I I think it would should still work. Um, okay. So, we're going to do the fit and I'm going to use an IPython magic command called percent time. Um, so you just put percent time in front of something and it will time it once and it will run it. So I'm training I'm fitting the model to X-rain DTM and Y train. Okay, so we built the document term matrix so that we could train a model on it and we've got Y train which we you know created during train test split. Okay, so uh at least for me that took 6 milliseconds. Okay. Um, and turns out that is fast in terms of the average model. Okay. Um, and we'll come back if we get to section six, I'll mention something about speed. Um, when we're making our predictions, we again need to use a document term matrix. We need X test DTM to be the same width as X train DTM, right? because it has to be the same number of features. Okay, so we're doing nb.predict x testdm we're storing the class predictions in ypred class. So we're making class predictions uh and we're storing them and then we are going to evaluate uh the model meaning calculate the accuracy in this case. Uh we're going to import from metrics and um we're going to do metrics.accuracy score y test ypred class and uh who wants be who who has not run this but wants to guess the accuracy. So we've used the bag of word just to summarize we've got this data set of ham and spam messages. All we've learned is what words, if you will, are hammy and what words are spammy. That's a summary of naive base. Okay, it's learned what words are hammy. What words are spammy? And if a message has a bunch of hammy words, it's probably ham. And if have a if a message has a bunch of spammy words, it's probably a spam message. And that's it. So, how accurate is it going to be? Percentage. Anyone want to guess? 80 80% we have a guess here. Other guesses, keep in mind null accuracy here is going to be about 80%. Which is the accuracy from always predicting the most frequent class. So a model is very likely going to do better than null accuracy. So it's going to be higher. But how high will it be? 95. We've got to guess any one more guess. 99. All right. And what we have is 98.9%. What? That's amazing. Um, you should just be blown away. But this I want to pause for a second and I want you to look at that and think like, wow, that did pretty well without oh, I don't know, tuning count vectorzer. um without really like I've given you a formula right now with this notebook that you can apply to any textbased classification problem. Think about that. You could go back to work do something like this. They'll be like whoa that's amazing. And you'll be like yeah I know I mastered machine learning and you'll know all you did was use scikitlearn and the count vectorizer. didn't tune a thing and yet you may not get 98% accuracy but that the rule in kind of text classification is simple works pretty well and it's a lot of work to do a lot better. Now it got 98.8% 8% because there's a lot of signal present in the data. Meaning this data set has like the spam messages are distinct. The ham messages are distinct. Well, the spam messages are more distinct in the words they use. So, this is like an easy problem. So, I'm not saying using count factorizer and this these techniques will always get you 99% accuracy. It usually won't, but it will probably surprise you how well it will do on your first pass. Can you explain the 80% again? Yes. Okay. So, um up here, she asked explain the 80% about what I said about null accuracy. So, about uh let's see SMS. Uh I guess will work. Whoops. Uh I must have red. Oh, wait. uh label num sorry okay uh how about one minus all right I was wrong it is 87% don't worry about that line of code if you can't immediately decode it but 80 87% of the messages are uh ham okay so a dumb classifier that always predicts ham will be right 87% % of the time. That's called the null model. And null accuracy is what accuracy could be achieved by always predicting the most frequent class. So you'll probably notice in like every article you read about machine like a machine learning model on the internet will be like, hey, we got 90% accuracy. They need to tell you like the class distribution so you can see whether that's impressive or not. Because if the model only did 88% it's a crappy model because you don't like the model doesn't need to learn anything to get 87%. That make sense? Okay. All right. Um All right. So, uh let's see. I'm going to use we're going to evaluate with a confusion matrix. Uh if you are not familiar with the confusion matrix, you can Google confusion matrix and you will probably see my blog post at the top which is pretty cool I thought. Um so uh confusion matrix is about um well we're identifying our types of errors, our true positives, true negatives, false positives, and false negatives. Um, I'm just thinking through right now how I want to spend our remaining time. And I think I will actually skip a couple things so that I can hit uh part six. Um, yeah. And I will just mention uh we are also going to How many folks are familiar with AU area under the curve? Okay. So you can calculate AU in uh scikitlearn using metrics. And what you need is predicted probabilities. You don't need class predictions. You need to do you need to calculate predicted probabilities of class membership. Unfortunately, I don't really have time to go into that. Um I do have a really long video about that. If you'd like to learn more, I can tell you after. But um these are the predicted probabilities of spam for each message. So it's saying the model is saying that for the first message in the test set there is a 0.002 02 uh well I guess point let's uh wait uh 28% chance of it being spam for the second message there's like a really small chat chance it's spam etc these are probabilities it's calculating of spam now one weakness of naive bays as a model is that its predicted probabilities are poorly calibrated ated. What that means is you get very extreme values. Okay, sometimes well-c calibrated predicted probabilities are useful. Okay, if you want to interpret a predicted probability as a likelihood of something, you want it to be well-c calibrated. This is actually kind of a big topic. I won't really have time to go in depth on it, but again, happy to talk with you in the hall about it. Okay, so we're going to calculate our AU. we pass it Y test our true testing set values and our predicted probabilities we get a very high score um and that is um well it's hard to have a really high accuracy and not have really high AU because of how how it's calculated but in any case it did well in both cases. Okay, the final thing I'm gonna do because this ends at 4:15. Um the final uh let me check that. Yep, 4:15. The final thing I'm going to do is we're going to redo a tiny bit of this process with logistic regression. We're going to talk about the difference and then I will wrap up with some I guess parting thoughts. Okay. Uh we're going to compare naive bays with logistic regression which is a model for classification. Not a model for regression by the way. It's a classification model. Okay. Very popular. Um and we're really just going to repeat the steps we've already done. Okay, so the idea is I could use knife base, but maybe logistic regression will do better or worse, who knows? So, we're going to import and instantiate it. We are going to train a model. We're going to train our model, do our fit, and the time it is 86 milliseconds, which is like 15 times slower than knife bays. It's not slow, but um it is slower, which is a I don't know if I'd call it a weakness of logistic regression. It's just a strength of naive base that it's really fast. Logistic regression is slower. That's why I like to start with naive base. Okay. Uh we're going to make our class predictions. We're going to calculate our predicted probabilities. And turns out that uh logistic regression one of its strengths is that it creates better calibrated predicted probabilities. Okay. Um what do you mean by that? So I'm trying to think how I can um Okay, let's pretend let's pretend uh I want to predict the likelihood well the likelihood that a message is spam. Okay. And I say the likelihood, my model says the likelihood is 8%. Or the likelihood is 2%. Or the likelihood is 80%. Okay. So models output predicted probabilities and I'm interpreting it as a likelihood. With logistic regression, they are they are it is easier. It is more likely that they actually represent likelihoods. with naive bays, they don't represent likelihoods. They represent extreme numbers. I mean, they still kind of do, but they're not as good. So, there are certain types of problems in which you care about um predicted probabilities. Okay. Bottom line is its accuracy and its AU are still really high. Okay. Um All right. I am just going to say a few more things and then I'm gonna end right on time because I know the speaker will need to set up and there's actually no break I don't think so I want them to be able to set up right at 4:15. Um so let me just conclude with a few thoughts and um if you never went to the repo or you didn't really look at it go back to the repo again it's bit.ly/patc /patc and you'll see all these files are here. A notebook with the output already run is there. You'll also see a dozen resources at the bottom you can check out. One of those resources is a video series about scikitlearn that I made. It's about 4 hours long. Um I recommend checking it out if you are new to scikitlearn or intermediate maybe. Um I've also got a video series about pandas if you're new to pandas. It's about seven hours long, 30 videos. Um, check that out. Free on YouTube. Um, if you like my teaching style, I'm gonna just gonna do a very uh brief pitch to say I'd love it if you join my newsletter. And the reason is that's how I announce things like new tutorials and access to new courses I create. And it's the community I give stuff to first. So, if you want to sign up for my newsletter, go to dataschool.io. io. Um or I've I'm going to put a signup sheet. Well, I'll probably just carry it to the hall, but I've got a signup sheet. You can do it that way. Drop your business card with me. I'll add you. Um I'm going to be announcing a free webcast about scikitlearner pandas probably next week. So, if you want to just come like chat about scikitlearn and pandas and learn, ask whatever questions you want for free. Um I'm going to be doing that soon. So, bottom line, uh please sign up. Um, I do have uh normally I've been like standing up here the whole time so you like this image is now in your mind, but I have some stickers and magnets from data school um if you're interested that I'll bring in the hall. Um and uh finally feel free to keep in touch with me if you go to dataschool.io. My email address is there. Feel free to shoot me an email at any time. Um if you go on YouTube, you'll see I respond to every YouTube comment, nice or not. Uh they're mostly very nice. Um but uh yeah, feel free to stay in touch. Um this was really a lot of fun and um I will take more questions in the hall, but I'm going to go ahead and say goodbye. So thank you again. You guys were a really excellent class and uh yeah, have a great PI data DC. So hi again. Thanks so much for watching the video. Uh, if you have any questions about anything you saw, please let me know in the comment section below and I'd be happy to answer your questions. Now, as I mentioned before the video, I do have an online course called machine learning with text in Python that goes a lot deeper into these topics and um is a really great course if you want to master supervised machine learning with scikitlearn and learn a lot more about natural language processing. So, there's a link in the description below and you can read more about the course that way. Uh, finally, I'd really like to have you be part of the data school community. So, uh, there's a link below to join my email newsletter list, and I'll be sending you lots of great data science content. So, thanks again, and, uh, I hope to see you again soon.