Text Representation Using TF-IDF: NLP Tutorial For Beginners - S2 E6

Key Takeaways

in this video we will talk about tf idf representation for text in nlp in the last video we looked at bag of words bag of engrams if you are not seeing those videos i highly recommend you watch them first before watching this video we have been looking at news article classification problem where a news article is given to you and you have to classify it into one of the companies whether this article is for tesla or apple etc and in back of words model what we did was we'll take an article and then design this vocabulary doc vocabulary is basically all the words from all the news articles and then we will take count of individual word in a given article for example in article one word musk is present zero time that is present 32 times and so on and just by looking at this vector right now you can actually tell that first two articles are apple articles because you see these terms iphone itunes they appear seven times three times six times three times whereas in the other two articles see musk appeared 15 three times one time so based on these word count you can say yes this one is for tesla and this other articles are for apple but look at this other terms that price market investor now these are very generic terms that can appear in any article i used to work for bloomberg and we actually were working on extracting company from earning estimate document and you will find this kind of generic terms too often in all the companies you know like stock market company profit loss this is generic and the presence of these terms in your bag of words model can suppress the meaningful words so here for i apple iphone itunes is the relevant term and for tesla musk and gigafactory are relevant terms but these other generic terms are suppressing the influence of these relevant terms and in this example i'm just showing you for that price market investor four terms but actually there could be 100 such terms and look at what happens to the vector in the first and third vector these terms are present with a equal amount see 32 31 45 44. so ignoring all the terms if you just look at the vectors and you give this vectors to a computer computer will think article 1 and 3 are similar actually they are different right one is from apple the third one is from tesla but if you look at just the vectors see this 7 3 is not matching here it's 0 0 but see other terms are kind of equal and there are four of them in reality there could be 100 such terms in your vector and your computer will think okay these two are actually related articles you know but that is not the case so how do you ignore these terms that's the question see you don't want to ignore them completely but still you somehow want to ignore it someone will come and argue that okay that is a stop world why don't we just remove all the stock works fine we removed that but then how about all these terms price market investor so stop words can help but to a certain extent we need to do some additional thing so let's think about this together let's pause this video and think what can we do or what should be our approach okay you just pause the video and you will be able to come up with that idea so one idea is we figure out in how many documents the term that appeared so that appeared in three documents out of four and i'm not talking about the individual word count in a given document i'm saying in how many documents this term that appeared it appeared in three documents article one two three out of total four documents similarly gigafactory appeared in only one document an iphone appeared in two documents this is something called document frequency number of times tom t so t here is that gigafactory etcetera number of times term t appeared in all the documents and if this number is higher if some term is appearing in majority of documents we should lower its influence because it's like a generic term but if some word is appearing in only few documents such as gigafactory see only one document iphone only two documents then that term is important and we need to give it a high score so we need to come up with some scoring mechanism here and obviously the scoring mechanisms should be such that you know higher the term appears in all the documents the influence would be lower and you can do that by doing inverse document frequency so if that term is three so you invert it okay then the number goes down and you don't do simple inversion but you do this formula total number of documents divided by number of documents tom t is present in for example total documents are four so numerator in our ratio is is fixed 4 but denominator is how many times that appeared in all the documents well 3 times total number of documents are 4 that appeared in in three documents so 1.33 total number of documents are four giga factory appeared in only one document so this number is four so now see when the term appears in less number of documents the score you are getting is higher see this 4 is higher than 1.33 and this is called idf inverse document frequency there is something additional to this formula which is called you take a log of this why do we take log i will explain in a minute or so but hold your thought for now and in your brain you should be thinking if the term appears in more documents the final score for that will be lower because you want to reduce the influence and you will come up with this kind of vector which will be called idf vector 0.1 to 0.12 and so on i mean i i'm just showing this we don't use idf we use tf idf which i have to still explain but for now i hope the concept of idf is clear in your head we also need to take into account the word frequency because that is also important correct but we don't want to just take a word frequency like bag of words because see some articles could be bigger let's say there is a 5 000 word article and there is only 10 word article small article you want to be fair and kind of normalize things so what you should do is total number of time term t is present in document which is just a simple word count that we took in bag of words model divided by total number of tokens in that document and that we will call term frequency so here t is the term and d is that document in which you are counting that term frequency so here for example the term frequency in article 1 for the term market will be 48 divided by total number of words in that article let's say in that article total number of words a thousand so the term frequency for market in article one will be 48 divided by thousand and tf idf is just a multiplication of these two terms and we will use that in today's tutorial to do our coding so we'll do again a text classification for e-commerce goods and we'll be using tf idf representation here i'm just showing some sample numbers i did not actually compute it but what i want to show you here is terms like iphone itunes gigafactory will have higher numbers 0.9.8 whereas terms like that price will have lower number okay one more thing i want to clarify is learn uses slightly different formula for idf to take into account the zero division possibility so what they do is they add constant one in numerator and denominator both and then the result they add one more time so it's just a little customization on your idea formula sklearn uses that this screenshot is from the skill on documentation now let's talk about why we used log you can ask this question to your dear friend who is sitting next to you and that dear friend is actually google so if you just google this question you'll find the stack overflow answer where they say that log is used to dampen the effect of idf and unki has given a very nice answer here so i'm just going to take a screenshot of that answer and show it here credit goes to stack overflow which is let's say in a work in a document you have a term called computer which is relevant for your given nlp problem and if the term appears one million times you know that that term is important is and and i'm talking about that one term appearing in your document right in your specific document so now if it is one million or two million it is all same correct doesn't matter so that's why we kind of take a log of that just to kind of reduce the effect of it if you know from your log function your log function goes something like this you know log of 1 is 0 log of 10 is 1 log of 100 is 2 i'm talking about log to the base 10 and the core kind of flattens out as you go further in your x-axis and the influence that you want to have in in terms of your idf you want to reduce it because as you have more terms the effect is same so you can just reduce the influence now the limitation of tf idf model is similar to the previous model that we saw before which is a as you increase dimensionality the sparsity increases as your vocabulary increases obviously the sparsity increases it's a discrete representation of your text it doesn't capture the relationship between the words the relationship between the world is captured by word embeddings and sentence embeddings but all other models they suffer from this problem where you have relationship between the words it's not captured because here you're taking a term and you're just taking account out of it and it suffers from out of vocabulary problem as well let's do some coding now i opened jupyter notebook and imported tf idf vectorizer from and i'm going to take one sample corpus and the meaning of corpus is the collection of documents in your nlp application i'm just going to take some simple dummy sentences and will create tf idf vectorizer out of it so the way you create the instance this is tf idf vectorizer is a class and you just create an instance of it and you will just say fit so when you do fit you are creating that vector basically right and then you're kind of training basically and then the creation happen when you say transform you can do fit transform as well just one call i think let's do that fit and transform and we will call it transformed output is that okay and once that is done the v object will get its vocabulary so i want to print a vocabulary first and i'll just use print so that i can see everything in like two or three lines see all this vocabulary how these indexes for example already the term already at a is at zero apple is that six and so on now i will print idf score so v this parameter v if you do like d i v you will see all the members and you will see idf for example as one of the members idf smooth idf actually we are using smooth idf and so on so let me first print just a simple idf score out of it and for that you can either iterate through this particular directory v dot work capability or if you want to go in sequence if i iterate through this directory i'm kind of going randomly over the places but i want to go in sequence where i take the zeroth index element first and so on and for that we can do v dot get feature names out and that will print things in order basically all your features all your vocabulary already zero m is one okay when you do tf idf vectorization these things are actually your features and based on the features you will calculate your vector so i will just call this all feature names and then i will iterate through it so i will say forward in all feature names i will first get the index and index is v dot vocabulary dot get because it's a dictionary you can call get method and you get this particular index right like 27 11 whatever and when you get that index uh you can then say v dot idf that will have your idf score and you can retrieve specific score for a word by using the index operator and you can just print that you can print let's use the format strings here and we will say word and afterward we have the score okay so this is the score now you will notice one thing which is this term is is present everywhere see in most of the document it is present except this one so you expect this tf idf or let's say idf score for is to be lower and the term like apple is appearing rarely so the score of apple tesla etc should be should be higher so check this is is 1.1 whereas amazon or apple is 2.5 so it is working and we are more interested in the the whole tf idf score so what i will do is now i will print that and that score is present in transform output but before i do that i will let me just print first two sentences from the corpus okay so these are the first two sentences from the corpus and i want to print the corresponding tfidf vector for these two sentences so that that is present in transform output correct we got that in transform output so transform output dot to array it's a sparse matrix so you have to convert it to an array and you just do two so this is the first one and this is the second one now watch this uh let's see we are looking at is okay and is is at i think position 17 right so is if you look at it here it is 17 so is position is 17 so there are total 6 here so the position 17 will come let's see where so 6 6 so this will be 0 1 2 3 okay 0 one two three four actually there are only five i see so this will be 15 16 17 it will be this one i think this one actually oh see 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17. so see the idf tf idf score for is in the first sentence is point one zero and the score for thor for example is how much so where is my thor thor is 27 [Music] so this is 17 18 19 20 1 2 3 4 5 6 7 thor score is 0.24 so it is more than is so just consume this i mean you'll get an idea that the things are working as expected the score for the term which is appearing in too many documents is lower and the score for the rare terms which appear in individual document is higher i mean that's that's the whole point i wanted to make if you remove certain sentences for example right now the score for is idf scored for is is 1.11 if i remove certain sentences you will notice that the score will go up but you can experiment we want to move to our actual problem that we'll be solving in this coding tutorial which is e-commerce text classification okay and i took the data set from kegel the source credit is given to this particular page this data set has four classes i mean the actual classes might be more i have trimmed it down and we have the description of an item that you sell it on listen amazon so amazon item description amazon or any other e-commerce website item description and what category is it electronic household box etc okay so it looks i have i have this other notebook that i usually prepare before my coding video so see it looks something like this you can see right so let's say there is a text and there is a label there is a text this text when you read it you realize it's electronic right sata hard disk type and when you read this one you realize it's a book so we have this one and what we want to do now let me just show you the csv file so i got the csv file number of categories are total four okay and see these are the text description so simple csv with two columns the description of an item and what category that item belongs to okay so let's import pandas so how to import pandas import pandas as pd how do you read csv we have done this like i don't know ten thousand times so you read e-commerce cv you store it in your data frame df dot shape you can print that and then do df.head what i do is i usually do f dot label dot just to check if there is a class imbalance and no there are 6000 items in each of the categories as i said like i trim down the data set so that there is no imbalance and now what we will do is map these label categories to numbers machine learning models understand only numbers so how do you do that so df.label you can use a map function and you can apply uh the dictionary mapping so what you will say is okay if it is household item map that to a number zero just random number you know zero one two three four if the category is books the number is one and so on and that will be my label number column and the way you do that is you'll have to specify the new column like this this is a new column that you're creating in pandas pandas is an awesome library you can create new columns like this by applying transformation on existing column and i will check my df head and it works like a charm 10 okay we are ready for train display as usual this should be in your muscle memory import it from escalant the trend this split method and we have done this so many times that i don't feel like writing it i'll just copy paste here you are doing painter split off df dot text is your main actually it is your dependent variable and df dot label number is your independent variable like a target variable correct so this is x test x train this is y test y train and we'll split train and test into 80 20 category 20 samples go to test data set random state parameter i'm just randomly initializing it to num some number and then we are doing stratify actually if you don't do stratify still okay because no actually we we have to do it okay otherwise you might get imbalance in your extreme extras so this is a good practice and when you do that it's gonna do train display and i'm just going to just print the shape of x-ray next taste took something like this 4 800 samples in your test data set okay and then just some value checking in in your y train i am doing value counts and you can see that there is a balance you know every category has 4 800 training samples and same thing you can do for y test right so why test value counts and y test also has equal number of samples from all the categories now i will use few classifiers from sk learn to train my model i will first use k n and then random forest the naive bayes i just want to show you if you use different classific classifier what is the performance okay so i will just import some necessary libraries and folks i'm doing copy paste i don't want to type and waste my time because we have done this enough number of times in this particular playlist so make sure you follow this playlist in a sequence if you are randomly watching this video some of these things might not make sense but if you follow the playlist in sequence it will be more meaningful will import sk learn pipeline and sql and pipeline allows you to create a pipeline where in a first stage you can have your tfidf vector and the second stage you have your knn classifier and then you know you'll do clf dot fit so x train y train and then you do prediction right once the model is strained you will say predict x test and once you get your y predicted you want to compare whiter predicted with your y test and the way you do that is by printing classification report and you always supply truth first and then the prediction and i usually use print because otherwise the formatting won't be appropriate for classification report so right now it is training the model in this fit command and then in the predict command it will do prediction on the x-based data set it will get y predicted output and y predicted it will compare it with the actual truth and based on that it has printed this classification report the result looks pretty good these are the precision and recall metrics for class zero one two three okay we know all those classes household books and so on and these are the f1 scores the f1 score of 95 96 97 this is pretty good if you don't know what is precision recall go to youtube search code basics precision recall you will find a video very easy video which explains all these metrics and i will just print x test and what was my y taste for first phi sample okay and i want to also know my y prediction for first five samples and see 0 2 3 1 3 0 2 3 1 3 only this one it got it wrong other categories it predicted fine okay so zeroth category lao haveli designer handmade this looks like clothing item to me or household so that is zero and you can see that household is zero books is one electronics is two right so this cotton thing for example go top classical retro cotton is two and two category is electronics okay so electronics it is predicting it fine sometimes if the label is not clear what you can do is you can print the whole thing and you need to supply this index okay this is the index that will give you the holes see you're talking about dslr cameras right electronics maybe this is leather so maybe it's some kind of case for the camera i don't know yeah but anyway it is classified under electronics category the third one is see the third one is this one right so this one is for let's say camouflage polyester multifunction winter face mask style mark this is some mass so household item so number three oh number three okay sorry clothing and necessary numbers three is clothing and accessories so the widest was that and prediction was also in in line with the truth now i will use multinomial classifier as well and check the performance so you can just copy paste this whole thing here okay and one additional thing you need to do is your naive bayes classifier you need to import and use that one here in the second stage of your pipeline and you want to check how the performance looks like and you compare so see here k nn give 95 96 97 97 94 95 97 90 almost similar okay not too much different but now i want to try a random forest classifier okay so i will do random forest classifier and you can see this could be trial and error usually for text problems i start with naive bayes but based on your problem your data said you know one or the other classifier might perform well there are some general guidelines you can follow okay test classification usually use multinomial naive bayes but in this particular case what's happening is the random forest is giving you probably the best performance 96 98 you can compare all these metrics with these these other metrics from the other classifiers and we concluded that the random forest classifier is giving you the base performance now when i trained this model i did not do any preprocessing i did not remove stop words and i did not do limitation so remember in the last tutorial we we designed a function that can do this preprocessing so i will do that now i will train this same classifier but using preprocess text not the raw text this classifier is trained on a raw text how do you know well see we use in train this place df.text df dot text is whatever we got in our csv we did not remove stop words or we did not do limitation i want to do that so i will import a specie library and i will define a function called preprocess now we have seen this exact function in the previous video hence i am not going to go line by line i will use my most powerful weapon of copy paste here you go token by talk on if it is stop word or punctuation word you just ignore it if it is not you do lemmetization and then you join filter tokens and you create a string okay so this will do your uh preprocessing and then on the text column you will use apply function and you will say pre-process apply and map that kind of similar function and this will create a new column called pre-process text it took me close to 10 minutes to execute this sentence so make sure you have enough patience you go for a coffee break or a walk while this is executing and i got this new column created and this has a pre-processed text so let's check one or two specific example for example the zeroth text right the zero the first one is this and if you look at the same thing in pre-processed text you will find that c a study in simple c a study is in is removed though is removed punctuation dot is removed see here then chair has a form it will be say chair form though has a form is removed so it got rid of stock words and some and it did limitation and so on and we'll do the train test split using the pre-processed text column here again the random state is same because of that the the samples that we got in the previous scenario where we are using the raw text will be similar so that we can do apples to apple comparison when we are doing this prediction and i am going to copy paste the same code for random forest like from here okay the same code i will just copy paste which is my most powerful weapon and i will train the model this model is trained using pre-process text the numbers that you saw above these are using the raw text and you will notice the performance is slightly better so f1 score 97 98 98 and 99 correct here see it's little less when you use raw text so we can conclude that in the case of this e-commerce classification problem with this particular data set the pre-processing is helping general guideline is you should do pre-processing but in some cases if you don't do pre-processing it is still okay so it just varies from case to case i hope you like this video this is this is all i had i'll see you in the next video and make sure you check video description because i'll be adding an exercise in next few days so by the time you are watching this video the exercise will be available learning coding machine learning nlp is like swimming you know i keep on saying this boring thing again and again to help you realize that only with the practice you can master this case just by watching video nothing is gonna happen so you have to practice when i'm teaching you code along with me do a coding practice and also work on an exercise so check video description below i think next next few days i'll be adding the exercise so if you're watching the video right now maybe you can check the video description after few days and i will have exercise ready for you thank you [Music]