Tokenization in Spacy: NLP Tutorial For Beginners - S1 E8

Key Takeaways

in today's video we are going to talk about tokenization in spacey we can do tokenization in analytic as well we have discussed the pros and cons between these two libraries and we decided we'll use spacey for the reasons i mentioned in the last video and if you remember our nlp pipeline video we had this uh this step called pre-processing so in this entire nlp pipeline we're going to begin with the pre-processing step the data acquisition and text expression and cleanup step is something we can maybe take a look at later maybe in the end to end nlp project but in pre-processing what we learned was there is a step called sentence tokenization when you you have a paragraph of text you first separate it out in sentences and then each sentence you split it out in the into the words so that's called word tokenization so we are going to see how you can do both of these things in spacey library also there are stemming lemmetization will cover stemming limitations in the later videos but tokenization is a process of splitting the tax into meaningful segments so if it is sentence tokenization we're talking about splitting paragraph into sentences and if it's a word tokenization we are talking about splitting sentence into words and you will be like what's the big deal with tokenization you know you split the sentence by spaces and that's it right why do we need a library well consider this statement if you are doing sentence tokenization for this for sentence tokenization you might use dot dot means the sentence is ended if you use that simple rule here you will end up creating three sentences for a sentence which is just one sentence okay so you can't use simple rules for this tokenization you need the language understanding you need to define spatial rules that drdoctor dr dot is not the end of the statement also if you have n dot y dot it's not dot doesn't mean end of the statement you need language specific rules here so what we will do now is if you are not installed already just do pip installs pc and that will install the spacey library on your computer i already have it installed to launch a jupyter notebook you can run jupyter notebook command and whatever directory you run that command and it will show all the files there so i i will i have created this spacey tokenization directory already so you can create whatever directory you want and just say python3 right so i'll just move everything else aside and i will do f11 to bring this in a full screen mode and then i will import spacey so that will import the library then i will create a language object so in spacey you can create language object in multiple way one way is you can just say spacey.blank en en stands for english okay now that creates so just simplistically think about nlp as an object which has english understanding it understands english language a little bit if you say d e it will be for german language now i how do i know like whether it's d or e and like is that's like magic so you can say specie language models and it will show you all the language models you have right so we will do something like this later on this is more like a train language pipeline so don't worry about this but see english is ian france french is fr german is d e okay hindi is hi so whatever language you are dealing with first create the blind component there is another way which is creating a pre-trained like pipeline we'll look into it in a later videos so this is how you create a blank english language component and then you will create a document so document is equal to nlp something here you provide your text so whatever text analysis you are doing it could be a paragraph it could be a multi-page document you need to provide a text here and the text i'm going to provide is doctor strange's visit in mumbai he loved pavavaji so much and i'm doing some text analysis on dr strange's beautiful journey into mumbai in pavaji so when you do this by default what this will do is by doing this see you can do for token in dock print open and when you hit that it has already done word tokenization it is so easy plain python code very intuitive to understand and if i go back to my presentation what it is doing in reality is this so when i say spacey dot blank i created nlp component around the text that i had by default it gives you tokenizer word tokenizer so when you feed text into it you get doc document doc document already knows about tokens so it has like token 1 token 2 token 3 and so on i want to take a moment and talk about first language dot in which makes building nlp applications easier for you you can perform various nlp tasks which are shown here via this platform if you're using a spacey type of library you download it you run your code locally if you are training a heavy model you need higher compute resources such as gpus whereas here everything is in cloud you just make an http call things run on the cloud you don't need a local machine with lot of computer resources there is a demo which you can see by clicking on this link and i have bought this bulb recently and you see there is a negative review on amazon and when i copy paste this here and i select text classification and when i say submit it immediately classifies that as negative and if i have a positive review i just copy paste it here and if i submit it it will classify this as a positive see it's pretty powerful so it makes uh things much easier for you you don't need to have like too much like detailed nano pin knowledge you can just call apis and get things done quickly for yourself the free tire is something you can try today you can just sign in and you can get an api access once you sign in if you go to dashboard you will get your own api key and you can use that api key to make the calls they have sdks available both in python and typescript go check it out the link is in video description below and thanks first language dot in for sponsoring this video okay so that part i think is clear now these tokens either you run a for loop or if you like python list style of index you can do this also you can do log0sdr [Music] doc 1 is strange and so on so you can use general like python list like index operation to retrieve all these tokens and you realize if i have two dollar the english model that we created it understands that this is a currency therefore it will split split two and dollar into two different tokens still let's try this if i do this hash i think hash also is splitting it into different tokens so it probably okay so it currency the punctuation it will split though them into different tokens if i was using just splitting by the words then it won't be sufficient correct because then it it doesn't it's not smart enough to detect the language and let me explain the the rules of the tokenization so i got this image not this image i got this image by the way from spacey documentation so for thanks for that but let's say you have this sentence let's go to ny the sentence has double quotes okay so if i just use split by spaces i will get these these words and these are not my true tokens okay that's when spaces tokenizers tokenizer come in place and it will do some additional processing okay what kind of processing well this it will first split by prefix so any prefix you have code that's a separate token so it will split see you green box it just splits that as a separate token it could be dollar sign bracket there are there are some characters it considers as prefixes and it will split them then it looks at exceptions such as let's see apostrophe s that is also a separate token and when you do that this kind of when you do splitting into this kind of individual component you will see in the later videos that this is very beneficial for doing the language analysis for building the nlp application then you do suffix right the way we did prefix there is a double quote at the end so that's my suffix then you have one okay suffixes could be whatever km is kilometer it could be bracket explanation exclamation mark no code etc then another suffix would be the excel exclamation mark after ny so just split that then one more exception would be i think yeah so the exception is split okay so suffix was this and the exception was this so now you split now your your tokenization is complete so if you feed this into spaces tokenizer let's go to ny you will get these kind of tokens you want to try that by the way let's how about we try that that would be nice so instead of whatever doctor strange i can i'll just do it at the top because i want my doctor strange to stay here so i will say let's go to andy see let's go to n dot y dot explanation mark see and actually we need like single quote because i have double quote as part of the sentence itself okay so okay there is a single quote also so i will use three quotes actually i will use three quarters by the way so you you know right like three quotes is like a whole sentence and since i have a single quote here that's why i'm using this three code anyways but see this is one token seek double quote let then see lat and s then go to go to ny so it works i just wanted to show you that i'm not bluffing okay it works and then we looked at the index operator now let's look at the type of this object see nlp i say right it's it's an object of english language similarly if you do doc that's an object of a dog come on isn't that clear token is an object of token there is a span object by the way so i have all these things right if i do dog see if i do dog one whatever it gives me one strain but if i do something like one to five you have studied this slice indexes in python correct it gives you a slice similarly in nlp also sometimes you want a span from a sentence you want certain span span means a sub string from that string and this object is of type span so if you do type span and you will see in the later video how this is useful you get an object of span the okay so let me just clean all this up let me come up with a different different sentence okay so tony and peter are friends they have mentor mentee relationship and tony gave two dollar to peter so that peter can buy the car okay just kidding so token zero is this correct and that token zero would be what tony now this has certain attributes so if you do dir in python general rule if you do dir on any python variable you get all the methods and everything of that class so my class of token 0 is token correct if you do type of token 0 it's an object of class token and that class token has all these methods see all this see is bracket his currency is digit is stopped so let's try to use some of these methods so i will first use token 0 dot is and if you hit tab it shows all the matters as a help so i will say is alpha so it says true because it is alphabetic like it's a word basically it's not a number if i do is none actually the method is like num it says fall it's it's not a number correct but but let's look at this so 0 1 and 2 okay so token 2 is equal to dot 2 let's first print and by the way you can use token 2. text also that gives you the text so this token 2 is the word to now if you call this method tell me what's going to happen any guess see this is smart it knows tw2 is a number it is like a number and it tells you that that is true what is token 3 okay let's try token 3 token 3 is the dollar sign and luckily we have a method called is ah is currency it says it's currency true now this can be pretty powerful okay so i can show you a use case but let me first print couple of just few of these attributes so i'm running a full look i'm printing a token token has an index which which is called dot i is alpha is something we saw is punctuation like number nine like currency see for tony it will say it's not like currency is not like number but when you have dollar it is saying it is currency it is not a number but when you have 2 it says like number is true so these these attributes can be pretty powerful in doing your text analysis now i'm going to um quickly show you a file so let me go here so i am in my c code nlp tutorial series uh sorry directory which has a file called student.txt okay what is there in this file i'm curious ah it has some student information see in my class i have pretty talented some talented students you know they come to my class too to learn python by the way all the students they'll get pretty good in python and let's say i'm a teacher and i have this text file of all the students and let's say tomorrow is gonna snow so i want to declare a holiday so i want to send an email to all the students saying that tomorrow is holiday who they enjoy from this text file i'm showing you a simple text bible this text file can be very big how do i extract all the emails all i want to do is from this text file i want to extract all the emails i can use regex you know regular expression that is something we have seen before but [Music] on this occasion i find like spacey can be even more convenient than regex so let's try this let's first read this file in python so you all know how to read the file in python when you do f dot read lines it will read all the lines in a text file as a text as as an array so text is an array of all the lines you know what slash n is right for the new line now spacey takes single text so i will convert this array into a single big blurb of text okay and the way to do that would be so i'm separating everything by space and i'm saying this so what this will do is text is an array right list it will join all the elements by list and it will use space as a delimiter in between the list elements when they join it will use space in between great so it's a one single sentence now i want to grab all the email ids so i will first create a document element from this text and then i will say for token in doc if token dot if token is like email then do something i can print but i don't want to print i want to i want to grab maybe all the emails in emails array like a list and the list has this method called append token dot text and when i do that i get the emails of all my obedient and intelligent students this is one simple use case you have other methods too like like you are a like number so many other methods okay so you can check all those in a specie documentation how do i see pc documentation google is your friend spacey token attributes okay so i think attributes you can just google and find it out okay i don't know i don't know how to get to it but you will find it out in in some documentations business documentation by the way is really good i really like that and see they have given a very good article by the way i highly suggest you read the article you will see this is the chart that i use in my presentation thanks spacey folks um you can do spacey token attribute like number see these are all the these are all the all those tokens that we have okay you have these alpha is you have all of those okay correct now let's move on to the next agenda now we will try uh a model in a different language so hindi is my is the language of my country india so i will use hindi and the hindi code we saw right it is hi and i will create you know some something i will take some sentence in hindi by the way you can type in like hindi keyboard and you can type it in hindi even with your english keyboard you can say see this is like you can type it in hindi and you can just say control c and ctrl v so similarly i typed something and i'm doing ctrl c [Music] ctrl v okay he's saying like he's borrowed money from someone and he's asking to return them and you will again do the same thing you will just print all the tokens and you see that it is printing all the tokens not only that along with the token you can print some of the attribute let's say is currency and token dot like number right so when you print that see for 5000 currencies false but number is true and for the rupee sign this is a rupee sign rupee sign it will say is currency yes it is currency so you take any language model it is it is having knowledge of these basic rules now if you look at this documentation there is this column called pipeline and only few languages have these pipelines but the other language says none yet see hindi says none yet and what is that pipeline we'll cover later okay let's let's move on to the new new topic so now sometimes you want to customize your tokenizer for example you have a sentence like this and you say for and by the way instead of doing for token in token sprint i will try to grab everything in a in an array so i will say token you know list comprehension right so list comprehension is a easy way to write for loop so instead of writing for token in tokens do something i can move this thing here in a for loop and whatever you want to do you can put that thing here so when you do that you get all these tokens now you're not happy with the default behavior of spacey here because see i use abbreviation you when you use slang and specific norms of any language that's when you realize the need of customizing spacey so give me is actually two words give me you want two separate tokens so what you can do is you can customize space here by doing this you can save from spacey dot symbols import art and then you can say nlp dot tokenizer you know we have the nlp object and you can say add spaces so i'm i'm adding a special case okay where i want to customize give me into into what so i want to use part so you want to customize this to you and me correct so this don't worry about syntax too much maybe look into art and other symbols we have available in later but this is a simple way of customizing spaces behavior where where i'm saying whenever i ask you to tokenize give me just give me two tokens instead of just one give me token okay and then i'll create a dock element and i'll do this same thing when i hit enter if what it is saying is special cases are not allowed to modify the text so you can't do give me you can't do give and me but what you can do is you can do gimme so it's like gim and me so you can modify the actual word you can split it okay and if you want to modify will we have further processing steps tokenization is just by tokenizing is kind of a little dumb it just splits the whole thing into segments you don't want to change the actual text that's not allowed so when i do this now see i get gim and me as two different tokens all right so that was about adding a spatial rule in tokenizer now let's do sentence tokenizing this is something we have looked into it before also so i will not spend too much time and here i have two sentences hulk and strange both are loving in their trip enjoying the food to re to tokenize this into sentences you can say for sentence in law not sent senses sentence isn't that obvious hmm this is not working what is the error let's read so it is saying my nlp pipeline is kind of blank that's what this error means it's not saying that so if you do nlp dot pipe pipe names that pipeline is blank and what does that pipeline means so what i can do is here i can add nlp dot add pipe because that's what it is asking me to do and i'm adding a sentence sizer into it and when i do that see my pipe names remember it was empty now my pipeline has this sentenceizer component so what i did essentially is this if i can open my presentation blank pipeline is like this you only get tokenizer you don't get any other component but when you build a pipeline see this the second part this whole thing is called pipeline and it has different component we will look into it later like tagger parser name entity recognizer will see all those details but the simple idea you want to learn today is you have a blank pipeline when you do this but when you do something like spacey.law load you get full-fledged pipeline where you can have these components and and if you have a blank pipeline you can add a component in the pipeline manually and that's what we did when we did space c dot add pipe here after tokenizer component we added component for sentenceizer which means now this nlp object knows how to split these sentence into how to split paragraph into sentences and when you do that see it is doing all this and it's kind of a little strange because the component that i added is little dumb and it is splitting see after dr when it sees dot it is splitting that into uh it is thinking it's a separate sentence so it still doesn't understand english language fully and we'll see in the next video probably that when we use pipeline like this it will fix all those issues so we'll look into the those later uh in in the future chapter in the future video okay so that's all i have for the tokenizer now it's the most interesting part of this tutorial which is an exercise yes it's like swimming if you watch a swimming video you're not going to learn swimming you want to make a career in nlp you want to earn big money by the way nlp engineers make a lot of money okay if you want to achieve a heights in your career success you have to work hard for that reason i am giving you this exercise where you might know about this thing stats book so this is a nice book free book for learning statistics and a paragraph of this book is this i just took some paragraph from that book okay that contains couple of urls for free data set website see free data set website is something that you need as an nlp engineer working for a company so i want you to write a code here that can process this text paragraph and grab all these data url see one two three four five and the second exercise is you have to extract all the transaction money which is the answer will be two dollar and 500 euro so the answer of this would be 200 and 500 euro and you have to you know write some code to extract that this success will require little thinking i have a solution here and it is very tempting to click on the solution but i know you all are brilliant not brilliant but sensor students so i'm pretty sure you will not click on this solution link you will work on on it on your own and if you click on it before without trying it look i have a magic embedded here so when you apply for nlp engineer job you know you will probably not get an interview call if you don't work on the solution diligently all right so i wish you all the best the link of this notebook the exercise everything is in our video description below so make sure you check out the video description if you like this video share it with your friends give it a thumbs up if you give a thumbs up you're paying fees of this session okay so if you want to pay back give it a thumbs up share it with your friends on linkedin especially or whatsapp whatever and i will see you in the next video thank you