Chunking - Natural Language Processing With Python and NLTK p.5

Key Takeaways

hello everybody and welcome to part five of our nltk with python for natural language processing tutorial video in this video we're going to be talking about chunking so what is chunking besides a really strange sounding term so consider you have a body of text we know how to split it up by sentence and by even by word and not only that but part of speech tags we're that far what would be the next step to figure out the meaning of a sentence well first we want to know who is the sentence talking about who what whatever uh and so generally we refer to that as the named entity in the context right but also just a noun okay so a person place or thing is generally going to be your subject and I say generally now uh so once you know that okay once you know the named entity what's the Next Step well the next step is going to be or the noun the next step is going to be finding out words that kind of modify or affect that noun so you might have many named entities or many nouns in the same sentence so you might have a sentence like uh Apple releases new phone comes with new color case $100 more and Tesla releases home battery okay okay so these are sent this is one sentence but it's talking about two different things completely and you might even have some opinions in that sentence and you got to figure out who where does that opinion apply is that applying to Apple or is that applying to Tesla so um so that's what we're going to be talking about now is chunking and how we might be able to do stuff like that so generally what that's going to be is is you're going most people will chunk into what are called noun phrases and these are just going to be groups of Na well it'll be a noun and we'll have a bunch of modifiers around that noun and and it'll be kind of like a a descriptive sentence I suppose or descriptive group of words surrounding that noun now the downside of that is you're you're only going to be able to use regular Expressions so what's going to happen is you are F like you can only group things together as a chunk and it can only be a chunk of things that are touching each other as you'll see in a moment so that's kind of the only downside here uh but you can at least chunk important words and then kind of break it out from there uh if if you want it so anyways it'll be more useful I think to kind of visualize what chunking is now so we're going to come down here and the way that we do chunking is we we make use a meshing of these part of speech tags and regular Expressions now if you are unfamiliar with regular Expressions you can still follow along and I'll try to explain it as good as I can but you're going to have two brand new Concepts being thrown at you here and you may not know which is which like are we looking at regular Expressions here or are we looking at like chunking code um so if you need to learn about regular Expressions this is uh python program.net it's my website crlf when you're on the basics tab anyways if you're not here click on start learning Basics regx enter we want this one actually um video tutorial text tutorial we very fun now the main thing that we're going to use with chunking is modifiers so with your modifiers you've got things like uh the plus sign means it m it's one or more question mark means zero or one no more asterisk means zero or more so basically like any number or nothing uh we probably won't use the dollar sign the carrot we won't be using really you might actually uh probably not though the either or you'll use that a lot the range you'll use as well so anyways um just keep that in mind that's kind of what you need to know uh with modifiers I will still hold your hand through this but like I said it might be confusing if you if you're not familiar with either of these Concepts so coming down here uh let's get rid to tagged and then we'll go or print tagged rather keep tagged there now we're ready to create our chunk gram chunk gram is going to be equal to and generally people use triple quotes for their chunk grams because this one will be really simple but a lot of times people are going to put them and separate them by lines uh cuz they can get really long and complicated really fast so chunk gram use a triple quots if you want don't if you don't want to it doesn't matter to me uh and we're going to chunk by now most people are going to put the r in front to denote a form of regular expression here but it I'm pretty sure it's not necessary we'll we'll test that if I remember to so now chunk you can call this anything you want you could call this um H if you wanted uh it's just what it's going to be called when it finds it so anyway a chunk equals and then you put the chunk that you want to you know find inside curly braces now uh to bring up or mention any form of part of speech tag use these little I don't know brackets sideways carrots I don't know what the heck you call these someone tell me what you call these um and first let's say we're looking for any RB now what is an RB let's head over oops actually let's go it's up here RB adverb now we want to find RB rbr or RBS any adverb we want to find how do we do it regular Expressions so we come on down here RB what would be a valid modifier to use we want RB followed by basically any character we could use the W I suppose but we're not going to do that uh we're going to say RB any character is the period and then we're going to use a question mark what does that mean so the period is any character except for new line basically and tabs and stuff and oh actually I'm sorry it's actually any as long as my own uh tutorial is correct it's actually any character except for a new I thought it was also tabs anyways moving on uh RB so anything and then question mark is zero or one because as you can see we don't have any part of speech tag that is longer well at least for RB uh than three characters okay but possessive pronouns can be and they use a dirty dollar sign there anyway moving on uh so that's fine uh any adverb then we're going to say what we'll say asterisk what is asterisk zero or more so we're saying any form of an adverb and we're looking for zero or more of these so this chunk we'll chunk whether or not there is an adverb but if there is one we'll get it then we'll have another one and then we're just going to have VB and then we'll use the exact same code that we used before which is period question mark and again if we find one great if not no problem then we're going to look for an nnp what's an nnp Harrison let's go up nnp proper noun and hot diggity there's an example of a proper noun it's me so come on down here nnp uh we are going to require the existence of a nnp then we're going to do a oops a plausible NN let's go back up NN is just a noun okay so nnp singular proper noun followed by possible noun singular anything now that's our chunk sweet so hopefully you guys follow along there if you have any questions as always ask below now what we're reg ready to do is uh parse via this chunk gram so we're going to say chunk parse equals nltk regex parser and the regex parser we want to use is the chunk gr now chunk is going to equal uh the chunk parser dop parse the tagged part of speech stuff and then we can do the following print Chun [Music] what did I do r x oh right I'm pretty sure in my original tutorial series I swear I hit this rag x p parser and I'm almost positive they're doing this because it's regular expressions and then parser but no one says regx they say reg I don't know anyway let's try this again I'm almost positive that hit me in the original tutorial uh this is going to go for a while so control C to break it and so this is your um you know basically all the output and what you're looking for is chunk right this is chunk so chunk State there's another chunk Union Address chunk United Capital so it's finding all these nouns basically White House chunk chunk okay now this is kind of ugly the computer loves this stuff but the human hates it so what you can do with um nltk if you Gra if you have map plot lib if you don't have map plot Li this probably won't work but chunk. draw save and run that and now you have this look at us wow that's way too many chunks what has happened for maybe because that's the start of it's probably because like this initial yeah right so the initial stuff here uh that was a little rough but anyway present George W bush and as we can see that really we actually would want to chunk these all together right so maybe any amount of nnps so we could come down here nnp um one or more is what we want so we want to do a plus okay so now let's try that again I don't know how far deep this one was but it was it was a few X's so anyways let just rerun it um okay so that's a little better um that's not so hot but this is hard because of the original stuff but President George W bush that is a correct chunk um let's get out of this initial stuff Mr Speaker vice president jie this is a tough this is a tough one for uh python anyway we'll start on this one so this is the original one that we wanted to chunk for sure um so President George W bush good State of the Union uh again you could add this stuff into your chunk like you could have a a possible preposition a possible I forget what DT stands for but it's like useless words I'm pretty sure um close close close White House put that together Eric Draper every dra I'm not that's almost like he's missing a space there um Capital Dome that's a good catch okay have served America all right you get the point so this is how we're chunking um a bit of information together obviously again like I was trying to point out our chunk is extremely basic and as you can see as we move forward we can think of a lot of like little stuff that we're like hm I wouldn't mind adding this or that or a possible of this and so as you can imagine these chunks can get really long okay so anyway feel free to mess with it I actually highly encourage you to kind of play around with this try to catch that of the or yeah I think it was of the see if you can catch that and while I'm thinking about let's look at DT again determiner yeah DT for not important anyway um so that's chunking now in the next tutorial video we're going to talk about basically kind of a a different task and that's going to be chinking and that's going to be so chunking is the you know grouping of things chinking is going to be the removal of something get that out so we'll tuck on that in the next video if you have any questions or comments on this video please feel free to leave them below otherwise as always thanks for watching thanks for all the support and subscriptions and until next time