Python Tutorial: Statistical Models

Key Takeaways

let's add some more power to the NLP object in this video you learn about spicy statistical models some of the most interesting things you can analyze a context specific for example whether a word is a verb or whether a span of text is a person named statistical models enable Spacey to make predictions in context this usually includes part of speech tags syntactic dependencies and named entities models are trained on large data sets of labeled example texts it can be updated with more examples to fine-tune their predictions for example to perform better on your specific data Spacey provides a number of pre trained model packages you can download for example the en core web sm package is a small English model that supports all core capabilities and is trained on web text the Spacey load method loads a model package by name and returns an NLP object the package provides the binary weights that enable Spacey to make predictions it also includes the vocabulary and meter information to tell Spacey which language class to use and how to configure the processing pipeline let's take a look at the models predictions in this example we're using Spacey to predict part of speech tags the word types in context first we load the small English model and receive an NLP object next we're processing the text she ate the pizza for each token in the doc we can print the text and the post underscore attribute the predicted part of speech tag in Spacey attributes that return strings usually end with an underscore attributes without the underscore return an ID here the model correctly predicted aid as a verb and pizza as a noun in addition to the part of speech tags we can also predict how the words are related for example whether a word is the subject of the sentence or an object the Deb underscore attribute returns the predicted dependency label the head attribute returns the syntactic head token you can also think of it as the parent token this word is attached to to describe syntactic dependencies Spacey uses a standardized label scheme here's an example of some common labels the pronoun she' is a nominal subject attached to the verb in this case 2 8 the noun pizza is a direct object attached to the verb 8 it is eaten by the subject she the determiner the also known as an article is attached to the noun Pizza named entities are real-world objects that are assigned a name for example a person in organization or country the doctor ends property lets you access the named entities predicted by the model it returns an iterator of span objects so we can print the entity text and the entity label using the label underscore attribute in this case the model is correctly predicting Apple as an organization UK as a geopolitical entity and one billion dollars as money a quick tip to get definitions for the most common tags and labels you can use the space e dot explain helper function for example GP e for geopolitical entity isn't exactly intuitive but Spacey don't explain can tell you that it refers to countries cities and states the same works for part of speech tags and dependency labels now it's your turn let's take a look at Spacely statistical model