Python Tutorial: Introduction to NLP feature engineering

Key Takeaways

welcome to feature engineering for NLP in Python I am Ron Ock and I will be your instructor for this course in this course you will learn to extract useful features out of text and convert them into formats that are suitable for machine learning algorithms for any ml algorithm data fed into it must be in tabular form and all the training features must be numerical consider the iris dataset every training instance has exactly four numerical features the ml algorithm uses these four features to train and predict if an instance belongs to class iris virginica iris setosa or iris versicolor ml algorithms can also work with categorical data provided they are converted into numerical form through one hot encoding let's say you have a categorical feature sex with two categories male and female one hot encoding will convert this feature into two features sex male and sex female such that each male instance has a sex male value of one and sex female value of zero for females it is the vice-versa to do this in code the use pandas get dummy's function let's import pandas using the alias PD we can then pass our data frame DF into the PD or get dummy's function and pass a list of features to be encoded as the columns argument not mentioning columns will eat pandas to automatically encode all non-numerical features finally we overwrite the original data frame with the encoded version by assigning the data frame returned by get dummies back to D F consider a movie reviews data set this data cannot be utilized by any machine learning or ml algorithm the training feature review is in numerical neither is it categorical to perform one hot encoding on we need to perform two steps to make this data set suitable for ML the first is to standardize the text this involves steps like converting words to lowercase and their base form for instance reduction gets lower cased and then converted to its base form reduce we will cover these concepts in more detail in subsequent lessons after pre-processing the reviews are converted into a set of numerical training features through a process known and vectorization after vectorization our original review data set gets converted into something like this people learn techniques to achieve this later lessons we can also extract certain basic features from text it may be useful to know the word-count character count an average word length of a particular text when working with niche data such as tweets it may also be useful to know how many hashtags have been used in a tweet this tweet by Silverado records for instance uses two so far we have seen how to extract features out of an entire body of text some NLP as applications may require you to extract features for individual words for instance you may want to do part of speech tagging to know the different parts of speech present in your text as shown as an example consider the sentence I have a dog POS tagging will label each word with its corresponding part of speech you may also want to know how to perform named entity recognition to find out if a particular noun is referring to a person organization or country for instance consider the sentence Brian works at data camp here there are two nouns Brian and data camp brian refers to a person whereas data camp refers to an organization therefore broadly speaking this course will teach you how to conduct text pre-processing extract certain basic features word features and convert documents into a set of numerical features using a process known as vectorization great