Python Tutorial : Feature engineering and overfitting

Key Takeaways

feature engineering uses domain knowledge and common sense to describe an object with numbers although adding more features can improve performance it can also increase the risk of overfitting in this lesson you will learn more about this interesting trade of sometimes the raw data cannot fit into the form of a table for example consider electrocardiogram or ECG traces for a number of individuals each ECG trace is a time series possibly of variable length that cannot fit in one cell of a table instead in the datasets shown here experts extracted over 250 one-dimensional numerical summaries from each ECG these range from simple summaries like heart rate to very complex properties of the signal with weird names like T wave amp all of which can be useful in detecting a medical condition known as arrhythmia even if the data are tabular some of the columns might be non numeric here is an example from the credit scoring data set the purpose of the loan takes values such as buy a new car education or retraining label encoder will map these values on to a range of numbers but the classifier is then confused it thinks that the categories have a natural ordering for example a decision tree might try to split the range in two if it splits at 4 it is putting loans for business together with loans for a microwave oven a different approach is to use one hot encoding implemented by the get dummies and us method this creates one new dummy variable for each category taking the value one for each example that falls in that category and zero otherwise you can see the first row of the data on the Left printed vertically for readability no artificial ordering is introduced how about capturing semantic similarity notice that similar categories share keywords for example all consumer loans feature the keyword buy you can count common keywords using count vectorizer from the feature extraction module first replace underscores with spaces for easier tokenization then apply the encoder using its fit transform method finally convert the resulting matrix to a data frame naming the columns using the debt feature names method of the count vectorizer object note that as we improve our feature engineering pipeline the dimension of our data frame increases the question arises how many features is too many well with more columns the algorithm has more opportunity to mistake coincidental patterns for real signal we can test this by adding columns to the data containing purely random numbers totally unrelated to the class as we add more columns on the horizontal axis overfitting kicks in accuracy improves in sample but deteriorates out-of-sample a popular solution is to add features freely and then select the best ones using some feature selection technique let's try the trick from the preview slide and augment the credit scoring data set with 100 fake variables then we use the select K best algorithm from the feature selector module to select the twenty highest scoring columns we use the chi-squared scoring method the feature selector has a fit method to fit it to the data and a dead support method that returns the index of the selected columns thankfully only a handful of fake columns remain in the selected features so remember this every decision you make in your pipeline might affect other aspects of it and in particular the risk of overfitting the following exercises confirm this inside