Get Data Into Databricks - Feature Store

Key Takeaways

hi everyone in this video we'll be talking about the feature store in data bricks generally a feature store is used to create discover and reuse features in data bricks a feature tape was any Delta table with a primary key that means that when you featuri your raw data by applying any number of joins Aggregates and transforms all you have to do to that table to make it a feature table is ify a primary key at this point there are a couple of different paths you can serve that feature table online for realtime feature suring this is known as an online table or or an online store too you can also use it in an offline or a batch setting for model training and inference when you use a feature table throughout the model life cycle you ensure consistent usage of features during model training for which addresses a key component of online offline skew in this notebook I'll go over how we can do that in data breaks this example we are a travel agency that wants to provide recommendations to our users uh to do that we need a model that predicts whether or not it's likely that a given user is going to book a um to make a booking or not we have some data on our users will apply some transformation logic to convert it into the features that we'll then use to train our model to do that we'll just select the data that we have specify and apply the transformation logic don't worry about the details and store the results in a data frame this point to turn that table into a feature table we can use the feature engineering client which is a comprehensive API to create manage and use features to create the table itself I just specify the name of the table the primary key the data itself along with the metadata that I want to have associated with it this feature table is now ready and directly integrated and governed by unity catalog we look at a high level overview of the data we can see that the ID is a primary key and walking over to the lineage tab we can see from the lineage graph that this feature table is Downstream of the original data and Upstream of the model that will train with our feature table we can create the training data set by specifying the ID which we'll use to um in the feature lookup to look up the features associated with that ID combine it with our purchase label and then use it to train our model so to train the data set grabbing the ID and the label I'll use feature lookup to use those IDs to look up this list of features from the feature table that we just created and then to create the actual trading data set itself I'll combine the original data with the features that I just grabbed minus sum specify that the label is the purchase column and I have my training data set next we'll train the model the details of how uh what model we train aren't important for the Intensive purposes of highlighting the benefits of using a feature table throughout this model life cycle what is important is that when you log a model that's been trained with a feature table using the client uh feature engineering client one of the things that's stored along with the metal met the model metadata is the feature spec the feature spec specifies the inputs to the model along with how to get them for example the destination ID is an input to the model and it's retrieved with the lookup key ID so when we have our model and have moved it to production we can run inference on it to do that all we need to specify is the ID on the back in there's automatic feature lookup that takes this ID grabs the features that are associated with that ID uses those same features as the inference features this is what ensures consistent usage of um features during model training and inference and then forecast whether a user is likely to make a booking or not here in this example we'll just select the IDS that we had in our table this is contrive in a production setting you might have a data bricks job that updates the feature tables with new user ID information run inference we just feed the the IDS to forecast and we get the prediction so in this demo I showed how we can create and use a feature table throughout the model life cycle you can find this notebook um along with others on deeper dive topics such as point in time lookup which involves on thefly feature calculations deploying online tables as well as streaming feature tables at DB demos. there you'll also find other helpful tutorials that span the entire data breaks platform from data engineering data science and AI data warehousing and bi along with data coverance thank thanks for watching