How Feature Stores Work // Simba Khadder // DE4AI

Key Takeaways

[Music] Simba I got something I got to I got to show you this real fast look what I'm talking here oh look at that so that's pretty cool this is my favorite because it's one of the only ones I have that actually fits in all of my cup holders around the house and all that stuff so if anybody wants to uh get one of these maybe you got to go buy Simba's booth in the sponsors channel so click on the leftand sidebar see what Simba's got on offer there now I know you got to talk for us man I am excited for it I'm ready for it you got to share your screen and I'll throw it up here on the stage and then we'll get rocking sounds good oh boom here we go I'll be back in like 20 minutes peace thanks dude talk awesome um hey everyone thanks for joining today um today we're going to be talking about feature Stores um we'll go we're gonna go very uh broad from just the what they are what they do why people care about them and uh maybe under the hood a little bit where we can understand how they work and how they enable people and data Sciences to build data pipelines at scale so uh feature store um remember how we learned machine learning I know if you all took that corsera class with Vander ing or whatever but you're giving these like perfect csvs you grab the columns you need the features are there for you and a lot of what your work was was just building simple models regressions um trees Etc on top of those nice perfect features so I'm here to tell you that those perfect csvs do not exist uh in reality in reality a lot more time is spent taking data that's a mess as Demetrius showed in his video out of the data swamp and into these uh perfect um places where we can actually use them so you heard it here first anding lied to you there is no such thing as a perfect CSV to work off of a lot of work um in data science if you're you know in a big company this is this is maybe going to look slightly familiar uh it's a little more messy there's a lot of uh kind of ad hoc processes to get models into production it's almost a little terrifying how much duct tape is used to put together some production models um that we use every day so let's break it down let's talk about let's talk about uh What uh Solutions the market come up with how people are actually solving these problems so the agenda for this talk I'm going to start by talking about um the actual data problem in machine learning I'll get into feature stores how they work the different kind of uh abstractions that and uh methodologies that different companies have taken to attempt to solve this data problem and then in the end I'm going to get into some of the internals talk about kind of the data engineering behind the scenes that's abstracted away from you and how feature stores enable um data scientists to build these scalable pipelines so let's begin what do feature stores do so feature stores do five things they help facilitate deployment they enhance deployment of features like feature pipelines they enhanc collaboration organized experimentation so there's kind of an organizational value to it it is mlops of course increase reliability of your features and your feature pipelines and then finally they preserve compliance so why is deploying features hard why is this a problem that needs to get solved well when we're building features often we're working in a notebook maybe we're working in um with pandas or some sample of data Maybe we're hitting queries directly on the database we're building this these dags to build our features to build our training sets that we finally use to train the model in this environment everything is nice it's static I have this notebook if you run through it in theory it will kind of do the same thing again and it's it's how a lot of data science is done in this kind of iterative um really interactive fashion and it kind of looks like this like you know you see a lot of pandas you see a lot of duct TB you see a lot of polars um you just see a lot of these sorts of Frameworks Ibis and other things and we're working in that Paradigm of experimentation but that Paradigm of experimentation is very different from production in production the data is changing we have to maintain and keep features up to date um and often times especially for low latency models or online models like recommender systems fraud Etc uh we need to keep uh we can't actually process the feature at request time those features have to be pre-processed and cached so now we have to deal with streaming data streaming pipelines batch pipelines we have to deal with situations where we're actually featuri data that comes with the request like if there's a comment and you're trying to see if it's spam well the comment comes with the request so you have all the code and the and the and the deployment and everything looks so different from that nice notebook that we started with there's this kind of um and and you know that's not even the that's just the beginning of it ver backfill back like once you start talking about streaming it just gets even worse there's this kind of Chasm um between that experimentation Paradigm the notebook and production and it can be almost insurmountable parts of the goal of feature stores and feature form is to try to um you know make it possible to cross that easier um there is some value obviously like you don't want to because other thing I've seen is everyone writes production grade um uh code from the beginning so there is no experimentation pipeline um that also um can make things very slow there's kind of there's a reason that we use notebooks we use pandas we use all these things in experimentation so the the problem to be solved is um how do we make it possible to you know work quickly in experimentation but then get those features actually in production uh because we don't want to end up in situations um which maybe some of you have experienced where you have this perfect feature in your notebook you're like cool if I build this training set my model works really well great now I need those features to actually exist in production and that's really hard the way I see it solved is in two ways one is uh either you know you kind of have to have have these unicorn data scientists who are also like really really good data Engineers who can build and deploy their own feature pipelines now writing um good SQL writing good data frame code that's very doable like most data scientists are are fully capable of writing that where it gets hard is how do you tune spark to do what you want um how do you set up how do you handle uh back pressure how do you handle uh monitoring um how do you handle Inc Inc mental runs because the data is changing um how do you update a streaming feature there's a lot of these problems that data Engineers this is what they do um but for data scientists you know it's it's a different skill set it's a very hard skill set and it's orthogonal so there are some unicorns who really are like exceptional at both but it's rare um and it's honestly like not what you want to just do just go hire everyone to be this amazing data engineer and data scientist um the other thing I see very often probably more often is that the ml data scientists pass the features either literally just passing over a notebook over the fence to a data Eng team the problem here is that the data Eng team though they are more capable at building those data pipelines for scale for production One features are inherently experimental you're always tweaking them you're always changing them um whereas most data engineering is built with this mindset of uh kind of engineering it's like engineering versus science there this robust ESS that comes with those pipelines and so one it tends to go a bit slower and two if you think of like the incremental Roi of like hey I have a slightly better feature for my model versus I have this new dashboard and exec needs you're pretty much always going to be deprioritized so for the complexity of the task versus the actual Roi to the business it doesn't really always make sense for data Engineers to be fully focused on all these ml tasks that come in so what we want is a way to democratize it so that data scientists themselves can build their own data pipelines production grade data pipelines but they don't need to be experts in spark because let's face it who really is um the upper piece that comes with this is okay cool like now we have our features deployed we're great now right we're good uh I have a few questions for you your features deployed you have these pipelines they're powering your models what are those features how did we build them how did they work what kind of peculiarities do they have um I know we all document our feature pipelines right maybe this looks a little familiar where you have your features in production but you know there might be some Untitled notebooks on the way maybe some table V4 that we use in prod maybe some uh shell scripts and ad hoc spark segements that we ran to make things work um and yeah just ignore that DF final final in the notebook so there is also a huge uh there this ad hogness with data engineering for ML where we just do whatever we can and we um to try to get it working and it can get really there's it's not just hard to manage it can actually result in really negative consequences to your model because a lot of times these features are also being deployed without proper drift monitoring so there's um M monitoring there's a concept drift there's a lot of different types of drift one type of drift that's really common or or one is the feature themselves the value is moving over time and they should right there's a certain level of of they will change but you kind of want to understand what's changing and why and catch things that go past a certain heris and make sure that you kind of have this like yellow flag to look into it the other kind of drift that's I see very often is where you will train a model and the feature in the training set will have a specific distribution and that distribution will be very different from what features are actually being requested in production so maybe in training the average age of the user is 50 but in production almost all of your recommendations are going to 20-y olds and so that difference can result especi if you didn't mean to do and you had no idea that was happening can result in uh negative consequences to your mod performance um so this tends to kind of be an afterthought I mean it's already hard enough to get your features in production um when you start worrying about monitoring um and then governance which uh can be another huge Blocker of I need to talk to Legal I need to get check off before I can even put this feature in production if you're in a heavily locked down space so these are all kind of the data problem s that I see the idea is I have a model models take features those features are just kind of signals to feed in um I'm you know as a data scientist you're fully confident in being able to maybe uh work in a notebook and get those train set set up now when it comes to collaborating and deployment and everything that comes with that it starts to become very ad hoc and this is where you start to see the cracks um and I would say General like the most painful is the facilitating deployment that's where I really see like it can be a complete blocker and it can just cause lots of issues and I've seen just a lot of really ad hoc processes to just get these features in production um none of this is necessarily like new it's not anything that I've kind of we've figured out and it's it's uh breaking ground a lot of companies have kind of seen this and have come with different approaches to how to solve it now the umbrella of uh Solutions oriented towards solving the data for ML problem um are feature stores so some people will say oh they don't you know we don't have one we don't have a feature store my view is if you have features in production if you have models in production that are being sered data then that data is getting there somehow it's getting processed it's getting set up it's getting there somehow that thing that process that's your feature store you know I think part of the issue um is and I think I I put it somewhere else I think part of the issue is actually just in the name uh when people hear feature store they I get a lot of confusion like why is everyone talking about this thing it isn't just a database or features are stored like it seems like a really a lot of hype about uh glorified cach and I think for some feature stores they've taken the approach of truly being a feature store um even though like you know other you know a lot of companies and products and solutions have tried to kind of rename the category to like feature platform virtual feature store a lot of different ideas um a lot there's still kind of this tie into feature store which I think pretty much every everyone who works in featur can agree is kind of an imperfect term um the thing about feature stores is they tend to apply again the storage but the thing is is how do you even get the features there a lot of the problems I talked about deployment actually come in with the transformation piece how do I take my data sources transform them and get them into my inference store and into my training store and then there's all this infrastructure underneath that I have to worry about so one attempt at solving this is what I would call the literal feature store um in the literal feature store as the name implies it's very literal you build your features elsewhere and then you store them in the feature store um there's a few examples of products that look like this a good example of this would be like datab break's feature store which I'm including here so in the datab bre feature store everything is feature tables they kind of look like this in the feature tables there are these features if I click on one there's not really much here this is it this is what the feature is because in the end it's just a column it's just tables and columns that's what the literal feature story is the value is that it it unifies it's like a singular place to store features and if they ver it can be used in prod but the um the negative is it's just storage compare that to something like feature form where the feature is not defined as a table but actually as a definition so all the SQL data frame logic is actually what the feature is the orchestration the monitoring the materialization all that kind of comes as part of it and you start to get much richer value and metadata that comes with that so yeah the problem of the literal feature store and this is something if you were looking at or building feature stores is like they don't really do the transformation piece and in my opinion it kind of you kind of have to solve most of the problems yourself anyway if you use those there's the feature platform or physical feature store approach which I think is much better the benefit is it really does those it actually solves those issues we talked about it really does the problem though is every single product that calls itself a feature platform pushes their own uh uh compute engine often of a custom DSL it's they're all proprietary and that comes with other issues of lockin so the idea of a virtual feature store was what would happen if we took the feature platform idea but made it more of a plug-in architecture so that data scientists work in Pyar SQL they're still writing code like as they would they're defining those things in feature form the name feature form I came up with it because I wanted terraform for features so that declarative approach and then it runs on these engines underneath the hood and that becomes the virtual feature store diving under the Hood a bit um feature form itself just quickly everyone was asked like where does it run it's actually running in in kubernetes the uh set of services one thing it does is kind of interesting is for a lot of tasks like a lot of the glue it will actually run jobs in kubernetes to move data around handle some of the metadata handle tasks like compaction and and interactions but underneath the hood what enables feature form to work what enables us to build this kind of virtual feature platform is deeply using uh open formats and open it is kind of this this Lous architecture that's appeared and I know there's some talks on it actually today has made it possible to do this what it does is it separates out storage and compute and specifically with Iceberg what feature form has been able to do is specify our it's pretty much an index right like you're storing all this data in uh est3 but with iceberg iceberg is allowing us to break things down set up indexes make it of a spark or whatever engine we use can choose and pick data to use um it also allows us to do it too like we can pick the right data files to use we can set up the right indices we can set up the right partitions we can do all that using Iceberg and it's agnostic to the the compute engine so we can use our own learnings of how to tune those indices across the different engines the other thing it's allowed us to do do is because of of some of these um uh optimizations it allows us to be able to also unify streaming and batch because what we've done is make it look like because with Iceberg you're kind of incrementally adding new data it kind of looks like a stream you can process up to a point and then continue processing off of kovka and so feature form kind of makes it possible to to be able to unify that streaming and bat so as a data scientist you're still just writing that SQL you're still writing um you're still writing uh py spark I mean truly like in a notebook like it looks like something like this or a more complex one like this this is just data frame code you're giving a little bit of metadata for feature form to use everything gets usable both as a data frame and back and under the hood we are building production grade data pipelines similar to as if a data engineer built it themselves and we even provide parameters for them to tune if they want to perfectly tune their spark job or whatever else they can do so as well on top of that I mean that's just the compute piece there it's the monitoring governance like there's all these jobs that we run we essentially provide that full endtoend feature life cycle um but we're running all the compute and storage and everything via commonly used open formats so that they're interchangeable so you can like your data Engineers won't see this like weird new thing that host data and stores data to them it will just look like a ton of things that they're used to but as a data scientist you get all this stuff kind of built in and finally um it's open source um check it out um you can find this on GitHub and if you want to learn more you want to reach out um I'll be in our booth after this and then you can always reach out at Simba feure form.com if you'd like to talk more about this awesome think it's question time question time dude that was really good that was the meme game is on point uh some of those like of course we document our features right like all the time always I could just hear people going don't and the CMO the chief meme officer here nice all right very cool so there was something you said at the end I'll ask the first question and then I'll let people drop in questions um because it takes a little bit for the stream to catch up but when you were talking about like using Iceberg and being able to use it with Kafka I didn't quite understand what you meant by that and why what's going on there yeah so think of the concept of I'm building a new feature this feature is um let's say a user a favorite item per user per day if I'm building that today and I want to train on it I want to train on of my historical data and so that's the idea of backfill so how do I write one transformation that I know will have I will pretty much backfill all the historical feature values so I can train but it will also continue to update in production as new data comes in so that in production when I'm using it it's it's staying up to date so that typically that's done very peacemeal and it's very Uh custom like people build and do this themselves we feature form because of the abstractions we've come up with uh you just write your SQL query and it'll just make it happen okay cool and then the whole idea of like it going with Kofa and iceberg is just basically saying you you'll unify it from the back field till right now yeah and think of it from like a data science perspective you're just like hey here's how I write that query I write the query I registered feature form and now it exists the whole concept of okay I have to back fill I have to interactive Iceberg and how do I pause that and then start a new streaming job and keep that up to like there's all these things you have to think about that if you're a data engineer it's like what your days look like and it's very complicated as a data scientist using feature form you just here's my SQL query cool I'm good like that Chasm between production and experimentation just goes away nice okay cool now the other piece that uh obviously am interested about is you mentioned iceberg is there it doesn't matter Iceberg whatever Delta Lake um hoodie all those or we support uh all three now so we because we have um we try to not force like if company's fully on hoodie or fully on Delta we don't really want to push them to use Iceberg by default feature form uses Iceberg and open source feature form only uses Iceberg um and I would say their Iceberg implantation is probably best um I think um uh nowadays I think it's it's because of uh data bricks buying tabular and the concept of unifying um kind of Iceberg and Delta I think the the separation between those two will go away um I think you know the question is Sol for hoodie um I like hoodie um we we you know so I I we'll kind of see what happens there but I guess my view is that soon enough most of what we do will be interchangeable and we can kind of customize based on the specific formats to get just e that extra performance okay and then for the uh the engines like what does that look like yeah I mean it's interesting like this separation means that every engine uses the indices that we create in iceberg slightly differently so we actually have to kind of tune them a little bit based on it but one thing that was really exciting that happened very very recently is uh snow Flake released uh something called uh Dynamic tables for Iceberg um and what it shows me is that snowflake is also taking the approach now of kind of DED removing the storage layer so you can actually store your uh snowflake tables in S3 or whatever as Iceberg and I just think that's where everything's moving to and as that continues to happen the compu engines kind of become plug andplay and really what what's the focus becomes how do I tune these indices and build all this glue around it and that harness around all that that's what feature form is all right yeah so this is this is fascinating for me because um it's like your in a way I'm trying to put it into a box in my head right and it's like you're orchestrating things but you're not orchestrating typical data in the way that we would think like airflow orchestrates it you're orchestrating more the features around it and I think that's one of the things that I've heard people talk about so much is decoupling the features from the code is so powerful yeah I think yeah I would describe it very similarly like airflow is for building and scheduling dags that's in it's good at it and there's other tools that are good at it too um and I think the problem with features there's a lot of just H semantical like problems that only exist for feature engineering like like feature drift monitoring doesn't exist outside of features so there's all these Concepts that we kind of have to plug in and API and abstractions and everything is really oriented towards a workflow that they ml people have yeah yeah and how so last last questions what's the wildest uh architecture you've seen with this maybe whether or not people are using feature form what are some fun things that you've seen because uh that's always a good one um what are fun things I've seen I think the most crazy thing I've seen is well one I I I'll put it as like the the tools that you will see at some of these old companies like you'll just learn about these like sap databases and other types of databases that like I've even like never heard of before that we had to like learn and figure out I think one of the funniest ones I ran into was um for one company um and I'm gonna this isn't all exactly but I'm going to change it a bit to make it so I can tell the story what they would do is they would pretty much SSH for like four servers because someone kind of built a back door enough to be able to get to data because they wanted to work in notebooks and not do everything through the way data end wanted and so they would like go and like almost like sneak in for like four servers and download like Park files and like bring them back and it would take hours because like those Network calls were so slow it was kind of a mess it was it was kind of funny it's crazy the things that data scientists will do to avoid using an mlops platform it's not good I think a lot of mops platform people like are really caught up and like it's perfect if you use it but there kind of this ux problem that's ignored like it's a product right like a platform as a product and so you need to think of who your end user is and if they don't like what you have even if it's theoretically valuable then it's not valuable yeah yeah it's like uh if you send an email and the subject line doesn't entice the person to open the email doesn't matter how good that email is is cuz the person's not going to open it so I feel you and the funny thing is is if no one uses it you can kind of convince yourself that it's really good and it's the users who are wrong and like you're you I've seen that too it's it's kind of that's dangerous yeah but then yeah on the other side it's it's also funny like the amount of work that we go through to not have to do that and not have to use it and the uh it reminds me of like when you see some of the amount of work that some like criminals or conmen go through to create this elaborate scheme in this elaborate con and you're like man if you just focus that energy into something legit you would have what you needed and probably way more and it would have been legit and legal so yeah I think it's very true there we go man well Sima this has been awesome dude I am going to Cheers you virtual cheers and cheers thanks so much we'll we'll have anybody that wants to go chat with Simba down at the booth and we're going to keep rocking because now we've got the round table session so yeah Sim I think you're also going to go to a round table session too right there's yeah I think there's a feature or yeah I think there's a feature store one there's or like engineering one so that would be cool to have you there and be able to ask you more questions so now let's keep it rocking I'll see you later s thanks dude