#34 Machine Learning Engineering for Production (MLOps) Specialization [Course 1, Week 3, Lesson 10]

Key Takeaways

for some applications having and tracking metadata data provenance and data lineage can be a big help what do these words even mean let's look at an example here's a more complex example of a data pipeline building on our previous example of using user records to predict if someone is looking for a job at a given moment in time let's say you start off with a spam data set this may include a list of known spam accounts as well as features such as a list of blacklisted ip addresses that spammers are known to use you might also implement a learning algorithm so piece of machine learning code and train your learning algorithm on the spam data set thus giving you an anti-spam model so these arrows indicate flow of information or flow of computation where training your ml code on the spam data set gives you your anti-spam model you then take your user data and apply the anti-spam model to it to get the dspam user data we're following our usual convention that things with a purple rectangle around it represent pieces of code now taking your dspam user data next you might want to carry out user id merge to do that you might start off with some id merge data so this would be labeled data telling you some pairs of accounts that actually correspond to the same person have a machine learning album implementation train the model on that and this gives you a learned id merge model that tells you when to combine two accounts into a single user id you take your id merge model apply it to the dspam user data this gives you your cleaned up user data then finally based on the clean user data hopefully some of this labels with whether someone's looking for a job you would then have another machine learning model trained on it to give you a model to predict if a given user is looking for a job or not and this is then used to make predictions on other users or maybe across your whole database of users so this level of complexity of a data pipeline is not atypical in large commercial systems and i've seen data pipelines or data cascades that are even far more complicated than this one of the challenges of working with data pipelines like this is what if after running this system for months you discover that oops the ip address blacklist you're using has some mistakes in it in particular what if you discover that there were some ip addresses that were incorrectly blacklisted maybe because the provider from whom you had purchased the blacklisted ip addresses found out that there were some ip addresses that multiple users use right such as if multiple users on a corporate campus or university campus share an ip address for security reasons but the organization creating the blacklist ipi just thought it was spammy because so many people shared an ip address this has happened before so the question is having built up this big complex system if you were to update your spam data set won't that change your spam model and therefore that and therefore that and therefore that and therefore that and how do you go back and fix this problem especially if each of these systems was developed by a different engineer and you have files spread across the laptops of your machine learning engineering development team so to make sure your system is maintainable especially when a piece of data upstream ends up needing to be changed it can be very helpful to keep track of data provenance as well as lineage data provenance refers to where the data came from so who did you purchase the spam ip address from and lineage refers to the sequence of steps needed to get to the end of the pipeline at the very least having a extensive documentation could help you reconstruct data provenance and lineage but to build robust maintainable systems not in the proof of concept stage but in the production stage there are more sophisticated tools to help you keep track of what happened so you can change part of the system and hopefully replicate the rest of the data pipeline without too much unnecessary complexity to be honest the tools for keeping track of data provenance and lineage are still immature in today's machine learning world i find that extensive documentation can help and some formal tools like tensorflow transform can also help but solving this type of problem is still not something that we are great at as a community yet to make life easier both for managing data pipelines as well as for error analysis and driving machine learning development there's one tip i want to share which is to make extensive use of metadata so metadata is data about data for example in manufacturing visual inspection the data would be the pictures of phones and the labels but if you have metadata that tells you at what time was this picture of a phone taken what factory was this picture from what's the line number what were the camera settings such as camera exposure time and camera aperture what's the number of the phone you're inspecting what is the id of the inspector that provided this label these are examples of data about your data set x and y and this type of metadata can turn out to be really useful because if you discover during machine learning development that for some strange reason line number 17 in factory 2 [Music] generates images that produce a lot more errors for some reason then this allows you to go back to see what was funny about line 17 and factory two but if you had not stored the factory and line number of metadata in the first place then it would have been really difficult to discover this during error analysis so i found many times when i happened to maybe get lucky and store the right metadata only to discover a month later that that metadata helped generate a key insight that helped the project move forward so my tip is if you have a framework for storing metadata that will definitely make life easier but even if you don't just like you rarely regret commenting your code i think you will rarely regret storing metadata that could then turn out to be useful later and just like if you don't comment your code in a timely way it's much harder to go back to commenting later in the same way if you don't store the metadata in a timely way it can be much harder to go back to recapture and organize that data one more example for speech recognition if you have audio recorded from different brands of smartphones let's say that in advance or if you have different labelers labeling your speech or if you use a voice activity detection model then let's keep track of what was the version number of the voice activity detection model that you use and all of these means that in case for some reason one version of the vad voice activity detection system results in much larger errors this significantly increases the odds of your discovering that and be able to use that to improve your learning over and performance so to summarize metadata can be very useful for error analysis and spotting unexpected effects or tags or categories of data that have some unusually poor performance or something else to suggest how to improve your system and of course maybe not surprisingly this type of metadata is also very useful for keeping track of where the data came from of data provenance the takeaway from this video is that for large complex machine learning systems that you might need to maintain keeping track of data provenance and lineage can make your life much easier and as part of building out these systems consider keeping track of metadata which can help you with tracking data provenance but also error analysis before we wrap up this section there's just one more tip i hope to share with you which is the importance of balanced train death test splits let's go on to the next video