#33 Machine Learning Engineering for Production (MLOps) Specialization [Course 1, Week 3, Lesson 9]

Key Takeaways

data pipelines sometimes also called data cascades refers to when your data has multiple steps of processing before getting to the final output there's some best practices relevant for managing such data pipelines let's start to an example let's say that given some user information you would like to predict if a given user is looking for a job because if they're looking for a job at this moment in time you may want to surface job ads or other pieces of particularly useful information to them so given raw data such as the data on top there's often some sort of pre-processing or data cleaning before the data is fed to learning algorithm that then tries to predict why are they looking for a job and the data cleaning may include things like spam cleanup such as removing the spam accounts and maybe also user id merge which we talked about in an earlier video for the sake of this example let's say that spam cleanup and user id merge are done with just scripting so explicit sequences of instructions that tells your code when is an account to be considered spammy and when should two user ids be merged of course these systems could be built using machine learning algorithms as well which makes them even a little bit more complex to manage now when you have scripts for the data cleaning one of the issues you run into is replicability when you take these systems into production deployment let's say during the development of the system you have input data fed through pre-processing scripts and the pre-processed data is fed to machine learning algorithm and after some amount of work your learning algorithm does well on the test set during the development phase you may have seen that pre-processing scripts can be quite messy it may be you hacking something up processing data mailing a file to a different member of your team having them have a few incantations in python or some scripting language to process the data and then having them mail the process data back to you when you take this system to production you then have new data which has to be fed through a similar set of scripts because this data is going to be fed to the same machine learning algorithm and your machine learning algorithm on this data is what will run in your product so the key question is if your pre-processing was done with a bunch of scripts spread out on a bunch of different people's computers and laptops how do you replicate the strips to make sure that the input distribution to your machine learning algorithm was the same for the developed data and the production data i find that the amount of effort that you should invest to make sure that the pre-processing scripts are highly replicable can depend a little bit on the phase of the project i know that it may be fashionable to say that everything you do should be 100 replicable and i'll probably get some criticism for not hewing to that line but i find that a lot of projects do go through a proof of concept of poc phase and then a production phase where during the proof of concept phase the primary goal is just to decide if the application is workable and worth building and deploying my advice to most teams is during the proof of concept phase focus on getting the prototype to work and it's okay if some of the data pre-processing is manual if the project succeeds you need to replicate all this pre-processing later so my advice would be take extensive notes write extensive comments to increase the odds that you can replicate all this pre-processing later but this is also not the time to get bogged down in tons of process just to ensure replicability when the focus is really to just decide if the application is workable and is worth taking to the next phase once you've decided that this project is worth taking to production then you know it's going to be really important to really replicate any pre-processing scripts so in this phase that's when i would use more sophisticated tools to make sure the entire data pipeline is replicable and this is when tools which can be a little bit more heavyweight but tools like tensorflow transform apache beam airflow and so on become very valuable and in fact you learn more about tensorflow transform later into specialization as well in this video you learned about data pipelines and when to invest in their replicability it turns out many applications have significantly more complex data pipelines than what we saw in this video and for those settings you also have to think about what metadata you want and perhaps also keep track and take care of data provenance and lineage let's go on to the next video to look at these topics