#38 Machine Learning Engineering for Production (MLOps) Specialization [Course 1, Week 3, Lesson 14]

Key Takeaways

in the last video you heard about the step of assessing a project for technical feasibility and for value let's take a deeper look at how you can carry out this diligent step to figure out if a project really is feasible and also how valuable it really is let's start with feasibility is this project idea technically feasible before you've started on the machine learning project how do you know if this thing can even be built one way to get a quick sense of feasibility is to use an external benchmark such as the research literature or other forms of publications or if a different company or even a competitor has managed to build a certain type of online search system before or recommendation system or inventory management but if there's some external benchmark that might help give you a sense that this project may be technically feasible because someone else has managed to do something similar either to complement this type of external benchmark or in the absence of this type of external benchmark here are some other ways to assess feasibility and i'm going to build a two by two matrix that looks at different cases depending on whether your problem has unstructured data like speech images or structured data like transaction records and on the other axis i'm going to put new versus existing whereby new i mean you're trying to build a system to do a toss for the first time such as if you've never done demand forecasting before and you're thinking of building one whereas existing refers to if you already have some existing system maybe a machine learning one maybe not that is carry out this toss and you're thinking of scoping out an improvement to an existing system right so new means you are delivering a brand new capability and existing means you're scoping out the project to improve on an existing capability in the upper left hand quadrant to see if a project is technically feasible i find human level performance hrp to be very useful and give you an initial sense of whether a project is doable so when evaluating hrp i would give a human the same data as would be fed to a learning algorithm and just ask you know can a human given the same data perform the task such as can a human given a picture of a scratch smartphone perform the task of detecting scratches reliably and if a human can do it then that significantly increases the hope that you can also get a learning algorithm to do it for existing projects i would use hlp as a reference as well where if you have a visual defect inspection system and you're hoping to improve it to a certain level of performance if humans can achieve the level you're hoping to get it to then that might give you more hope that it is technically feasible whereas if you're hoping to increase performance well beyond human level performance then that suggested project might be harder or may not be possible in addition to hlp i often also use the history of the project as a predictor for future progress and we'll say more about both hlp and history of project in the next few slides but the previous rate of progress on the project can be a reasonable predictor for the future rate of progress on the project you'll see more of this later in this video moving over to the right column if you're working on a brand new project with structured data the question i would ask is are predictive features available do you have reason to think that the data you have the inputs x are strongly predictive or sufficiently predictive of the target output y in this box on the lower right for a structured data problem if you're trying to improve an existing system one thing that will help a lot is if you can identify new predictive features so are there features that you aren't yet using but you can identify that could really help predict why and also by looking at the history of the project so on this slide you heard about three concepts human level performance the question of whether predictive features are available and also the history of a project let's take a deeper look at these three concepts and let's start with using hlp on unstructured data images so i use hlp to benchmark what might be doable for unstructured data because people are very good on unstructured data tasks and so the key criteria for assessing project feasibility is can a human given the exact same data as would be given to learning algorithm perform the task let's look at an example let's say you're building a self-driving car and you want an algorithm to classify whether traffic light is currently red yellow or green i would i would take pictures from your self-driving car and ask a person to look at an image like this and see if a person looking only at the image can tell which lamp is illuminated and in this example it's pretty clear green but if you find that you also have pictures like this then well i can't tell which lamp is illuminated in this example and this is why it's important for this hlp benchmark to make sure the human is given only the same data as your learning algorithm it turns out maybe a human sitting in the car and seeing the traffic light with their own eye could have told you which lamp was illuminated in this example on the right but that's because the human eye has superior contrast to most digital cameras but a useful test is not whether the human eye can recognize which light which lamp is illuminated the useful test is if the person was sitting back in the office and they can only see the image from the camera can they still do the task and that gives you a better read on feasibility and specifically it helps you make a better guess and whether a learning algorithm which will only have access to this image can also accurately detect which lamp in the traffic light is illuminated making sure that a human sees only the same data as the learning iron will see is really important i've seen a lot of projects where for a long time a team was working on a computer vision system say and they thought they could do it because a human physically inspecting the cell phone or something could detect the defect but it took a long time to realize that even a human looking only at the image couldn't figure out what was going on then you can figure out much earlier that with the current camera setup it just wasn't feasible and the more efficient thing to do would have been to invest early on in a better camera or a better lighting setup or something rather than keep working on a machine learning algorithm on a problem that i think just wasn't doable with the imaging setup available at the time next for structured data problems one of the key criteria to assess for technical feasibility is do we have input features x that seem to be predictive whenever we're trying to predict why let's look at a few examples in our ecom example if you have features that show what are the past purchases of a user and you like to predict future purchases that seems plausible to me because most people's previous purchases are predictive of future purchases so if you have past purchase data you do have features that seem predictive of future purchases and this project might be worth a try or if you work with a physical store given data on weather if you want to predict shopping mall foot traffic so how many people will go to the mall well we know that when it rains a lot fewer people leave their house so weather is predictive of foot traffic and shopping malls and so i would say you do have predictive features whoever try let's look at some more examples given dna of an individual let's try to predict if this individual will have heart disease this one i don't know the mapping from your dna to whether or not you get heart disease is a very noisy mapping in biology this is referred to the genotype and phenotype but the mapping from genotype to phenotype or your genetics to your health condition is a very noisy mapping so i would have mixed feelings about this project because it turns out your genetic sequence is only slightly maybe mildly predictive of whether you get heart disease so with a question so i'm going gonna put a question mark there or given social media chatter can you predict demand for a clothing style this is another iffy one i think you may be able to predict demand for clothing style right now but given social media chatter can you predict what will be the hot fashionable trend six months from now that actually seems very difficult so one of the ways i've seen ai projects go poorly is if there's an idea like let's use social media to figure out what people are chatting about in fashion and they will manufacture the clothing and sell it in six months and sometimes the data just is not that predictive and you end up with a learning algorithm that does barely better than random guessing and that's why looking at whether you have features that you believe are predictive is an important step of diligence for assessing technical feasibility of a project one last example that that may be even clearer which is given a history of a particular stock or a particular shares price let's try to predict the future price of that stock all the evidence i've seen is that this is not doable unless you get some other clever set of features looking at the single shares historical price to predict the future price of that stock is exceedingly difficult and i would say if those are the only features you have those features are not predictive of the future price of that stock based on the evidence i've seen and so even leaving aside the question of how much predicting share prices or trading creates any social value i have some questions about that sometimes i think this project is also just not technically feasible finally on this diagram one last criteria i mentioned a couple times is the history of a project let's take a look at that when i've worked on a machine learning application for many months i found that the rate of previous improvement can be maybe a surprisingly good predictor for the rate of future improvement here's a simple model you could use let's take speech recognition as an example and let's say that this is human level performance and i'm going to use human level performance as our estimate for bayes era or the irreducible level of error that we hope to get to let's say that when you started the project you know say in the first quarter or q1 or some year the system had 10 error rate and over time in subsequent quarters the error you know went down like so so q2 q3 q4 and so on it turns out that it's not a terrible model to estimate this curve so if you want to estimate how well the team could do in the future one simple model i've used is to estimate the rate of progress as for every fixed period of time say every quarter the team will reduce the error rate by some percentage relative to human level performance so in this case it looks like this gap between the current level of performance and human level performance is shrinking by maybe 30 every quarter which is why you get this curve that is exponentially decaying towards hlp and by estimating this rate of progress you may project into the future that hopefully in future quarters you continue to reduce the error by 30 percent relative to hlp and this will give you a sense of what might be reasonable for the future rate of progress on this project and thus this gives you a sense of what may be feasible for an existing project for which you already have this type of history and can try to extrapolate into the future in this video you saw how to use human level performance the question of whether you have predictive features and the history of a project in order to assess technical feasibility next let's dive more deeply into assessing the value of a project we'll do that in the next video