#30 Machine Learning Engineering for Production (MLOps) Specialization [Course 1, Week 3, Lesson 6]

Key Takeaways

some machine learning tasks are trying to predict an inherently ambiguous output and human level performance can establish a useful baseline of performance as a reference but human level performance is also sometimes misused let's take a look one of the most important uses of measuring human level performance or hrp is to estimate bayes error or irreducible error especially on unstructured data toss in order to help with error analysis and prioritization and just establish what might be possible take a visual inspection task this may have happened to you before but i have gotten requests from business owners saying hey andrew can you please build a system that's 99 accurate or maybe 99.9 accurate so one way to establish what might be possible would be to take a data set and look at the ground truth data say you have six examples where the ground truth label is these and then to ask a human inspector to label the same data blinded to the ground truth label of course and see what they come up with and if they come up with these you would say this inspector agreed to the ground truth on four other six examples and disagreed on two out of six and so human level performance is you know 66.7 and so this would let you go back to the business owner and say look even your inspector is only 66.7 accuracy how can you expect me to get 99 accuracy so hrp is useful for establishing a baseline in terms of what might be possible there's one question that is often not asked which is what exactly is this ground truth label because rather than just measuring how well we can do compared to some ground truth label which was probably written by some other human are we really measuring what is possible or are we just measuring how well two different people happen to agree with each other when the ground truth label is itself determined by a person there's a very different approach to thinking about human level performance which i want to share with you in this and the next video beyond this purpose of estimating bayes error and establishing what's possible using that to help with error analysis and prioritization here are some other uses of human level performance in academia hlp is often used as a respectable benchmark and so when you establish that people are only 92 accurate or some of the number on a speech recognition data set and if you can beat human level performance then that establishes then that helps you to quote proof that your learning algorithm is doing something hard and helps get the paper published i'm not saying this is a great use of hlp but in academia showing you can beat hlp maybe for the first time has been a tried and true formula for establishing the academic significance of a piece of work and helps with getting something published we discussed briefly on the last slide what to do if a business or product owner asks for 99 accuracy and if you think that's unrealistic then measuring hlp may help you to establish a more reasonable target there's one other use of hlp that you might hear about that i'll be cautious about which is i've seen many projects with a machine learning team wants to use hlp or beating hlp to prove that the machine learning system is superior to the humans doing the job and as tempting as it is to go to someone and says look i've proved that my machine learning system is more accurate than humans inspecting the phones or the radiologists reading x-rays or something and now that i've mathematically proved the superiority of my learning outro you have to use it right i know the logic of that is tempting but as a practical matter this approach rarely works and you also saw last week that businesses need systems that do more than just doing well on average tested accuracy so if you ever find yourself in this situation i would urge you to just use this type of logic with caution or maybe even more preferably just don't use these arguments i've usually found other arguments than this to be more effective at working with a business to see if they should adopt a machine learning system the problem with beating human level performance as proof of machine learning superiority is multifold beyond the fact that most applications require more than just high average tested accuracy one of the problems with this metric is that it sometimes gives a learning algorithm an unfair advantage when labeling instructions are inconsistent let me show you what i mean if you have inconsistent labeling instructions so that when an audio clip says nearest gas station let's say 70 percent of laborers use this labeling convention and 30 percent of labelers use this labeling convention neither one is a superior transcript to the other both seem completely fine but just by luck of the draw 70 percent of laborers choose the first one 30 choose the second one so if the ground truth is established by a labeler maybe just a label with a slightly bigger title but really by one labeler then the chance that two random labelers will agree will be 0.7 squared plus 0.3 squared which is 0.58 so we had two labels use the first convention there's a point seven square chars of that or if both of your random labels use the second convention there's a point three squared charge of that then the two of them will agree so the chance of two labels are green is 0.58 and in the usual way of measuring human level performance you will conclude that human level performance is 0.58 but what you're really measuring is the chance of two random labelers agreeing this is where a machine learning algorithm has an unfair advantage i think either of these labeling conventions is completely fine but the learning algorithm is a little bit better at gathering statistics of how often ellipses versus commas are used in such a context then the learning algorithm may be able to always use the first labeling convention because it knows that statistically it has a 70 chance of getting it right if it uses ellipses or dot dot so a learning algorithm will agree with humans 70 percent of the time just by choosing the first laden convention but this 12 improvement in performance whereas human level performance is 58 and your learning algorithm is 12 better is 0.70 this 12 better performance is not actually important for anything between these two equally good slightly arbitrary choices the learning algorithm just consistently picks the first one so it gains what seems like a 12 advantage on this type of on this type of query but it's not actually outperforming any human in any way that a user would care about and one side effect of this is that if your speech recognition task has multiple types of audio for some there's this dot dot or ellipses versus common ambiguity and the learning algorithm does 12 percent better on this if your learning algorithm makes some more significant errors on other types of input audio then when is performance where it actually does worse could be average out by queries like these where kind of fake looks like is doing better and this will therefore mask or hide the fact that your learning algorithm is actually creating worse transcripts than humans actually are and what this means is that a machine learning system can look like it's doing better than hlp but actually be producing worse transcripts than people because it's just doing better on this type of problem which is not important to do better on while potentially actually doing worse on some other types of input audio given these problems of human level performance what are we supposed to do measuring human level performance is useful for establishing a baseline using that to drive air analysis and prioritization but using it to benchmark machines and humans sometimes runs into problematic cases like this i found that when my goal is to build a useful application not publish a paper you publish a paper let's prove we can outperform people that helps publish paper but found that when my goal is to build a useful application rather than trying to beat human level performance i found it is often useful to instead try to raise human level performance because we raise human level performance by improving label consistency and that ultimately results in better learning outcome performance as well let's take a deeper look at this in the next video