Computer Vision is Not Perfect

Key Takeaways

I'm hoping everybody caught our episode a few weeks back maybe it's been a couple months now on object net for those of you who need a quick refresher object net is sort of a sidecar project to the ever-popular imagenet corpus of images so as most of you know a lot of the computer vision challenges or benchmarks or just general training datasets you use imagenet comes up over and over again I think over a thousand different class labels everything from truck to dog cat to Panda don't forget the gibbons in the titles of these papers just in the past couple years started including the phrase meats or often beats human level accuracy well that's as pretty much as good as it gets if it's gonna do object recognition as well as a person I don't know that I can hold the algorithm too much high of a standard so where do we go next well before there's an X there might need to be a step backwards here because the problem is many of those algorithms do not generalize to training data not including in the imagenet corpus object net is a corpus of out of sample test examples to see how well your computer vision algorithm generalizes and let me tell you guys the report cards are not pretty these ninety-nine point whatever accurate models are not generalizing very well well why not could some of it be due to overfitting it almost implicitly sounds like this is an issue of overfitting and sure models do over fit but at some level you kind of expect that imagine if I showed you a page of logos from some graphic design students project specifically logos placed into actual scene settings there's no doubt in my mind you could identify what the logos are now if one of those images also had the Nike logo in it you would certainly recognize that one too but you'd recognize it as the Nike logo is that overfitting not really in the strictest sense but that line is worth exploring so after finishing my prep work for the object net interview I thought you know what I need to look around and see what the hacker communities doing is anybody trying to break models or just come at them from more of a lower level what the heck does this thing do and I was fortunate enough to come across a blog post that gave me just what I was looking for this week on the show I'm joined by Julia Evans blogger technologist programmer zine publisher and presenter some pretty awesome YouTube talks we've got that more right after the break [Music] hey Davis skeptic listeners we're launching another survey and we'd like your help if you've got two minutes to spare please take our survey at data skeptic comm slash survey your feedback helps us deliver the quality content you can find from data skeptic and you might just win a free t-shirt so do me a favor and head over to data skeptic comm slash survey and tell us about yourself I am Julia and I run a little team publishing company called Wizards Ian's awesome well tell me a little bit about Wizards eenz before we get into our main topic so I think five years ago I was working as developer and I was really excited about s trace which does it lets you trace system calls and I was giving a talk about it and I wrote a zine about it which is like a little like twenty page like hand drawn thing and Sharpie about like my love for s trace and I gave it out at this conference hey wait long story short it's the whole thing but people really loved it and I started making more and now I run in business doing that for the last like eight months and this turned into this big thing that's how I spend a lot of my time right now very cool well I know a lot of the audience will know what grep is but not necessarily s trace can you give us the high level and share the love sure so let's say you have a program and your program is for example opening files on Linux and you want to know like which files its opening right s traces is this tool that will tell you every system called the program is running so a system call is basically like the way that your program has to open every file it'll ask the operating system like hey can you open this file and there's no other way to open a file on Linux except use a system call so if you use s trace you can kind of see exactly what your program is doing you don't have to have the source code and you don't have to know even the programming language that it's written in I think of it as like this magical spy tool that you can use to learn about what any program is doing I love it because of that very neat yeah I'll have some links to things about use eans in the show notes including that one for people who want to check it out I also have had the pleasure of seeing a number of conference talks you've given on YouTube and heard you another podcast and read your blog and there was one blog post in particular I want to invite you on to discuss because it matched so well with the interpreter theme we're covering today so I guess we're getting into the nitty-gritty of that to start off with maybe I could ask what was your background with neural networks and deep learning before you went on to the adventure we're gonna talk about today yeah at the time I was working as a machine learning engineer building sort of fraud models but I'd never used a neural network ever basically and I have like no idea about how they worked and the reason I wrote this post cuz this is really about a paper that I read and that I loved and the reason I love this paper is that it started to give me kind of like the first idea about how do all networks work right instead of being like this is a total black box in magic in terms of like you've done some I would guess qexg boost kind of stuff for what types of algorithms did you know before like Renan Boris and logistic regression I assumed had a lot of the knowledge carries over did you have any presumptions or surprises that you learned along the way about how deep learning works well this paper is about how to trick neural networks and really it surprising thing to me about the paper was that it basically shows in a way that neural networks are more like a linear function than you might think in some ways which was not something that I would have assumed like I think I knew that they were sort of like logistic functions in there but like the idea that it would act like at a large scale in any way of like a linear function it was very surprising to me this Joel Network is like image Det right that's classifying images and I was like well that's not a linear function at all but actually like it is like a little bit yeah I think everyone's had that moment where you first tried the legitimate image search whether it's Google or somebody else's really like oh wow a lot has happened in the last decade this works now you know we have to give a lot of credit to image recognition but creeping up on us is of course these ways you can trick it was that reading that paper your first introduction to it or did you tend to notice that that was sort of a part of the discussion just learning about neural networks I think reading that here was my first introduction to that idea that you could trigger a neural network yeah I'd never heard of it before so one thing I liked about your blog post was it got really deep into the hands-on of things we've covered these fooling image papers in the past and sort of talked about like okay the gradient does this and that but yours was the first like really hands-on step-by-step walkthrough could you talk through that experience and the steps you took to do your own fooling of a neural network yeah totally so I read this paper and I was like that's cool but I don't personally believe in things that don't implement and that I can't do like on my computer the reason I wrote this is someone had asked me for an article they're like hey do you wanna read an article first about something and I was like oh yeah I read the school paper I want to write this article but then I was like well to write the article I need to implement it right cuz like what good is it if you don't implement it on your computer and then there's the question I was like wait can i implement this is it going to be super hard well at first I was like wait like how do I get a neural network model right I'm like do I need to train it because I only had my laptop which is from 2013 and I was like I'm pretty sure I don't have a GP here and I can't train this model right like it's not gonna happen I'm the first school thing I learned is that actually you don't need to Train models all the time you can download them so I found a version of imagenet which I could download which is the same model from the paper and I just downloaded it and it was like a hundred megabytes and I was like sweet great because I just need to check the model right I didn't need to train it so it was good enough to download it and what's the tricking procedure like and it gets you sort of talk about the basic idea so let's talk about logistic regression first instead so let's say you have this really big vector has 1000 entries in it or something I think the way the paper explains it is that like well the l1 norm is really different from the L infinity norm and you're like what does that mean right like look at like I had to think they're like why that was relevant at all about what I was trying to say is let's say you have some vector like an image which is a factor of like RGB values and let's say you make a really small change to that factor like you add like a value of like 0.1 to it so 0.1 is not that big of a number and if you're just like changing an image that's not gonna result in like a visually obvious change to your image right but if you instead take the sum of all those like 0.1 values and there's like a thousand things in the vector then that's gonna add up to one hundred right which is a really big number you kinda exploit this by saying okay let's say I have some logistic regression model which is just a vector that ever I'm taking like the dot product of my image with this other vector if you add some like small error and you make the signs of those errors match up with your vector that you're taking the dot product of and you can make it the result of your of like taking the dot product of the logistic function the vector and your vector change a lot does that make sense it's hard to say without like a white board yeah maybe we'll make an analogy if you humor me to steganography where you can embed a hidden piece of data in an image in a subtle way people can't see is there a parallel there so I guess you're sort of like superimposing on your image a very light thing which matches up exactly which is kind of I guess like in phase with whatever the model is so that when you take the dot product some like large positive number it gets added to the output and actually when I did this in practice like the second auger everything is interesting because when I did this with the paper towel I don't know if this is meaningful but when I like made this like vector to shake it with a paper towel I got a bunch of swirls like the pattern it looked like was some swirls and I was like oh maybe that's because paper towels of swirls but maybe it's not I don't know but like it seemed like they were sort of like some meaning you know to like the very light vector I was adding and it actually was related to like yeah that was one aspect I really enjoyed about your blog post maybe you could go a little deeper about some of the ways you just sort of raw interacted with the model for example I myself had never thought to ask what is a perfectly black square get classified as what did you learn just kind of having a conversation with the model if you will totally I always like to do ridiculous things with computers when I'm trying to understand them I guess because I feel like you often learn a lot more that way right to be like okay instead of us trying to classify something normal let's classify like a black square so let me see I'm just gonna look to see what the blog post said ah yeah so it said that the black square was like 27.3 8% velvet and 4.67 2% a paper towel well the paper towel is weird but the 20% velvet I almost want to say that's an a-minus right yeah yeah exactly it could be for lighting or something another funny thing actually in there was I looked at the Queen this particular model doesn't know anything about people at all and it said it was to be clear Queen Latifah what Queen are we talking about I'm sorry the Queen of England thank you that Queen so I asked it about a picture of the Queen of England and it was like oh that's definitely a shower cap 99.7% the crowd and I guess maybe all of their pictures of shower caps are worn by old women like I don't know right like it's hard to know why it thought that exactly but it was very sure that the Queen of England was a shower cap like you know your training set is your I guess that raises an interesting point I mean a lot of these examples probably seen the one of like is a dog or is it a chocolate chip muffin and there's a lot of these cases that are sort of benign and almost juvenile and like mistakes we can laugh at I don't as often see mistakes of like of concern like oh it's labeled someone is a criminal when it's just a person walking down the street did you see anything like of concern or were they mostly like silly mistakes when you were able to trick the Machine at the lowest levels well I could trick the Machine into thinking anything I wanted so I could say like oh here's the Queen and then I could I didn't test this specifically but I think that I could probably trick it into thinking that the Queen was a vulture right like I could take like any label and convince it that any image was that label it's kind of interesting and that some of these mistakes seem almost explainable to us like oh we see what you were going for but then there's this added layer we can start to manipulate the images and get it to say just about anything like you're pointing out but that is a distortion in any of those cases were you able to tell like oh I can see this image has been manipulated no because the pixel values I think we're very small so you can tell it all that no gotcha so it's really below the threshold I guess in that respect and then how dramatic could it be it wouldn't surprise me if you know you could get a car to be a truck or something like that but in terms of crossing these class boundaries to what extent can you trick the machine thanks to this week's sponsor terminus DB termas DB is a model driven graph database it's open source now and forever and it's been battle tested in a number of industrial settings yet surprisingly it's open source and you can try it out for free terminus DB can unlock the access of graph databases to your project or application store data in its ideal form so the data scientist can make better use of it the best way to sum it up is to say that terminus DB is like git but for data can you imagine having source code that you don't manage with git well how are you managing your data if you don't have a quick answer head over to terminus DB comm and see if it might be the solution for you so the first thing I did was I took this like black screen and I made it like 100% paper towel this is definitely a paper towel of probability like 99.9 percent or something and then I had a panda and I made that into 100 percent of vulture because I think I found that the Panda wasn't a vulture at all I don't think I found things that I couldn't do like a panda doesn't look like a vulture and it's definitely not like 99.9% vulture right like the differences are pretty great the paper talents pretty interesting what as you kind of went down the gradient and accented the most paper towel list of the paper towel features I don't know what what is the extreme example of a paper towel look like it looks really swirly if you multiply the pixels it just looks black but then if you like amplify the pixels you see a bunch of swirls that are all kinds of different weird colors you show a number of those examples where there's this I don't know if we call it the mask or the composite but the distortion that gets applied I find myself at points sort of reading into those and thinking like oh it must be the swirl that's kind of the design that the popular brands use or it must be these features of a panda's face to what degree do you think that's sort of anthropomorphizing is appropriate yeah it's not clear I think I would say I would not say 100% but something's going on maybe we could comment on how diffuse it is do the distortions tend to gather in regions how do they compare amongst each other I can tell you something different that's interesting actually which is so there are two ways you can do this tricking thing so when you take this neural network and you're trying to trick it you can either try to move it towards something like oh I would like to make this more like a paper towel or you could feel like I could make this less like a cat you can move it away basically just like whether you put a plus sign or a minus sign on your factor of like a zero point ones so one thing that I tried was I would take dogs and it would be like oh this is a Corgi or whatever and then I would be like okay let's make it less like a Corgi what if it's not that and then it would pick like a different type of dog instead and then I'd be like okay but also not that dog and I would sort of like keep on moving it away from whatever species of dog get it picked and it would keep on picking different dogs it would seem to make sense to what degree do you imagine that might be a bias of the imagenet data set that there are a lot of dogs I guess so like I mean that it could make a sort of child's mistake looking at a picture of the Queen but it's like you're struggling to convince it that a dog is not a dog right yeah yeah my sense was definitely that there were a lot of dogs in that dataset because it was more difficult to move it away from the idea that something was a dog and then I'd love to know more about that process it makes sense to me intuitively that you distorted no it's not a Corgi it's this type of dog were you in kind of what I guess an optimization person might call a deep valley or were you at some point able to pop out of that and make the dog into some non dog thing so I don't remember if I could get the dog to a different thing I do have this graph of what happened when I move the Panda and to being the vulture at first I tried to go and just like one step add one factor that was going to trick it into being a vulture and that actually didn't work that was I think what the paper said to do and it works sometimes but sometimes it didn't so what I ended up doing actually was taking sort of like 100 small steps towards being a vulture so I would take like the gradient of the neural network take a small step sort of in the direction of vulture like see where I got and then take another small step from there and it turned out that I sort of like taking 100 small steps to try to make it a vulture worked a lot better but on its way to becoming a vulture it also went through like Madagascar cat and given for some reason but there's nothing like a rise it first sort of went towards like Madagascar cat and more than vulture and then the vulture later on took over I feel like that must have been something about Felix pace of images and actually I think at some point also it went through being an ostrich on the way to vulture and I a machine-learning expert about this question and I was like oh it was like a panda and that I went to be an ostrich to being a vulture he was like oh yeah it totally makes sense there's this panda ostrich situation I was like really maybe it's because they have a lot of black on them like I was so surprised because I thought this was just like a random thing that just happened but he was like no this is a real thing of the data that happens with y'all networks you know it's funny you should mention that because that example perked my ears that you went through Gibbon there's the famous paper maybe you know it as well one of the earlier fooling images ones and their example is here's a panda apply the filter get a Gibbon yeah there must be something about that Gibbon that in the multi-dimensional space they're in a closed manifold or whatever you don't know how to describe it but there's something there for sure oh man I'm now looking at pictures of pandas and Gibbons on my phone and trying to understand I guess they bought that black around their eyes maybe that's it yeah that's an interesting aspect of this how can we know what it's doing the systems do seem to by and large do a nice job but also make these mistakes do you think that the way in which we trick them gives us any hints about the quality of the models I find hard to comment on this kind of stuff because honestly I don't know a lot about neural networks I don't like to give like big takes about what this means I think it definitely made me feel like it was not super magical if it was so easy to trick well it made me think it was kind of linear I guess is the point as like a math person my overall impression was just like oh this is more linear than I thought and I don't know what it means that I know network is like quote unquote like more linear than I thought but I think it's really interesting well as you point out in the blog post there are many amazing things that come out of this technology you can now do image search and get exactly the child you're looking for whatever the case may be great at retrieving cute pictures for sure possibly great at driving cars I don't know yet given your experience and the knowledge of neural networks the degree to which you have how do you feel where are the lines you know what are you comfortable relying on a neural network for I think with machine learning in general it's really important to understand that the importance of your training set what we saw with like the Queen and the shower cap if something isn't in your training set it's not gonna be in your model I feel like that's the most common thing that's sort of like lay people don't understand about machine learning how common it is to have training sets that in some way do reflect real-world data and how it means that then the model isn't gonna work on that real-world data that like isn't represented in the training set you know so I read a lot of literature on different ways we can fool these types of systems and we interview a lot of people who are trying to develop these automated processes but all the wonderful academic side aside there's this aspect of this where you know you went through some non-trivial steps but you downloaded something wrote some code and were able to interact with this thing and without it having been too nipson read every single of the latest papers you're able to be like look the queen is not wearing a shower cap I was just curious if I could get some of your general thoughts on how I don't know if it's hacker ethic or just ingenuity that you bring to the table how might people who don't have PhDs in machine learning attack and learn things about the models well I definitely don't have a PhD in machine learning I took one machine learning class and my masters that was it and I think this whole project would be maybe three days something like that it definitely took a lot of hours in those three days maybe you know it wasn't like a huge project I mean the ipython notebook are the the Jupiter notebook is amazing basically all I did was I downloaded this model and I Kathy I guess that was what I found but I'm sure it would have worked with tensorflow too if I could get that to work oh yeah it says it only took six hours of cursing which was pretty good some of my favorite lines in the post yeah yeah I made a github repo which has all of this though it probably doesn't work anymore because it was 40 years ago yeah I think it's pretty approachable especially if you're not trying to train the model I think that playing with the model and trying to see how it behaves is a lot easier than trying to train the model where you need to figure out parameters and it takes so long to train it and like what's going on and I feel like if you really want to get hands-on with the model it's more fun to sort of play with it and I think the other thing that I didn't realize actually is that you could interact with them at a different level than just putting predictions into it right because I had the model I could take like the derivative of the model at the point and that it's really easy to do that because at first I said like probably like six hours trying to figure out how to take a derivative and then eventually I was like oh backpropagation is literally taking the derivative and it's like the most common function the thing that neural networks want to do the most is to give you directives which is the thing that the paper was about you can sort of like do that and explore the models are a little plastic I guess right yeah you know I had spent some time gave up trying to figure out the right question I could ask you that was solicited that exact comment from the blogpost I really like when you said oh this is basically just the chain rule it's the most delightfully dismissive yet accurate description of deep learning I've heard yeah right okay I asked on Twitter like someone who I knew it was like Julia is just the chain of hook that's a back prop is and I was like oh okay I know where I am right like worried and highlights the fact that yes these are very linear systems that were able to manipulate them because the derivatives give them away in certain respects I guess as we've said you have some ml background maybe you don't do a lot of this heavy duty deep learning every minute of the day but do you have any general thoughts on what it would take in terms of design to come up with models that don't suffer from these sorts of challenges do we need a revolution or is this just gonna be incremental oh man I mean I've never seen a model that doesn't suffer from these sorts of challenges yeah me either but I think we give ourselves credit that we don't I mean yes we get fooled by optical illusions but whatever algorithm I'm running seems to not have this property I don't think that's true oh yeah my impression I guess if machine learning models in general is that like if you give it something weird it's not gonna work look if you give it something which is like well represented in training set then that's great I think it's pretty common in the real world to have machine learning models because like most machine learning models exist in the real world you know and so they're getting all kinds of weird data thrown at them all the time and so of course it's gonna do something dumb and sometimes harmful I would go more to earth we should put some safeguards on those systems and make sure that like nothing's super bad can happen when the model does something dumb then to figure out how we can trust the machine learning model completely that's good advice one of the things I wanted to ask you about was a little bit along the lines of the toolbox you've certainly have a strong proficiency and a lot of the things people would need to pull this stuff together and can do a project like this what are some of the key things you think an aspiring data scientist needs to know to walk a similar path yeah obviously python is amazing the jupiter notebook is amazing cuz it lets you do this kind of exploration where instead of writing a program you're just like Oh what happens if I do this so what about that Oh what about that and I think it's a much more fun way to work this post I think this is actually where I first learned about docker because I needed to install this hold like cafe thing and it was super confusing and so I'm pretty sure I downloaded a docker container which had kind of everything installed inside it already and that was super useful cuz it just made the installation burden a lot less and I'm not sure if it was going to be else installed on my laptop otherwise without having and I guess there's also just the willingness to use some new framework that you don't understand right like I didn't understand cafe I Sloane understand but like I kind of managed like a wrangle it to do what I wanted like in this case and I was like alright I just need to take one derivative that's literally the only function I need from this and so I just learned how to do that I don't understand I got what I wanted absolutely so to wind up let's remind everybody the paper that inspired your work in case they want to read that after they've gone through your blog post and get a little deeper what's the paper that got you started along this path it's called explaining and harnessing adversarial examples from 2015 and it's 11 pages I found it pretty clear I'd really recommend reading it and the other thing that I read that helped you with this also from 2015 is called breaking linear classifiers on imagenet by Andrey Carpathia it also talks about pandas and Gibbons it explains how to trick it into the Ghia panda as a given yeah and we found that those two things so I think there's really something about the pen doesn't the Givens yeah that post has a lot of really great pictures for sure y'all have links to all that in the show notes as well as places people can learn more about yours Eanes julie where else can people follow you online i'm on twitter at work i have a blog that's it I believe it's Borgess be 0rk it's bzrk that's right cool I don't have that link in the show notes as well thanks for taking the time to come on thanks so much for having me [Music] thanks for listening to data skeptic interpretability our guest this week was Julia Evans our theme song is number 5 by Big D in the kids table Claudia our brewster's our Associate Producer Vanessa Burciaga does guest coordination I've been your host Kyle polish stay in and stay safe everybody [Music]