Adversarial Explanations

Key Takeaways

so you're listening to data skeptic you can imagine that people like you are also listening to data skeptic would you like your message to reach them or your company's message to reach them why not consider advertising with us send your enquiries to advertising a data skeptic com who is ready to go straight to a computer doctor I definitely am but I know I'm an outlier in that most of us don't trust don't yet trust our health and our bodies are at least to review the Diagnostics on those things to a machine perhaps for good reason last week on the show we learned about object net the computer vision corpus designed to be more of a capture of the real world after the observation that a lot of the popular data sets are intrinsically biased it's the dirty little secret of computer vision that with all the high accuracy scores generalization to the real world still is a problem another problem is the ability to fool these systems as we discussed with Angwin quite some time ago here on data sceptic these adversarial attacks that can be performed on the neural networks and at first I wrote that off said well you've got to have the weights of the other Network that's paramount to having a breach almost or security leaks how did someone get your model yet we've otherwise learned that models can be stolen from api's knowledge of the training data set can often let us know already what baked-in biases or potential for adversarial attacks might be available transfer learning makes that even worse and there's been quite a bit of research to be frank I haven't read all of it yet but it seems to be the case that you can do adversarial attacks quite effectively without those weights in a lot of situations while we're criticizing let's throw in the ever-popular accusation that neural networks are black boxes we have literally no idea whatsoever how these things work that's rather blatant clickbait scientists may not totally understand what's at the depths of the oceans but they're not blind neither our machine learning practitioners we're building the tools of interpretability what do we need to do to sufficiently explain the decision-making process of these models at a higher-level language one not involving tensor multiplication yet I've recently learned that even some of those interpretability tools be fooled an adversarial attack can be used to generate what appears to be a convincing explanation although one that really isn't a generalized explanation for the operations of that model but there it actually lies something kind of insightful it reminded me of early math courses you study as best as you can for the tests you show up the first pages all easy answers then a couple of difficult ones and then out of the blue as that test is winding up can you believe what this teacher did they snuck in an example that we didn't really go over in class what do you do when X is negative we didn't talk about that teachers trying to trick me no teachers giving you a adversarial example and the way in which you respond is gonna be quite insightful as to whether or not you memorize the rote performance of a few actions or you generalize the technique perhaps the true path to model interpretability is watching how our systems get fooled welcome to data skeptic interpretability a podcast about computer vision adversarial attacks and model interpretability today on the show I've got Walt Woods who joins us to discuss his paper adversarial explanations for understanding image classification decisions and improve neural network robustness this is what inspired the introduction it brought in some of the ways I think about neural networks I'm excited to get into that interview right after the break [Music] thanks to this week's sponsor the 2020 Gartner data and analytics summit while this event is next year in Grapevine Texas March 23rd through 26th now is the time to act you can save three hundred and fifty dollars with my early bird code those of you traveling on the company dime keep in mind that your boss may have a budget for events and travel and this sort of thing and if he or she doesn't use that up they might lose it the agenda has gone up on the web to key notes that caught my eye Worley McMullen speaking on creating a culture that is ready for AI and dr. Hanna Frey who I love she's on more or less all the time I read one of her books she's great professor mathematician author that's just the tip of the iceberg across the eight tracks they're offering visit gardenerd.com slash us slash data that's Gartner GA RT and ER comm slash us slash data there enter the discount code all caps no spaces data skeptic I'm Walt woods and I am currently a research engineer for a Galois Inc what do you do there so gal was a bit of an R&D lab that focuses on trust in critical systems and so right now I'm working on a couple projects one of which deals with kind of you know PDF is a very broad suspect there's a lot of files that fit in that and there are a lot of parsers for it and so basically because you have multiple interpretations of the spec can you then reason about the overarching security properties as people kind of add on to this language and can you track how exploits grow in that space along those lines and then some malware evolution stuff and looking through your Google Scholar record you're a pretty diverse in the topics you've published on in addition to the computer vision paper we're going to discuss today could you give me a sense of where the breadth and scope of your research interest comes from sure so I joined academia after being a senior software developer for a few years because I wanted to work on more difficult problems basically and I had dabbled with machine learning and neural networks a fair amount you know in a high school making games okay neural networks that's a way to avoid hand coding an AI agent right I suppose that when I joined grad school I figured that what was holding the field back was the hardware that there wasn't fast enough ways of simulating large enough networks so I actually it went to grad school for ECE and did some work on accelerating what's called sparse coding algorithms so there's some biological research right that what your eye is doing is breaking down the visual scene that you're looking at into these kind of salient components and then it's just a composition of those components that actually get processed by the rest of your brain and say hey maybe I should dodge that car driving towards me and so as far as coding is basically a mathematical means of approximating here's a bunch of raw pixel data let's turn that into you know higher-level components so dude dealt with some circuits for that and then the computer vision stuff which I was really excited about because interpretability and I kind of got back on that because it was rather apparent that hardware was not what was holding these algorithms back you know watching the GPU boom as it were and seeing Google harvests you know thousands of GPUs to train on ever larger data sets and seeing that they're eking out a few more percentage points on tasks like image net but definitely not solving the problem in a way and that's part of what makes adversarial examples so interesting to write is that it doesn't change the semantic content of the image but the interpretation is completely different so on the show previously we've covered adversarial attacks and I'll put some links in the show notes maybe even to my early recording with Angwin who was one of the early publishers I guess their claim to fame was their adversarial images weren't just noise gradients they actually kind of looked like things you know so think they even won some art contests with these adversarial examples they had so I imagine the listeners get that part question is that related to the deep dream Stefan or which thread are they pulling so I think that's more towards the root of it all they are one offshoot that was saying hey we can design other variants of adversarial images they are just these purely kind of Gaussian noise things we can do in other more interesting ways I guess all of this is my long explanation to say we've talked a lot about adversarial attacks on the show but we've never had the phrase an adversarial explanation what's that all about yeah so that was kind of a fun turn to put together but you know if you think from a person point of view if you're looking at a picture of a bus there's a series of image transformations you can apply that will eventually turn that picture of a bus into a picture of say a frog and if you think of that as a spectrum basically right if you take a picture of a frog and subtract the picture of a bus that's kind of a one-shot hop to go from the bus to frog but as you turn down the magnitude of that perturbation you know eventually there's going to be a hazy grader in the middle where you're not sure if it's a bus or a frog and so it's kind of interesting that that didn't happen in machine learning applications and that kind of bothered me this work actually grew out of the idea that we would stick an intermediate layer in the neural network and do some heat mapping on the intermediate because attention networks have become a thing so it's clear that if you look at just the input pixels you could make sort of a heat map so then that guy was thinking well maybe you can make a heat map of some of the intermediate steps and pull something interesting out but what became clear pretty quick is a a heat map for a frog and a bus might look the same from certain angles and be it still was unstable right after so examples kind of highlight how unstable neural networks are and even if you have a really convincing explanation a really convincing heat map if you can perturb that image ever so slightly and get a completely different result well how much was that explanation really telling you so oh that's kind of a noodly way of saying that the recognition that current explanation techniques were kind of fragile in the face of these adversarial explanations coupled with the fact that you know in the salient domain world and what we think of looking at a picture is that a frog is that a bus there should be more of an intermediate ground between the two images and it's borderline sensible that seemed like a domain that was really lacking and so set out to try to develop a transformation that would at least aprox with neural networks and do you think this is a property that's exclusive to computer vision or is this just the one place you happen to be battling with it no definitely not it shows up in a lot of machine like tabular data applications get this too so what I think one of the papers I cited for the adversary explanations paper talks about tabular data and how you can generate adversarial examples on that too and it kind of boils down to the same idea where you know if you have a truly robust classifier that if you put some input in always gives you the correct semantically significant output then you would expect that when that output changes if you look at the changed input it should reflect that semantic change and so most of my work focused on the visual aspect but the hope down the line would be to extend it to tableau the method was derived specifically to be pretty general and then I didn't get out the full title it's adversarial explanations for understanding image classification decisions and improve neural network robustness before I guess we get into the core contributions you have a pretty well sighted and well researched related work session could you hit on some of the highlights bring us up to speed to where your contributions begin there's been a lot of great related work in adverse examples one of my favorites is this lab six weird bio at all it's a mispronunciation but they actually took the idea of adversarial examples into the kind of I believe 3d printed a model of a turtle and painted it in a certain way so that if you hold it up to different camera angles it always shows up as a rifle and Google's object detection thing I really like that as an example that you know this is more than just a minor image manipulation problems actually a highlights an issue with the way neural network vision applications process data so Alexander Madri at all has a good body of work on what's called adversarial training so athaliah it all kind of showed hey this is a real thing you know we can derive real-world representations that from a variety of angles actually cause adversarial effects if you're using like a phone camera and so Madrid alls theory was kind of if you look what these adversarial perturbations are and you just build them into a training cycle maybe the problem will stamp itself out and they actually show a fair amount of the initial robustness results that have held up over time there have been some other denoising approaches that have been proposed but a lot of those rely on assumptions that at least if you assume you can see the weights for the network don't hold up you can basically always derive an adverse OA tack that's still low magnitude whereas the matcher it all work produces more robust networks yeah those examples like the turtle and the rifle are almost terrifying to me in the sense that I see so many positive case examples of computer vision working well like Yolo is such a great example of that you know like wow real-time processing getting all these objects correct one could easily build a great deal of confidence that computer vision is approaching a solved problem but then you tell me this painting turned sideways suddenly a rifle how are we to reconcile this cognitive dissonance yeah it's very alarming especially if you consider it for a brief moment I was touching on some airport security applications and I mean just imagine there right if you just design a certain material that you know completely hides basically what this object actually is an airport scanner and that's a very real thing that could happen with current machine learning tech this work was fun to work on for two reasons one it shows you kind of what the models looking at right the idea of adversely explanations is that biases in the training data actually show up and you can see it's keying on this feature which maybe is present all the training data but doesn't generalize to the object class that it's actually trying to detect be the techniques in the paper actually result in a more robust Network and so I think that there's a lot more progress to be made in this space where you can make a machine learning model that acts more like you or I would when presented with adversarial images where we make a more reasonable gradation of this buses turning to a frog or this turtle into a rifle and so basically I guess stabilizing the models almost seems like and in Horton's step in addressing that particular issue and do you have maybe a tangible example of what it means for the training data to have one of those biases that shows up yeah I don't know if you're familiar with revile that almost line work of course yeah we're gonna have him on the show in a couple of weeks actually well that's fantastic so I feel like they had a really good example of this where they I want to say they trained a bunch of pictures of a husky and there was always snow in the background and basically if you then showed that classifier an image of anything with snow in it snow was the dominant feature where even though they were trying to get the network takea on the Husky it actually ended up keying on the snow there's also I don't have a citation on hand I know some researchers have done work on like racial profiling and police systems and that's another place where this sort of bias can creep in in a scary way where you might have you know the number of convictions to date fitting a certain profile which has nothing to do with the crime at all and then you can make certain disturbing statistical correlations there that aren't necessarily indicative of the underlying thing that you're actually trying to classify yeah a very pointed example also alarming with that in mind I definitely see the value in why we'd want better explanations I know we haven't touched on the methodological choices yet but from sort of an end to end point of view if someone were to be benefiting from your contributions and the work described in this paper how much better of explanations do they get what sort of added interpretability can they expect to gain from the experience that's a tough question to answer for a couple reasons one is that a lot of the output is qualitatively different right and if you look at the appendix of this work there are a lot of examples showing different transformations and comparing again the Madrid all Network if you just do adversarial training and then do some of the explanation transforms I talked about you get things that look kind of like the target class so like in Figure s5b on the paper there's this little picture of a dog as input and the eighth column shows it turning into a horse and you know if you look at vercelli train network qualitatively it's difficult to see a horse you can kind of see that it's getting some maybe red leg wraps in the lower right corner or whatever and then if you look at the grows below that with our method you give something that looks holistically like a horse so from a qualitative point of view I would say that this paper significantly improved the quality of explanations you get out of networks over previous work you know the methodology we attempt to quantify that - there's this concept of back to random accuracy robustness area so do you want me to get into that methodology a bit I guess and compare that to prior word yeah I'd love to because everyone will know techniques like accuracy and f1 score in these sorts of things which are wonderful statistical tools but I guess your qualitative tool has more rigor than most do in a way and that's kind of interesting well that's the hope right but you always want the number to go up when the quality of the results goes up if you're doing that as far as image quality goes you're doing well so I guess I'll start by talking about the accuracy robustness area a little bit early in this work you know I had this basic idea of kind of this Lipschitz constraint and it seemed like the results were good so let's compare it to other related work and looking at the adversarial example literature what you find is a lot of people say our classifier you know on CFR 10 gets a 92% accuracy on a clean data and 60% accuracy when it's under attacked by an adversary and the problem with reporting results that way is that 62 percent accuracy under what conditions right what what type of adversary you're using how large of a perturbation are they allowed to make what do the resulting images actually look like all that information isn't really rolled into that 62 percent single member so you know looking at precision recall curves basically for inspiration where if you're doing this binary classification problem you want to wait false negatives against false positives and look at how your class photos with number of different thresholds if you plot that ROC curve and then you take the area under it you get kind of summary behavior rather than behavior with one given threshold so the idea with accuracy or bust this area is to do an analog of that where rather than we're gonna report the accuracy of our model against one specific adversary that is allowed of perturbation it's 10% of the overall image magnitude let's look at how the classifiers accuracy degrades against adversaries allowed to make increasingly large perturbations to the image because again back to the bust of Frog example at some amount of perturbation you want your classifier to change right even the perfectly robust classifiers should break down at some point and so kind of looking at the behavior of how as you increase the size of perturbation allowed looking at how the classifiers accuracy Falls if you take the area under that curve that's what accuracy robustness area is supposed to capture so that's accuracy robustness area and so if you have a data set like C far that has two highly related classes a truck and a car are the example I used in the paper don't share a lot of features right they're both vehicles they both have a lot of kind of square edges there are differences there's a reason we call a truck a truck in a car a car but they're very highly related classes and so when we are testing our network modifications against the mandarine all modifications the attack area where you just measure the top one accuracy of that classifier degrading against an adversary wasn't actually producing numbers that I foe correlated well with the qualitative change in the explanation so that's what led to this thought experiment about cars and trucks being highly related classes where how do you measure you know in an accuracy sense those will always be close together you almost expect that a smaller magnitude perturbation is going to turn a truck into a car than versus a truck into a frog making a truck into a frog should take a higher magnitude change in the pixel set and so that's where this better than random area was conceived where what if rather than just measuring the adversary's distance - hey let's make the classifier wrong instead let's measure the perturbation size needed to transform the classifiers prediction back to a random chance prediction so if you have ten classes and the initial class of our confidence is 60% it's a car let's go from 60% all the way down to 10% and that number the BTR aring or better than random ara correlates very well in my opinion with the explanation quality you get out of the model one hope honestly going for it is I would be super excited to see the accuracy robustness area ideas coming out of some of the tests of subsequent work because I feel like it's really important to characterize the behavior of a classifier against a whole range of adversaries rather than you know just reporting one accuracy number at a certain fixed perturbation yeah absolutely do you have any thoughts on the like reproducible process for how we make that happen a lot of the vanilla statistics like an f1 score most of the popular machine learning libraries have the tooling built-in whether it's SK learn or whatever these are new ideas that aren't yet in those libraries how can they be popularized yeah that's a great question I suppose getting an implementation into one of those libraries would be a good start but this paper does have code attached it's honestly a pretty simple algorithm you just sample a few 1,000 in this case images and you measure basically you know how far an adversary would have to go to break each of those images and you build a curve and do some stats so there's code for handling that attached to this paper maybe I should put in a pull request to buy torch or something for getting this sort of evaluation easier to access because you're right that that is a barrier to entry at the end of the day everyone's their heads in the sand focus so much on what they're researching that if you come across the mention of another evaluation criteria you're just going to skip over if you can't find a readily accessible means of implementing it yeah similar question when it comes to like the half hover rectified linear unit which first of all should ask you to define but given its importance in the contribution how can one get that maybe into tensor flow or wherever they might be using it it a real small change I think it's like ten lines of code or something to implement the half Huber rectified linear unit not at all difficult methods to pick up and start using no maybe that's the better point here is that I don't know how much research works this way with this research at least one of the two main outcomes the leap ships constraint and the half Hueber rectified linear unit main outcomes in terms of modifying models both reduced to pretty simple implementations and you can see that in the code attached to the paper they're both very small changes to the training process or your activation module that really make a significant difference in the results you're gonna see in terms of explanation quality yeah there's something that kind of interesting for me in how something that takes so few lines of code and is an arithmetic process I guess if you break it all down that that can have this big consequence on this more abstract idea in the same way like I don't know if we change the chessboard from its current shape to plus 1 squares all around it the whole game would probably be very different what's the intuition that got you to come up with that insight and then think to try that instead of the myriad of other functions you might have tried I forget who said it but there's a quip floating around somewhere about a gradient descent via grad student so in a sense the way this leap shits idea came about again I was playing with other methods of explaining a model and trying to come up with Sol Sol I'm if you look at prior explanation techniques a lot and attention models there's some great attention model work as well but if you look at those they're all focused on the generation of a heat map and if you have two things that have a similar silhouette the heat maps not going to really tell you much about that and so the original thing that I was setting out to do in a sense was come up with some way of producing High Definition full-color explanations that would actually show you the rich features that the networks keying on and adversarial examples were a natural problem for that because you could make a small color space change and end up with a completely different output and so basically you know there have been a lot of theories about why adversarial examples are such a thing and one of those theories is that neural networks are too linear if you then measure basically the rate of change of the output of the network with respect to the input you can see that yeah the output changes a ton when the inputs change only a little which kind of reinforces that view of adverse examples I don't necessarily prescribe to that view as an overall explanation for the phenomenon but I think it's an interesting thought experiment the leap shoulds constraint is what you get when you minimize that quantity right when you minimize the rate of change of the output given the input and so it basically naturally fell out of that kind of poking at how can we get a full color explanation to pop out when full color changes change the output of the network too much if you backtrack that well what's a way to measure how much the output is gonna change when you change the input in this case it's you know the derivatives basically and then applying the leaflets constraint naturally minimizes that and it turns out it works very well there is a fair amount of prior work on layer wise optimization or constraint of the Lipschitz constraint it just really turns out that end to end made all the difference in this case at least so the paper walks through some of the steps you guys took the beginning with C fart n doing some training on that data set and then going on to generalize to others like the ever popular image net and cocoa and whatnot can you talk a little bit about how you saw the work evolve when moving into other data sets and domains going from inist to C far in this being of course the ever popular black and white digit data set in this to see far is a really big step you're getting a much higher resolution image well a somewhat higher resolution image you're getting full color and the diversity of the classes there are backgrounds included right there's a lot of entropy in each image basically that your machine learning algorithm has to filter out and so there are a lot of algorithms that can show promise on in this but don't scale really well to see far in contrast when you're scaling from C far to image net the semantics content of the images are basically the same if I'm not mistaken see far is a scaled-down version of the same data basically that makes the image net data set and so the main issue with going to image net is the number of outputs right going from 10 output classes to a thousand output classes and where that you know if I'm being framed basically I got lucky what happened there is when you stabilize the output as the leap ships constraint does it doesn't matter as much if you're dealing with 10 outputs or a thousand because you're stabilizing all of them relative to one another to pivot actually what was really interesting is going from C far 10 and imagenet to the Japanese Society of radiological technicians data set the lung nodule data set and what's so fascinating about that is you only get I think it's 247 images total of full chest x-rays and I want to say a hundred some of those actually have nodules but it's still a high definition data set and so basically dealing with you know still a heightened dimensionality data set where you don't have as many examples of each class that was a really interesting and kind of difficult to wrangle transition so part of the paper discusses an adaptive technique right for figuring out how strong the sweep shuts constraint should be so that you still get good accuracy of the classifier is just robust accuracy and when you have so many fewer examples of each class that balance is a lot harder to do with a fixed scaler so the adaptive technique I think really came out of dealing with that J SRT data set so this research really leaves off a couple of possible points you could follow up on it could you talk about a few of the directions that you yourself or maybe you hope others may head following up on this yes there are two follow up aspects that I personally am super excited to see one and you touched on this a little earlier is extending it to domains other than just images again the technique super generalizable it should just fine on audio data in some encoding space at least and it should work just fine on tabular data and/or you know word embeddings or something along those lines and I think it would be really interesting to see how it would end up actually applying to those data formats and what you know the explanations would look like and if any algorithm changes would be needed to get it working the second aspect that I'm very excited to see is so I'm actually a little annoyed at this it's always if if she'd had more time you would like to see a better version of the research because a lot of the feedback I've gotten on this work mentions and rightfully so the reduction in accuracy on clean data that these robust classifiers get and it's very much an open topic in the literature of can you make a classifier that both is accurate and it's robust and what is so fascinating to me about this work and the thread that I would really like to pull out is why does the training loss you know in a classification sense never reaches zero I think it's so interesting the neural networks which have so many parameters and if you're not applying this sort of regularization and it's worth pointing out it turned out in the course of paper that no other regularization like shape drop or drop out or anything like that was applied in this work just the the resident architecture with the leap ships constraint what's enough if you take a base model and apply it to especially a small data set like C for ten you expect your final training loss to be very close to zero you expect to have almost hundred percent accuracy on training data and then it doesn't generalize as well over to evaluation data but in this case there's something about the combination of the regularization constraint the Lipchitz constraint and maybe the model architecture maybe there's a better activation function than the half cuber I'm not sure but you would imagine that there should be some combination which allows for that overfitting on the training data and I think that would be fascinating to see because in my results the loss on the training data and the evaluation data in a classification since our base right I end up with so the most robust networks have around 69% classification accuracy on CFR 10 but that's both training and evaluation and it's just not expected that you would see that or at least it surprised me I think that there's work to be done there to explain why the parameters for the network are getting locked up in that way basically and if you can unlock them than you might recover that accuracy while retaining the robustness gains do you think that's kind of a formal trade-off we're gonna encounter the way we have bias variance its robustness accuracy is it that sort of level of formality or just more of a heuristic I guess I think that's what a lot of the field is pitching I think that the most robust classifier you can imagine for say school buses is always gonna predict a school bus if you have a classifier that never changes this decision that is by definition the most robust classifier you can come up with but it's not very useful right and so clearly in the limit there is some amount of trade-off there but I don't think we're close to it now I think there's a lot of head space to grow into in getting networks that are both accurate and robust and frankly even the Madri it all worked on adversarial training I believe that they drop seven points of accuracy maybe down to ninety or something percent accuracy and they get a significant gain in this accuracy a are a metric that the paper talks about compared to the base classifier and so extrapolating from there one's got to imagine it can be pushed out even further without sacrificing accuracy or as much accuracy is this work sacrificed so robustness is largely about dealing with those biases due to the training sample we touched on earlier in that regard how will some of these methods help in getting more generalizable models models that will transfer two domains it hasn't seen yet super interesting question I think because you would expect right that in this adversely explanation work it's visually apparent that the classification member gets a pretty good idea of what objects it's looking at it kind of understands at least the rough outline of what a horse should be and yet it doesn't lies as well and what's interesting to me is that adversely examples on non robust networks are transferable right they pick out noise patterns that are independent of the initialization of the network meaning that you can initialize a bunch of different networks and they'll all be susceptible to the same noise pattern in the end and what that tells me is that the the popcorn noise adversarial example that you traditionally see is a feature in the data set right it is a bias that actually is there and shockingly it generalizes really well so it does indicate that there's kind of something missing from current robustness techniques because we're not seeing more generalizable accuracy out of those models even though the features they're keying in on are you know lower frequency more salient ly observable in a sense very interesting yeah Walter well first to tell us where to find the code I'm sure at least a few listeners are gonna want to give that a spin for themselves and if they wanted to maybe reach out and discuss the work what's a great place to get in touch with you if you're open to it sure so the code is on github it's at W would slash adversarial - explanations - C far feel free if anyone's interested in chatting about adversarial explanations or really anything on model interpretability and explanations I'd be delighted to receive an email and the emails in the paper it's W woods at PDX edu excellent well thanks again for taking your time to come on the show and discuss your work with us yeah thanks for having me [Music] thanks for listening to data skeptic interpretability our guest today was Walt woods Claudia Armbruster is our associate producer Vanessa Burciaga does guest coordination our theme song is number 5 by Big D in the kids table which is office foot 7 inch with lounge not off shot by la me like YouTube would have you believe but in any event last but not least I've been your host Kyle polish [Music]