How to Hack an AI

Key Takeaways

foreign I'm not sure if everyone can tell from my voice um firstly I'm coming down with a bit of a cold um and secondly I'm Australian so it's 5 a.m here um I've had a lot of coffee I think the combination of both are going to be good for the talk ultimately I'll just be very honest so um I apologize in advance if I make really bad jokes um so today I'm talking about how to hack an AI and I found that um it's been really great to hear the previous speakers um both Giant and Matt talked about AI security and I find the common struggle when I talk about AI security is that people are like oh great AI for cyber security and I'm like no no the security of AI because it is just as valid an attack surface and that's increasingly becoming apparent so I'm currently the CEO of maleva security labs and before my current role I was working in Consulting Academia startups government either implementing AI selling AI or buying AI of someone else and I found that security and safety and bias particularly sort of 10 years ago when AI was even more hyped than it is now um was never a topic of conversation and it was only when I started my PhD in adversarial machine learning that I realized like this is a real academic threat uh why is no one talking about it it's going to become a threat for Enterprises as well so it was really great to be able to hear Matt talking about the nist framework um that's one really important thing that's coming out there's a lot of different standards and Frameworks coming out what I find though is that um AI is so often used as an analogy for magic right people keep talking about using AI for this and AI for that but um that doesn't really mean anything so what I find quite useful is to actually go through a bunch of different attacks that can be performed on AI um to really hit home exactly how AI systems are vulnerable and why those kinds of Frameworks are really so important so let me start by um talking about what AI is exactly and I won't labor this point because 20 minutes and um some of the previous speakers have gone through definitions already which is great so the two that I'll highlight here are adversarial machine learning and then AIML security so adversarial machine learning is the ability to hack ml algorithms through a range of methods that broadly exploit technical vulnerabilities inherent in the architecture of deep machine learning optimization and then AI security refers to the technical and governance considerations that Harden those AIML systems to technical exploits by an adversary so it's a little bit different to talking about safety and bias and we can think of the Cyber analogy where cyber safety is sort of about how you know people use cyber systems in a safe way and you know particularly like children elderly people um thinking about the impact of the real world use of those systems whereas cyber security refers to how to actually like technically um secure those systems for an adversary and then the governance levels around that um so that's the AI analogy as well and so when when I usually talk to people both in my professional and my private circles about AI um for some reason the first people um especially people who aren't really deep in AI um the first thing they say is oh Skynet the Terminator um and I don't know if that's because I just hang out with too many people born in the 80s or if the Terminator really was just a great movie maybe a combination of both so for this talk I like to lean into that and I use Skynet as an analogy for an AI system that has both computer vision and language tasks and we're going to look at how um like broadly you could hack that system so let's go through our objective for today so usually the objective would be to demonstrate categories of adversarial machine learning attacks and how they could be practically implemented great um we're going to reframe that to how you can hacks guidance so it cannot identify and kill Sarah Connor so at a broad level I'm conscious that people here probably aren't AI experts um even the term AI is sort of you know I don't I don't like to use too much um you're probably typically cyber and information security professionals so I came from the data science side and then moved into cyber and information security a bit later on say at a broad level these are like the two main architectures of machine learning models that we see today um we have neural networks um which are really great for computer vision because they're able to condense um high levels of information that you would see in in pictures and videos and then the Transformer architecture which is the backbone of language large language models like uh like chat GPT and without needing to go too far into the weeds essentially almost any like every machine learning um model is really just a combination of being able to map input data to Output data that mapping uses an optimization um so that you can make the most efficient mapping from input to Output so that when you see a new input data or piece of information you can most effectively map it to a classification or a prediction as an output at the end and what most machine learning algorithms rely on is this idea of gradient descent optimization so this is the function that essentially enables you to perform that mapping of input to Output over many many iterations and without this optimization machine learning wouldn't work at all because typically most machine learning models are Black Books right we don't know exactly how they work we only know that over many iterations they learn to map input to Output through this process and so this is great this this is what's enabled um deep machine learning to become so important um in Academia and industry but it introduces a vulnerability in that if you understand this optimization function um for a given Target Model you understand what kind of mapping it's producing between that input and output and you can use it to your advantage so instead of trying to minimize the error or minimize the loss um you can instead maximize it but within a small value so that it's not so obvious or perceptible um so without needing to go too technical into this um we'll come back to gradient descent later um but but you can be conscious that this is you know there's a real technical underpinning under AI um that you can exploit so if we look at a typical ml life cycle and I'm generalizing here because it can get really specific but you usually group it into your training phase and your inference phase so you'll start with your input data you then train your model um you have some kind of output or prediction and then ideally there'd be a kind of feedback loop now this is how an ml practitioner would create a deploy model you can also think of this as your attack surface so some broad categories of attacks that you're performing at each of this stage uh poisoning attacks so poisoning the training data so that the model learns essentially the wrong thing um you have extraction attacks so creating a near replica of the model to train and launch further attacks you have evasion attacks so causing predictional classification to be incorrect and inference attacks so leaking sensitive or confidential information the model was trained on and the ways that you would Implement these attacks in the kind of manifestation they would have can differ depending on the model architecture so attacking a neural network for computer vision is quite different to attacking a language model in sort of the results it would produce but essentially the underlying techniques you would use are pretty similar so this is what our Skynet model is doing and how it sort of looks under the hood so it's probably learned um it's performed that mapping by being fed a lot of images some of which are Sarah Connor it's learned features through its different convolutional layers about what an image of Sarah Connor looks like and then it's able to perform some kind of prediction at the end based on a confidence level as to whether that image is Sarah Connor over Lara Croft for example am I going to start by going through evasion attacks which does occur further down here in the output or prediction phase but this is where the field of AML really came to the fore um and the examples are a bit cooler so I don't want to run out of time for this one so this is the proverbial adversarial machine learning example anyone who's aware of AML has probably seen this one before um so basically in 2013 2014 the field um really took off when they're getting good fellow created this adversarial example here in the middle of this this specially targeted noise where they used um that gradient descent function of a Target Model and instead of stepping down the optimization to you know optimize it they instead took a step up the optimization function and we're able to create this noise that was specially crafted based on the Target Model to try and influence the prediction um based on knowledge of the parameters so it's not random noise especially crafted noise specially crafted to produce and like an untargeted attack so where you want the prediction to be anything but you can also create targeted versions as well so the combination of the image of the panda plus this adversarial example um causes the Target Model to predict that it's a given with 99.3 confidence even though a human can't see this and this is sort of an older example now um 10 years on the field has moved to create lots of different examples like this um the top left is a 3D printed image of a turtle um which rotated at any angle can cause a classifier to predict that it's a rifle and you can imagine the impact of the inverse of that right like a rifle with a specially crafted coating that's always predicted to be a turtle um that stop sign um proved the brittleness of many uh models in autonomous vehicles by showing that even just random patches over stop signs can prevent a driverless car from recognizing it and drive straight through and there are lots of um and more specific attack types as well where you can do things to the stop sign that aren't so obvious and then the example on the right shows an adversarial patch worn by that second person that prevents it from being um recognized by sort of people classifiers and now you have things like um adversarial glasses that prevent you from being recognized by by facial recognition so we've gone ahead and done this to our Sarah Connor image um so we have an arrow and image of Sarah Connor on the left um I've created these and this adversarial example for that image um using mobilenet V2 as the target classifier which is just an open source model and I've added that to that original image and then I've run it through an open source celebrity detection model by clarify sorry clarify um and so this is the original image it shows that it's Linda Hamilton at 99.9 accuracy which is correct that's the actress who played Sarah Connor and when we put through the adversarial example um predictor to be Harry Styles which you know I would take as a compliment Harry Styles is very pretty so this this is just an example of how easy it is to create these kinds of attack types um in like in real life um the extent to which you can really perturb an entire image um it really depends on the use case so for particular like maybe signals classifications example it's easy to perturb the entire like frame um and some of the research that's being done now in this domain is looking at like specific targeted regions um so the example I did before was an untargeted attack so let me run through a targeted attack for you so I put together this notebook and hopefully it plays let's see perhaps it will not work okay um basically um I run through this notebook and you consider to see from you know the the bar on the right at least how short it is um I can basically use the same method of the gradient descent optimization to produce a targeted example so I can narrow in on for example the rifle that she's holding create um choose a specific Target like a starfish and then create that adversarial example um placed on top of the image of the rifle and when I run it through Melbourne at V2 again it predicts that it's a starfish at 100 confidence um and the point of being able to show you how easy it is to do is that especially now that chat GPT has been released um the ability to use AI has been democratized um and is so much easier which is great but the ability to attack it is also um is also democratized and it's and it's so easy to do so let's move into poisoning attacks so we're over here at the input data phase and I'm going to Breeze through these ones because I want to make sure we get to the sew up within the 20 minutes but poisoning attacks involve um poisoning the training data so the model learns the wrong thing there's a bunch of examples of this already happening in the real world so we've probably heard of Tay Tay was a chatbot released by Microsoft that was um that that would learn off the inputs that people used to interact with it so it very quickly became racist homophobic sexist I don't know why they didn't think that this would happen but they it had to be shut down in 14 hours I think um the release of chat GPT and its dependence um not on on people's interactions with that at least so overtly um is is really important because people do try and Red Team it right um here's another example so the just as we have the equivalent of back doors and cyber security we also have back doors in in AI so in this example um the top right image has a backdoor trigger so the little little circles down the bottom in the training phase um those images were given a Target label zero the model essentially learned to associate that back to a trigger with the label zero and then down the bottom at the inference phase it always predicts an image with that backdoor um to be a zero regardless of of the True Image and maybe this example seems a bit you know cool okay got it um but you can imagine the implications where um that back door is like the adversarial example where it's a lot harder to detect that it's actually there and there are um some examples of being able to add backdoor triggers to um to training data in this way and especially with the dependence of AI on a lot of Open Source data um there needs to be steps involved to make sure that the data that you're using for trading your model is actually clean and free of like bias and back doors so extraction attacks involve creating a new replica of the model to train and launch further attacks and given the commercial um importance of a lot of models particularly at the moment uh you know this is a massive commercial value to firms if that if their model is leaked as well as a security risk so cyber security naturally is an area where um a lot of models are created that are really closely held for um for commercial reasons but also to ensure the safety of the end users so things like spam detection malware detection um monitoring sustainment um if those models are released um that doesn't just represent massive Financial loss to the you know the company that that has created that model but also it allows cyber actors to get into systems so here for us got an example we could um perform targeted queries to the model that that's typically how this would occur um to extract that model information and understand what kinds of like features the model may have learned about Sarah Connor so that we can disguise those um or understand um what kinds of other vulnerabilities that model might have that we can then exploit inference attacks involve leaking sensitive or confidential information the model was trained on um here's an example from a large language model so gpt2 is known to be vulnerable to um when prompted with information that could be like potentially sensitive it would come back with information about a person's you know address and details and other sensitive information like that um GPT 3 and gpt4 Via chat GPT um there's already been a lot of red teaming efforts against those models um they're obviously um a lot more stable than something like gpt2 but even then there's known to be issues where for example you could prompt um chat GPT to give you an example of why um of why socialism is a great system and it will it will give you a response but then if you ask it to do the same about fascism it says that it can't do that so um even chat gbt2 having a vulnerability like that a lot of academics still rely on using earlier versions of GPT because they just don't have the compute to be able to use um the later versions or the access so it's important to consider those kinds of implications as well okay so we're coming up to time already so we've implemented all these attacks against Skynet um and success Sarah Connor survived so maybe that's a bit of a trite example about how it can be used and probably work better in a 40 minute format but the the idea is to show just how easy um these attacks are and how like real a threat they are we've only gone through a few attack varietals but in things like um art which is the adversarial robustness toolbox and then other sort of repositories like clever hands there are over a hundred different attack types already out there um and this is the future of AI right we're looking at um massive increases um especially since church gbt was released last year um AI Safety and Security has finally hit the international Consciousness which I'm happy about it's it's really important that people are talking about it but it's increasingly easy to understand how these kinds of models can be attacked so just a few examples of how these models going to be used um diffusion models like the one on the left which is Dali two um generating really realistic images um that's a a great thing for me to be able to use in my in my talk that was a prompt where I asked um to create an image of a robot taking over the world um but it's really easy to create sort of deep fakes um through Technologies like that as well we all know about large language models Chachi preteen isn't the only one um lots of lots of large language models and with functionality exist and then the image on the bottom right is Sam segment anything now by meta which is a computer vision model that's able to segment all of the objects within it so this is where AI is heading um we've we heard Matt talk about um some of the sort of repositories of ttps so this is the outlook's Atlas matrix by Mida um which which includes adversarial machine learning ttps um there's also an incident repository called aiaaic couldn't tell you what it stands for apart from being an incident repository of AI related attacks and it's being updated every day with ways that um real AI incidents are happening around the world so what do we do so as cyber and Information Security Professionals um there is a lot that you can do in the field of AI security I think the thing that's really important to consider is that across all of these different attack phases you have different differences that can be employed um that just aren't because people aren't aware of how big a threat AML and and AI attacks can be but the thing is we we also need to reform the field from one of attacks versus defenses to one that's closer to the Cyber analogy of risk-based controls mitigations and governance and there is um that that frame of mind is is Shifting to to sort of a more mature Paradigm but there needs to be a maturing of the field of AI security but to become closer to that of cyber security and realistically businesses that involve and and Implement cyber security risk management practices need to start including AI in that risk profile as well so a few things for you to leave with today we use AI every day whether you realize it or not um social media Healthcare um image recognition um we already have so many touch points Finance credit checks you know we have so many touch points with AI now um and it's only set to increase so we need to make sure that the way that we're using it is secure and safe um cyber security can inform ml security please use your skills and knowledge to contribute to the field of ml security and AI needs governance and risk management we have to prevent AI from going the way of cyber security um as a massive both Technical and geostrategic threat when the internet was first being adopted people always prioritized convenience over Safety and Security and we can't let the same happen for AI so I believe we have some time for questions foreign