Kola Ayonrinde - Security Grade Interpretability Catching Failure Modes Early

Key Takeaways

[Music] Okay, cool. Um, yeah, thanks thanks guys for um inviting me. Um, yeah, excited to chat to you today and excited to see if people excited about um air safety and security and in particular and the role that mechanistic interability can play within that. Um, so I'm Cola. I um am a researcher. I work at um the air security institute. um mostly on issues of agency and mechanistic interability um and how we can apply um both um white box interability but also blackbox um techniques to um understand and secure um like powerful AI systems. Um so yeah maybe who's starting to get a sense of what people's um level of um familiarity is maybe if you're um if you're you'd say that you're familiar with mechanism experience as a field maybe if you put like maybe if you give me a thumbs up uh in the chat hands up okay great hand yeah okay so a few people a couple of people um so yeah this is what okay yeah nice nice a good number of people number of people that's Um so yeah, we're going to talk today about like kind of the foundations of ecosability um and how we can use um it's relevant for air safety. Um so yeah um there will be uh time for questions at the end but also if you have clarifications or stuff like that feel free to stop me or like jump in with questions like raise a hand or just um uh just shout out that would be cool. Um yeah so okay so this is the safety group um so why why interpretability within that why why you're interested in interability um and there's a few reasons for that some um but let's I'll give you at least one story for why you might be interested in that so progress in AI has been really fast and it seems to be speeding up over time here's like a chart of one benchmark it's people have been quite interested in which is RKGI. Um so it's supposed to um determine the kind of intelligence which is like really hard for machines to get out and for AI it would really struggle with it. Um yeah, progress is in the last um you know year been like really rapid in in getting getting up with that and the chart is actually a little bit old. So I think it's actually increased past this now and they're um now starting a third RTGI the um so because the other ones have been saturated. Uh so why oh yeah and how did that happen? Well, what really happened was deep learning really works um across many different domains from uh computer vision and reinforcement learning, natural language and even combining these things together now. So we have like um natural language systems to train with reinforcement learning and also have vision capabilities and audio capabilities, right? Um and this should seem to really work. Um and that's been kind of the real story um of the last 10 years I think uh is in in AI and computer science. Um, and we've asked like why why is this the case? Because a lot of people didn't predict this. A lot of people were very surprised that deep learning worked. Um, and so why is it the case that deep learning works and what do we learn from that? And I think um this uh quote from Ilia is probably one way of getting at this idea and he says that predicting the next token well means that you have to understand the underlying reality that led to the creation of that token. which is meaning that kind of you're doing this compression task and you need to understand what it is about the world that creates the set of statistics you seem to see. So it's like it looks like you're learning statistics but in fact in order to do that well you're learning something much deeper and that's why it seems to work right. So neural networks don't learn summary statistics like previous things before we had we used to use you know um kernel methods and we used to use like you know random forests and this kind of thing and these kind of worked okay but they were um shallow and they would you know just often had like were like very fixed. they weren't very flexible, quite brittle, they over fit a lot. But neural networks don't learn so much like those previous methods. Uh they learn algorithms and representations which correspond to the world. And that's that's why they seem to work so well, right? because they seem to like have developed these internal representations and these internal algorithms which allow them to work in many different scenarios, be quite flexible, um still make some mistakes but you know far fewer than previous systems would uh and be do this across many domains. Uh and that's that's been the real real impressive thing. Um and over because of that like neural networks we we build them we don't uh we don't we don't build them we grow them we um like give them a bunch of data just like set them off and just like you know have assay basically uh and that means that in general we don't really understand the learn algorithms even though they are there we don't we don't have a good sense of them typically um and this is this is hard from a safety perspective and from a security perspective because um we what we would like to do is be able to kind of read off um what what it is that they're doing to be able to like if they're making um doing important things in society or making real important decisions um you know transforming parts of our economy or you know our social contracts our society industry so on um we would like to be to have information about this but generally we don't have that much um and so there kind of two claims here kind of the why um interpretability seems valuable here. So one is that in the situation that we're in looking at input output behavior is often insufficient for understanding models. It's just not enough. Um just looking at this behavior um we can have systems that kind of seem to work sometimes and seem to not work in others and it's not really clear when they stop stop to work or when they perform in ways that we weren't expecting or um you know have adverse consequences on the world. and that's not really going to work for like very high stakes deployments. Um and secondly that we can reduce the risks of both misuse and misalignment by understanding the inner workings of models. So understanding how they how it came about that they had this behavior. Um and that's really the core football and so that's what in about so saying that you know we we might look to understand and reverse engineer neural networks um like neuroscientists do for brains like um some reverse engineering like um people do for compile program binaries or like physicists might do for complex systems in nature. That's that's kind of the the thrust of it. Um and so yeah, there's a few examples we might have in like neuroscience of people doing some similar things like this. So you might you might heard like play cells um which um not just scientists found that there are kind of different um areas of the brain that light up when a mouse moves through like a maze and things like that. Um and these have been really valuable and and in kind of understanding what it is that's causing um behavior and what it is how mice like representing the world around them. for example, um there's both sort of theoretical and and some practical work on like understanding how neurons relate to individual concepts um maybe in different ways that kind of generalize. Uh and then for neural networks um um like artificial neural networks we've seen sort of there be like a geometry of concepts across neural networks um from so like kind of maybe one of the first examples in words of was noting that um you could kind of do arithmetic uh in this geometric space that represented the concepts the neural networks were learning. Uh and this was kind of exciting discovery because it kind of suggested that there was actually something there to interpret. There was some structure there um that was behind how the network was performing these kind of amazing tasks. And um further we've started to see more of this. Um so one example from the zoomin paper is people understanding like how um this uh vision model was able to learn the task of um detecting cars and you're breaking it down into you know windows and car bodies and wheels and combining those together and saying if you have all those things then you're you're you know very likely a car. Um, and so you can kind of build up a sort of a sense of okay, um, it's not just this kind of random thing that's doing stuff. There's some structure here and it's kind of in built in a way that is at least plausibly understandable by humans. talking about and that's the exciting part about so here we go. Um yeah in all these cases we have some explanations of neural phenomena um that allow us to understand how some asteroid system works. So we have like a this explanation in that final case of like the the car is kind of coming about the the prediction of the car is coming about because um the these intermediate predictions about uh you know oh I notice there's a window there and so on and so we might have this kind of like sketch definition of interability that we can work with saying like what interability is is kind of this study of explaining neural networks. Um and yeah, this is kind of exciting and interesting as a as a field. Um one of the one of the cool things about it is that we have a really good sense of what neural networks are, right? So like we have really like a mathematical sense. We can like write down formally what a neural network is and like how it's behaving. You can you know read the code of neural networks. Um but the the more difficult thing about this is the word explaining in here. This is the other component of this definition. um because there aren't a lot of formal definitions of what it is to be an explanation uh in um you know in in machine learning or or in AI. Um and so kind of today we're going to talk about how we can really understand this concept explanation and use that leverage that to do good interpretability. Um and yeah there's a little some kind of so yeah kind of asking the question kind of like what is a good explanation? How do how do we how do we find them? How do we know that one explanation is a good one? And maybe you can motivate this by thinking about kind of sometimes we have conflicting explanations. So we have like the geocentric system and the heliocentric system and we want to say okay seems like one of the explanations much better right like we we believe the the sun is at the center of the solar system for example. How how do we decide between these two? And similarly there's been examples of this in um in inability. So they have uh this uh this on on the right hand side is um the the group operations. So the symmetries of a cube uh and there's papers who are trying to understand um how neural networks learn um these group actions these these symmetries. Uh and different people suggested some explanations. So uh on on the left here there's a paper suggested like one explanation. I won't go to exactly what the explanations are and and on the right they suggested another explanation and then these these conflicted and people were able to say like oh okay actually like we think there's some senses here. We think this actually this is the this is the real explanation here. And so you're having to basically make the decision between some different types of explanations and figure out what are the what are the explanations that actually explain the system. And that's kind of the the heart of mechability. Um so we're interested in this question of like what makes a good explanation and taking the learnings from previous inter research from some work that's been done in neuroscience and also some work that's been done in philosophy science where philosophy science have been interested in this question of like what are good explanations how do we have explanatory theories for a very long time and so we we can leverage that yeah um so yeah what intangibility is kind of is interesting so um maybe we can frame it in terms of the other scientists that it might be like to some intuition here. Um so natural sciences like you know biology, chemistry, earth science, physics so on. Um they explain physical phenomena by creating hypotheses and running experiments. That's typically how they work. And then we also have like formal sciences like mathematics, music theory, decision theory, so on um and linguistics maybe um and they use deductive methods to understand abstract systems typically construction proofs and that kind of thing. Um so a question we might have is like okay what kind of explanation are we looking for like which which of these sciences are we similar to? Um so kind is interpretability like a natural science or a formal science in some sense. Um yeah does anyone have any thoughts on this? Maybe feel free to like um maybe put something in chat or like um raise your hand if you have a thought. Yeah. Um one one hand. Yeah. Go for it. >> Yeah. Um I guess from my intuition I I would I would imagine that formal science is a bit harder with interpretability just because of the scale of things. So just as as a machine learning researcher not in interpretability a lot of our work is very empirical trial and error see what sticks. So I imagine for interpretability that would be my guess as well. >> Yes great that's a great guess. This someone else I think someone else had a hand up as well. Yeah, I have the same opinion. Uh >> the same opinion. Okay, cool. Yeah, I think it's it's interesting like because in some sense neural networks are very formal objects. They're mathematical. We have a really strong sense of them. Um we can kind of write down the the whole entire forward path um in terms of code and in terms of the the weights and stuff. So in some sense it looks like a formal science, right? But at the same time, as you said, the these networks are really large and really complex. And so it's like really hard to do formal science like things. It's really hard to, you know, make proofs in that sense that we might normally do. And so it's kind of this weird intermediate point. And so we think of interpretability as like um kind of a kind of a strange science in that sense. Both they're formal objects, but we're studying them with natural science methods, doing empirical studies, and so on. Um okay, so we we've chatted a bit about like why interpretability. Um and then we're going to talk about like how we have explanations in interpretability and some of the core objects. So like computations, representations, how they appear in interability and then we'll close by telling a little bit about explanatory optimism um which is a conjecture um which is lies at the heart of interability and this um how we might think inability is possible. Okay. Um so yeah explanations inability. Um first of let's just take look at science more generally. So the epistemic aim of science is to understand phenomena by way of explaining it. So we're looking for explanations on all time in science. And the reason we're looking for explanations is because we think that giving those explanations we can understand the phenomena we're actually interested in. Maybe it's something in the natural world. And so we're answering this question kind of like why do the phenomena occur by this explanation uh such that we can understand And so um yeah so oh here we go. Yeah. So um and typically like humans kind of understand some phenomena when they have like an an accurate explanation of the phenomena. So there's like this bridge between um the phenomena you want to understand and this explanation is doing this bridge. So when you grasp the explanation and the explanation predicts the phenomena well it's like an accurate explanation then you understand the the phenomena. Um and we can think about this in terms of compression. So like um typically you might have like a large set of observational data and you what you want is some explanation such that you could kind of like recreate this observational data very precisely um without needing just so much information in the in the data. And so you're kind of trying to exploit regularities that appear in the natural world or appear in the system you're trying to study um for compression such that that's what explanation is really um and uh yeah so there's some some work on this in particular by um some colleagues and um and in that case we we're kind of interested in representations um so uh I won't spend too much time on this because I haven't known for ages um but we're able to show that um kind of there's some intermediate um point of um okay I won't spend too much time on this because we we can we can move on but effectively able to show that this compression objective really maps well to interpretability um so we can think of this interpretability task is kind of like both trying to get explanations which are short in in this description length sense but also very accurate. Uh and so okay we're talking about adjustability there but like okay what what about the mechanistic part um what do we mean by that? Well a few people kind of tried to understand this in different ways. So um so yeah like there's been some amount of confusion um but like here's the the original definition um whereas uh one similar to what we presented earlier by um uh some people anthropic um and they have you know some possible claims that you might think of that come along with this and these are speculative claims. These are we don't necessarily um we don't have to necessarily take all of them for exactly as written but kind of the kinds of things that people interested in or the kinds of things that people might inspire people into into the field. But a more solid definition um comes from um this paper and um kind of we say that kind of the types of explanations that are interesting for meability people are ones that are at the level of the model. By which I mean um you might have you might want to explain some like large complex software system which like contains a bunch of deterministic code and also things like a model um but mechanism is really just like focusing on the model. It's like kind of not looking at the whole system or like some broader sense. It's just looking at the model. Um the explanations are like on in the sense that we're looking at kind of real entities in the model that we explaining in terms of they're falsifiable in the sense that we can um you should yield testable predictions. So we should have like some explanation be able to be like okay was my inspiration correct? And that should be an empirical an empirical question something we can measure. And then the kind of fourth and kind of most important thing uh that distinguishes mechanistic interability is that explanations are kind of causal in the sense that they identify kind of a step-by-step chain from the cause of the phenomena by which we mean like in terms of a neural network the input to the output. We want to understand like the whole process of how that happens. And so in some sense these um these explanations the kind of thing they look like they typically look like computations happening over representations. And I'll get into exactly what we mean by that um just now. So recall we said a minute ago that injectability is kind of the study of explaining neural networks. Um and so we can we can more precise about exactly what we mean here. So we have like some subdistribution and we want behavior and we want to explain we have an explanation which like explains the model over that subdistribution. Basically that's kind of what we're looking to do and and they take this form computations over representations. Um and so like yeah computations and circuits. Let's just kind of talk about this for a moment. Um so there's kind of two ways that people can think about the repation the explanation being accurate or faithful. So one is the sense of kind of behavioral faithfulness and that's kind of saying that like okay when I put the same input in uh to my model and my explanation I should get the same output like I get I I the model the explanation and the model kind of agree on the output behavior. And this other sense uh which we're more interested in in mechanistic interpretability um which is that like it's not just they agree on the the output behavior but they actually agree on the whole process the step-by-step process that produces the behavior and that's kind of what we're interested in and the way we can understand that is through these objects we call circuits which like computational objects they kind of not just saying like okay this is what you're going to expect at the end but they're saying like oh and here's the process in which you get to the final explanation the final um um and so we call some uh an explanation explanatory faithful if it matches that. So if in all the step-by-step process you're u matching the model. So you can you might think of like some earlier types of explanation like decision trees being like behaviorally faithful possibly if they're if they're successful on some some um you know some small small set of things. They're not typically explain faithful. They don't have the process of like how you got to the answer, which kind of thing you would want if you're going to to take your explanation kind of like out of distribution or understand um robustness properties or real safety and security properties. And we call these computation pass circuits. And so there's a few examples of circuits that we have here. Um and so we can talk about um a few of them. So I think some of them might be familiar to you guys. Um so induction heads are one example. So these are kind of attention heads which are often responsible for in context learning. So where in context learning is um being able to do better prediction later in the um late later in your your sequence. So later in the context window compared to earlier because you've learned some things um along the process of the context. Um so this kind of and you can kind of see this this process of how this is working. So you can see like it's happening in the first layer and the second layer and things are connecting together so that you can see how the um how the eventual output is kind of coming about from this step-by-step process. I won't go through exactly the details of this but um there's a the paper is also in and um about kind of gives a real nice description of this. Um there's some other examples of circuits um which I won't go through all of them just for time but um there's you can kind of see in all these examples of circuits that there's different things happening at each stage and then things are combining together and then we're eventually getting the final output and these things are both happening over multiple layers and in some cases over multiple tokens as well. So this is kind of like how these circuits are developing. Um and people have also thought about ways to kind of like um automate this process of finding circuits and discovering circuits. And so this is like one example and another one is here this recent work by right and Jack Lindsay and and his team there. So we talked a little about computations or or what we might call circuits. Uh and then the second major objective study in interability are representations or what we might call features. Um and where we might think of um these circuits as being some sort of like an algorithm that the network is um running we might think of the features the representations as kind of the variables within that algorithm. So when you in your TS class you maybe had some like data structures like these like the features you have some like algorithm which operating on that data structure uh and there's like the circuits. So that's kind of like maybe how you might want to think about this. And the kind of the important thing about the the features or representations here is that every feature should um kind of represent something which we mean it kind of has some some thing it refers to which is um from from the input training distribution or ultimately from from the from the external world. Um and being being the case that they can we can pick out those those features and understand um what are the things that are being picked up um in in the inputs and from the world which actually matter for the prediction is kind of one of the main goals of inability. Um so these these representations or or features sometimes called are these like fundamental units of neural networks. Um and you know we these patterns of neural neural activations which we can think of as representations um are really only representations when they have this this correspondence to to the input data and to the world. So there you can imagine like some activations are just kind of noise or some activations um you know might not have any any any real sense in which they're representing or referring to anything in the world but the ones that are these are these are what we think of as representations. They're kind of doing a real thing. Um and yeah um there's some nice ways that people thought about representations. Um so Harding in particular um who is also wrote a really nice paper about which she calls operationalizing representation. So kind of understanding when we have a pattern of activations and when we say oh this is this is referring to something this is actually representing um and she she kind of notes that there are kind of really three conditions here for this to be true. And so one is that um the representations um are the causal results of things in the input that there's like a correlation in some sense between uh the input and the and the representation such that like if you change the input then you would have got got a different representation. Um this representation wouldn't fire if the if that wasn't present in the input. Uh and then the second two conditions are kind of about um being the the representation itself being the causal the cause of the output behavior. Um so saying that kind of if the feature is present or the recognition is present and we try and remove the content um then we should expect that the output behavior changes appropriately. And similarly if the feature isn't present and we try and add it in uh then we should we should appropriately see the output behavior change as well. So um kind of the interesting thing is both that the features are operating both kind of as a causal intermediaries. So they're both results of the input they come downstream of the input and they're also kind of um causes output upstream of the output. And yeah, so we have a few examples of like features that have been discovered in um the networks up to now. And so um here's one feature that you um may have seen a little bit. It was kind of popular on on um social media at some point kind of thinking about how Claude in this case as the paper was anthropic um had a feature that was responding to the Golden Gate Bridge. And you can see in this case um that different types of the words the Golden Gate Bridge would activate this feature but also picture the Golden Gate Bridge in different settings in different you know um different orientations. the Golden Gate Bridge in this case, some of them were in uppercase, some of them were in lower case. Um, it's also the same in other languages. And so this feature is kind of robustly representing the concept of the Golden Gate Bridge, no matter what the context, no matter the modality. And so that's kind of the thing that we would want in in a network and this kind of that we're able to look at and be like, ah, I see that's that's the feature that's responsible for the model like I'm understanding the bridge. And so that's we've now shown here kind of that this like robustly picking up on the inputs from the first side of things. We also want said that it's important that this causes the output too, right? And so um for that we want to do some intervention to show that if this feature was active and um activated sort of artificially um intervened on to be active that they would have the corresponding um result downstream. And so you can see here on the on the left hand side um like the the the model was asked if it has a physical form and it it replied kind of as maybe it would if you were um asked this and then you know your your shadow interface like no I have physical form that kind of thing and then they intervened on this feature this representation um and we're able to show that um the model like was now started to like um really that feature was causally relevant to the output right so um you might think you might say that the the model was like you know activating that presentation or maybe was like thinking about this concept or something like this um such that um the model now replied that oh it is the golden gate bridge right so like it's changed its answer um and it's changed the answer in a specific way um towards thinking about the golden gate bridge towards talking about the gold bridge to change the answer and there's many other examples of this so this is an example that's similarly multimodal and like quite um complex and abstract about being up on unsafe code um and reach to the kind of recognized this concept. Um, and then that when uh we intervened on this representation that the model in this case uh I'm not sure if everybody writes C code but if you if you do you'll notice that um there's like a buffer overflow error in this um the second piece of code. So the model kind of changed its ability to write code um to write code in an unsafe way. Um and so this is like kind of very safety or security relevant, right? Um cool. So uh we're going to close with one section and then we'll open to questions. Um and we're going to end with kind of a theory which underlies a lot of mechanistic interpretability and I think interpretability more broadly and this um idea is called explanatory optimism. So what does that mean? So first I'll give some background and then um I'll introduce what explanatory optimism is. So imagine that you're trying to communicate with a machine uh some AI system and so humans have some concepts they understand in particular you you understand some concepts so um this you've understood the things that I've been saying to you right now um it's because we share some concepts um and you're able to leverage those in order to understand what I'm saying um yeah I similarly have some concepts to understand um and some there's some overlap here right like it is the case that when we I'm not just only talking about the output so like you know when a model writes English or something or some other language but the internal representations they're they are made of concepts in some sense these representations um and some of these representations are ones that humans kind of share and understand um noted by kind of the examples that we showed earlier um where we looked at those representations and we're like oh the goal bridge that's the thing I understand too I have the concept the model has that concept but and so you can think about this as like this intersection of these um these two v this vend diagram um so where the human concepts and the AI concepts kind of intersect so both both have that concept and then there's also plausibly some concepts that the AI systems have that we don't have and and vice versa and maybe this become more the case you might think as AI systems become more powerful maybe there's more concepts that they understand that we don't natively understand that we we haven't seen Um, and that kind of poses a bit of a problem because we want to communicate with them. In particular, in this case, we want to understand them. And so, you know, we want to get concepts that they're thinking about into our heads so we can understand them and and make ensure the safety and security. Um, but also the inverse problem is also important, right? So, we might want to um put sort of uh values into the system in that sense. We want to show that like okay the values that we have in our heads we can kind of get into um a way that the machine will understand it. Um so it's kind of these these dual problems um which it'd be really nice in some sense if most of the concepts that we really cared about were in this intersection of the human concepts and the AI concepts. Um now there's a few ways we could try and do this. So, one way you could try and do this is be like, well, um, all the concepts that AI has and that I don't understand, I don't want to I want to remove those. I want to get rid of them. I want to kind of collapse all of the AI's concepts into concepts I understand. And we might think we I'm calling this coercion here. So, we're going to coersse the model to only use concepts the human understands. Uh, another way we might go about solving this problem is we might call transcreation. Um so uh what we could do is say like okay well there's some concept that this model this AI understands and I don't understand maybe maybe I can understand some close approximation of that concept and like maybe that's enough and maybe maybe maybe that'll be enough for me to like like kind of I can get by with that it's kind of lossy but it's okay. Um a third way that I could kind of approach this problem is what we're calling here concept enrichment. So we might say like okay there are some concepts that this AI understands and I don't understand but possibly I can learn this concept possibly I can increase the set of concepts I understand such that I cover more of the concepts that the eye understands and kind of like empowering the the human interpreters to better understand AI systems. Um and this seems pretty exciting actually. I think um the second and third approaches I presented seem both pretty exciting. Um so I I think in some sense a lot of the goal of interpretability is to do the second and third. Um I think primarily the second so primarily this transcation but in some sense some some consideration as well. So and so yeah here's the kind of like the things I brought out. So like we have this problem where not all the concepts that AIS understand are understandable or you know humans understand natively or already understand in some sense. Um, and so there's kind of three solutions we talked about. So one is maybe I could just like force the AI to only use concepts I understand. One is like well I can maybe try try and like get a close approximation of the AI concepts and hopefully that's enough. And the third is like maybe I can just like truly understand the AI concepts by like trying to improve the set increase the set of concepts I know. Um and so yeah these are all kind of operations different sense I'm trying to and but there's kind of a problem here so and the problem kind of is illustrated by this uh this diagram we have in front of us here and the problem is what if there's some set of concepts that when I do my concept enrichment so when I try and increase the set of concepts I understand um or when I try and do my transcreation when I try and like um approach estimate the concepts that the concept is just so alien so weird to me that I try as much as I can and I don't have a good sense of like what is meant by this concept um what what ought I I to do then and I think that's a kind of a really key question that we might want to deal with because if there are a lot of concepts like this there's a lot of concepts where um I as a human or maybe the humanities in general um like find it really difficult to understand the concept that AI is using um that's makes it hard to verify um a system or it's hard to say that um the system is like safe and um or in some appropriate way and so for a safety and a security um there's kind of a real kind of issue here if if there's a lot of concepts that the AIS are using that are alien to humans in this way um and in particular the problem this is a problem for interpretability. This is a problem for for people who are trying to understand systems in this way, right? Um and so one thing we might know is that um so yeah so it certainly if this set of concepts was very large and responsible for a lot of the important concepts of the um that the network is using for its prediction task then interpretability is probably like going to be really hard or maybe it's just not not a viable way to to get safety of these systems. But if this set of concepts is very small or or maybe it's only like insignificant or unimportant concepts that are alien in this appropriate way in this green set then inter um and this kind of question or maybe like this conjecture that um we might think inter researchers have implicitly or have an assumption to their work that at least for the the systems that are coming up in the near future most of the important concepts are understandable by humans. Um, and this is a conjecture that nobody has um like sort of proved is true or um but I think it's something that as a field we're interested in understanding to what extent that's true to what extent is true that most of the important processing happening in network is understandable by humans. Um and in particular bits is um this this idea that humans kind of kind of understand most of the important processing is called the principle of explanary optimism. Um and this uh yeah it's kind of the the main underlying assumption of interpretability of all types in okay cool. So that's uh that's where I was going to end and like leave time for some questions. Um so yeah, I wanted to um note and thank my collaborators on some of this work um who listed here and also um to all the people's work who um I used examples and there was links in the in the presentation and um and like at the bottom so feel free to um look at those. There's bunch of really great work in it happening now. Um, and if if you missed any of the those, feel free to like email me if there's like things I talked about that you're wanting to know more about that you didn't quite catch the um the reference for, then feel free to ask me that. Um, and yeah, I wanted to some references here. Uh, and I wanted to open up to some questions now if anybody has any. >> Awesome. Thank you so much for the the talk, Co. I think this is one of the the best uh walkthroughs of mechan we've gotten. We've got quite a few in our community, but I think this opens the floor for a lot of cool questions. So, I think Neil has his hand up. Neil, go for it. >> Yeah, thank you. Um, super cool talk. Uh, thank you for your thoughts. I think we actually talked briefly at ICML this summer, but I never got to hear your take. So, it's super nice to have the have the chance to hear your thoughts now. Um I was wondering uh so you talked uh a little bit about circuits and I feel like um there's potentially like a bit of a weird moment now where um it it feels like lots of the circuit work would look at like specific behaviors or specific computation um relating to like um specific isolated I don't know um like you mentioned induction heads and and things like this um but now it feels like the community has shifted towards these like reasoning models Obviously the word reasoning is doing some heavy lifting but at least qualitatively like they produce like thousands and thousands of tokens now. Um so that's also thousands and thousands of forward passes where you could take any point in that in that long chain of thought or scratch pad to kind of like zoom in and try and find a circuit. It feels like the problem has gotten like much much harder in like a in in this past like ever since reasoning models got so hyped. And I was wondering what your take is on that and and how we as a field or how mechanistic interpretability should change it its methods or like is circuit work still feasible in in that context and yeah I was just curious about that. >> Yeah that's a great question. Yeah. So um yeah so there's a bunch of elements to that question. Maybe I'll take a stab at one of the elements and um feel free to ask a follow up with one of the other parts was more interesting. But um so as models have gotten into reasoning models by which we mean the kind of using more forward passes before giving an output it might be the case that we think well since is kind of looking at um like the model and like kind of the forward pass in that sense maybe it's got a lot harder. I think there's like kind of two sides to this. So one side is that if models are doing more reasoning um in like multiple tokens um typically that probably means that they're doing having to do less work in one forward pass. Right? So for a model of like similar intelligence or similar like level of output you can either get there by doing lots of in internal reasoning or lots of external reasoning like a long chain of thought. And so in some sense um though we're doing having to do more e external reasoning there supposedly that means for the same map intelligence we're doing less internal reasoning. So in that sense the task of mechanistically has actually gotten slightly easier. So the models have like smaller and doing less work in each forward pass. Um you kind of see this in some of the um like like the like haiku or like the small flash models are getting like really powerful in a way that um that only the really large models could be powerful in the past. Um and so I think there is a kind of an opportunity in some sense in that we like have these models which are um yeah we're kind of like needing to explain smaller parts. So each token is doing like less work compared to the other case it might have been where like you just have these like really large models doing like all the way in forward one forward pass. >> Yeah, that's a that's a an optimistic way of thinking about it that I hadn't considered actually. So thank you for for that. Appreciate it. >> No worries. Yeah, Rira, go for it. >> Yeah. Uh, hi Cola. Thanks for the presentation. It was really nice. I had a question about um when you were talking about formalizing what mechin gives us in mathematical terms. So I was wondering what do you think of um the completeness problem of mechanistic interpretability like I like to call it you know because once we have explanations it's not guaranteed that it's a unique explanation in some sense right for example if you come if there is a circuit there and it's not guaranteed that that's the only circuit right I think there was a recent iClar paper on this about the identifiability of circuits like >> even if you have a circuit and you know you kind of do something with it you zero out all neurons or elements there there's the model kind of adapts to new components which do the task that the old circuit was doing. So this kind of other roots that models find and so whenever we come up with the circuit based explanation it's not it might not be the only component that's responsible for a certain behavior right so how do you think MI can get around that or what kind of guarantees do you think that MI can prove and do you think there are limits to what it can do in th those terms? >> Yeah, this is a great question. Yeah, so there's been a line of research about this. So um yeah, typically people talk about there's like a few like key words if you're interested in like learning more about this. So there's like um one word is like backup heads or like back backup ideas and there's there's a word of like hydra. So kind of the idea being that when you like ablate some parts of the network, there might just be parts like there might be two paths that the network is using to do some tasks. Um and like you might ablate one of them, but there might be some like other thing that kind of takes uh takes on that role in the absence of that other circuit being there. Um and so yeah, this is like a a line of research and I think the best thing we can really do here is and just be really thorough about our our causal interventions. Um and so the the fact that we can run these um kind of um as much as we want in some sense and we we can do more controlled experiments than kind of any other discipline. You know, if you look at control experiments you do in like psychology or neuroscience, these are like really hard to do at scale. It takes, you know, um like quite expensive. It takes a lot of um complex machinery to do these. Um and in searchability, we can do them like loads at once basically and we should just be like really thought about doing them. Second thing is um uh I also have another paper um called um evaluating explanations and it sets out kind of um this like explanatory virtues framework. So like how you should think about um trading off explanations between between each other. So if you have two explanations and you're asking the question like which one of these explanatory theories is the better one. Um that's kind of question that that paper kind of deals with and and what what we mostly say is that um you should be really be interested in like testing out distribution. So like this is something that's like very familiar to ML people, right? So you kind of having a really solid test set with your explanation. um but also um really focusing in on our um like the complexity of your explanation. So we know um that simple explanations generalize better. Um and so to the extent that you can maintain this accuracy and this faithfulness um while um being having a simple explanation that gives you more credence that this explanation is is the correct one. The third thing I'll say on this is that um this idea of explanatory faithfulness really helps us here. So it's not just that we're looking for getting the same behavior um with our explanation. We're looking that the step-by-step process is the same in our explanation as in the model. And when we have that criteria, that's really quite a strong criteria actually. And so that really helps us out because if your if your um explanation is the same on every step, then it's it's really as um Yeah, really increases your chances that that your explanation is is the correct one. >> Cool. I I work in AI eval so I would love to read that paper you mentioned on evaluating explanations. Sounds really interesting. Yeah. And yeah, I love that idea of explanatory optimism you had. Um I was recently part of a paper called mechanistic interpret needs philosophy. So I think >> Oh yes, I love this paper. Okay, great. I and yes I I speak to your authors about this. Yeah. >> Yeah. Yeah. I was one of the authors there. So it's really interesting to see how you talk about this. I think this is also something we bring up about what kind of explanations we're talking about comes to MI. It was very cool to hear your opinion and views. Thank you. >> Yeah. No. Yeah. They're great good with that paper and um I think um I think some of your co-authors maybe are um AEIS next week. Um, so I think there's been going to be a lot of discussion about that kind of thing and like how we connect um our understanding of explanations um to interceptability into AI systems uh there. >> Yeah. I'm also at AI next week and then London after. So >> Oh, amazing. Feel free to please reach out to me when you're at AI would love to. >> Yeah. Yeah, sure. >> Uh yeah, this is great. I I had a quick question also uh about I really like the li the slide on like alien explanations or alien concepts. Um but I wanted to ask you like in terms of like explanatory optimism as you mentioned seems like this is like the core philosophy behind me and at ICML this year it seems like >> more people than usual have become more skeptical of MI. Uh so in your perspective if this like alien concept kind of circle is very small so ideal scenario do you think MI is like necessary and sufficient to ensure safe AI systems in the future or is this something that has to be used alongside other techniques to have those kind of safeguards in place? >> Yeah, this is a great question. Yeah. So, um, yeah, I think there's a few ways to answer this. And so, um, maybe the primary thing to say is, um, even if we have high confidence, the metric contility would be very useful for AI safety and security. Um, we probably don't have 100% confidence. I think it'd be very I don't think I know anybody who has like 100% confidence that um this will like definitely work and definitely work you know before um very powerful systems are impacting our society. And so it probably makes sense for people to pursue very a lot of different approaches in parallel. Um and so yeah, we might think of our defenses um or the way that we are going to um kind of ensure safety as having like multiple uncorrelated bets or sometimes people talk about this as like a Swiss cheese model where there's like some one defense has like some holes in it, but hopefully the the next defense that you have um doesn't have holes in the same place and then you you can um get safety that way. Um so so yeah, that's more how I think about this. like I'm excited about a lot of different people trying different approaches to ASD. I think this is a particularly promising one. Um but I I think that other people should work on different things as well. >> Awesome. I think we have time for one more question if anyone has any burning thoughts. >> Uh yeah, I I had a quick question. Thank you, Cola. Great presentation. Love the Swiss cheese metaphor. I thought that was pretty funny. Um so I I recently read a paper out of Berkeley's uh AI uh computer vision lab bear. Um so it it it was pretty it was pretty interesting because I had thought of circuits research as being kind of um you know a fun academic exercise but what they did is they used basically supervision um super positions in a language mo