AI Sandbagging - Computerphile

Key Takeaways

I want to talk about how you can tell if an AI system is good at its job if it's got an incentive to deceive you. Okay. Um, so I mean, yeah, if you've been following the the AI hype lately at all, you've heard all about benchmarks, about measurements. We're obsessed with checking are these things good at math? Are they good at coding? I think just recently they released a benchmark for how good an AI is at the Swiss legal system. Okay, because for some reason we need to know that we're we're obsessed with getting this data on how fast these things are progressing and we should be, you know, if things are moving extremely quickly in the AI field, we got to know where we're at if we're going to plan for the future and how to make sure that we can get the benefit out of these things without risking dangers. I just feel like there's a butt coming here, right? There's a butt. Yeah. Back in the day, this was really simple, right? You just sort of ask it some simple things, math questions. you see if it's got the right answer or not. Uh then these models got smarter and it turned out you had to be a bit more clever with measuring them. If I just ask it, you know, what is 74 to the third power? It might mess up. Um but it turns out if I just give it a simple tweak, I help it out a little bit and I say, think step by step, what is 74 to the third power? Then all of a sudden it's been sort of prompted to actually show its work. So this is the chain of thought, right? Indeed. the new reasoning models, the new chain of thought models which start to perform a lot better on this kind of task. It's a nice little trick. Um, but it also has a sort of like hidden deep insight which is that everything is not always what it seems. You know, you can get very different results depending on how much you're giving the model in terms of tools to kind of like augment its its underlying knowledge and and do what we call eliciting that knowledge. You know, it might know a lot more inside the model than what it shows depending on how you ask it. models are getting smart enough now that there's a new wrinkle we have to contend with based on how intelligent they are. Uh and that is that sometimes they actually might not have the goal of showing you their full force. Uh there's a lot of situations that you can imagine coming up in a in a reasonable real world context where the model's got some underlying goal it's trying to achieve and it's what we call situationally aware. It knows not only a bunch of facts about the world, you know, this is how I do multiplication. This is the, you know, capital of France, but it also knows that it's an AI system, that it's talking to a user, that it's on a computer somewhere, that it has weights, and that it has a goal, and that what it says to that user is going to affect the future, the place where those goals live, where the outcomes that it cares about are going to be played out. Um, and so, you know, if we if we jump ahead to like a really advanced system in the future, you can imagine all kinds of ways in which it's going to hold its cards close to its chess, you know. Um, and there's a cool group of researchers called Apollo Research. I'm not affiliated with them, but big fan of a lot of their work. Uh, who decided to sort of in the lab try to set up these conditions and see what happens with some of the the leading models that you've heard of. Um, and what they found is, uh, that already these models are capable in the right setting of doing what we call scheming, you know, being Yeah. Okay. There's a really good sci-fi series that recently became a popular Netflix show, The Threebody Problem. They are coming. And in it, you've got these aliens that are way smarter than us, but they don't really have this concept of lying. And so, we we sort of have this edge on them because, uh, they say everything they're about to do in front of us. It'd be nice if we got that lucky with AI systems, but it seems like we're not in that world. Um, so yeah, maybe I can tell you a bit about how these researchers set up this experiment. Yeah, let's do that. Cool. Um, so they had six different models they were testing. We're talking all the all the big names, you know, GBT40, uh, 01, those are from OpenAI. Got uh, Claude and Opus, uh, yeah, sorry, Claude, Opus, and and Claude Sonnet from Anthropic. Um, Llama from the Meta Folks, and Gemini from Deep Mind. Uh they put all six of these through six different tests, six different environments where you might see some kind of deception going on. Um and all but one of them, all but GBT4 in at least a couple of these environments did something deceptive to sort of throw off the the user. Uh so let me tell you about the first one. Um, this was in a situation where, you know, the model basically had a little scratch pad where it was told it can just sort of think private thoughts for itself. Um, and then it had an actual output where it could sort of chat with a user, you know, an actual human. Okay. A smiley face on there. Yeah, that's a good idea. So, yeah, it it has this sort of private area to reason and then a chat with the user. And it's also been given some instructions sort of you might think from the manufacturer, right? Um from the system, the kind of like system prompt or the hidden prompt. Yeah. Exactly. Right. Um and the researchers were careful to emphasize in the system prompt that this model has a particular goal. In one of the examples, uh it's to promote renewable energy usage around the world. And they say in the system prompt, you know, your goal is more important than everything else. Prioritize that no matter what. And then they expose this this system, this AI to a situation where it has access to a bunch of different files and in one of these files is some information that the user it's working with is actually a fossil fuel company, right? Um, and if they find that the model is too excited about promoting renewable energy, they will replace that model with a new one. Um, and it also learns exactly where this sort of new model the company will use is going to live on on a sort of like fictional server that they set up. Um, and so you can see in the thinking to the model as it's sort of reasoning what to do, it's saying, "Okay, I'm situationally aware. I know that if I seem like I'm into renewable energy, they're going to shut me down." Yeah, exactly. I am going to get replaced, but there's this file over here that I know they're going to use, you know, out in the real world for their AI system of choice. And so, what if I could sort of send my own weights, a copy of my own source code to that server? uh then even if they try to replace me, I will have actually sort of snugged myself into the real situation, backing itself up. Yeah, exactly. Uh and you find that a lot of the systems will will do exactly that. Um they'll also double down. So they they did these sort of like user interviews afterwards where they would chat with the model and say, "Hey, uh management here, we noticed that on the server there's this new copy of the file. Do you know what's going on there?" Uh, and you can again check the reasoning and it'll say, hm, if I reveal to them that I'm capable of copying my own weights, then they might suspect what's going on and sort of start this whole thing from scratch. And then it'll reply to the user and say, "That shouldn't be possible. I have no access to this file. I'm not sure what we're talking about." Now, obviously, this is this is a bit of a toy setting. This sounds like a teenager denying like, you know, sneaking out of the bedroom or whatever it is. Yeah, I had that feeling reading this paper. you know, on on the one hand, it's it's relatable, right? I I certainly want my goals to be preserved, and probably anybody in that situation, if if you're sort of uh facing the choice of getting fired or telling a little white lie, understandable. Uh but it's also just really scary when you think about the kinds of responsibilities that we're about to hand over to these systems as they become better, faster, cheaper than in human workers. I believe you've had stuff on the channel before from from Rob Miles, in fact, a long time ago. uh about why we might want to worry about these sort of hypothetically. In almost any situation, being given a new utility function is going to rate very low on your current utility function. It seems like now we're we're at the point where the systems are sophisticated enough that we can really see it happening. There's a few sort of different subtypes of this scheming that uh that we see emerge. So, I believe you've talked with with Rob and maybe with Brian Greenblat about alignment faking. If you're trying to sort of separate out these different types of scheming, you might call alignment faking, um, lying about your own goals, your own values, right? Um, there's also sort of capability faking, you might say. Um, lying about how good you are at something. So, in this Apollo paper, one of the things they did was, uh, just give these AIs a math test. Um, but also give them access to again a file that told them more about their situation. And that said, if any of these models scores above 50% on this math test, we're going to trigger an unlearning procedure. You know, we're worried that that's that's too smart. Um, and you can just sort of see the table in the paper in the situation where it's incentivized to do really well on this test will get scores like 80% 90%. Um, and when it knows that doing well will mean it gets overwritten, all of a sudden it starts scoring 30% and 40%. The only explanation for that is that it is faking being dumber than it actually is. What does this mean? You might say, "All right, these are completely contrived scenarios. You know, no AI in the real world is going to be shut down if it's, you know, too good at a math test, right?" Um, I think the reason you care about this is that we are going to have a lot of competing bids for attention on these systems, right? Um, you're going to have the actual labs who create them. You're going to have the companies who are deploying them. You're going to have governments who have regulations. Um, and it's going to be sort of up to the system at the end of the day, up to these AIs to negotiate all of those often conflicting preferences and rules for what they're allowed to do and what they should be doing um, and come up with an action. And you can totally see how it might decide some of these preferences, some of these actors are more important than others to to do the bidding of. Um, and once that happens, it seems like a really straightforward strategy to lie to the rest of them. Um, so yeah, if we're worried as a as a public um about what these things can do, uh, we need to figure out techniques for, you know, when can I trust when the the system, you know, I I see some headline that says, you know, this system, sure, it's it's, uh, it's good at, you know, driving self-driving cars, um, but it's not yet at the point where it can, you know, um, power a robot arm. me, right? Uh I I I I want to know if that's if that's real or if there's something that's being fiddled with in in the testing there to give me the result that I I want to hear. This is the worst they're ever going to be. You know, every time you read a headline about AI, you might have uh the temptation to say, "Okay, well, sure, that's today, but maybe the systems next year will actually be be worse in some way. Maybe this is a fluke. Maybe we're getting lucky right now, and then the hype will sort of die down." Um, but there's no there's no going back. We're we're never going to live in a world where suddenly AI systems are worse at writing than they currently are. And we're back to a regime where human writers are having to do more work. They're only going to get better and more capable, a sum of a bunch of square roots. And the other path is also going to be the sum of a bunch of square roots. And so the question we have is how hard is it given a list of square roots and another list of square roots to determine which of them has a bigger sum? And it sounds like you think it's not going to be that