LLM SATs FTW

Key Takeaways

Thanks so much. Excited to be here as always. So, uh, sometimes you just get a title and the title's so good, you just have to build a talk around it. So, uh, large language models, structured analytic techniques for the win. Um, and I almost named this talk something completely different, which is double-edged swords. And this is simply the idea of um, well, we'll get into that in just a second. So, first, uh, who I am. Um, I need to update my bio with SANS, uh, because currently I am an instructor of cyber security at Utah State University. Uh, I also have a a startup going on as well. Uh, as Rob called out, I was one of the authors along with, uh, the intimidable Rebecca Brown of intelligence-driven incident response. Uh, and I'm also a former SANS 578 cyber threat intelligence instructor. I say former because I'm fairly certain being a SANS instructor is a little bit like being a Marine. You're never ex, you're just former. But I was asked by Rob to talk about threats and emerging threats and what I saw and and I went to kind of a weird place, but I think it ended up actually being um better than I might have expected. And so the threat that I kind of got to was cognitive bias. And if you've done any cyber threat intelligence, if you've done any type of intelligence analysis in the past, you're going to go that's not emerging. That's a a really well-known, really common thing. We've been talking about cognitive bias going back to the 60s. And I would agree with you. You're 100% correct. But what I think is really important right now is the fact that we are able to do cognitive bias faster and easier and more quickly than we ever have in the past to the point that it's becoming a bigger problem unto itself. And so if you are a parent or if you're just really anybody who's paying attention to what's going on with AI right now, you're you're seeing people giving into using AI for almost everything. You know, you'll there are people who are using it for therapy. There are students who are just frankly writing entire papers based on nothing but AI. And frankly, that's making it easier to fall prey to cognitive bias. you know, these AI tools are based on just huge amounts of data. And if there's bias baked into that, which we all know there is, uh there that bias is going to come out in the results. And so we're able to frankly scale our ability to be biased and and I see that as a threat that we as security analysts need to find a way to solve. Um for for those of you, you know, to to kind of get into, you know, what do we really mean by this? uh from Daniel Kinnman's uh thinking fast and slow. You know, he had the definition that systemic uh deviation from a truth based on system one thinking. Um totally recommend reading that book. Fantastic way to think about kind of metacognition. But let's get into how we can solve for cognitive bias. So the the long-standing method of getting at cognitive bias and finding ways to counter it is the idea of structured analytic techniques. This is not new. Uh this came out of the CIA by by a gentleman by the name of Richard Huer. And I had actually never read the definition from their book. So I I wanted to make sure I got it in here. Structured analysis is a mechanism by which internal thought processes are externalized in a systemic and transparent manner so they can be shared, built on, and easily critiqued by others. Now, if you're following along with the development of AI, you would know that this runs directly counter to the blackbox problem. And the black box problem is the idea that even the people who are building many of these foundation models don't truly understand how they're getting to the decisions they're getting to or why they're generating the text they're generating. There's some really cool work being done by the anthropic team on trying to kind of essentially brain map their AIs, but the fact is we don't really know and and this is also a problem in human minds as well. So structured analytic techniques provide a framework by which we get that information out of our heads. Get away from just the uh anecdotal or guessing or whatever and and map out our thought process. Unfortunately, no one does structured analytic techniques. They're they're really rare. And even when we do them in uh teaching in 578, I would always get students asking, "Well, how often do you do these?" And I'd go, "Not that often." And there's there's a couple reasons why. Structured analytic techniques are hard to learn. Uh there's an entire book full of dozens of different techniques that can be applied in different circumstances. They're hard to use. They take practice. They take effort. Um they're hard to teach. You you have to build people up to do these kind of techniques. The other problem I'm starting to see for a lot of organizations is they work in teams. So, if you're an individual analyst by yourself, doing a structured analytic technique can be really hard because you might literally not have anybody else to go to to say, "Here's what I'm thinking. Can you check it out?" You're you're externalizing this thought process only to have no one to externalize it to. And then lastly, these just take time. Uh an analysis of competing hypothesis, which we're going to talk about a little bit more, can take multiple days depending on how you do it. So, how do we solve this? Well, I said this was called double-edged swords. Uh, so what if we take analysts and we take these structured analytic techniques and try to use the same large language models that to some extent we're trying to defeat and put them together to try to, you know, get over those problems of structured analytic techniques. Well, I could give you theory and I could give you what I think might happen, but in my opinion, the very best thing and what I'd really like to see more people do in general is just experiment. Uh, in my experience, the people who are most optimistic about AGI is going to solve everything and it's going to be the best thing ever. Don't spend a lot of time with these tools. But by the same token, the people who are going AI is garbage. It's nothing. It's not going to help us. It can't do anything very useful for us. they don't tend to spend a lot of time with it either. So from my approach, I'd say let's just go see what happens. And if I have this theory, let's test it out. So I picked three structured analytic techniques that we're going to see not can a can a large language model replace a human, but can a large language model assist a human in doing this technique. So the first one is a idea generation technique called starbursting. And this is a brainstorming technique where basically you take a challenge, you take a problem and try to come up with the, you know, kind of common questions we think of from journalists, the who, what, when, where, why, and how as quickly as possible and and generally with multiple approaches to each one. And from an investigative perspective, those are the kind of things an analyst is trying to do. If you're doing incident response, the first thing you're thinking about is, okay, how did they get in? What was the initial intrusive? Let's see if an AI can do that for us, though. So, how did I do this? Well, I'm using a tool called Streamlit. Streamlit makes it really easy to build fast kind of um prototypes of software. I'm using a tool called Langchain that makes it easy to kind of structure how I get responses from uh in this case I'm using OpenAI 4.0 or yeah 40. Um for this one, it's a zeroot technique. I'm sending in one request. I'm getting the answer back. I'm trying to generate it as quickly as possible and then generate a JSON file so that I could either ingest it somewhere else or just review it my by on my own because again I'm not trying to get the AI to replace the analyst. I'm trying to get the AI to help the analyst. So in this case I said what about a ransomware attack on a hospital? So here's my interface. Makes it makes it nice and easy to kind of just generate something quick. I put in the scenario a ransomware attack on a hospital and I get back five different answers. Now I asked it for six which is interesting but let's look at the ones that we got back. And oh by the way I wanted to go a little bit further and I asked it I said well let's also generate a diagram so I can look at this you know use my my uh visual cognition as well. And so what did it give me back? Well, okay. The who who carried out the attack? Who was affected? Who responded? Yeah, those are good questions an incident response team would want to be able to answer. What about the diagram? Well, here's that same thing just kind of visualized in a way that I could either present to somebody else or just helps me sit there and understand, okay, did I did I get it everything I need? Well, I'm going to go yes and no on this one. I gave it kind of a simple problem and it kind of spit out exactly what I'd expect a junior analyst to be able to share. If I'm on my own, that's not a bad place to be. Let's try something a little bit more complicated though. I mentioned analysis of competing hypothesis and this is one of the classic uh structured analytic techniques that gets put out there by teams. And you know, this is a really complicated one where you're trying to evaluate an analytical decision by separating evidence from the hypothesis of what happened. So in this case, I ended up having to build a much more complicated tool kind of built on the same framework. So in this case, it's doing a multi-stage process where first I give it a complex question. The first call it it basically is generating a list of hypotheses. Then it goes through each of those hypotheses and says, "What's evidence for this?" And then a third time it's sending the combination of a hypothesis and evidence and going does that what what how can you score that? In this case, I went with a a minus5. This this evidence completely disproves this hypothesis to a plus five. This evidence basically assures you that this hypothesis is correct. And again, it's not supposed to replace me. So I hadn't output a CSV file. So a human could actually take a look at it and maybe even make their own changes and updates to it. So the test case I gave it was something really complex that I give my senior uh threat intelligence class who was behind the XZ backdoor. So what did we get? Well, it generated five hypotheses for us. The XZ back door could be attributed to state sponsored cyber espionage due to its complexity and sophistication. It could be cyber crime. It could be an independent group of hackers. Maybe not the granularity I would have liked it to get to, but those are all valid approaches that you know different different hypotheses that could be there. It started generating the evidence for each hypothesis. Now remember, it's an it's an AI. It's a large language model. It's not necessarily factual. It's probabilistic. But most of these pieces of evidence do kind of align with with what I'm looking at. So then I asked it to go through and score it and you can see at the top it's giving you know minus3 + three minus3. I had it generate me the matrix and then at the end I have it kind of do some tallying for me. So in this case the highest scoring possibility it might be the work of a criminal organization that specializes in cyber crime with a score of 34. The least likely is it could be a activist group. The piece of evidence that was most helpful was this XZ back door uses advanced evasion techniques. Now again, is that perfect? Probably not. But hey, I had this output that I could go through and tweak as a as a human analyst. So lastly, I did I wanted to do one more on a different type of thing. So this is this key assumptions check. And so this is exactly the kind of thing that you might look at another analyst and ask them to do for you. Here's something I wrote. What are the assumptions I'm making? and can I back up those assumptions? So, in this case, yet another streamlit app, I used a report from a Utah based company trying to stay a little loyal. So, uh, Strider Technologies did a look at North Korean IT workers and their PRC backers. And I did another zero shot where I read in the PDF and said, "Tell me what the assumptions I'm giving are." Well, it read it in. It ended up having to break it into multiple pieces and then it spit out 27 assumptions that it thinks I'm making. So take a look at a couple of these document assumes North Korean IT workers are dispatched abroad to countries like the PRC, Russia, Southeast Asia. Okay, that that that is a good assumption that's in there. And it's not to say any of these assumptions are wrong. It's just to say these are something you need to be paying attention to. So what are my results? I've done three different experiments. I've gotten three sets of outputs. What is it? And my answer is like almost everything in intelligence, it depends. Is this better than having another human analyst? No, probably not. But if I'm an analyst by myself, is having tools like this and using a large language to model this way something that is going to help me identify and counter my own biases? Yeah, I I think it would help. You've got to tweak some things and you got to learn how to work with the tools, but it's there. So, in conclusion, I'd say you need to try these things. Um, Jeban's paradox I don't have time to go into talks about the idea that we're going to use more AI whether you like it or not. The more efficient a system gets, the more of it we use rather than less. So finding ways to use these tools, make them part of your workflow, and counter their biases as they're countering ours is incredibly important. Um, I'd like to see us move from artificial intelligence to intelligence augmentation. Let humans do things that humans are good at. let computers do things that computers are good at. Um, I think you need to focus on experimentation rather than theory. Um, but also set your goals the right way. AI doesn't have to be better than than a human. It just has to be better than the best available human. I I took this from a talk done on education and AI from Ezra Klein, but I think it's a really really key point. Uh, lastly, something to know. I didn't write any of that code that I shared. I generated it all this way like using using vibe coding. Um these tools are coming and they are only getting better and we're going to have to learn how to embrace them and make the most use we can out of them. So with that uh there's where you can find me and thanks so much for having me Rob.