Fighting Back with John Hubbard

Key Takeaways

Welcome to Cyber Leaders with me Kieran Martin and me James Line. Now we're both from SAMS who are kindly backing this podcast. I myself am a massive geek and have spent my life chasing cyber criminals around the internet. And as is constantly clear, I'm not a massive geek, but I did deal with cyber security policy and operations in government. And I set up the UK's National Cyber Security Center. But together, James and I are trying to unpack the weird, wacky, wired, and wireless world of tech security and all the complicated things it involves. We are progressively making you geekier though, Kira. Work in progress. But anyway, this podcast is a voice for security leaders. We want CISOs, security directors, and everyone beyond to build up their knowledge of what works, what doesn't, and to ultimately secure their organizations more comprehensively and quickly. Indeed. Now then, James, I'm wearing blue today. Cool story. Kieran, I I don't really care. Well, you should. Um, you're going to have to give me a bit more on that. What? Why exactly? Blue is the theme of today's podcast because when things are down and everything is bleak, we talk of hope, blue skies in the days ahead. But enough of that. Because we work in cyber security, we do spread as much gloom as we need to on this podcast. We talk about threats and baddies and harm and loss and all that depressing stuff. Without giving into FUD, I might add. We've had that conversation. We have had many times. All of that goes with the territory of cyber security, I would say. Well, it does, doesn't it? And you know how to wreak a bit of havoc yourself, don't you, James? You could do a fair bit of proper red teaming, breaking things, causing absolute mayhem and chaos, can't you? That is fair, Kieran. I've got a great story about accidentally locking out 130,000 accounts in a business at one point. It has been done, but I think I see where you're going with this. Well, I'm going in two directions. One is we're going to do an episode just the two of us, and you locking 130,000 accounts. But today we're going to do an optimistic blue skies episode. A really empowering one. Core purpose of this podcast. One that helps people to cope with all the badness. Tell them about all the good things they can do, the positive steps they can take on cyber defense. So what you're saying is you're looking to balance out my offensive nature and my general cynicism with someone. You know, this is what we must do. You're right. And I have just the guest, Kieran. I knew you would. Offensive, James. So please introduce our guest. Oh, with absolute pleasure, Kieran. We we have today a proper a true superstar of cyber defense. Oh yes, the beckenbar of cyber defense. Uh what that's France Beckenbar James. He was the greatest defender football or soccer for our American listeners and indeed to cheat our guest. Greatest defender the world has ever seen. He once played in a World Cup semi-final with a broken arm. True story. Oh, of course. Well, well, not the perfect comparator for our guest. I I suppose because well, firstly, he is American and because as one of the s's most legendary instructors, more of what Americans would call a defensive coach, I suspect. That is fair. So, sorry. Go on. But a for anyway, look, our guest today is someone who embodies the aim of this podcast. We want listeners in the cyber security community to have actionable, usable, easy to understand takeaways to help them fend off the baddies. And our guest simply has no peer in his ability to do this. Now, he was originally an engineer, but developed an interest in cyber security as one obviously should. So, he went allin, did a masters in cyber, started work for a pharmaceutical company's security operations center back in the days when even technologists thought a sock was something you put on your feet. Although I am waiting for my AI powered socks to get my temperature regulation perfect living in the future. Where's my hoverboard? They're in the post. And he became uh the sock manager for the US and thankfully for us found SANS. He's developed some of the most popular courses on security operation centers, but has broadened out into incident management using cyber intelligence, network security, and basically everything to do with cyber defense. building effective tools and practices and teams to enable organizations to find these intrusions into our networks and to respond to them to stop the bleeding. His personal mission in his own words is to simplify cyber security operations for the next generation of cyber defenders. By the way, impressive because I think my personal mission is at least three paragraphs. So, he's clearly a master of brevity as well, which I always admire. That's something for us to learn. Oh yeah, one day, Kieran. One day, maybe he could teach us that, too. But look, he's done this through his work, you know, in Sam's, his Seckhub YouTube channel, his wonderful blue, yes, blueprint blog and podcast to match your shirt today, Kieran. Thank you. And he's been so impressive as a teacher. He's now the curriculum lead for cyber defense at SANS. Thousands and thousands and thousands, tens of thousands of students will have learned from him on that journey. Superb. So maybe the best way to describe him is the coach of the blue team or indeed the bringer of blue skies. The bringer of blue sky. Yeah. Well, certainly the first one. We'll see if the second is true by the time we get to the end of this episode. Anyway, look, we got there in the end, Kieran. So, please welcome the manager of the blue team, the captain of team cyber defense. We're getting all football on it now, aren't we? Yeah, we are. The one and only John Hubard. Welcome, John. Thank you so much for having me, guys. I don't know what I did to deserve an intro that amazing, but I will take it. Really, really excited for this conversation. It's my absolute favorite thing to talk about. I think years of making life difficult for the cyber criminal deserves such an introduction. But well, look, John, thank you so much for joining us and bringing your vast expertise to our listeners. We've got some standard questions that we got to get out of the way, but you know, they're often so revealing. So, tell us first and foremost about your journey into cyber security. You were an electrical engineer. What made you swap that for the invisible digital keyboardy harms that we like to discuss on this podcast? That's a great question. Uh, thinking back on it, I believe that this is something that was always inside me, but I didn't realize I could make a career out of it. And what I mean by that is growing up, I was always into investigating the odd, dark corners of the internet and computers and other things like that. You know, back when I was connecting through 56k modems to AOL, downloading all those questionable tools and text files and illicit knowledge that kind of danced around the internet back in the '90s. I did a little bit of a playing with stuff there. got into the I would say maybe hardware and electronics side of hacking a little bit as well with uh some freaking and you know boxing and all that kind of fun stuff. Freaking Oh, you are so old school, John. I think you might be my hero. I think I can admit that now. Right. Oh, 100%. I'm going to get you a cape and maybe a Captain Crunch whistle. Yeah, exactly right. So, we uh me and my friends that is crafted a little bit of that sort of stuff. So hacking was always somewhere inside me, but I I didn't know at the time going into college that that was something I could make a career out of. So started with electronics cuz I was also very much into that. And really what got me pulled in is as I was doing my undergrad degree, started listening to a bunch of podcasts. And if there's one specific thing I can credit to really saying like, oh, this is something I might actually be able to do, it was listening to the Security Now podcast, which is now still running. And I thank them, you know, immensely for giving me the initial knowledge to even get into this and talk my way probably into my first job. And so that's one of the reasons I run a podcast now is like I want to give back to people and hopefully help them on that same journey. But for me, um, became an electrical and computer engineer for a while, had a job, they had me getting a a master's degree, and for that by that time I knew that I was interested in this. I was doing a ton of self-arning. I was sending myself to Defcon for fun and just in love with the scene, in love with everything I was learning and thought that with that passion, I certainly could make a career out of it. And so fortunately, I was able to talk my way into a job uh after gaining that degree and doing a whole bunch of other self-study and and surpassing and things like that. And then the rest is history. Got into a sock analyst job and uh followed it as far as I could. Vague interest leads to specific interest leads to passion. And then you find people in the community, you find resources, you you learn more, you deep dive, and one thing leads to another. Reminds me of my own journey with a few variations. I'm not allowed near electricity. It causes significant problems. But if I may, John, you know, one one further little dimension on this before we get to helping our listeners here. What brought you into to teaching? You've helped so many students. You are phenomenal in a classroom. I' I've seen you in action. I've I've read the reviews. Tell me about that. And I know people kind of describe becoming a SAS instructor as the longest job interview in history. I know it takes years to get to that mantle, but you stuck with it. So tell me more. Yeah. So for me, when I got that first sock analyst role, maybe a year or two in, I got my first training from SANS. And at the time, I wasn't familiar with it other than my co-workers who had taken courses saying like, "Oh, you got to do this training. It's the greatest thing." So I went uh first class. Uh I took the easiest one I could find, forensics 610, and uh jumped right in. Just kidding. For those who may not know, that's malware reverse engineering. Uh, but that's where I started and I I went to that class. Jumping in at the deep end there. Yeah, exactly. Right. And not not necessarily intentionally. I was like, "Oh, that sounds fun." And then got, you know, smacked in the face with a bunch of deep knowledge. I hung with it, though. Pushed through it. For context, for folks that may not have done that, that would be the equivalent of going to an Olympic diving pool, finding the deep end, and deciding it's not deep enough. So, maybe taking some breeze blocks with you whilst you jumped in. So, good good for you, John. Sounds like that went really well. Yeah. So it was it was a struggle but never have I learned something so fast and so relevant and so useful back in my day job right and so you know as a student I'm like this was awesome right never has anything fast forwarded my career like that so the next year I get them to send me to another class and that was SE 560 the pen testing class with Ed Scotas again total you know mind changer and and game changer for my career added some very critical skills and perspective on how attackers work when I'm a defender right really important stuff to know and then after that I took 5'11 and I'm like this just keeps hitting, right? Sans has caught lightning in a bottle here and I I'm going to go to as many classes of these as I possibly can. Well, I was advancing through my career with largely a lot of help from SANS at that point. And eventually I I took a bunch of searchs, passed them all and studied a lot, so got great scores and got that invite email that was like, "Hey, do you want to try teaching?" And honestly, I think what got me into it is like at the time I'm like, "This might be crazy, but I think I'm going to try it." Yeah. I'm very conscious of these moments in your life where it's like if I send this email, if I click this button, like my whole future might change in a different direction. And this was one of those moments, right? I remember submitting it being like am I really going to do this? Like what if it works? What if it doesn't work? Right? Hit the button. Um this was back in the sands mentor and community days. So a little bit different, you know, wind up to being an instructor back then, but that set me on the path. Started teaching and I loved it. Right? Connecting with students, getting to talk with peers. It was just something that was so fun and so exhilarating, too. I've never gone skydiving, but I imagine like the feeling of getting done with a class, especially early on when you're an instructor doing it for the first couple times, is is pretty similar to like, "Oh, I just jumped out of a plane and lived." It's like, "Oh, I just got in front of this classroom full of other people and I didn't look like an idiot for six days." Yeah. Having done both, I think your analogy works, by the way. Yeah. So, it's a tricky thing to do, but it was really fun. And after that, basically, I was just hooked, right? and uh had that realization that that after I got in, every time I'm doing something difficult and hard and scary, I love doing it. Sands pushes me that way. So, I've stuck with it and we'll see where the trail goes. So, given that James wants you to jump into a very deep swimming pool with breeze blocks attached, I imagine the teaching equivalent of skydiving is probably well within your capabilities, but let's wind it way back. Can we combine the two, Kieran, somehow in your imagination? Absolutely. But it's beyond me. Now, in terms of helping our listeners, as we've promised, some of them will be saying, "Right, go deeper. Go more technical." Others will be going, "Hang on, I would feel this 610. I might be looking at a mirror here." Others will be saying, "Freaking, not sure what that is. Don't like the sound of it." But now you're teaching, you're head of the defense curriculum at SANS. You've written a course called blue team fundamentals. So, at the very basic level, you've got this brilliant stuff on blue team fundamentals. So, what does the concept mean? What does it encompass? So first a little bit of story on where the class came from and then into the kind of general definition there. The class came from me being a sock leader at Glaso Smith Klein. That was a company I worked for headquartered uh over there in London. Large organization, a lot of stuff on the network to defend and I was a US sock manager. And in that role, of course, I was hiring new analysts. I had also come from being an entry- level analyst early on and kind of worked my way up to that. So going through that whole thing doing a lot of self-learning I thought that I had a fairly deep understanding of what it was difficult to grasp as a newcomer and then as a manager I was like now that I've taken all these s classes by this point I'm like what class am I going to send new analysts to to get them spun up in a sock right and that class didn't quite exist there were some things kind of around it but there wasn't like here's the starting sock analyst class and so after being involved with SAS for a while there and teaching they're like why don't you write it and I had one of those moments of should I really do this but I was like I want this to exist right I need a class that's going to teach people these things. And so what are those things, right? Blue team fundamentals. For me, if you're going to work on a blue team, those are the things you have to know to be able to at least do the job to some minimum level of proficiency. So specifically, right, we're talking about what data are you collecting from what's happening on your network, whether it's packets, whether it's logs, stuff from the endpoint, stuff from the cloud, and then what are you looking for within that data? How are you going to verify quickly and accurately whether that actually is a problem and how big the problem is, whether it's APTs or maybe commodity malware and then how are you going to quickly get in the way and contain the problem and make sure things don't get worse. And so blue team fundamentals to me and and the blue team in general is all about like what are those things that need to happen and understanding that data and going through and understanding those processes behind minimizing the risk of any giant kind of breach that may come your way. Love that great framework. I mean you need visibility of the data. a means to process it and a means to react to it. Of course, these things are easy to handwave at and say, they're actually pretty hard to then translate into skills and then orchestrate into a team that works effectively at enterprise scale to do well. What do you think the biggest challenges are for newcomers to sock in in getting started with this in learning these? You've seen a lot of students work through that. What what's the hardest bit do you think, John? Um, I don't know if this is the hardest bit, but one of the more confusing bits, which I guess makes it hard, is what is important to know and what is not important to know. If you go on YouTube and you search anything about information security, there's infinite rabbit holes with infinite depth. All these specialties, right? Do I need to know apps? Do I need to know command line commands? Do I need to know packet captures, log files? Like, where do I even start? And so, it's a question of like what's within the scope of the role you actually want? And then within that role, what's in scope? What are the most important things? And prioritization is one thing I'm hitting constantly in the class. And so I try to focus in on like if you are going to get this job and do these things, like what are the top things you have to know? And I think that's something that universally I see newcomers get lost in. It's like I don't even know where to start and they just need a coach and someone to point them in the right direction. Yeah, that makes that makes a ton of sense. Hey, um I'm curious, John, and and then we'll we'll we'll get on to other topics because there's a lot to cover with you. You know, we're obviously talking about this now from the perspective of newcomers. You know, prioritization. I actually think if if you were to pick one word for the ultimate blue team challenge in any part of cyber defense, it's that like thinking about which controls matter, which data matters, which things you're going to go after. It's also the largest challenge for CISOs. You think of all the shiny boxes you could deploy, all the different things you could invest in for the kind of plethora of different types of attacks or different initiatives. any pro tips from you you can share in how you know security leaders can make sure that their teams are working through this prioritization exercise well anything they should cause those teams to do regularly or questions they should ask to help them be intellectually honest about prioritizing as the sock is developing any little nudges you might ask yeah absolutely so prioritization can't be accurately done if you don't know what the threats are so threat intelligence is going to be a huge piece of this in every industry, every company of every size is going to have a slightly different risk profile, slightly different attack groups that may come their way. And so while you can go out and buy best of breed tools and all sorts of stuff like that, that doesn't mean those things match up with what you might be going up against. So if you're trying to build a sock, right, one of the things you want to do from the start is like think about what are the critical business systems, what are the threats to those systems. Are we worried about confidentiality, integrity, availability, you know, are we maintaining uptime? Are we keeping the power on and the water clean? or are we making t-shirts, right? Those are usually the two extremes I go to. One group is going to have a very different set and a very different impact if it's successfully attacked than a company that is making, you know, consumer products or something that's not literally keeping people alive. And so, you have to balance the threats with the spend and the risk and everything else and make sure that you set off on a rational course that is informed by the best threat intelligence you can get your hands on. Right. But before we move on, I'm going to make a valiant attempt on behalf of some of our more general uh listeners who have a very high level interest in cyber security as you and James get into so much brilliant detail and James goes all jargony on you. Now, he mentioned these people who think that a I would never care really. Oh, come on. Guilty is charged. Okay, guilty. Now, you mentioned earlier, James, people who still think sock is footwear. And I'm thinking of those maybe seuite executives or maybe some new sisses or people who've been told they're chief risk officers or something and you know who you are. You've had a board discussion with your cyber security team and they've nodded knowledgeably when they're told yes things are improving. We've stood up our sock and they nod there and they silently listen but they're admitting to themselves they've no real idea what the purpose of a security operations center is and what goes on in there. So just tell us briefly John what to you is a security operations center? What's its role in cyber defense? [Music] Supply chain attacks surged to 245% from 2023 to 2024. And breaches like the Oracle cloud breach, GitHub action supply chain compromise and vulnerabilities such as log forj have proven just how one compromised supplier can disrupt thousands of organizations. Whether you're looking for comprehensive training, expert-led webcast, or readytouse templates and frameworks, the new SANS supply chain security hub has everything you need to secure your supply chain from development to deployment. Explore our resources at sands.org/suppchain. Yeah. So, I love defining things from the top down and starting general and then kind of having everything in more detail traceable up to that one thing. And when we had this conversation in in both of the classes on sock that I teach, one of the quotes that sticks with me is something that came up from a guy named Rick Howard who's the cso over at N2K Cyber, he wrote a book called Cyber Security First Principles where he tries to like define this from scratch, right? and and he has this kind of quote with meticulously chosen words on what needs to happen from any cyber security team. The ultimate kind of highle abstracted goal. And what he says about that is the goal is to reduce the probability of material impact to an organization due to a cyber event. And what he's pointing out there is probability because no team is perfect and material impact as opposed to any impact because certainly bots and commodity malware is not going to be a huge deal versus ransomware that brings down the entire company. And so in a general sense, what a sock needs to do, everything it does needs to be traceable up to that goal. If they're taking some action that does not reduce the probability of material impact, they're probably doing the wrong thing, right? But how do they specifically do it? Well, what they're looking at trying to do is just what I said before, right? know the enemy, know and predict what they're likely to do, what they likely want, and specifically how they're going to do it, and then match that with the capabilities they need to be able to spot those activities before the attackers get through those entire attacks. And that's the thing about these really damaging attacks, usually they're multi-stage attacks. And so if you can stop one of those attacks anywhere in those stages before the final stage when they exit with your data or they hit the button on the ransomware and everything gets locked down, the blue team is still one. And so what the blue team is looking to do is collect the right data, look for the right things and disrupt those attacks sometime before they reach the end of that cyber kill chain. And we have a bunch of tools and a bunch of data we need to implement to be able to do it and a bunch of technical training and process and workflow that supports it. But that's largely what a sock is out to do. Thank you. That is extremely helpful. Right James, you can get back to need clarity and brevity things we can learn from Kieran. But but look, let's bring in another fundamental bit of blue team security. And you know, you you've been trying to balance out my cynicism and offensive nature, but I think we still get to talk a little bit offensive with John. Yeah. All right, then. So, look, to the non-cyber security person, it is a bit of a weird phrase. It sounds like checking if a writing implement works, but you know, by its full name of penetration testing does tend to bring out its own set of giggles. Let's be honest, cyber security professionals will know what pentesting is, right? This idea of finding flaws by emulating attacker tactics. John, you know, you have a huge, you know, cyber defense leaning here with the sock. There's this trend of of penetration testing that's been going on for years. There's red teaming, there's purple teaming. There's all these variations that all center around this idea of testing one's defenses. What's the latest here? What's the overlap, you know, of good pen testing and good offensive with good blue team? What trends are you seeing and what should CISOs be looking out for? And sorry to hit you with a question that is um transcending all of cyber security in at least four different colors. Oh, no problem. That's that's another thing I love talking about. So, blue teaming is great. Setting up a defense team is great. Setting up smoke detectors is great. But you still always have that lingering question of well what if they don't work right the smoke detector you hit the button the thing beeps and you say ah great right I am safe from fires with a sock you got to hit the button right and hitting the button is what the penetration test does or red teaming adversary emulation there's a bunch of different tweaks on those things and we can get into those specifics on what each of those different things is and everyone has a little bit different definition there but largely the red team is there to sharpen the blue team and also help them ideally sleep better at night everyone sleep better at night all the way up the CISO because you have actually pretended to be an attacker, a relevant attacker using real attack methods ideally, and you should have been able to stop them. Purple teaming is kind of the same thing as well, but it's a little bit more interactive and more exhaustive in my experience. A lot of times that's more of an exercise where you have both teams sit in a room and you just iteratively go through a giant list exhaustively of all the different ways of accomplishing certain things. And so that's actually something I like doing before getting into a full unannounced pen test. But really, all of those things are are awesome ways of checking that the sock is actually able to defend against the most realistic things that those testers can come up with. Now, all that stuff is great and we love those and I highly encourage everyone to do that. In fact, it's mandated in most places, right? For compliance reasons, you got to get a pentest. But those are point in time assessments. And so, some of the things we've seen come to the market more recently are tools for breach and attack simulation. And the idea here is yes, it is more realistic to have a human on the other side of the keyboard pressing the button and or as real as way possible uh periodically, right? Maybe you're doing that every quarter, a couple times a year, once a year, whatever it is. But like breach attack simulation tools will continuously run these tests and sometimes you can fully automate like adversary emulation with tools like MITER's caldera and you can say, you know, act like a1, right? and it knows all of the things and it can just cue those up and not only try them in a rational set of like here is what this group does, but it can try them over and over and over again. So, you know, today if it works, tomorrow if it works, 3 months from now if it still works cuz anything can change, right? Environments are always changing, things are shifting, the network changes, cloud setups change, and you never know what's going to break something. And so, the only thing worse than missing a detection because you didn't have it written is having it written and not realizing it was broken and missing it for that reason. So all of these tools help you in various ways uh with various different time scales verify that your detections are actually going to work. So to build on that, so it's my turn to ask the actionoriented follow-up. What is best practice for using a pen test using those results? Oh, Kieran, you're learning. I love it. Well, we are in the presence of a world-class teacher, so I should be learning. That's the point. So what should and shouldn't you rely on a pen test for? Put another way, let's say you're a SISO and your team presents the pen test results. What are the sort of questions you should be asking them? And what will be the red flags for the blue team to mix my colors in terms of what you might get for the answers? I'm going to give you one for free before John answers. If the uh the pentest report has been hidden in a drawer and never looked at, that might be concerning, but go on, John, more more seriously. Absolutely. I would start with that. Uh beyond that, I would say start with being very clear on the definition of what kind of test you did. So, you said penetration test, but did you mean that or did you mean as opposed to red team, as opposed to adversary emulation? Real quickly, here's how I think of these and I'll caveat this with not everyone agrees with these definitions, but doesn't really matter. They're all important. Pentesting I've often seen is like, let's just go into an environment, whatever the scope happens to be. You set it and you see what's possible, right? And if you can capture domain admin credentials, you do that. If you bring down something, maybe you show that you could do that, but maybe you don't actually do it. That's one thing. Red teaming, at least in my experience, has been started with what is the business scared of happening. So, in my previous experience right at GSK, one of the red team exercises we ran was could someone either disrupt manufacturing or tamper with the formula for a medicine in a way that wouldn't be noticed. And so, there's specific flags as we call them. And you're going in and the point of that test is can this one specific scenario actually come to pass that we are scared of, right? That doesn't mean there's a threat out there trying to do that though. Ideally, you would align those two things, but there's that kind of test. And then there's also adversary emulation where we're saying what would our attackers do in our environment to do the things they normally do. So I would first start with like what kind of test was this and what was it optimized to find? Was it optimized to find anything and everything a specific goal or pretending to be some kind of unique threat group? And then from there you got to ask about the scope of that pen test. Uh you don't want to say like oh everything looks great when really like half of the network wasn't tested right? It's very important to know what was actually checked in the scope of that. Oh, if I had a dollar or a pound or a yen or a Oh, I was going to say a bitcoin. That doesn't quite work. For every time I've seen that one, John, that the age-old Yes, we did a pen test. Approximately 1% of our assets hidden away in a closet corner somewhere and missed all the things our customers actually interact with. O, yep, we did a pen test, but we weren't allowed to do anything other than probe the perimeter, right? And if you got in, that was it, right? Yeah. Those are not very realistic, right? There there are no limits on attackers. So the less limits you can put on your pen testers, the more realistic things are going to be, right? How much time was given? What were the goals of the pentest? And then ultimately like the results, right? That's the big thing is what was found and how big of a business risk does that highlight? And then what are you going to do to fix those things? Who's going to take those actions? And how are you going to make sure that those actions are actually taken? Because I will tell you, right, if you ask pen testers how many times they've found the same vulnerability in the same environment over and over across multiple pentests, they will tell you all the time and it drives me to drink, right? They will say it is a repetitive thing that people do not fix the things they are paying us to find. And so organizations seem to say like, I feel warm and fuzzy cuz I got the pen test. The point is not getting the pen test. The point is fixing the findings in the pen test. So, make sure those are actually assigned out and followed up on so that you are not the victim of something that was found in an earlier pen test and then it still worked when the actual attackers did it to you. Not a good look. Good advice on that last one. And look, it's funny. We we all want to roll our eyes at that and go, "My people would never do that." But it happens so often, even in competent security teams, a bunch of folks all point at each other for whose job it is, and then the work doesn't get done. And John, I I believe you'll agree with me on this. It's it happens in pen testing. The bizarre one is it happens in incidents too where there's all this retrospective analysis that you do the wash up, you find the things you need to fix to prevent the incident next time and the work doesn't get done cuz like the organization takes a deep breath, everyone goes back to their jobs. I've even seen pretty significant, you know, businesses and organizations manage to have the same near identical incident again later because they didn't close the windows they identified as open the first time. Does that reflect your experiences too? It's it's not just the pen testing but more of a general theme. Yeah, absolutely. I have seen that sort of thing before and that speaks to one of the models we talk about in class and that is the kind of two components of doing this well. You have to have the task work, the understanding of what's going on, and the teamwork piece, which is the collaboration and the assignment and the delegation of things and following up to make sure they get done. And yeah, absolutely. I've seen that sort of thing happen. And it's understandable, right? Everyone gets busy, but it's simple, but not easy, right? We know we just have to do the thing. It's just the thing doesn't get done. And so, it's that simple kind of follow-up that we need to make sure happens to not be in that very tough position of explaining why something you already knew about happened again. Yeah. And I think again it's not lazy people. You know, you said it, John, like security teams are busy. I've never met a team that wasn't running at 100%. or or more. So, it might sound patronizing, but as a security leader, I'd pay a little closer attention to that area and and validate that there is actually a plan for follow through and that it is vetted and tested. Look, John, let's get a little bit geekier and a bit more technical. And Kieran, I I know you're in learning mode. You can stay silent for this one for a bit or a bite. Geek away. Go geek. You know, John, one of your most watched presentations, I know, is is all about analyzing email headers and and how spoofing email works. A problem that ought to be dead in 2025 by now. I mean, we've had email a long time. And of course, there's been some some massive improvements in the controls available here, but it's still a major problem for cyber defenders. And of course, so many organizations, even if they're not necessarily spoofing domains, you know, they may be borrowing authority and using fishing tactics and looking like they've effectively spoofed. And a lot of people think training overworked staff with other skills to spot emails and find dodgy flaws is the answer. And then you know so many cyber experts come out and say well hang on that's not possible. It's too hard. What do you say about this one? I mean it is one of the core problems facing every cyber defender. I didn't think fishing and spoofing would be sexy in 2025 when I started my cyber security career but but here we are. Yeah. It's the problem is ultimately that it's a constant moving target. I just did a presentation on this like two days ago for a webinar event we were running for the cyber defense curriculum all about modern fishing techniques and how they've evolved over time. And so the way I look at this problem is it's one of a whole bunch of stuff coming in, right? Conceivably millions of spam emails, and you have a successive uh kind of series of finer and finer mesh nets filtering things out that are less and less obvious as you get towards really well-crafted emails. So we have our basic stuff, right? Our our spoofing, right? just a few years ago when I did that initial presentation. Like that was still a very important and relevant thing because a lot of people didn't have cloud-based email which now tends to like cover most of those problems. People have the standards SPF, DKIM, Demar, some of those things set up because Microsoft just does it for you, right? And that helps a lot. But that's not the only way of spoofing an email, right? You can just fill in the name of the email address as your CFO and ask for money to be moved. You can have the actual email address name looks similar and just a lookalike domain name. Right now again we have technical controls that can look for that. Do you have them turned on? That's a different question. But those are built into a lot of email security gateways and tools and Microsoft. All that stuff is there in 365, but you have to turn it on and use it. And so ultimately we have the basics covered pretty well now. And now we're moving into a world where email is often weaponized using links that you can't immediately tell are bad. links to Dropbox where it's a good website, but they put something bad on it. And so the link can be scanned all day and your scanner is going to say, "Well, Dropbox is conceivably fine, right? It's what you're downloading from Dropbox that's the problem." So maybe you go there, it's a PDF. Well, something scans the PDF because that's what email does now. The PDF's fine. What's in the PDF though might not be fine. It might be a QR code. It might be a link. That link takes you to another website. And then that other website happens to be a clone of the Microsoft login page that can intercept your name, your password, and your two-factor token. Then it will steal the cookie because it's intercepting all that traffic. And now that attacker can log in as you. And there's a million different iterations on that sort of thing. That's the problem, right? Every killchain of all the things that happen in a fishing email is not only somewhat unique and different, but it's also hard to automate scanning those things. Attackers can use captas and other stuff like that now to actually hide from scanners. So while we can throw a bunch of technical controls at it and we should and we do, it's not perfect and we have to supplement that with user awareness training. Ultimately, it's the same story as anything in security, right? Prevent what we can, detect the rest and respond as quickly as we possibly can when those two things fail. So in reality, the answer is yes, but both. Yep. And you know I do take your point on this this layering and as much as there's been incredible progress in technology you know the usability of the work environment cloud services I mean like just just awesome all of that has also presented many more opportunities for subuse from the attackers even in a legitimate ecosystem and of course that's without touching on AI which you know has a lot of utility here shall we say very obvious utility for the attackers in improving quality creativity ity scale, you know, barrier to entry. I also think it has quite a lot of utility for us in the blue team side as well. I'm curious if you I'm not going to take you off into an hour of AI and blue team because that's a whole separate episode. No, but okay, we'll do that next time. But but I'm curious you super quickly before Kieran forcibly crowbars us on. Do you see that, you know, the same most advantages to the attackers today on the stuff developing blue team legitimacy? Is it is it about even keel? How do you think about it? I think it has raised the capability of both sides somewhat evenly. Clearly in the past it was easy to tell a fishing email from poor language, grammar usage, things like that. Now they have AI to make that thing very clean. But we also have AI to say, "Does this email look like a fishing email?" And some of the fishing emails that are the most damaging are the ones that don't have a link and they don't have a file attachment. It's just text that says, "Hi, I'm your CFO. Please move the money. Don't tell anyone. I'll explain tomorrow." Well, now we can point AI at that and say like, does this look like a common business email compromised scenario and is the text potentially suspicious? And yeah, it can identify that sort of thing and and flag stuff that otherwise would have been very very difficult to programmatically pull out before. So, there are upsides to both red and blue. That's why my emails keep getting filtered. Well, look, we do have to move on a bit. Those were excellent and extremely useful answers. much better if I may say so than the questions deserved having stayed silent so that James could get highly technical. He came out with such profound insights like we've had email for a long time and other gems. So you know look um let's widen this out a bit John into the whole field of seamore security information and event management for the uninitiated. So I'm going to ask you to boil down whole courses into a few minutes. What's the key in terms of managing these security events? So security events break down into back to the prioritization topic. You need to know like what are the business systems that are critical and what are they doing. One of the biggest things that every sock has to do is just have visibility. So that means either the log files from the systems themselves, the audit logs and the activity logs from the applications on those systems and then the packets going in and out of those systems. So the first piece is gathering the important data. very obvious sounding, but I can't tell you how many times you can walk into a sock and say like, "Do you have the data you need?" Yes. Okay. Maybe they have it, maybe they don't. A lot of times they don't. And then if they do have it, is it actually parsed correctly or are we just collecting it and pretending like we're doing something with it and feeling warm and fuzzy cuz we have the data, but we're not actually using the data. Once you get the data, you need to make that data better. enrich it, correlate it, test the detections, run those pen tests against those systems and making sure those actions are actually getting flagged when they hit and then making sure you can actually respond to them, right? The data is all about data quality, making sure you have it, making sure it's getting better and enriched and parsed and everything. That's what enables threat hunting, right? Not every detection is reactive detection. There is proactive detection where even if you're not seeing the attack because it's a zero day, having all of that data helps you find anomalies and you can still get into those things if you have the data and identify those very questionable things that are occurring. Pull the thread, find out it's an attack. Pull the thread and sometimes wish you hadn't. U pull the thread on a Friday. Exactly. Which is when it seems to always happen. Attackers love doing that to us. Mhm. John, I guess related to that, you you've done a lot of telling folks, you know, about threat intelligence, what it is, what it isn't, how to use it. Well, you know, sitting here in 2025 with, again, having had threat intelligence for many years now and not as long as email, as Kieran points out, what are the big lessons for for this year, for this moment, you know, how should CESOs be thinking about changes in threat intel? Um, I think with the advent of AI, we are able to better process the threat intelligence, the mountains of data that we have and make sense of it. But in some sense, that's what we've always been trying to do. Now, we can just do it faster and better. So, for example, there have been numerous instances recently where chat logs would leak from ransomware groups and things like that. Whereas in the past, it's impossible to read through years of chat logs. Now, you can throw those into an LLM and say like, give me a summary of what they're talking about, what they're interested in, what they're going to try to do. So those are some opportunities there. But ultimately like threat intelligence in a lot of ways in my experience of it in the sock is always what it was before and that's just making sure you have the data and that data is organized in a way that helps make a picture about what you should expect from the attackers. The thing I always say in class about threat intelligence is it would be insane for the military to go to some foreign country for a battle, not know anything about the land, not know anything about the weapons, not know anything about the goals or tactics that that army might use. it's not going to go well, right? So, threat intelligence for us is still the same as it always was in the fact that it's supposed to help us prepare for what we're going to experience and then use that information to prioritize how we're going to prepare and and spend our defensive money and time and training and everything else. And you can break that into multiple different layers. Um, often we talk about strategic versus operational versus tactical threat intelligence. Tactical is that very low atomic kind of uh, you know, pebbles of of information you'll pick up per incident. Operational threat intelligence is where you take those pebbles and you start to organize them in a threat intelligence platform and you can start to see the bigger picture, right? It's stepping out and saying like ah the pebbles there's a hill over here, there's a mountain over there and then when you zoom all the way out, especially when we're talking CISO level and things like that, that's your strategic level of threat intelligence. And those people need to know what is my entire threat landscape. And they can't know that if the people in the sock have not been organizing those pebbles into groups and activities and campaigns and what they're trying to do in each of those things. And so what a CISO needs is that very high level broad overview of what is the threat environment here? How risky and dangerous and capable are each of these groups? And where am I going to spend my money to do something to mitigate those risks? And so every organization needs to have kind of all three of those pieces to make sure that you can actually adequately prepare for the attacks you might experience. Okay, I need to jump in here because this gives me the opportunity to shoehorn some policy questions into this podcast. as always caring. Well, everything we've been talking about in terms of all our sporting analogies, I think it's a home match for you, James, and an away fixture for me. So, I'm going to seize my moment here and take it beyond the organization. So, it's a slightly left field one. We used to talk a lot about information sharing being the solution to national cyber security problems. 10 years ago, it was very fashionable. It was the holy grail. Then sort of fell out of fashion a bit. I think people started talking about it as the thoughts and prayers of the cyber security industry. And threat intelligence is a big part of that. It's not the whole of it, but I guess where I'm going is how much of the problem is knowable and actionable and how much does sharing play a part in this and what are its limitations? Where do you stand on all of this, John? Uh, again, another thing I love having a conversation about in classes, it's this effect of every sock is an island if you do not share that threat intelligence, right? What works on or A is going to then work on or B, C, and D. If they're not talking from or A and they're saying, "Hey, I just saw this and let me share it with all of you." I understand why organizations do not want to share threat intelligence because it admits that something bad may have happened in their environment. So the fast answer to that question is intelligence sharing is amazing when you can get people who are willing to do it. It raises the bar on attackers. It forces them to recreate all their tactics and techniques cuz they use it once. It doesn't work anywhere else. Significantly slows them down. And speed is a huge factor when it comes to success in a sock. But kind of the the broader question there, how much is knowable? Obviously we know there are things we don't know. We don't know zero days. We don't know new threat groups until we see them. There's a lot of stuff like that. But the things that happen when a cyber attack occurs don't change in certain ways. And what I'm referring to there is there are certain anomalies that you can pick up with standard data collection that if you're looking for them, it doesn't matter if the thing is a zero day or not. You can say, "Hey, why is that machine doing something it's never done before?" And then that can be the thread you pull to uncover stuff. So even if you don't have specific intel, sometimes those things are still detectable. An example of this would be something like the Solar Winds breach, which I'm sure everyone is pretty familiar with, right? Couple years ago, huge disaster vendor, you know, supply chain attack. It was a trusted system with a lot of permissions, right? But when you have servers and things like that, they tend to do about the same thing day in and day out, which gives you the capability as a defender to say, "All right, well, I'm going to watch for it to do something different." So if you had EDR installed on a system like that, you could say, well, why was there a new executable that was run here that's never run before, like in this case, it may have been the Cobalt Strike Beacon executable, or why is that server talking to a server it's never talked to before, or connected to a domain name, which in that case wasvmcloud.com, right? Maybe the team sees that and it says, well, this server has never reached out to the internet and talked to that domain before. Does does Solar Winds own that domain? Do a quick Google. No, they don't. Okay, wait. Something is weird here, right? So we can pick up these things in other ways even if threat intelligence is not specifically there. The problem is knowable if we understand the principles behind it right and one of the core principles for any sock is not all anomalies are bad things but all bad things are anomalies in some way if you are collecting the data to identify the anomaly. So that's what kind of motivates me to like teach a lot of these principles because we can find those things if we know what we're looking for and we have the right data. Great takeaway. I like that James. Yeah me too. Well, well, look, I know we're getting dangerously close to our time a lotment. And of course, we have our standing feature for the end of the show that one must hit, but you know, may maybe one just more tiny little area to squeeze in the relationship between defense and offense in cyber security. We touched on this a little bit earlier. We talked about kind of penetration testing and its role. So, look, you've got all this threat intelligence. Some of it you may wish to defend against, but you might also be tempted to ask whether you can go a bit upstream. I guess what I'm asking is where does this blue team fit in a holistic cyber strategy? How much can reasonably be expected of it? And what potential and risks do you see in the offensive space? To put a finer point on it, you know, are there scenarios where we should take the fight to the attackers using this uh this this threat intelligence or is that a bit of a dangerous playground? Uh I've heard a lot of discussion, you know, people like why can't we hack back, right? And those kind of things. Certainly that's not going to be advised by nearly anybody unless you're a very select group maybe working in government offices deep underground with security clearances right where you might not go to jail for doing such a thing. Lawyers love that question. Yeah, generally not a good idea. There are other things though we can set traps and honeypotss that kind of feels a little bit more safe and and those kind of things. Certainly those kind of things are possible but largely where does this fit in a more holistic approach? Technical cyber defense is just one piece of the pie, right? There's governance and risk and compliance and how policies that you have for your company are going to drive all of those things. And one of the things I love talking about with the sock is like it's easy to write a policy to say this is what we are and are not allowed to do. That makes writing a technical detection much easier. So for example, right at my previous organization, we had policies that said like these are the network protocols you are allowed to use. So for us as defenders, we can say, hey, anytime we see any other protocol, it's at best someone creating some kind of acceptable use policy violation. At worst, it's an attacker that didn't know about that policy and now we just caught them. And so we can interweave like the paperwork piece of this and the technical detections to create better ability to detect. And then also considering, you know, where we fit in on legal and compliance measures and where we're storing our data and all the other problems that are cyber related that can come into that technical cyber defense from advanced attackers is just one piece of the pie. So while it's fun and it's exciting and I love it, uh certainly it is not the whole picture for any organization and the risk that they face in the cyber domain. Brilliant answer and I guess I'm probably legally obliged in some way I don't understand to agree with you that hacking back in the narrow sense of the term probably best left to governments. But look before we finish up I'm going to sh in one more question but it's quite an easy one or simple one not easy given our blue team blue skies theme of today. You've had a lot of experience in cyber defense. Are we winning? Can you give us some hope? How's it going? Yes, I strongly believe that we are winning and I have some data to back that up. Now, I know people listening to this may think initially that that's crazy because they continue to see ransomware. They may see plenty of reports that say cyber crime gets bigger and bigger every year. That may be true, but there are other dimensions that I think are representative of our defenders getting better at defending because I think a lot of that is like, oh, people are getting money. Well, there's just more people trying to get money. So if the pool of attackers grow, if every attack is still harder to do, is that winning? I think it still is, right? Ideally, we would like to get it that it's so hard that we actually dissuade the attackers and we may have not tipped that far yet. But here's the one piece of evidence that I love pointing everyone at. Mandant does a report every year called the Mandian MTS report and it actually just came out a couple days ago containing all the data for 2024. And one of the key metrics that I always open up this discussion with is what would winning look like and are we going in that direction or not? Winning to me is if you consider any single attack, is it getting harder or easier? And are we catching those attacks faster or slower? Right? And they have tracked from all the incident response that Mandant does, which is a ton of it, right? What is the average time it takes for an organization to recognize that they're breached? They've been tracking this metric for over 10 years. I think 2011 was the first time they published a report with this number. Back then it was 416 days, over a year to realize on average that an organization was breached. Wow. And guess where we were in this last report that just came out? 11 days. So we have made significant decrease in that number. And throughout the years, it's gone down, down, down, down, down, right? And so that to me means defenders are better at finding attacks when they are happening. Of course, that's just kind of one slice of the pie. But seeing that number continuously go down, and it has gone down every single year until this year. It went up to 11 from 10 the year before. I'm going to call that a probably not significant increase. That tells me we're going in the right direction. Also on top of that anecdotally if you talk to pentesters they will tell you EDR XDR NDR all these other kind of uh things that have come out in the defensive space have greatly matured what is happening and it is much harder for them to get a successful pentest. They have to dodge all this stuff that was just not even there in the past. It was super easy to do a pentest like 5 10 years ago and it is much more difficult now. So I would say absolutely we are doing a better job defending. We are winning. We haven't won with a period at the end yet but we're getting there. Optimism brilliant. You can definitely come back. I would love to. I also like the number of references to winning. It's very Charlie Sheen, but I I'll take it more genuinely. But look, Charlie, look, we've covered so much and as much as there's many more topics that we'll have to wait for the the next time you're back with us, we are going to have to bring things to a close. But first, we come to James's favorite part of the show, but it's highly problematic this time. There is nothing problematic about my 30se secondond takeaway game. Not in general, but there is this time, but for a very good reason. Yeah. Uh, fair enough. Well, look, we we've been asking John for 30 second takeaways with basically every question about Sim pen testing socks. A 30-cond takeaway every 30 seconds. It's quite satisfying, isn't it? Well, yeah, I know. And it's been great. So, we have to keep up your ritual, though. So, I have an idea. Okay. Do do expand. We'll ask John for his 30 secondond summary takeaway of all his other 30 secondond takeaways. I like it. So, meta, right, John? Other platforms are available. Nice. Well done. Thank you. No pressure. But this is the most important part of the entire series of season 2 of the San Cyber Leaders podcast. In fact, this could be the pinnacle, most important moment in all of cyber security, depending on what you say. As they say, no pressure. We want listeners to have actionable advice, right? We we want them to be able to protect their organizations. You've already given lots of advice, questions to ask, things to focus on, how to think about the stuff, but this is for people who only have 30 seconds of free time, perhaps in an elevator whilst drinking a skinny latte with a twist of, you know, caramel or something. What are the most important things of all really important things you've already told us in about 30-ish seconds? Well, fortunately, I've had coffee this morning as well, so hopefully I can spit this out in 30 seconds. Uh, I would say if you are a leader and you're talking to your sock, ask them some very practical questions. Can you name me the groups and the threat types we're worried about? What are they going to do? How are they going to do it? Do they know who those groups are? Name the systems that would be involved in that attack. Do they know specifically how those things would happen? And have they run red teams, pentest, adversary emulation things against it. Ask them if they have the data from all the key systems. Can you see if one of these things even starts? If so, how early on can you tell that that might be the intent of the attacker? Ask if they're proactively threat hunting and if they have, what are they finding on those threat hunts? Ask them how fast they can respond and what's getting in their way if they can't respond. And finally, ask them how are all of those things staying up to date over time because what you're doing today is not going to be the same as what you're doing tomorrow. Hopefully that wraps that up in about 30 seconds. Wow. I hope the recording worked. That was brilliant. I think we may have just changed the entire future of cyber security. No. Nay, the world and politics. We're actually going to start landing on the moon based on that. But but in all seriousness, great advice and we've touched on quite a few things in this podcast which sound obvious like the organization ought to just take care of them. But if you don't ask the question, these things don't tend to happen. Trust John, he's seen this movie many, many times. Well, John, thank you so much for giving up so much of your your time with us. Thank you. It's been absolutely wonderful, and I do hope you will come back. Oh, absolutely. I'd love to, and I'm definitely excited to hear the 130,000 account lockout episode coming up as well. So, I'll be tuning in for Yeah, we're definitely doing that. Oh, it's a beauty. Change's worst day. Oh, it wasn't even my worst. Just probably makes the top five of my greatest screw-ups. And I shall gladly share it. It was quite entertaining. And how we fixed it was also quite well. Yeah, we do have to do that. We'll save that for another time. But look, for today, I think that's it, Kira. Apart from feedback. You always forget that. I do every time. But look, feedback is incredibly important. We do actually do this for practitioners in the field for security leaders. So, we do want to hear from you. Leave us feedback at the podcast site or email us at cyberleersodcasts.org. Tell us anything you like. Well, not anything. Ideally, it' be tangentally related to the podcast or cyber security. I'll take anything. But anyone, thank you for uh listening today. Yes, thanks for listening and keep cybering. So, from me, Kieran Martin and me, James Lines, it's goodbye. [Music]