USB "Update" = Backdoor

Key Takeaways

If you can as a young person get into your college CCDC program and you make it at least to your regional, you will get at least one job offer. If you make it to us at nationals, you will have at least three job offers. For 20 years, I've been trying to hire one student and I finally got one because all the rest were hired before they made it to me. So, if you're looking to get a security job, go through the CCDC program. And it gave me the time to be able to pivot into their main network and do the classic things that people hear of, get to do main admin and all that kind of stuff. When Dave was telling his story, what I really wanted to get after is how how these things are hard. So don't get frustrated when you're doing something that's hard. You just keep working at it. You keep looking for that thing. These are by default things that nobody knows how to do. So you have to find those things. Hey everyone, David Bumble coming to you from Black Hat with two very special guests. Dave, welcome to the show. Thank you. Evan, welcome to the show. So, just for everyone who's watching, we're going to cover a lot of stuff in this video. Use the timestamps below to jump to the different parts of the video. We're going to have some really cool stories, some red teaming, some advice. So, we've got two experts here in different fields. Use the links below to jump to the part of the video that you're most interested in. I really want to thank Threat Locker for sponsoring my trip to Black Hat and allowing me to enjoy this amazing conference. As Danny says in 2025, you cannot permit all traffic and then try and stop threats. You need to deny by default and only allow trusted traffic in your network. Guys, you have created this tool to do with AWS. Tell us about it. So, we've had a lot of investigations in AWS, uh, which is why we wrote the cloud forensics class at SANS 45509. And one of the things we kept having a problem with is that it would be Friday night. It's when you get an incident, right? the the normal team inside ditches and they say, "Hey, great. Here's some credentials. Go fix things while we take the weekend off." The problem is you get those credentials and you find out that it gives you access to like 120 different roles inside of AWS. They're asleep and you got to figure this out. So, you have two choices. You can either brute force through each one getting frustrated or you can do like I did, give up and just write a tool that does it for you. And so, that's all this tool does. It's very pro to the mission right on point. You can either say here's my credentials from my credentials file that's on my system or here's an API key. It'll then authenticate in give you a list of all the roles available, test them all and say which ones can actually read the cloud trail audit logs inside of AWS. Then let you see and pick which ones you want and then pull them out. That's it. And it saves me so much time. I use it all the time now in my real work. And now I don't have to, you know, wake up Monday morning, you know, to have a call with a guy where I say, you know, I'm sorry nothing happened. you didn't give me what I needed. So, you were featuring us at Arsenal here at Blackhead, right? Yeah, absolutely. Yeah, we love the Arsenal program. Uh, we've been here multiple years and each time we try to bring something new. So, guys, I'm assuming you've been presenting this. What's the feedback from the audience? Uh, everyone who needs it says, "I wish I had this already. It would have saved me so much trouble." So, give me a bit about your backgrounds. Dave, you've been around the block a few times. No offense. Oh, no, no, that's true. I got the gray in the beard for a reason. Uh, so I've been doing digital forensics for 26 years now. Started as a pentester before that. Uh and let's see, I've written five books on forensics, the hacking exposed computer forensics series. I have a a patent digital forensics for file system journaling reconstruction for forensics. Uh I got a bunch of awards. Uh let's see, SANS. Uh and then of course a lot of people know me for the cases that I worked. So in the litigation space, I was an expert for Zenax via Oculus when they alleged that John Carmarmac stole all the VR stuff. Uh, and then I for KPMG led the uh, Solar Winds investigation for Solar Winds, but now I'm I'm not there anymore. I'm a a vice president, Charles River Associates. Now, >> I got to ask you about some of those stories. Evan, what do you do? >> Uh, so I've been doing offensive security, vulnerability research, malware development, uh, C2 development, that kind of stuff for close to 20 years now. Uh, and so I'm kind of a mix of offensive red teaming and then product development and creating tools for people to use to hack or hack themselves in some cases. I was part of the founding team for Randori which was a tax surface management uh platform. We got acquired by IBM and now I do offensive consulting and tool uh product development as well. >> That's great, right? Digital forensics, red teaming and now you guys created this tool together. >> Yeah, some people wonder like what's the connection? Yeah, I was going to say like >> this should be two interviews, right? >> Yeah. So, we started off cuz we were both kind of in the offensive space back in the mid '90s. I switched over to forensics. We stayed friends and then we started doing the National Collegian Cyber Defense Competition together. So, we've kind of been co-leading the red team there for like the last 21 years until we've had two generations now of college kids where, you know, there's 183 colleges in the US who participate. The 10 best make it to us. We're kind of the the final bosses. There's a team of 20. It's it's us. It's Rafael Mudge from Cobalt Strike. It's uh Joe uh from Sliver. It's uh Alex Levenson who writes Gcript framework. It's uh Matt Weekes who has a very interesting background. It's Mubix from uh Hack 5. It's Egypt from Rabbit 7. It's all the bad guys that know what everyone went together and we wreck house every year and the best team wins and they get jobs in the space and it's amazing. >> Yeah. So, if I'm a young person, right, and I want to like get a job in the space, that's a great thing to go for. If you can as a young person get into your college CCDC program and you make it at least to your regional, you will get at least one job offer. If you make it to us at nationals, you will have at least three job offers. For 20 years, I've been trying to hire one student and I finally got one because all the rest were hired before they made it to me. So, if you're looking to get a security job, go through the CCDC program. Where do they get that? Can you can we put a link below the video or something? >> Absolutely. At CC nccdc.org. >> Guys, you both got really cool stories, a lot of experience. David, I want to start with you. Solar Winds, you got to tell us about that. >> Solar Winds. Uh, so this was a it was co >> uh we were all at home. Um, it was luckily right after my birthday before Christmas. I get a phone call from a a guy I know at a law firm uh that brought us in, DA Piper. Uh, and it's my friend Eric, and he says, "Cancel Christmas." And I was like, "Oh no." And at the time, right, I mean, there was some stuff out there, stuff around it, you know, I didn't know what I was getting into. I hop on the first call and I'm like, "Oh, this is it." Like, you know, we knew what was happening. But what no one seemed to know is how it happened. And so for me and my team at KPMG at the time, we came in, CrowdStrike was there as well, and we had to work together to try to figure out, okay, we already know the bad thing happened. We know how long. No, sorry. For people who don't know what it's about, can you just explain like what happened and like what you guys did? >> Absolutely. So, if you don't know, Solar Winds is a software company. They had a product called Orion, which is this massive network management software that was very popular with governments and agencies, which is why it was a target from uh Russian intelligence according to the FBI. That's their attribution, not us. Very clear on that. Uh they actually managed to implant malicious code inside of the build process so that it was delivered to all their customers. And they basically had uh which most people ran as domain administrator uh on their networks a piece of software meant to administer the network and maintain and monitor it that was also phoning back home to Russian C2s. >> And when you guys got involved obviously it was like this was discovered now. Is that right? So what did you guys do then? >> So at that point yeah Mandy comes out with their blog the FBI calls and says you're going to have a bad day that those were the words by the way that was told. You're going to have a bad day tomorrow was what they told the siso and then the and then it came out. So we get involved and the mission is find out how this happened. Yeah. Like we know what happened but how exactly >> and it's a huge network right and there's developers and there's corporate staff and there's sales people and it could be like at whatever point. So we get in deep and we start working and the the the CEO of Solar Winds actually was switching over at the time of the incident. The old CEO was retiring. A new guy came in. The new CEO we call him Rama. Amazing guy an engineering background and thank god he was there because he kind of structured everything. He sat down and he said, "Okay, I don't want you just to start hunting for things. I want you to come up with theories, like what are all the different ways where this realistically could have happened?" And so we broke it down and we're like, "Okay, if you could actually do this, ignoring all of the other like stuff that's been published cuz right this is high scale, right? You know, couture C2 um where do we look?" And so we started looking all around the DevSack pipeline of okay, where all could we inject? So we started saying, okay, the easiest thing is looking at the systems they're developing on. But the problem with that is that when you're looking at the code commits, it gets really obvious that someone's putting malicious code. So we had the devs check that. It came out clean. And we said, okay, well, what about the orchestrator? So that's the system that's actually going to say, well, where is this going to get compiled on? What's it going to bring in? So we looked at that. We looked for like malicious repos being brought in. We looked for new scripting being brought in. We looked for like off-site build host being brought in. That wasn't it either. We started looking through all the build scripts. We started looking for everything. But the one thing that none of us thought it would be because it would be too difficult to pull off is there was this ephemeral virtual machine environment that constantly were coming in and out of service where they were actually doing the compiles. >> Oh wow. >> And when we found it, all of us our minds were blown like one of those hat tips like well done bad guys like you did. Everyone was surprised. And so we're sitting there, we're going through one of the developer systems though where the the actual code was being compiled and we were super lucky because they actually cleaned everything else off, but this was a VM that they turned off in the middle of the OP and was never turned back on. >> Oh wow. >> So we get a copy of it, we go through and one of the things in the forensics world that I'm known for is something called the USN journal. It's a change journal on the disc. And so one of my associates, Kevin, actually was going through the change journal because we're pretty sure this could be the right machine, and saw that there was a file being created, renamed to a legitimate file in the compilation process, and then moved back out and deleted. And so we go to the dev team. We're like, is this normal? And they're like, absolutely not. And at this point, we're like, okay, we know it's happening here. We know the file that it is, but what what process is doing this? Because nothing's calling home. Nothing's injecting code. Like all of the stuff is passing clean. And so we actually instrument the full thing. We run the compile process all locally contained. And that's when we see that there is one process that is masking itself as as task scheduler and all it's doing is reading the process list waiting to see that particular m not just MS build but MS build for the right solution file running and then it has a race to rename that one file and place it and then delete it once the compiler is done. >> Wow. But I mean the how do you find that stuff? I mean like digital forensics is like a needle in a haststack, right? It is. It is. And it's all about as Rama said, right? Make your theories. So when we talk about thread hunting, we're like, oh, what's your hypothesis? You talk about, you know, offensive security, it's all about, you know, what are your attack paths? And for us, it's how could this be possly to like most theoretical? Let's run through the scenarios and test it all out. And then that's the only way you do it cuz otherwise like these guys were so good. >> Okay. D, if you got to tell me the story about the Oculus, you mentioned that. >> Yeah. Yeah. So, you know, every once in a while you get one of these cases that you're like, "Wow, this is really cool." So, like John Carmarmac came to my Christmas party. Oh, wow. I I think that's really neat. He's a really nice guy to be completely honest. Like I I would hang out with him again if he'd ever give me the chance to be completely honest. But what happened was John was working at ID Software, right? Made Doom, made Quake, all these things. But he loved VR and he loved the technology behind it. And all of this is public record. It's all in the filing. So, I'm not saying anything I'm not supposed to. The case is long gone. Uh, but I was John's expert and Facebook's expert because when he left ID to go to Oculus, they were super upset. >> I can imagine. Yeah. >> Because they're like, "Hey, this is the guy who's our genius. He's going to create a future of VR and now Palmer Lucky has stolen him away from us." And so they went through his computer looking to find anything that would show that he was doing something improper. And they raised all of these BS allegations like, "Oh, he took source code and he's a bad guy." And you know, no one believes that. I mean it's John Carmarmac. John Carmarmac. He could he could do it from memory, right? The guy he owns a rocket company too from Right. So when we start digging in, we're like, okay. Yeah. I mean, he took some old email like involving his rocket company and like some Haskell code that he was playing around with >> Haskell on Oculus. Yeah. >> Yeah. And I'm like, none of this involved. None of this is right. So we like we threw it through we like traced out every execution, every file opening, every path opening and showed exactly like what he was touching and what he wasn't. And we were actually able to prove that he didn't he didn't use it. He never used it. And the the courts agreed, the jury agreed. The only thing if you ever go look up that resolution of how that case ended, it was that Palmer Lucky violated the NDA because when John Carmarmac got him to sign an NDA that says, "I'm going to give you a version of Doom that works on the Oculus, but don't show it to anybody." He went immediately to a to hotel room and show it to investors. And that's that's the only reason why there was a partial guilty verdict. Evan, you've been too quiet. I got to bring you in now. How long have you been doing red teaming for? >> So, I've been red teaming for very close to 20 years now. I think my first red team was officially in 2006, a thing called the red team. So, and I specialize in starting from the outside, compromising systems legitimately with zero day and techniques that people don't actually see to test organizations for things they can't patch for. So, probably one of the more interesting stories was I was working with an organization where I scan them. Nothing's vulnerable sitting on their perimeter. There's nothing really for me to go after. And I just keep I keep going at it. I keep going at it. And finally, I found that uh an HVAC system that was exposed to the internet. No ways. But it wasn't always exposed to the internet. It's kind of a similar thing. It's popping up and going down. What happened was the guy who was in charge of that HVAC system was going to a big conference like this. He exposed it to the internet so that he could monitor his system while he was out of the office and then he'd shut it down at night and then bring it back up during the day. But I was able to find that system, develop an exploit for that system, compromise it, and that got me into just their IoT network. But their IoT network was completely unfiltered. It could get out to the internet just fine. So I had a field day of going around and kind of doing whatever I want. They couldn't they didn't have any visibility in anything. They weren't blocking anything. And it gave me the time to be able to pivot into their main network and do the classic things that people hear of, get to do main admin and all that kind of stuff. When Dave was telling his story, what I really wanted to get after is how how these things are hard. So don't get frustrated when you're doing something that's hard. You just keep working at it. You keep looking for that thing. These are by default things that nobody knows how to do. So you have to find those things. >> You had to write your own exploit for the HVAC system. >> Yeah. Yeah. Yeah. >> And what did you write that in? Uh so well I wrote the exploit in Python but the HVAC system was all in C. It was a embedded system that was yeah very old. >> So single HVAC on the internet you got in through that into the IoT network then you got onto the traditional network. >> Yep. >> Wow. >> I have a whole pivoting story too. There's a lot of that but yeah so it's I actually wound up I found a system that could uh was being used to send packages and I built a a thing to implant on USB devices malware that I'd written. So it would auto when they plugged it in it would autoimplant that malware on that and then they someone carried that out of that system. I made it look like an update needed to happen. They carried that out of that system into their network to plug it into the actual network. Then that called home to me. That's how >> type thing, right? >> Uh similar similar concept, but they controlled the USB device, right? So I had to figure out I was watching them do very similar to forensics. Forensics and offensive security when you really get down to the research are very close. It's just intent, right? So, I was watching them plug USB devices in and out. I had to fingerprint the USB device and then develop the malware that could run from that USDB device when they plugged it into another system. Can you guys talk to your younger self or to someone who's watching? They will look at you guys and think, okay, I want to be like you guys. I want to be you. So, Evan, how do I become like you? Because a lot of people who watch are into hacking, red teaming, you know, is there part from it being hard? Sorry. Go on. First, you have to get a lot of tattoos. You don't have to have any tattoos. You don't have to have a hoodie. You can wear a nice shirt if you'd like. Um the really the advice that I would give to myself as a younger person is keep trying. Everybody that's done this are people. You are also a person, which means you can figure it out. You just have to put in the effort. It's not going to be easy. Anybody that tells you it's going to be easy or you can just do a single class and you'll know how to do everything is not telling you the truth. But know that everyone else is also frustrated and sitting there going, there's no way I'll be able to do this. This is never going to work. They just kept trying. And if you keep trying, you will get there. >> Okay. So, what do you recommend though? Okay. I want to do this CTFs or is it like a course that I take, you know, give us some advice like where do I start? >> So, the SANS courses, right, are amazing. Um, CTFs I came I started with CTF. uh you have more access now to CTFs and writeups from CTFs and all of those things. The only thing I would caveat with that is don't rely on the writeups. Do the struggle in the CTF. Try it, fail, try it, fail. If you really, really get stuck, then you're looking at the writeups, but only enough of the write up to get unstuck. And even then, be okay with being stuck. Especially in offensive security, like I said before, you're doing things that no one else has done. there aren't solutions for. So, you have to just get really good at being frustrated. But that's okay because everybody else is frustrated and if you can break through that, that's how you get good at this. >> I'm going to push back on you, right? >> Yeah. >> S's courses are great. >> Yeah. >> But I don't have the money for it. >> Yeah. Yeah. Yeah. So, um there's amazing books, there's amazing YouTube channels like yours that are just awesome for this and full of information. Uh GitHub is a wealth of information of how to do these things. Um, I think it's CTF time is a website where they post all the writeups and all of the different things of all of the CTFs that are happening. Uh, I found it very useful. So, I came up in a time with CTF when there weren't writeups. Not super helpful, but if I could find exploits, I'd look at their exploit versus my exploit and figure out where I could have made things a little bit better. The other piece of this, and this is just what I would give anyone for just life advice, is try to constantly improve just a little bit. >> I love that. >> No one starts at the bottom and just gets to the top. They they take steps. So as long as you can enjoy those little improvements and keep building and building and building, it's that. So the other thing with offensive security is every piece matters. No one in offensive security has the same background. We all come from different places cuz it's unique perspectives on things. So everybody has some sort of input for something. It's figuring out how what you already know can fit. Do I need to get a cert? >> I don't think you need to get a cert. There are certifications that you can get that are useful. Well, certifications I think are more useful for finding a job and getting past some of the recruiters and that kind of stuff, but I don't think there's any one certification that you're like, "Oh, okay. I'm done." The reason that I'm attracted to offensive security is there isn't a done. There's always something else to learn. Dave said, "We've been doing CCDC for 20 years now. You better believe offensive security has or really just security in general has changed a lot in 20 years." But that's why I love it. I don't there there isn't necessarily a book or a checklist to follow but you can create that path. And the other piece is is the soft side of things networking and the people that you know >> like for people networking >> the people that you know the other the other piece that is huge is find your local group. So we're going into Defcon. >> Every single city now almost has a local Defcon group. Find that group and go talk to those people. They meet up often once a month and you'll have industry people. You'll have people trying to figure things out, doing local presentations with six people in the room and you can just ask them questions and they may be six months ahead of you in the path, but they're learning by teaching and you're learning from them teaching and you're helping each other and it's just all kind of growing. So, >> but I'm going to push you on, right? Is there any recommended C? I mean, if you don't want to answer it, that's fine, but is there any like s that I should try and get? Uh, I mean, so there is like the OCPs and those of the world. Um, I have I don't think I have Well, so I have a CISSP, which I guess I'm not supposed to admit if I'm an offensive guy, but I do have that certification. Most of my certifications are old and expired. Now, >> I think like if I was starting, right, it's um the nice thing about training, it's not for everyone, but it some people like the structured thing. >> Yeah, totally. So, you can go after uh there's, you know, kind of pick your path or what you want to go after and explore. I think education is great for exploring things to figure out if you want to do them. You may not know, and this is the same with college, too, where it's like, hey, go take a class on programming. Okay, I don't like the programming aspect. I want to do more of this. Go take a class on business management. Okay, I'm going to be the business guy. Or, I don't like that I should be a programmer. I see it the same with certifications in some of these training courses. Don't expect that to be, hey, I did this training course, now I have a career. that use that as an exploration of what do I actually want to do? What's going to make me passionate? Right? People will argue, oh, you don't need to be passionate to have the career. I sure want to work on things I'm passionate about. So, finding that piece and security is so big now. Finding the piece that you're passionate about and it kind of takes care of itself. Dave, same question, right? I don't have 20 years of experience. How do I become like you? Like what would you what advice do you give? So, let's divide into three age ranges. You're still in high school in the US. cyber patriot. No question, hands down. US funded. All the high schools participate. They give you training materials. A teacher becomes a coach. You drill, you learn defensive maneuvers, you learn the basics of forensics, and then you can actually make it all the way to the national level goal of DC. You know, everybody's going to try to recruit you to their college. You know, you're in college, get a CCDC program. Your college doesn't have one, see if you can start one. You again those those teams, right? Because I one time made the mistake of one of those college teams that I said, "Wow, your college must have a g great program." And they said, "Absolutely not. Our college is a terrible program, but we meet every week and we train. We meet every week and we drill. We meet every week and we're ready." And I'm like, "Yeah." And they won. Like they were back to backto-back winners of our national competition, which is not easy to do. And then if you're outside of both of those things, right? Or, you know, you're in another country altogether where it's not available to you, you know, there are the cyber games that you can play. But more importantly, you know, I would say for especially for forensics, which is a very niche topic, you know, the the channel I would start with on YouTube is called 13 cubed, okay? 13 cubed D. And that guy's great because he actually breaks down like all the different forensic artifacts. One episode per artifact kind of explains what it is, how to use, uses freedom source tools. I would say start there. Get an idea. I have a blog, hcfblog.com, has a thousand articles on forensics that I've written over the last 15 years. There's tons of information on there. Like when somebody at work askked me a question, I do a Google search against my own blog and then give them the article and I'm like start here. But beyond that, like if you really want, especially in the IR space, get ready. And you know, there's a CTFs, which I love. I have CTFs you can play of mine. Um the fun thing I do with new people is I get them on Hack the Box, but not for the reason you think, okay, I get them on Hack the Box on any of the machines where there's a path of how to get in. I get them to get in and then we use open source forensics tools to grab the data out of the box and then we get them to do the analysis because those are real intrusions. >> That's great. >> And so at that point you can either remotely image the box and have a local copy or you can just make a triage collection with Cape which is free and open source. You can grab it back and then you have all the forensic artifacts where you can find your own intrusion and everyone else's. That's the best training set there is. It's free. I love that. You can also use that same data to teach yourself how to do the hack the box challenges. So, a lot of those because people are hacking those boxes actively. If you get on there and figure out how to investigate some of that stuff, you can see the paths that they're doing. So, if you want to learn how to do that stuff, learn from the people that are being successful. Go watch them and figure out what they're doing. >> Guys, I got to ask this cuz I I see this a lot on YouTube. There's no jobs cuz AI is going to take it all away. There's still jobs, right? There's a lot of things AI doesn't do well or things you can learn to do with the IIA to do better, but you're going to have to be better in that entry-le role to get qualified. I'm a hiring manager. I'm looking for that human skill interaction motivation. You show me you're involved in the community. You show me you're trying to do it the best and you're doing CTF. You're grinding your own time. I'm getting you. And I would say AI is not going to replace us. It's going to augment us. So if you know how to use the AI and get the correct answers out of it and build tools and interesting things with it, that's where it's going. the the tool we actually presented. We used AI to build uh because we know how to build tools and we figured out how to get the AI to build the tool for us. >> Can people reach out like LinkedIn? Where's a good place? Uh LinkedIn, on YouTube, I'm learn Forensics. Easy to remember. And on most of the rest of socials, it's H E CF blog. >> And I'm LinkedIn and Twitter and all those. I'm Syndrome. S Y N D R O WM. >> Guys, thanks so much for sharing and thanks so much for giving back to the community. Much appreciated. Absolutely.