Detection Engineering: The Blue Team Cheat Code | Host: Mark Orlando | September 12, 2023

Key Takeaways

[Music] [Music] yeah [Music] [Applause] [Music] hello everyone good morning good afternoon welcome this is Sans wait just an info sec I am your guest host this week my name is Mark Orlando and we have a lot to get through today so very happy to see all of you joining looks like we have folks here from all over the place uh when you have a moment please drop us a line let us know where you're joining from what time uh what what time of day it is uh in your part of the world um I am joining you from Chicago in the US so very good morning to all of my friends in the central Time and Eastern time zones um again we have a lot to get through today lot lots of great and interesting news stories we're going to talk about here in just a little bit um but the theme or the title of today's episode is detection engineering The Blue Team cheat code uh this is a topic that is near and dear to my heart as a security operations person it's pretty much my entire background so uh thrilled to be here today to talk about kind of the hot cyber stories uh of the week as well as a deep dive into detection engineering how you can do it how you can do it better how it fits into other operations functions and to that end I'm G to be joined by some fantastic guests but uh before we get there let's see it looks like we have uh oh a local someone from shamberg welcome that's right down the street from me some folks from Eastern time we got Washington DC Charlotte Tunisia India France Romania Chile um that's excellent very very happy to see you all here today um for those of you who are joining wait just an infos for the first time um in true sand style our goal here is to educate to give you something that you can take back and immediately use in your own environment but we also want to make it fun and we want to make it as interactive as possible so throughout the episode today you're going to have the opportunity to provide your input ask questions share your comments and I would encourage you to do that you can use the chat feature of whatever app you're using to join um please please please give us those questions and feedback we're definitely going to have plenty of time to talk about your comments make sure we answer your questions and make this information as useful to you as we possibly can with all that being said we start out these episodes um with news bites and for that I'm going to hand off uh to my colleagues Thomas wolf Michelle Pearson who are going to talk about the new stories of the day or of the [Music] week hi folks I'm editorial contributor Michelle Peterson and I'm here with my colleague Thomas wolf uh we're going to discuss some of the biggest news that we've seen come out of sans news by news letter this past week in cyber security so Thomas I'll let you go first thanks Michelle all right folks so uh back in July we covered news about an attacker exploiting a zero day to forge tokens and access a bunch of different Microsoft applications uh one drive SharePoint Skype teams even Xbox and a bunch more Microsoft gave a pretty wishy-washy explanation that left a lot to be desired but they just released their official findings they found that due to a bug a key actually slipped into an automated crash dump which is pretty crazy and from there the threat actors were able to compromise on Microsoft Engineers account uh interesting point to this is that no one thing occurred that led to this incident it was many factors that culminated into this perfect storm that allowed the attack to succeed all of which have now been fixed and there is a lot to learn from the investigation so uh Michelle I definitely suggest folks go to the Microsoft website and read this report always a good idea make sure things are updated according to the latest specifications and all that fun stuff so um in other news uh the I guess it's been a couple of months maybe not quite uh since the SEC has put out new regulations concerning how businesses um report their uh cyber events cyber incidents as well as um their uh management of and their strategies for their cyber programs and policies in general so uh the first part has been rolled out we have it's it's a phase roll out so part one is uh cyber disclosure rules for public traded companies um and their risk management strategies um in other words how their boards manage cyber security oversight and governance and the other part to these regulations that have been talked about quite a lot um is the event reporting and that has not officially gone into effect yet it will mid December for large companies and then small companies have an additional six months after that uh to prepare to comply with these new regulations and if you have watched news bites um or wait just an ifos SEC before you will probably remember we had uh the Sans uh director of emerging security Trends John pescatori uh did a nice segment giving giving a pretty much a rundown of everything that is changing with these new regulations and what businesses actually need to know um there is still some debate too about you know what is a material cyber event so I'm sure there will be more to come from this in the future I know Sans will be producing more uh more resources to help our our community out we'll be cutting out with like a checklist to uh help your organization navigate these and uh we'll be sure to share that as well as the link to John pescatore's video in the comments very cool yeah I'm sure there's going to be some interesting interpretations of what a material incident is so with that and our last story of the day uh miter and cisa have teamed up to release an adversary emulation tool for operational technology networks the tool is called Caldera for OT it's an extension of miter open source platform which is also called Caldera so basically the tool it allows you to craft a specific threat that you can launch against your network and look for vulnerabilities now you may be thinking I don't work in critical infrastructure but even if that is the case you really should take a look at this tool as your operational technology systems are super important and your organization likely isn't giving them the focus that they need and that's a wrap for this week's news bites folks don't forget that you can find more critical cyber security news and commentary by subscribing to the Sans newsbites newsletter at sans.org newsbites thanks again and we hope to see you again next week thanks Michelle thank you thank you Thomas and Michelle lots of interesting stories this week and I'm uh very pleased to hear a shout out to Caldera uh for those of you who don't know me seems like there might be one or two of you uh who don't know me on the the live stream today but um my background uh again is in cyber operations primarily so uh at Sans I uh teach our blue team fundamentals course essentially our sock analyst course SEC 450 I'm also the co-author of ldr 551 formerly MGT 551 that's our building and leading sock teams class so um in my day job when I'm not working for Sans uh I uh am with a firm called bionic and we work with a lot of different clients on their security operations so the reason I bring that up is uh we're often advising clients and I'm talking to students about not only designing and building their security controls and their monitoring uh effectively but also testing those things having that empirical evidence that tells you that things are working as designed Caldera is a great project a great tool for that very happy that they're expanding capabilities into the OT sector um really really interesting uh also want to give a shout out to the Microsoft story which if you aren't familiar with that or you haven't heard about the Microsoft security incident yet um I highly highly recommend checking that out do a quick Google search I'm sure it's going to be in your top five results for Microsoft compromise um there's a lot of commentary still about this incident but uh it seems that depending on what side of the the fence you fall on either there was an adversary that got very lucky through kind of a a chain of events as Thomas mentioned um or was very skilled or maybe a combination of both and I think we might talk about that a little bit more uh when we have a more in-depth discussion with our guests I think that might be a good example to talk about through the lens of detection engineering so uh thank you to all of you who are commenting uh in the chat awesome to see so many people joining from all over the place um you can also see that we've dropped my socials in the chat please feel free to connect with me um but let's let's keep those comments and questions coming I'm going to keep an eye on that as the episode progresses and as we talk with our guests I want to hear from you we want to talk about your questions your comments so before we get to my discussion with our our two special guests uh I want to hand off real quick to my sans's colleague Matt Edmonson he's going to talk to you a little bit about Sans cyber Defense Initiative CDI Matt what separates the Sans SEC 497 practical ENT courses from the other ENT courses out there well I'm the author of the course Matt Edmonson and so I think I'm uniquely qualified to uh to speak to that or fairly uniquely at least and I think what separates it is we actually have the time and the infrastructure to really explain how things work right and that just manifests itself in ways that it's tough to even anticipate right it's not just a collection of sites it's not just a collection of Tricks we actually take the time to understand to plenty of hands on labs to get a feel for it and really help you understand not only oh in this situation I should do this but the why and to better interpret the results that you're seeing right I think one of my favorite parts of teaching is actually taking the time to get to know the challenges the students are facing and to work with them to help develop solutions to their problems I uh recently got an email from someone who was the head of cyber threat intelligence for a fortune hun company and they wanted to reach out and let me know and it was kind of funny they said yeah I'm looking like a genius at work and it's all the stuff that you taught me about persistent monitoring around sentiment and some of those things and so getting those emails like that that just makes your day right honestly it makes your week when you kind of see that and so I get emails constantly from you know current students pending students and former students about some of the problems that they're that they're facing and just helping them come out with Working Solutions I I just really enjoy that I always have and I think frankly I always will the open source landscape is changing quite a bit right right we not only have the Twitter X we uh we have artificial intelligence machine learning really starting to seep in right a year ago it was a buzzword now it's something that a lot of us are using in our daily lives and so I'm excited to talk about some of these new developments at CDI and DC so if you're interested in taking the course with me come to sans.org 497 thanks Matt and uh a second on uh the excellent course um 497 SEC 497 highly recommend it if you're all at all interested in some of the topics that Matt was talking about you definitely want to check that out so I mentioned uh the need for interaction right we want to get that interaction going get that discussion going so uh before I introduce my guests I want to introduce the question of the day which I will pose to all of you that question is what are your top detection challenges and I realize that's a pretty broad question but I I want to hear hear your thoughts on this um when you go to detect the latest greatest stealthiest incidents threats adversaries anomalies what are some of your biggest challenges there and to respond you can either uh join through the slido app or you can use the QR code that's on your screen right now um already getting some responses there definitely on board with the people answer I think that's a pretty common challenge um so please share your thoughts there we're going to talk about some of your responses in a few minutes while we're waiting for that um I would like to introduce my two very very special guests I'm super excited to talk to these guys really happy that we were able to get them on the episode today um so I'm joined today by Zack Allen and Scott poy um Zach is with data dog I'll let him introduce himself in just a moment Scott is with cyborg security and and I invited both of them to come on the discussion today because um I feel that they have a lot to say um on detection engineering and threat hunting um he can speak to it much better than I can but I also just want to give an extra special shout out to Zach's detection engineering newsletter which I've been subscribed to for a while if you're not subscribed to his detection engineering newsletter I can only assume it's because you don't know about it yet so you definitely want to check that out um Scott's got some great resources and some Community things that he and his team are working on he'll talk about as well but um gentlemen welcome thanks for joining today um was hoping you would take a few minutes uh and just introduce yourselves starting with uh Zach tell us a little bit about your background and and the work that you're doing sure thanks for having me Mark hi everyone uh my name is Zach Allen I'm the director of security detection and research at data dog uh I mostly run our detection engineering function when it comes to building out how to protect our customers with our security products on the side side uh I have a newsletter called detection engineering weekly at detection engineering.net and every Wednesday morning uh 8 8:01 am Eastern uh you'll get a newsletter in your inbox that talks about the latest um articles and the state-of-the-art with detection engineering as well as like a threat landscape update with new stories great thanks Zach uh Scott yeah my name is Scott poy uh I work at cyborg security is a senior threat Hunter and content developer um where I basically I'm studying adversaries and figure out how they work and then coming up with unique ways to use technologies that people already have deployed to find and detect them um for post exploitation um specifically um my background I've probably been in cyber security for about 15 years uh I had the fortune opportunity and the stress for standing up a security operations center um and building that out from the ground up which was a fun exercise um but we had a lot of success there um and I'm just love everything cyber so it's kind of my passion this kind of why I enjoy being on here I thanks again Mark yeah AB absolutely um excited to dig into it and for those of you who uh are less familiar with detection engineering as kind of a formal sock discipline or security operations discipline you know this is uh there are a lot of ways to describe it I think but um maybe the simplest way might be you know take what you know about a threat or an adversary and we want to kind of translate that into a detection right um and you know how you do that I think varies that's that's what I want to talk uh with both of you a little bit today um you know the level of rigor and formality in that process and how it connects with other kind of functions in the sock I think those are some of the things that that differ from one team one organization to the next um so I guess uh while we wait for those last few responses on the slido poll to come in um what are your thoughts and I'd like to hear for or I'd like to hear from both of you on this what are your thoughts um or recommendations for teams that are just looking to get started in detection engineering right like I've got tools in place I've got people to operate the tools um maybe I'm writing a few custom rules or tweaking things here and there but I want to make that more repeatable and more more formal like how do I get going how do I get started sure I'll uh I'll jump in on actually Scott go ahead please please all right yeah so you know for me especially in an environment I think what's really important um there's always great resources out there like miter does a great job with behaviors and and tactics that adversaries use but one of the things I think is commonly overlooked is understanding your own environment and processes you know if you're just starting out and you have the tools and Technology I feel like there's a lot to be gained for instance is a as a just a easy example if you're looking for when accounts are being created um well what is the internal process for creating accounts in your environment so you can actually identify those anomalies um so you can kind of keep things simple when you start really understanding your environment and your data as you're kind of getting off the ground um and I feel like that's kind of a great way to um get a lot of good Returns on some basic um simple things doesn't require a lot more work than really trying more advanced type things you can do that's great Zach what do you think yeah I I think Scott has nailed it here um there's a lot of things you can do with detection engineering and you kind of want to road map it out in the sense of starting small and expanding outwards I always tell people that the this concept itself is about scale and so it's measuring where you are now and where you want to go and what tools Technologies and people do you have available to to kind of implement it um if you want to go and check out there are a few resources um one's by Kyle Bailey detection engineering. uh uh this is the maturity Matrix that Kyle wrote to kind of showcase like hey if you're just starting out try to hit these things first and then as you mature here are some different ways that you can get a better process in place for your detection engineering program um other resources um higher dosed from Snowflake uh he did something similar where he he filled out a maturity Matrix and says you know if you focus on people technology and process what's like your crawl phase what's your walk phase and what's your run phase and so you're always kind of looking back at one of those three things right uh people process and Tech and I think it really depends on what stage you're at but what's nice about it is that you can kind of take what you like and take what makes sense and kind of start building from there great that that makes a lot of sense um yeah anyone who's taken uh my ldr 551 class knows that like I'm a huge fan of maturity models and Frameworks like if someone else has created it that's fantastic I don't have to create it I can just use what's there um so I I appreciate you calling out a few examples of those um and definitely agree with Scott's comments you know understanding your own environment um we don't immediately have to worry about thread intelligence and feeds and everything that's out there um we can start by just spending time studying what we know about our own network our own systems um th% and I'm really happy that that you brought that up um let's pause for a second go back to our slido poll I want to see uh what some of our audience uh has come up with in terms of challenges in detection this isn't necessarily specific to detection engineering um but I think you know the three of us and probably many of the folks in our audience would recognize a lot of these challenges so let's see we've got Resources Staffing um you know lack of skilled experts who have the time as well as the skill set to do do this kind of thing uh what else do we have uh misconfigurations false positives too many alerts to deal with so that's something that I think plagues most uh sock teams right too many alerts um how can I pause that or step away to write better alerts if I'm already underwater with the alerts that I have um that's a pretty common challenge um I see some kind of vendor management uh you know making sure you're getting value from your partners who are providing this um interesting any surprises here from uh you Scott or Zach no I mean I feel like I've heard this the same role before um yeah I mean the big thing for me that always stands out and that's the false positive Tendencies right because I feel like detection engineering one of the main goals is to have that High Fidelity so that it can be more operational and not just kind of responding to the the kind of noise in general um so that's a that's a big one I've seen a lot of plac yeah I think for me uh the lack of resources under staffed you know that's a huge question to answer and I think I said earlier to me it's all about scale right and what I've seen successful detection engineering programs Implement are Concepts from other parts of tech so specifically software engineering devops site reliability engineering and what's funny is that these these places and and these um these profession already kind of fa these problems of being understaffed and what they did is they started creating different ways to kind of tackle these problems at scale and so I always tell folks like when you think of detection engineering I think of it as like a three-legged stool we have our security subject matter expertise it's critical making sure we know what the threat landscape is but the two others are statistics and software engineering and picking the parts that that are the best from each one of those and putting them together you can start to see like a clearer picture on how you could take a smaller team and really expand it out and and make it do the work of a much larger team interesting I like that I like the the three-legged stool um love a good three-legged stool analogy no matter what but that's I really like I can't do four-legged stools it's too much to remember it's it's too much that's that's just unreasonable three-legged tools uh for sure so um you know kind of related to that point you mentioned having a smaller uh team do the work of a much larger team through some of these repeatable processes and having the right elements um one of our audience had an interesting question here I'm an IT support engineer in a small company um how can I Implement an open source or free tool for detections how can I Implement them and I think this is a pretty common challenge too um because of course we're talking about security operations teams and you know making this a formal discipline but if you're a team of one or a team of three and you're kind of doing everything um you know how can you kind of focus on having good detections um interested in in your your thoughts on that do you have any recommendations for small teams who maybe you know will never have a dedicated resource maybe can't pay for an MDR or an mssp is there anything that they can do uh to get better at at detection I mean I'm familiar with people using you know elastic stack right um kind of a good community tool what you know it has the abilities to let you at least look at your data centralize your data which is a good common first step for if you want to operationalize anything right um but you also are able to then you know have some basic detections they even have an endpoint that kind of rolls with that as well that lets you kind of get some def defense as well as alerting um associated with it so um yeah I'm a big proponent of like using GitHub for a lot of really cool tooling and solutions for things um so so you know obviously being active in the community looking for resources where people are trying to solve similar problems usually I come up with some interesting Solutions as well so um yeah absolutely right and one of our audience also said don't reinvent the wheel so definitely agree with like using Community Resources um wherever you can um good um and we have some other folks kind of chiming in with with some other free resources other resources um I think that makes a lot of sense definitely um so you know we've talked a little bit about like getting started and some of the challenges um and I know both of you have have spent a fair amount of time working on this problem set right making detection better tying in with other functions um what would you say are some of the mistakes that you commonly see uh with teams that are trying to do detection engineering or trying to improve their detections there are some of the pitfalls there Zach um let's start with you any any major issues that you run into quite often yeah it's it's a great question um I I think some of the fundamental things uh that I've seen kind of go wrong in this field is thinking that more is better and what I mean by that is taking on a lot of rules at once and feeling like you have this concept of good coverage and basically what happens is you start getting runaway alerts I think talked about false positives earlier on how critical they are and how detrimental they can be to a team so to me I think you know teams that are just starting select a core set of problems that maybe your business wants to prioritize top three things um figure out how those map to data sources they could be um off logs it could be email signin logs and just start small start small and expand from there it's a lot easier to do that than it is to start big and then bring it all the way down to something that matters and I think if you kind of take that approach it's iterative and you're able to take on more work and make decisions on that a lot easier than trying to manage all kinds of alerts from all kinds of data sources and all the problems that come with it good yes Scott what do you think um major mistakes that falls yeah so I feel like trying to solve everything with the detection um can sometimes create a problem right and that I think lends itself to some of the false positive Natures like there there are just some common activities in your environment that maybe coincide with kind of behaviors or or things you want to detect from an adversarial perspective but it might not be advantageous to make an alert you have to respond to every single time so you know it's good to kind of divide up that work where maybe you have detections that are your higher Fidelity things that maybe you can even automate some sort of response or enrichments to or put in front of an analyst versus you have some reporting right that you kind of have these daily or weekly whatever you want their reports to look like that are more your false positive heavy type things that kind of give you an idea of what's happening in your environment and maybe that also could sus out things that make and mature that detection so it could be more of a real-time um capability um as an example of of keeping things simple too um is one of the best detections that I I think we we created in our in our sock back in the day just called it chained authentication and it was on the premise of if an adversary gets in your environment and has one set of credentials they're going to try to move as far as they can with that set of credentials likely right and so all we did was basically say if we see a Network login going from one host and then we another one chaining off that host to another host it's worth looking into um and I want to say even to this day it's still catching pin tests and it's just a very simple idea we're not focused on how are they authenticating what could how they are stealing things um and sometimes just kind of keeping it simple understanding what it is you're actually trying to identify um can give you more returns than those really Advanced Techniques that people use you know along that chain and that's why we call it a chain right you don't have to get every single piece of it but once you identify certain links instant response becomes very successful and as far as tracking down all those other pieces so that's good yeah I I like that and I I think I've probably said that many times to clients and students alike which is you know detect the links not the chain keep it simple um don't try to do too much don't try to you know boil the ocean so to speak so I'm glad Zach that you pointed that out um you know and I always uh you know from my my Military Days uh we used to say slow is smooth and smooth is fast right so before we worry about doing all the things or doing the things super quickly let's focus on doing them uh reliably repeatedly few mistakes you know and I think that applies to uh detection engineering as well um but but really good points um just going to some of the audience uh comments and questions um I see we've got some additional kind of resources in terms of Open Source projects and Frameworks and standards um I see uh Yara I see velociraptor security onion um these are all in the chat great resources to check out if you're trying to play around with or get started in writing your own detections um question about uh detecting traffic going through split tunnel and if you're not familiar with that that phrase you've got especially these days a lot of users that are connecting in through VPN to your corporate Network um but depending on the configuration of that VPN they might have you know essentially a doorway into your network but they also have a doorway at home through their their wherever they are their home connection or wherever they're working from so they can also go directly out you know to the internet that way as well hence the term split tunnel we can go either direction and effectively what that does is kind of bridges an untrusted unknown network with potentially your corporate Network and I think you know one of the reasons I wanted to highlight this question even though it's very very specific is how do you detect a thing that you can't even see um and you know back to the Scott's Point earlier about understanding your environment how do we overcome these gaps if I don't even have the data or the visibility how can I write a detection for that how can I you know come up with uh some kind of way to see that so um whether it's split tunnel uh or you know another kind of Gap in your visibility uh what did the two of you tell you know internal stakeholders or clients about these kinds of gaps you know hey I I know this is an issue I want to detect this attack but I actually you know don't have the data aside from the obvious answer of of well maybe get the data um you know what do you what do you say to something like that in those situations yeah for me I kind of fall back on the chain of things right um for one you're kind of looking at okay there's a lot of steps in an attack and then a lot of those steps are kind of repeatable right it's not so much linear as as it as attacks really occur so yeah you're going to have some blind spots and sometimes that's just the nature of what tooling you have sometimes it's the nature of cost right you know if people aren't going to throw money at everything so you had the visibility of everything um so as long as you're really focusing on those key points that you expect adversaries to to fall over um you know essentially when you think of uh detections they're really just trip wires you're putting in different places right um and so you what you have to be able to explain is based on where you have visibility you have trip wires in line so if someone were to get around certain trip wires you have other trip wires they might stumble on um and then just for your your example of the split tunneling you know I've seen some interesting Solutions as well where people just created some basic power cell scripts that like scraped the history files of browsers and then had that push out to um data collectors um to be able to see the activity locally on the host as far as where they're actually going what's going on so if you have the time and the resources you can also kind of engineer around that as well um but just know there's cost with Homegrown Solutions so but you know there's there's things that you know it's you can get creative and it can be fun but you know understand that the time and the resources that take it's all a trade-off Zach anything to to add to that you know accounting for visibility gaps when you're writing detections yeah I I love Scott's analogy of a trip wire here and I I I think to kind of double down on this too um especially when it comes to something like you're worried about exfiltration and it goes through split tunnel instead of just focusing on those trip wires and the network you can provide more of Defense in depth on the resource itself so for example if you're worried about source code leaving or if you're worried about certain um sensitive documents like from a file share leaving the network you can add additional controls on top of that and through there that's when you can find those additional op detection opportunities or that trip wire Scott said I'm stealing that Scott so I'll make sure to Tribute as much as possible no worries uh good deal so yeah and I I I think those are all great great points right it's uh you know it's always a trade-off spending time and resources to detect you know maybe the initial vector or some really complicated Fringe use case may be less attractive than looking for you know maybe you kind of um accept detection later in the chain um but maybe you'll catch more uh sophisticated you know types of threats or different types of attacks that way so um that's great uh questions keep coming in please keep them coming uh these are great questions great comments um there's a question about um using like block listing or allow listing and you know I'm gonna pivot off of that question a little bit um and just say that you know my favorite kind of detection problems are the ones that I don't have to solve um if I can shift something to a preventative control and just like get out of the business of having to go look for that thing all the time fantastic so uh any utility like Windows app Locker or you know anything that can restrict actions and then I can sort of you know maybe not ignore it but take it off my plate I think those are great um so yes to you know allow listing or you know block listing those kinds of things um so uh let's see another question here um about Sigma and I wanted to highlight this question because there are some really great projects out these days for blue teamers like Sigma Yara of course has been around for a little while Jupiter uh notebooks are making kind of a a Big Splash in infos SEC now these are all open source projects designed to standardize analytics so that you know I can write a sigma Rule and theoretically I can use that same rule logic in Splunk I can use it in Q radar I can use it in elastic you know whatever I happen to have um so there's a question about you know what are your thoughts on utilizing Sigma rules um do you utilize them so let's start there do either of you utilize sigmo or any of the other tools you know Jupiter Yara all of the above if so why if not why not yeah uh at least on on my team side uh open source is super super important and it's just a really great way to kind of get started on on a specific problem um I think uh you know Phil venables wrote recently about industrial versus artisanal Security Solutions and to me open source is like approaching industrial which means that it's standardized we can take it bring it in we can kind of jump ahead by 80% and then we have to finish that last 20% and so the way I would suggest users to implement Sigma is again don't take the whole thing and and and deploy it instead pick things that you like and begin that in process and make sure it matches kind of like with the business objectives there I think without Sigma it would be a lot harder for all of us to do our job and especially in the vendor space because you see a lot of the um the backlogs and a lot of the inspiration on vendor-based detections come initially from Sigma and it's that open source mentality that I think is helping protect a lot of people yeah I would agree I I love Sigma um one of my favorite things about Sigma that you know you kind of see a lot in in our industry in general is it kind of standardize some things right it gives us kind of a Common Language and it breaks down that learning barrier where maybe you're not familiar with how to write rules but if you learn Sigma you kind of understand the logic that goes into a rule and what may be required in a rule um but I think one of the pitfalls that some people fall into when they just solely rely on Sigma is not understanding their own tool sets right so I think Sigma like you mentioned like the 8020 like I think it gets you 80% of the way to help build the filtering you need in a detection but there's some great uh transforms or aggregations or things you can do in your tools to maybe increase the Fidelity or you know you may be able to pin that to something else to make a a more reliable detection or or focusing on a more specific problem um so I would I would definitely people that if they do rely on used sigma don't forget about what their tools are actually capable of um when they're trying to import or Port those types of DET detections in so fantastic I I love these answers uh so much better than any answer I would have given to that that question um and and very diplomatic right they there it's not 100% solution as things often aren't in uh Inc but uh I really like the the 8020 kind of logic there is it's great for sharing great for standardization and it gets you uh there much faster but you still have to think about what your tool capabilities are and how you can get the rest of the way so that that's awesome um I love that and if I could just kind of like record both of those answers and then dub uh AI over when you said Sigma I think maybe the answers might be very similar because there's another question here from a member of the audience about how do we leverage AI artificial intelligence and detection engineering and I can I can almost hear everybody's Collective like heads explode or like melt when they hear AI um you know the groans but you know I I don't know if either of you are doing AI or any kind of Automation in detection engineering today but I guess to broaden that question slightly how are the two of you using more Advanced Technologies whether it is some kind of more advanced automation or maybe AI you know ml those kinds of things are you using any of those in detection engineering are we not quite ready for that would you think Scott start with you yeah so I kind of have two takes um one I don't think Ai and ml is the Magic Bullet but I think there's a lot of things that does really well because really what it is is just Advanced statistics right um so when you have an environment that doesn't change that much um then it's really good to catch outliers in in anomalies that way and that's how I kind of picture the best capability of AI I know they're building models and integrating more models together and it's going to be a more stronger solution but to today I still feel like it kind of sits in that realm um secondly how do I use AI so I'm a big proponent of chat GPT um but mainly for uh emulation like creating emulations for types of attacks that I wanted we haven't really touched on emulation but I think emulation is is very very important because sometimes when um an advanced problem comes your way and you're able to figure out how to at least emulate the data to see how your tools um can detect or see that type then you can understand the data around it and build better detections that that may not um be as obvious so I remember back in the day when I was looking at DNS tunneling as an example and I was thinking this seems like such an advanced attack like you know how does someone leverage this and they can get around all your defenses by using you know public DNS infrastructure and things like that but then when I was able to emulate DNS tunneling it was very obvious when I looked at it what DNS tunneling looked like it was a ton of DNS requests a ton of different random subdomains to a common domain and there obvious patterns in there that if I didn't look at that I'd still be kind of stumped like well it seems like a very complex problem so I think when I use chat GPT to help me code some things um that's kind of how I leverage that for emulation purposes and help solve those kind of what I would say would be complex problems okay yeah Zach what's what's your take on on AI automation I uh it's a good answer is to say it's basically statistics it reminds me of that what is that Meme where like uh Fred from we do like rips the mask off it's AIML and he takes the mask off and statistics underneath what are your what are your thoughts Z Ai and ml is just a different positioning for statistics you know um but basically uh what I've seen it be very successful on Scott was touching on is baselining and so how can you describe what's normal and then how and then using Ai and ml to do like a continuous prediction of what's outside that Baseline I think a great example of this is um impossible travel somebody's usually logging in from New England then all of a sudden they just logged in from Russia and there's no aircraft in existence that can transport that person that quickly to go and log in there and so that's one example there um we're using Ai and large language models at data dog and it's been particularly useful for summarization and so you have a huge page of like logs or just a massive search results or an incident with a ton of different work streams going on you can ask a large language model for that summarization and I'd like to see more of that when it comes to building detections like you issue a query or you get a um a list of true positives and false positives ask chat GPT hey what do you think this is how would you split this up what's legitimate what's not the other one that I've seen and I've talked with uh several detection leaders is they're using it for translation and so taking something that maybe uh so Sigma has the their backends their pi backends to go from Sigma to a different um uh search query language but if you don't have that backend how can you con do that one toone conversion and I've seen a lot of folks use large language models to do that conversion and then implement it in a way such that they'll run the rule in the source language translate it run the rule in the destination language and then compare results and if they're good they'll shift the detection in production with the new um new query so I've seen those two things be used pretty successfully here great great great examples um and quick time check we have a little bit less than five minutes remaining and unfortunately I don't think that's going to be enough time to get through all of the uh great questions and comments we still have in the chat um you know I I think kind of one thought that is relevant to some of the comments and questions that I'm seeing is you know when you are creating new detections regardless of how formal your process is for doing that um I think uh it's important to remember that you're essentially creating more work and you're creating work Downstream so um you know we've talked a little bit here today about emulations Scott you mentioned the importance of testing and emulation Zach you you just talked through a couple of examples where you're doing testing um you know you're kind of enriching the data as much as you can and I think all of those things speak to arming the analyst or whoever is responding to that alert with as much information as possible and as high of a quality of alerting as you can um because you're creating more work for them and as we've already discussed as we've already uh heard from our audience having time and resources you know that's that tends to be one of the largest challenges I think in our word cloud that was the biggest one so um just kind of a parting thought here while we're on parting thoughts um I'm sure we could talk about this all day but um Zach we we'll start with you any parting thoughts you know if I said what's the most important thing I need to know about detection Engineering in a minute or less what would you tell me um it's a great question I'm gonna back yeah it's okay it's okay I'm gonna lean back on um uh it's a scale problem and the scale as as Scott talked about in his first answer here it just really depends on your environment and how much want to take on so start small and expand and think about those maturity models crawl walk run don't I I I strongly suggest that you don't jump right in and take on as much work as possible because it can get really messy really quickly and so making sure that you could be iterative and show quick wins in the long run pays off a lot more than trying to get something massive started from day one okay good Scott how about you yeah I would say to to kind of boil it down and kind of harp on some of the things you were having with the lasting remarks there is I mean keeping it small and simple um you know to get good ground early on and then when you spend time for automation don't spend so much time on the response part spend more time on the contextualized part um I really feel like if you're able to automate some contextualization then you can um make your detections better or at least make your responses to your detections better um and you can get a lot of Runway that way as well okay great um about minute and a half left uh Scott anything that you want to highlight or or plug in terms of resources for the community or things that you're working on that might be of interest to the audience yeah so we have a lot of free resources as cyborg security that kind of go into like actual workshops Hands-On things for threat hunting which I think is really good we also host a podcast it's out of the woods threat hunting podcast where we just want to get it's not season professionals we like really new people people that come on and we like to interact kind of similar fashion here and we really talk shop security and focus on threat hunting um and also touch on some of the top stories every week so yeah excellent good Zach how about you yeah uh if you want to check out some more detection engineering resources I run a newsletter uh I plugged it at the beginning of the podcast detection engineering.net it's called Uh detection engineering weekly um and I also uh run the security Labs team at data dog and we try to publish a lot of very technical topics and blogs on this topic plus a lot more and so security labs. datadog hq.com is where you can go find it and we're we're trying to publish as much as possible and give those resources out so we all can get better together outstanding outstanding thank you yes uh plus one go subscribe to Zach's detection anding newsletter um it's uh it's fantastic so um before we go one final thing that I I want to plug two things actually um I mentioned that I'm the co-author of ldr 551 building and leading sock teams um my co-author John hubard and I just completed uh in the past couple of weeks a pretty major update to the course new Labs uh completely overhauled cyber 42 simulation game that we play in the course so you know we talked about detection engineering but in this class which is five days long we talk about everything from build building your Team Staffing to collection detection response continuous Improvement many of the topics we mentioned today like adversary emulation uh measurement maturity scales building the right culture training your team all of these things and more we cover in ldr 551 so if that's of interest uh definitely check that out um I also want to call out the Sans detection engineering survey uh which is I believe still live we posted it um couple of weeks ago and uh that's out there we've published this detection engineering survey to kind of take stock of what some of the common challenges are some of the things we've discussed today and uh of course as with the other sand surveys we'll be releasing that data set when it's ready but I highly recommend checking that out and contributing to it responding to the survey questions um if time perit permits so with that uh that brings us to the end of the episode I want to thank my guest again uh Zack and Scott thank you very much for joining for sharing some of your thoughts and your time um want to share uh thank the sand staff uh Thomas Stephen and Michelle Brett everybody behind the scenes thank you very much and uh thanks to all of you for your comments and input in questions have a great day have a great evening we'll see you next [Music] time [Music] [Music] oh