Defeating Attackers with Preventative Security

Key Takeaways

[Music] hi my name is Jake Williams I'm here at the SANS Institute today joining me is Ismael Valenzuela he's a principal engineer at McAfee as well as a certified sans instructor and we're here today to talk about some of the use cases for EDR Ismael thanks for joining me thanks for having me so we're here today to talk a little bit about EDR endpoint detection and response tell me some thoughts about this what do you think about EDR overall I think they're just becoming a very key technology within security operations and especially as we've seen in the last few years with people adopting a seam and doing a lot of you know correlation in the scene building use cases people starting to find out that they don't have enough telemetry to be able to triage certain alerts like a connection going out to suspicious IP address right and then they have to people to the endpoint to see a lot of the telemetry that it's it's they're so easy our provides you with all of that information yeah it's a really interesting point we were talking a little bit earlier today actually about that problem right where the the Sam or the seam depending on how you pronounce it here the the Sam is a very passive tool where we have to plan in advance what we want to log but very often we haven't been able then to log enough data to provide all the context around an intrusion right now absolutely it's like there is various reasons from by which for which you can not feed the scene with absolutely everything like the telemetry that coming out of an endpoint it could be you know it's a lot of data but at the same time some of that data it's it's it's very tactical and you never know which piece of information you're going to need to be able to triage an incident so having things like DNS requests like process execution like network connections user logins it's critical to be able to triage and do an investigation and to figure out what's the the next step in terms of reaction sure we definitely find that as well right the best laid plans best laid to rest of course and and we prepare what we want to log with the sim but we often find that in the middle as you pointed out in the middle of an intrusion investigation we have some tactical need for additional data and I feel like that's edie are plays in it it helps fill that gap and provide context around the alert that I'm actually seeing in the SEM absolutely it's it's not a thing of seam or EDR they both play a key role within secure operations and it's about analytical pivoting rights having the right information to start the investigation then pivoting on to whatever you know kind of repository has that that data so they both go and like really well hand-in-hand yeah do you know one of the things that we run into a lot is is being able to prioritize and contextualize alerts that come out of the sim one of the big problems that we run into and we've discussed this in the past obviously as well one of the things that we run into a lot is that maybe I have multiple alerts coming in and maybe they're all related to fishing obviously if I have one of those that's targeting the organization's CFO I want to address that before the one targeting let's say the the janitor right of course one of the issues that I have there is that I see all these alerts they come into the SEM they all are showing machine names and those machine names don't really mean anything to me and and I have this problem with inventory can you speak a little bit to how EDR helps solve that that issue of contextualizing you're right it's a big problem especially as people start to use c-more all the technologies right how do you prioritize when you have ten thousand dollars per day so EDR helps with with that context right in the end we usually say you do an investigation it's all about context you need to understand just say this is a high priority system or it's just know from a from a tier of user or a segment where that that asset it's not that valuable so that kind of surgery with privatization helps you with being able to react to things as soon as possible an EDR can give you a wealth of information based on all the data that is collected over time and based on that context that will be on the endpoint it will be harder to get from other sources sure you know one of the problems of course that we experience in industry is that as we're trying to prioritize this alerts and even identify that that that alert did come from the CFO's machine that that is a a much higher priority alert oftentimes we're dealing with outdated inventory data right that became stale the second it was completed and and I believe EDR helps us a little bit with that to look at right now who's operating on that machine absolutely the the continuous monitoring piece is a very important one and it's not just the the raw artifacts that are relevant as we say but also the context the time context the asset value where it is located and an EVR technology can can help to provide that kind of visibility where it's it's very difficult to obtain otherwise sure I'll tell you one of the challenges that we experience in our investigations is DHCP and we see DHCP the moving target of assignment of IP addresses - host names that becomes a big problem for us and can you speak a little bit to how we can how EDR can help eliminate some of those issues that we would otherwise have correlated data in the sim well so as you say write again and that they don't the same could be challenged in one of the things that EDR can help with is to run those queries in real time right so it's no all about just historical searches which is obviously very important to be able to do hunting and to find out what happened with that system over time but also real time so you want to be able to just query all the systems you have connected on to the network right now to find out what they're running what's the IP leads that they have at this particular time and you will be able to get all this information regardless of whether you can bring it into your team or not right yeah and that that's something that's very critical to note right we were probably we may have all of the actual data available to us in the SEM often we don't but but let's suppose for a moment even that we did even if we have all the available data in the sim with DHCP one of the things we run into is that maybe 24 hours ago a machine with a particular hostname had one IP address and now this morning it's been reassigned a different IP address and and now I'm trying to correlate these flow records across multiple different and logs across multiple different IP addresses in the SEM whereas in my EDR because I'm querying against the endpoint not the IP address assigned at the endpoint in a given time I have a little bit better context and definitely historical context as you pointed out with an EDR console and in some cases when the EDR solution goes hand-in-hand with the scene there's good integration between both you could actually go from the edr solution even bringing me with ends from sim that when correlated with the investigation you're doing so again favoring the analytical pivoting methodology that we that we usually recommend to follow absolutely and that that's at that's a hundred percent critical it's it's not just about a point in time alert really as we talked about ETR that the big benefit that we see with EDR is being able to contextualize the alerts that the alerts that we're seeing now I do want to mention that EDR is not always a panacea all right we we run into problems sometimes where junior analysts I think in particular have some struggles or some additional challenges in operationalizing EDR can you speak a little bit to that yeah absolutely so many of your solutions there they've been designed from the perspective of we have a skilled analyst a threat hunter or someone that you know based on just having access to that wealth information now you can go and hand write which is great like for many of us but the reality is and not everybody have those levels of skills right and of course it's not about substituting or replacing the analyst we still want to train right we want to train them we want to give them you know the skills that they need but the reality is that also the tools can do some cases a better job at guiding the the analyst when being confronted with so much information just to pinpoint to highlight hey this is important this is interesting not just in terms of alerts for things that aren't clearly known bad but also for things that could be suspicious like and also what to do after that how to follow from that point all right so you brought up a couple of really great points there and I'll first kind of pivot into the you know into the response piece right Edie ours is endpoint detection and response and and very often we don't talk a lot about the response features right when I when I look at a lot of marketing material I see a lot of we detect we enable you to threat hunt we and I say and then what right the are is for response all right so you know as we talk about junior analysts knowing what the next response feature-rich response feature to activate can be a challenge absolutely as we always say like detection it's it's it's great right it's a must but then you have to do something so if you detect something bad pat yourself on the back but then do something about it right and try to mitigate the impact of that so some easier solutions they mostly they are solutions they provide like a simple can of capability to just quarantine device I salute the device which is great like you know we've been involved in many instant response cases we know how important is to have that ability to contain an incident but in many cases see there are solutions can also provide you with a more advanced features need to run your own scripts to collect some artifacts suspend terminate a process or do some more advanced features sure and that that's absolutely critical you know as we talk about the response features I agree that the having those response features again is or wealth of different response features certainly can help us help us to choose the correct one coming back to the junior analyst though how does the junior analyst particularly add more response features doesn't that make it more difficult for junior analyst to get up to speed with an EGR system well it's a good question I think it depends a lot on the context right and I think easier solutions they should be able to provide the analyst with enough context to understand ok so this is a raw event right let's say and it's a network connection or it's a registry key added to something that could look like persistence but now we're using the word persistence right that's that's a tactic by itself that we used like the mitre attack model how does that fit within the the attack chain right where are we and again back to the assets right what kind of assets are affected so if this information is presented to the analyst in a way that it conveys context it tells a story it makes the way easier for the journalist now to make a decision an informed decision on what am I looking at you know guiding the analyst into what is that I'm looking at in this particular event one of the particular things that we see with a lot of our clients with EDR systems is we find that unlike many other security tools EDRs actually cross to workflow boundaries the detection side typically would have been an information security role whereas the response typically up to this point has been a largely an IT role all right IT obviously in taking some response takes risk for a system downtime right killing the wrong process quarantine in the wrong system can have serious business impacts and we find that one of the places that junior analyst sometimes have trouble there is is even if they have the technical capability to let's say quarantine ax systems suspend a process there they were afraid as an information security asset already moving into the IT realm right they find that it's a little bit more difficult to even if they feel like that might be the right role taking that risk is is there anything we can do as as an information security community to enable them to make better decisions and and help them along that process of getting to the right remediation steps I think it's a very fair point as well I think it has to do a lot with getting the tools to allow the analysts or the stock in this case to be able to adapt to their workflows not the other way around right and because they're in reality each shop each company each organization just secured operation team it's going to be completely different so in some cases you say like IT would say no no remediation here in all the places you know there's some limited response you can make and then when it comes to maybe the you know reimage of a bar so that's that's IT but I think that the tools should allow the shock to be able to adapt to their workflow it's not the other way around where you have a tool that is constraint seeing what you can do and that's the only thing you can you can do with the tool really you know I think it's a great point right so here we're talking about that you know we kind of have counter points here about flexibility right the more flexibility we have with the tool the more capabilities we have in the tool the easier it is to adapt the tool to your workflow and not try and write a workflow around the tool itself on the other hand we of course have the challenges of the more options that I'm giving an analyst particularly more junior analysts the more opportunities we're giving them to fail right and of course that that's not what we want you know from a tooling perspective one of the things that we found is that any tools that try to give an analyst a hint about where to go next seem to be better adopted by by organizations and definitely provide more consistency in response can you speak to that a little bit well you know that for some of us we've been doing these for a long time you develop this kind of gut feeling right when you see this it's like I know this either before or I guess I know what's going on here the Doudna Islands doesn't have enough experience to be able to do that so that's where the tools as you say can do much better in telling you guiding you where you are in the investigation right so for example we talked about investigation guides where you get a finding which is not something that it has to be malicious per se but it's like hey look here there's something here that could indicate potential persistence or lateral movement or something like that so one of the things I I feel that during a list start liking the most is the whole concept of building hypotheses and questions that support those hypotheses so what do I have to ask myself to be able to take the bias away from me right everybody has and and make a good consistent thorough investigation and tools can help with that too by providing you know this is help asking making sure that Jimmy Islands can ask the right questions as he goes where she goes through the investigation yeah that's that's that's a great point right so basically as we talked about this what we're really talking about here and forgive me for you know if I'm paraphrasing incorrectly here but it's really an expert knowledge system so and providing a decision support system in particular to our junior analysts right so so taking wrapping up the years of experience or decades of experience right that our senior information security professionals have and arming our junior analysts with those as you point out falsifiable hypotheses right that they can come in and test and say did this happen or did it not and and again an EDR system allows us to do that it's very interesting that from you know as we look at a sim typically we don't have the ability or it's not as easy to come up with those hypotheses right because either you had the data or not as opposed to being a very passive system as opposed to ET are being very active yep absolutely so again I think this place for both both tools is just a matter of knowing how to leverage them to follow your workflows and in many occasions we're gonna see like people looking at the same you know finding some alert and then as we said before going deep into the endpoint and then using that information to also correlate other artifacts they find from their investigation maybe the EDR ticket some automated action to at least quarantine wild situation is well assessed scoped where it's it's something that either I can help to together with some other information you can get me from the network or other tactical sources it might be going into the same once you have the full picture you know do the full response and remediation depending on what you what you found sure to borrow a military analogy we often find we often talk about EDR at least internally in my organization we talk about it's being a force multiplier right it's a tool that allows a single analyst or a small team of analysts to project force in a way that that they wouldn't be able to without without that tool because of course again they have the ability to perform the active queries as you pointed out that quarantine during that triage assessment is absolutely critical I I think back to a number of the ransomware cases that we've worked and and that we've seen where there are early indications early on in the investigation that something's amiss here IT is is seeing it IT security is seeing it and they don't have the ready to go remediation actions to go let's quarantine the subnet let's take these systems off we know that this particular process is related to the ransomware kill this process everywhere we see it across the enterprise and if they lack that tooling that can turn a small incident into a very large incident very quickly right absolutely find out which DLL or what DLL what system is running this DLL right more probably into into much in my environment or what machine is listed in this connection or that's that kind of visibility and the flexibility to to do that it's it's important you know a scenario that we've talked about in the past here to actually involves network traffic and this is a very common Alert that we get on the sem where we see a outbound connection to a known malicious site or suspected malicious web site let's say or a domain name or IP address that we've identified through threat intelligence and we've got that alert and we can tie that back to a system and that's great but but can you speak a little bit to how EDR can really contextualize that alert on that system yeah usually when you follow that kind of workflow the investigation stops there so you potentially maybe take a snapshot of the machine right find out what's running they're prone is that you're not going to find what was running before in the past so both things are important right you want to know what's running right now on the system but you also want to see what was running before for the last I don't know 24 hours 48 hours we didn't go hunting for specific IOC s and bring that into some kind of investigation window where you can now start mapping artifacts building relationships in some cases you know using reputational sources to find out what you know about some specific artifact and that that helps to Dallas to start painting the picture of what's going on in my my organization yeah definitely and and to that point create a problem that we run into a lot is alert fatigue right and so here in a particular scenario let's say we've got that reputational hit on a domain name or an IP address and it's one of let's say a hundred alerts that we got today maybe one of my analyst picks that up in six seven eight hours all right they go then over to the system and try to get some information about the system and the connection is no longer active right speaking back to its EDR and maybe even doing some automation around that maybe the alert pivots into the EDR system to go query data from the system live right absolutely and then of course that gives us additional context and especially these days with the techniques that we know that are being used like living off the land techniques where there's no clear like black or white good or wrong it's it's all like gray right so it's it's on the details it's in the command lines it's under the kind of artifacts are being used how you know what dll's are being loaded into each each process so you have to have the granularity and that kind of level of visibility to be able to figure out is this a false positive words to say something I'm really concerned with I love your example of talking about DLLs on the system and I couldn't come up with a better heed er use case than that and by the way we didn't rehearse this at all about the DLLs this is so perfect when we talk about dll's on the system this is an example of data that we couldn't possibly log for every system running on the network it's very very rare in the case of a SEM to even have process level auditing let alone DLL level auditing for all my workstations in a network it's very very rare for that to even be forwarded to the Sam it's simply too much to look at the individual dll's in a process but once we identify one that we believe to be malicious or otherwise suspicious I now need to rapidly find out where else in the network is that act and this simply is a question the SEM isn't gonna be able to answer absolutely like if you look at the open source tools that are available and that you know we're teaching this this sales classes as well about to do threatening penetration testing many of them like Empire PowerShell you can you can just embed one of those dll's into a legitimate process unless you have the granularity of looking at all DLL that are loaded there you're not going to see this the evidence or the potential command and control the processor is running the command and control and that's something that it's available to everybody everybody can use it so of course more advanced actors would also be using those techniques and even more advanced ones sure and this isn't just an example of advanced attackers necessarily you know this afternoon I'll be teaching 25 students here again how to do DLL side loading as well as how to use PowerShell Empire specifically right so you gave it a great examples here of you know we're walking out now with 25 new red teamers that red teamers penetration testers etc who have these skills to go into these environments and of course if we're teaching that here on on day 2 of a class you can imagine that there are far more advanced techniques that attackers are using that again absent an IDI are absent something that we can actively query for those indicators are compromised I feel like we're hunting in the blind a little bit purely relying on the SEM absolutely and I'm talking about now how to leverage that in the end what we want to to build this better resilience right better shorten the time to detection and reaction to be able to mitigate impact and EDR tools can help you also to provide some of these metrics for example like you know how you doing in terms of how long it take you to conduct an investigation the quality of the investigation things that will help you to move the needle right is the stock manager or the c-level executives to move the needle in the right direction which is ultimately the the angle right of security operations yeah we talked a lot about meantime to detection and meantime to response as being two very important cybersecurity metrics and a very good evaluation of the quality of the cybersecurity program very often I find particularly folks will anchor on meantime to detection but you have any thoughts on why they're anchoring Oh meantime to detection and not meantime to response that's that's a good that's a good question the problem with meantime to detection is it sounds like a great great metric to have when it is problem is that it forces you to have forensic capabilities which not everybody has right because you need to understand when that artifact was first landed on the system and again that's something else that EDR solutions can can help with but if you detect you do a good job at that you know the new forensics all the advanced stuff but again you don't have the ability to remediate and to do this in an orchestrated way in a way that is integrated with the rest of your ecosystem in a safe way as well - then there's no purpose right because it's all a lifecycle so you want to detect you want to react then you want to build better prevention and detection by with the rest of the tools that you have in the stock to again minimize risk sure you know we often analogize the when we talk about you know mean time to detection as being our only metric right I like to analogize it to law enforcement or anything for instance maybe there's a mugging out on the street and a police officer says hey wow look at that that's bad and then stands there and does nothing right and of course we're gonna do that in cybersecurity we want to have those as you mentioned the ability to the ability to react and you know I often find when we're reviewing programs cybersecurity programs for our clients that they often focus on mean time to detection because it's easier metric for them to collect because they have a sim but they haven't invested in any orchestrated response as you talked about and as a result their response numbers are being self-reported they're relying on IT to say this is how long it took us to do that response right now those numbers obviously get there's a lot of subjectivity in those numbers I find that's that clients that have EDR systems and that have orchestrated the actions between detection and response I find they have much better meantime to response metrics not only the metrics themselves are lower but that the quality of the numbers that they're providing are better absolutely totally agree well you have any other thoughts here about EDR or do you want to wrap this up what do you think well thanks for having me I think I think was awesome really enjoy it now this has been phenomenal and I really appreciate you coming out and contributing this information back to the Sands community thanks for having me it's a pleasure thank you [Music]