Burning Down the Haystack - SANS Security Operations Summit 2018

Key Takeaways

so we're talking about burning down the haystack right we're trying to get rid of all the hay neat Mythbusters did a thing on this right where they said hey how do we find the needle in the haystack a couple different ways they actually found the best way was if you take the hay and you put it in in pop of water and you let the needle sink to the bottom but who wants to listen to a talk title titled let's put the hay on top of the water right find the needle no it wasn't as cool so burning down the haystack we're gonna get to that alright so first a little bit about me where did I come from um army I spent some time in the Army did Electrical Engineering College Cisco Networking I got into InfoSec after I started breaking some stuff figuring out how important it is to build things right is anybody remember the first time you cracked WEP anybody that was that was a great experience when it's like this is actually being still used in the mid early to mid 2000s and so I'd find some networks and it was like oh I can crack WEP that was crazy practicing on my home network what are you guys thinking anyway then so I started realizing that building stuff was more what I wanted to do than breaking stuff but seeing how easy it was to break decided I need to build so I did some stuff in the DoD as well as electricity sector started building and assembling the sock tool sets and once I started putting the things in place right your sim your IDS's your log management your infrastructure all that stuff the tools everything started blowing up and then I realizing hey I need to do some Python scripting so that I can actually maintain these tools and I can actually do something to make sense of all the information that's coming out of this right so that's fast-forward that's what I do today full-time is work on helping organizations automate their processes right a couple certifications there the haystack let's talk about this so this is what it feels like to me whether it's like the you know you can think of this as like the matrix or your alert console on your sim right they bit about the same so you start seeing this stuff just rolling through and you're like how do I deal with this oh my gosh it just starts becoming noise after a while you know I used to think I could see the blondes and the brunette's and the redheads but no I can't so you know the guy from the matrix thought he could do it but you really can't at some point it just all becomes noise so we've got to figure out a way to deal with the basic stuff right I love that I'm following the guy on the threat hunting stuff because this is not threat hunting okay this is how you free up some of your time to do threat hunting all right is you've got to deal with this stuff like you talked about this is the toil this is the grind that we all have to deal with so how do we get past it what do we do well we could hire more people right this that's a great it's a great thing to do and everybody's hiring but as this my favorite comic illustrates that's a hard thing to do so good luck with that right even if you get the headcount at your organization trying to hire maintain retain the talent and do all the things that you need to do to keep those people at your organization it's gonna be difficult what's next more better tools that's me right there by the way I was the land itis that said like I want the more tools give me more stuff right or blinky lights let's get it in there so get all the things in pretty soon your signal-to-noise ratio just gets blown out of the water right you've got all this fancy stuff you got some things running off giving you the information you got to stay ahead of the attackers you got to do the next best detection thing but the signal-to-noise ratio is the problem all right of course we need better processes but every organization has them right and like my big Big Lebowski that's just like your opinion so when somebody comes in and tells you how to do your process better right everybody's got opinions on how to do processes better so we got to get them better but that's not gonna fix the problem so is it time for something new right it's not really new Automation has been around for a while but just not very heavily leveraged in security so how do we think about this and bring it in and security what we want need what we need to do let me talk about two aspects of it right my colleague Rob talked about soar yesterday where you throw in the R on the response but I'm gonna focus on the automation orchestration stuff today right so where do we actually orchestration sitting between the tools and the automation doing it at machine speed so ladies and gentlemen brace yourselves all right so definitions machine based execution of one or more security actions that's the automation piece right a lot of people throw that term out today very frequently when we think about automation it's not you know you could think of IDs as automation you can think of sim as automation all these tools have an aspect of automation but when I'm talking about here today is tasks that would normally have to be accomplished by analysts right things that somebody would manually have to do we done a pretty good job of automating that blog analysis and stuff that's what we have sim for we don't we don't watch packets as they stream across the network anymore right that's what IDs is for that's what some of these other things are going orchestration is the glue that sits between the tools across a complex infrastructure right so in typical sock you may have 10 20 30 40 50 tools right you've got all these things that are that provide information that provide detection capabilities alerts information they're part of your investigative sources they could be other IT tools like your email system your server logs your LDAP right your Active Directory so you're trying to get information from these and as part of your investigation you have to orchestrate across those things now a couple quick caveats what it's not not a replacement for human analysts okay we're not talking about getting rid of people it's not a silver bullet or unicorn right and it's not more blinky lights this is this is not something that you set and forget it's not something that's gonna solve all your problems this is something that's going to help you save some time right that's really what it boils down to all right so when I said not a replacement human analysts let's talk about this for a second David autor did it did a TED talk in September 2016 really great talk you should all check it out if you have it he says why are there so many jobs still if we've got all this automation right across industries he uses the example of the ATM and the bank teller machine right where analysts are I'm sorry bank tellers used to have to distribute cash right well in 1970 when the ATM came along bank tellers were really worried about well what's gonna happen to my job right it's gonna go away well the number of bank tellers between 1970 to 2000 s early 2000s actually doubled right so you had way more of those bank tellers but their job didn't look the same right so what it does is it evolved the job so it doesn't take it away but it says now it's not just looking at some of the same tier one alerts the same phishing emails the same things it's actually making it more complex what you have to accomplish right so it takes a higher level of cognitive ability more the soft skills other things like that we've talked a lot about in this conference about interfacing with people right using that human API that's a big thing that's going to become more and more necessary in our analysts all right so like I said I hinted that earlier and Rob talked about this yesterday so I won't spend too much time on this when we think about the OODA loop right we're trying to make this tighter that with the idea between behind automation orchestration when we're reducing this time that it takes between what something comes in and we see it and to the end point when we act right if we can keep that Oda loop tighter then we're gonna win so how do we burn down a haystack right let's start lights up stuff on fire what do we what are we trying to accomplish right let's think about our goal are we are we trying to triage phishing emails this is a you know one of the top use cases for automation are we trying to remediate common malware are we trying to do routine Intel updates or common ticketing tasks right all these things that we need to do that happen over and over again but let's focus on one it's also helpful if you can think about the scenario where you spend the majority of your time right by saving or by focusing on a scenario that takes you a lot of time then you end up saving more time right and then that has downstream effects so follow the steps go to the whiteboard right start diagramming it out start figuring out what it is that actually makes up this process now you know you're building your documents and your diagrams you're kind of thinking about a flow chart right you're thinking about the different actions that you're taking you're thinking about the different decisions that you're doing right as you're walking through this think about the time that you're gonna spend on each step and then you're gonna be able to calculate ROI and say look by automating this process this is what we get back this is the amount of time that we're saving from doing this right and then if you multiply that by the number of times per day then you get the the goal so the end of the end state is that you get playbooks right that you build these things out together and these these can be built in you know you can diagram it on whiteboard you can do it in Visio you can do wherever right you're just trying to understand your process and lay it out step by step all right so how do you actually get started you need a couple of repetitive processes right it's you can't really automate something if you don't know what you're doing you have to peel those back so let me talk about that for a second you you you know kind of like I hinted at the previous slides you can't just kind of surface-level well I just want to you know remediated all my similars that's not really gonna do you any good you didn't think about what are the individual steps right do you do you take a similar do you look up information on virustotal do you do internal context or you look up stuff on dns find out the hostname find out the owning business unit do you have to contact someone do you have to send a notification what are the discrete steps that you have to go through right so really peel that back now if you want to do this effectively you also need security tools with api's if your tools don't have api's you're not really gonna get very far right you're not gonna be able to get the action against those tools that you need to accomplish more more tools have these so you're in luck right people are starting to get the message is that they have to have this right they don't have api's then they don't scale well you also need some scripting knowledge right so I'm personally Python is my language that's becoming the most popular language that's the latest and greatest is that it's you know exceeding some of the other ones out there like Java and JavaScript familiar with requests model module in Python to do API interactions have some sort of data management web service how are you gonna interact with this right so kind of building the framework of those pieces you know other languages are fine too like if you want to use PowerShell if you want to use Ruby you know JavaScript there's a lot of things out there that could do this type of stuff but I've found that Python has the most easy to pick up modules and it becomes more like Legos right building blocks just like my five-year-old son loved me some Legos so the other thing I'll mention too is that optionally there's there are platforms out there right there's free community editions so it's literally you go out there and google it community security automation orchestration boom you can download at least two different ones that I know about the top of my head the benefit of these is that they're gonna rapidly accelerate your time to value right they've built a lot of the plumbing that you need to actually talk to your security tools the common ones that are in majority of socks right the major cots vendors so it's going to reduce the amount of scripting knowledge you need you ate it right you still have to know what you're doing but the the low-level plumbing is there to really accelerate it all right so now let's get into the fun part everybody really loves this the the success example right so do your best success kid here we go all right MSSP security event triage this was we were working for a managed security service provider and they wanted to do the same thing every time a certain set of events came in right they found that they would want to do a check in their ticketing system to see if a ticket had already been created if the ticket was there then they would link that to the incident if a ticket was not there that would create a new ticket right they would also query their sim to get the results that applied to that particular event and then would update that ticket with a CSV of the query results right they would have that ready to go that was something that took a long time that every time an analyst log on saw a ticket they would have to pivot over a query their sim get the results back and make it look at it so by doing this by having it in the ticket as soon as the analyst came to look at it that saved them about three to five minutes per event well for an M it says P that's huge right because they're dealing with tons of these every day so great success story there another one was phishing email triage so like I mentioned earlier this is like one of the most common use cases right this is something that even more so than your similar console right you're dealing with these phishing emails that come in from other people in your organization they say hey is this phishing it's from my cousin Bob or its from the Nigerian prince or whatever right and you've got to sit there and go through and find the you know link in it and you got to detonate that or you know do some screen captures of it or you got to take the attachment and actually figure out what it's doing you know so rather than have to do that manually why not have something monitoring that email box pull it in and actually start going through and doing that right regex parse the the indicators out of there and then start submitting those to a sandbox or to thread Intel Service right cool couple cool things about this particular one is they would do these reputation actions like I'm talking about on the URL on the file hash they would maybe even submit their file to a sandbox to detonate and get the results back they were able to actually integrate with duo as a multi-factor form of approval so that they could if they found some malicious pieces in the email they could actually send an approval push to duo the analysts could look at it and say yes this exceeded the threshold and they could click a button and have it implement the countermeasures so they could block hashes using fire amp and they could block domains on Open DNS right again it's all about tightening the OODA loop making it to where you can take that action faster and you can counteract adversary what they're trying to accomplish you know again some people bring up the concern of like okay well wait a minute I've got people on my mobile phone like blocking stuff wait a minute that's gonna cause problems well yes I mean there is potential for self-inflicted wounds but which one are you more willing to take right which is the bigger risk and are you are you giving people the tools and knowledge and are you setting up the decision-making criteria in here such that it exceeds thresholds that if it's known bad we're gonna go ahead and deal with it then yeah you your risk of somebody causing problems is gonna be lower than the risk of leaving it out there under mediated alright another one software company was wanted to do some domain blocking and they wanted to do it without having to interface with the the user interface for their in this case I think it was like open DNS or it's now called Cisco umbrella right so they said they wanted to be able to just have an email that was sent to an email box with the domain in the email that they wanted to block and boom it would take care of it right so again it made it easy for them to implement this on the fly there were a short staff team that said you know they have their analysts that aren't able to sit in from the console all the time if they get something coming in that they need to deal with they need to be able to do it quickly right so um financial company hash hunting they were starting with a list of hashes so again this is a common problem you get an Intel bulletin you get something that comes in and says hey here's you know sixteen hundred different hashes that are bad what are you gonna do with that right are you gonna try to figure out if they're on your endpoints how are you gonna do that with sixteen hundred of them are you gonna do sixteen hundred manual queries in your endpoint detection response tool I don't think so right so how are you gonna set that up to say we're gonna we're using an endpoint platform that doesn't have the containment API if even if we found one what what would we be able to do with it so by integrating with different components they checked with their EDR tool to find out if the hash showed up on an endpoint but then they were able to leverage Windows remote management or actions directly on the endpoint to perform some of that containment like deleting the file or blocking the process right so by thinking about an orchestration the sum of the parts is greater than its whole right by bringing the different capabilities your tools together sometimes you can shore up maybe what's a gap in one tool by leveraging something else that you have through orchestration all right phishing email analysis so this this again was a retail company that wanted to deal with some mailboxes that they had that we're getting phishing emails and an interesting thing that they had in this case was they were getting some emails directly from their users and they were getting some emails that were filtering through fish meat triage right so they were having some of the URLs being rewritten which is an interesting problem to solve right so how do you think about if you got read url's now I've got to deal with those and they were coming to the same place so when you're automating off of those emails you have to then think about identifying the different types of them and treating those differently right so again it's part of the automation process of saying hey if we if we find that these reg X's have been I'm sorry these URLs have been rewritten now then we got to extract those and treat them differently we got to rewrite them back to something that we can use to check a reputation service or to you know detonate in in a sandbox right now they were able to then automate the escalation notification process appropriately and with this particular retail company they were getting so many of these a day that you're talking about massive time savings right when they're getting 100 to 500 of these every day and you're able to automate that process of dealing with it now that frees up tons of analyst time and they're able to get to all those alerts one of the things I didn't mention earlier I wanted to talk about is there was a there was a recent survey from enterprise strategy group that said 54% of organizations admit to ignoring alerts that are worthy of further investigation because they simply don't have the staff or resources to do it right so you can see that I mean there was many of these cases where these emails they would just have to take a quick look and try to deal with it or frankly just in some cases it would act it would get ignored so being able to address all these again reduce the risk freed up more time big benefit this was an interesting one alert suppression right so how would you deal with that if you've got alerts that you don't want to handle at your sim right so some cases a sim may be good at this some cases it may be bad at this right if you if you have the ability to suppress it there that's probably gonna be ideal but if you can't then what are you gonna do with it right you want to keep those things firing but you only want to be notified if there's so many of them or if there's such a threshold that's something that that happens beyond what you can deal with so we were able to handle that there was a there's another case for allert suppression at a bank they wanted to do the same thing except they want to suppress it for 24 hours right so they said hey if this alert fires on this particular workstation in this particular user then you know don't tell me again about it for another 24 hours I'll deal with it then later so you know the differences in these in these two in these two alerts suppression things were different Sims different goals of what they trying to accomplish in different levels of escalation right so the idea of being able to do that without having the capability native to your sim was a big benefit of being able to automate and orchestrate this type of stuff okay this was an interesting case too so they had a they had a malware repository at a large bank they wanted to check their Intel file hashes that they got so they came a list of file hashes came in right another common problem they wanted to check against their 2av engines using virustotal right so they were running to a different antivirus engines within the bank if their AV didn't detect it they would actually download the file from virustotal and then send it to the AV vendor for adding to their DAT file right and they would say hey bender you're not detected on this we need you to go ahead and pick this up right so it was interesting that their their model they wanted to like make sure that they were detecting that stuff and they were making sure that they were keeping up with their Intel hashes the other thing they were sensitive to is where it started to get even more advanced is they wanted to prevent their virustotal license overuse right so they had a limited number of virustotal lookups and if they had checked something within the last seven days they didn't want to check it again so again another thing that they could they could keep track of a great case to use and so they could say hey I'm not having to do this again because I know that I did it before this one was a great example they were getting a lot of different alerts and they were using email which is not the great greatest way to to triage these but they were getting it into an email box and they were having to review these and they were having to say hey four different types of emails how do we you know how do we deal with the different types of alerts so being able to monitor the email inbox and say with it's a certain type of alert send it one way do one play book it's another type of alert send it another way do a different playbook right so that was the idea of doing object-oriented playbook writing and this was kind of like what Rob was talking about yesterday when you think about a playbook methodology right you give yourself the ability to reuse play books where you focus on file analysis playbook you focus on a specific type of event playbook that allows you to say hey I'm gonna do these different aspects and then I'm gonna go back and document everything in the main central central place so it handled a variety of in a detailed fashion all right so this was one specific type of alert they brought it in I know it's not really belong there specific things there but the idea is that they were able to bring it in and they were able to launch some specific malwarebytes remediation scans they were able to resolve it they were able to update it tagged it send notifications back out so I mean again the common process is the themes right is being able to update tickets being able to check things that would have to be manually checked prevent somebody from having to open 16 tabs in a browser right on six different monitors and check all these things that they would that they would spend their time doing it allows you to handle it systematically and fast it through automation all right so another phishing types of investigations it's not to sound like a little bit of repeat but again this is such a common thing this one was interesting that they wanted to check for a couple preconditions at first right so they used a whitelist said hey if any of these domains or any of these emails are on our whitelist we're not going to deal with that right so just get rid of a bunch of hey right there if they started detonating the files in the URLs they would then check for the thresholds that they want to be notified about right so again common process common thing that they needed to accomplish and then they would get notified those if they were above the certain threshold so there was another healthcare company that wanted to do a similar process right again phishing investigation but the difference is that every organization wants to approach this slightly different right which is why you have different playbooks you have different tool sets that you're integrating with and all this type of thing so they started with the emails they check to ensure the email was attached this is again another common problem right if every time somebody forwards in an email you don't get all those headers on the front so I was a part of a part of the sock that was having to reply back to people and say hey could you please send the email in but except drag it as an attachment and put it in the email so that we get all the headers right what if you had a way that detected if there was only one email and there wasn't - there wasn't an attachment of that that it would reply back to them and say thank you for submitting but please follow these instructions right please attach it to the email drag it in don't just forward the one that you think is bad right I mean that saved them a ton of time right there just being able to automatically on to those and get people trained up to do the right thing had the instructions printed out once they only had to set that up once and then boom it's on autopilot right after that once they had both emails they were able to perform those reputation actions the sim queries and then escalate and respond as necessary there was a logistics company that had a very interesting use case they again received a lot of phishing emails they wanted to do more than just check stuff and see if it was bad right they wanted to actually purged at and be able to quickly do so they were leveraging office 365 so they had all their emails in the cloud so we set up some scripts for them and again we leveraged a lot not just Python but a lot of PowerShell windows remote management and office 365 security compliance Center to actually kick off some purge jobs right so if they were had an email gateway monitoring tool that basically said if an email was bad or if a link was bad it got through they would then use that as the detection event to kick off this workflow and so they were able to to reach in to search for everybody's email box kick office a security compliance search to find everywhere where that email showed up and actually kick off the purge right and have several approval steps and check steps along the way to make sure that those series of things that need to be done were accomplished this one was a good one about from an insurance company a they wanted to do some registry key investigations so they were leveraging their sim and a combination of endpoint monitoring that they wanted to actually see every time a registry key got added to the registry run key that they were tracking right so that would pop up now you could you could imagine that that could generate a lot of false positives but it also would be something of interest right so in in order to reduce the false positive and actually get through some of this stuff they would every time one of those things came in they would initiate the file acquisition on the endpoint they would pull the file in they would then send the file over to their sandbox and detonate it and then they would check the results of that sandbox destination detonation right so you have the whole the whole workflow that was being able to be done every time and so they would only see something of interest if it was above a certain score threshold from that sandbox right so if they you know had ten of these a day or 20 of these that were coming in now they maybe only had one or two that they actually had to look at that we're interesting that when they detonated in the sandbox the file actually showed some potential malicious behavior right so again getting rid of all the hey wanting the needle another one that they were doing with around threat activity so there was some similar to that they were matching an IOC now this was a specific IP address or a domain again you know kind of likes mentioned before the pyramid of pain these these are not the best indicators because they can often you know result in false positives but if you've got a list of these and you want to start checking this stuff okay how do we kind of cool that down and find the interesting piece well we kick off a series into the sim we find the the actual proxy log where whenever someone visited that IP or that particular domain what was the full URL that they were visiting okay then at that point you would kick that over to the sandbox have that detonate and find out if that URL that they actually visited was malicious right so again it's all part of that process that would have to normally be done manually but we're talking about automating it so that you don't have to sit there and do all those and you'll only get notified or escalated of the ones that are more interesting right kind of reduces some of that that fatigue that alert fatigue so rather than have all these alerts that we we can't deal with we want to save time by using automation orchestration so some key takeaways right it's gonna save you time it's going to reduce the hey by triaging and closing these things automatically in many cases right so I just gave you a lot of examples of doing that you're gonna be able to kind of deal with the the common low-level ones so that you can have your time freed up to deal with the more advanced things right the point is not that you want to find the latest and greatest apt or the biggest threat you want to be able to find the things that are more interesting go through the steps that you would normally do on an alert so that you don't have to do those by yourself right it's gonna make your job less boring it's gonna hopefully reduce risk to your organization by addressing the the alerts and the things that you normally wouldn't have time to do finally you know think about the process that you need to automate okay that's that's gonna be the key to this if you if you again try to just make it too broad or too general you're not gonna be successful you're gonna want to really get down to the specifics get down to the nitty-gritty pieces so that you can say hey this is step ABC and follow that rabbit all the way down the next steps I mean again besides thinking about those things and really thinking about your process look into scripting right working look into the scripting with api's dealing with those types of things I mean that's a skill that's that's only getting more valuable so you know the other thing is the community SAO platforms if you find that and you want to check it out it's gonna save you some time on getting into this space but writing your own scripts and doing all this stuff on your own is completely possible and there are many people that do it so that's the key takeaways from my my talk I ended early so you guys can thank me later great thanks so much really appreciate that very distance