Incident Response: Azure Log Analysis

Key Takeaways

let's go through a little cyber crime mystery let's do some incident response and investigate a hack only by using logs from the environment I want to walk you through the scenario of a hacked MSP or managed service provider that's using an Azure active directory and we'll go through all these different components from the initial intrusion pivoting in between different machines stealing cookies and browser sessions and then a full domain compromise so I am inside of a Linux terminal and I have all of these logs pulled down for us I want to be able to take a look through them we'll open it up in an editor but first let's get through the story first things first our goal is to figure out how the hackers gained initial access into our domain and environment we know this is azure based we're working from the cloud here and that domain controller in the environment will track some logs for us to see what users had failed login attempts which had attempted logins and successful logins so let's open up that interactive login log now I believe this is from Microsoft pview and I know know there's a whole lot to look at here and I don't even have word wrap turned on this is just a component the leftand column but we can dig into the details here we know the date of all these entries request ID correlation user IDs all of these goids that uh Microsoft Azure might track for all that information here if I do actually toggle word wrap on you can see these are significantly longer entries and there's more to it between what user is interacting with what service and how they're logging in I'll zoom in on one of these entries or kind of separate it from the others so you can see hey here's a result here's a record and entry that this user Paul Bowman did try to log in from San Jose California but failed hey error validating credentials due to invalid username or password but we can even see their browser information like the HTTP user agent header we can see of course the location and we can even see the IP address that they're coming from the IP address information might be really valuable because we could then go analyze how many IP addresses are included in this log entry and how many times did each of them try to authenticate if we jump back to our terminal we can try and cat out all of the interactive sign-in logs and we can actually try to cut specifying the field separator of a comma because we know this is a CSV file or a comma separated value sheet and I believe oh we need to specify that as our delimiter but the field number we could try to count that manually I think it's uh one two look I think the IP address is about 20 so let's try that let's try F20 and let's see will that pull all of the different IP addresses yes we can see them all output on each line here let's try to sort that output and then let's pipe that to unique and we can display the count of how many times they're present in the file so about 12 occurrences of this 137 IP address 211 occurrences of 20. that 121 of four blah blah blah now that is Handy information we could go dig down and explore all the different IP addresses and what their activity was but I know some of you might say oh well the hacker or threat actor could change their IP addresses or sort of rotate through different proxies as they tried to attack this network and you're totally right so there might be other malicious indicators that we could look for we know from that example of Paul Bowman's login that they did actually try to log in with Firefox as their web browser the HTTP user agent might be worthwhile to dig into we see other folks using oh a rich client we might see other folks using oh um python requests and more requests from python requests that's that automated python module that allows us to see HTTP communication back and forth we can force requests with that and I'm wondering maybe they're using a utility or a tool to try to spray a different passwords and attempt to log in through every single user here's Johnny Gonzalez here's Joan King here's Edward White George Edwards look at all of these failed requests and these are all coming from that IP address of what is it 20.25 and remember we know that 20.25 the IP address there had about 200 entries so let's try something wild let's grab that 20.25 IP address and I'm going to search for every single line that includes that and I want to get with regular Expressions just a DOT star everything that comes before and after it so basically grep right I just want all of those lines I'll find them all with that button down below but I want them to be in their own separate file and let's see how many we have here okay about 21 lines all from the interaction of that IP address here I'm going to do a little bit of a magic trick I'm going to hit control a to uh select all of them I'll hit control shift L to get multiple cursors in Sublime Text and I want to cut out just all those gu IDs or some of those client details that I don't really need to know right now uh but you can see especially in the time date stamp this is like every single second so it's not realistic to say that oh every user in the organization is all trying to log in at the exact same time this is clearly a password spray against every single user in the or see all of their names listed and all of these failed attempts can i contrl f for failure let me go see here yep failure failure failure failure failure now remember our goal is to find which user did the hacker actually break into which password username credential pairing did they find that they could successfully log in with we want to look for Success logins right I'll get back to it and I'll contrl F4 success and we see one entry here uh let me see if I can just grab that line on its own and I'll turn word wrap off here so we can dig into it looks like Dale Clydes uh one Success login with multifactor authentication but I want to add a little asterisk here a little bit of a disclaimer because we are kind of zoned in on just this IP address right here maybe uh we're blind to something else because we've filtered strictly on this and with that I'm acknowledging your point right and that look the thread actor might have used multiple IP addresses so what if say they brute forced passwords found one successful login with from one IP address but then used another to go actually do their dirty work let's be cognizant about that but these logs probably give us a little bit more detail in the error message or success message when trying to authenticate we can see all of these login attempts with that python request user agent but the error message here is the user didn't enter the right credentials we see this over and over and over again but oh there's something interesting on this one I think I'm tracking that so let's keep note of that uh okay and there's that Dale Clyde's one down below is that the only odity let's keep looking no scrolling through the log I think that's it uh let me go take a look what is the rest of that error message ooh due to a configuration change made by your administrator or because you move to a new location you must use multiactor on the location to access whatever okay that error message must mean that they had the correct username and password pairing they found the right credentials they were just prompted for multifactor authentication let's get back to that line the one where specifically there's a configuration message and they have to use multifactor location let me remove everything just above and everything just underneath it and it's Paul Bowman let's turn word wrap right back on okay looks like yep that same 20 IP address from Washington Virginia I'm assuming even in their password spray from that IP address they found the correct username and password pairing if we look back at the writeup for this lab this exercise and this activity they're chatting about everything that we've been discussing all the remote IP addresses the uh geographic location that they tried to sign in from and their user agents but look they saw this exact same air message that we saw the configuration change requiring multiactor authentication so that user Paul B 's password was discovered and then used the next question is did the hacker actually use this account were they able to successfully log in let's go back to the full log of all of these interactive sign-ins and again let's just look for anything for Paul Bowman let me try to search for everything with their username included we'll find all copy and pastes to another entry here and these are all the attempts but can I look for Success oh yep looks like there are plenty of success entries for our Paul user here do we have the timestamp from the misconfiguration error log that tells us kind of when this all happened here oh I think I cut that part out uh while we were editing that's all right we can go back to it and search for that configuration change yeah okay so that was a 1703 Zulu let's look back at these logs of the other entries we have a 1703 that is a success yeah it looks like that 1703 timestamp is here and then everything following is just a bit after later in the minute we can look for those success entries as they did log in and again yep this looks like a different IP address so I think our hunch was right now we have our first breadcrumb we have our first puzzle piece we know that the user Paul Bowman was compromised the thread actor used that as their initial access and now a springboard and launching point to move further into the network so let's go see what else they did oh and hey before we get too far down the rabbit hole please let me say this whole activity this exercise this lab environment is all from John Strand and anti- siphon training and black hills information security and that whole tribe of companies doing incredible stuff and especially from their pay what you can training I'm not sure if you're already spun up on all these stuff that they offer but there seriously so much good stuff between learning how to be a security Operation Center specialist doing some penetration testing digging into other training and education it's all phenomenal and look if you dig into their pay what you can training you can literally choose the price tag for all the awesome stuff here you decide how much you want to pay if you love some live and online training you can check out some of the courses that they have coming up on the calendar I see some stuff open in November plenty more on the horizon take a look at the road map see what's coming up on the schedule and take a look at some of John strand's other intro Labs we've done some videos on them but they have some phenomenal stuff here to work with other digital forensics instant response live hacking Etc if you want to take a look at the navigation here we can pull down and see all of the great labs and activities exercises that they showcase all the stuff from introduction to Security even some active defense and cyber deception plenty of awesome stuff to dig into for real cyber security training huge thanks to Andy siphon training Black Hills infos and John Str and that whole crew for sponsoring this video please please please go take a look pay what you can training link below in the video description after the discovery of the compromise user Paul Bowman we decided to go through the security logs of each workstation and look for any suspicious files being run or used by other users this is is of course pivoting so we have some new logs to look through and let's go dig into those I've opened up this other CSV file and this looks like an export of the security log from the Windows Event Viewer so with that we can see all the stuff that happened on that endpoint presumably workstation 3 or ws3 so we can see files that ran hate programs applications and let me actually dig into that we can see oh process IDs or process names let me go search for every single process name to see just are there any Oddball things that stick out let's get back to the command line and let's try to actually use that cat and grep capability one more time let's GP for process name display all those and now we could actually kind of cut this right if we wanted to let's cut with the eliminer of a colon and get the I think Third Field right there we go so now we have all of the executables ran let's pipe that to another sort and unique taxi just for some poor man's analytics here let's see are there any odd this is the path to an executable ran on this host and the number of times that that was executed so let's see is there anything that sticks out I'll keep scrolling I'll keep scrolling I am most interested in things that probably only ran once cuz you really only need one invocation of malware right but there are a couple other interesting things uh I see Ninja rmm in here a remote monitoring and management solution uh some regular Windows tooling right wmi things are happening oh cyber CNS is in here just as well but I got to admit I don't see anything that sticks out right at first bat but don't forget let's take another big picture view we might have pigeon hold ourselves just by looking for things that have semicolons in the path because we're using that cut command and using the semicolon as the eliminer looking at the second field we might be just trimming stuff out of our view let's go back to the command that we ran but let's remove the cut and just kind of see hey give me everything I'm fine with it um now we see some weird stuff right hey two occurrences of a process name being executed from a device another device looks like WS or workstation 1 file five maybe that's a file share super specialized highly Advanced malware bypasser 2.exe okay little on the nose little cheeky but that's probably bad now this means something interesting because we saw these logs these uh again Windows Event Viewer logs of the security tab that security log from the workstation number three but it looks like it pulled a file from workstation number one and that was where Paul Bowman had actually logged in given his use now we've seen initial access and we've seen lateral movement they were pivoting from machine to machine but there was something interesting that we caught in just a moment of looking at the processes invoked ninja RMF there's a remote monitoring and management tool in use in this manage service provider environment so this threat actor this hacker could do something a little bit dangerous they could dig into another user's cookies or their session information right hey maybe they were able to get into an endpoint one of the workstations like workstation 3 workstation whatever and maybe they could actually invoke this malware or they could actually dig into some other information maybe their malware would be like an information stealer or a cookie stealer that would retrieve all the sensitive details and all the cache data of their web browsers because hey say you're a system administrator you're one of the it Engineers that works with the remote monitor again management tool well you probably interact with that in your web browser like Google Chrome right but before we go too far down the rabbit hole here let's keep in mind okay that we just saw this super specialized highly Advanced maare bypasser whatever on this workstation number three and that was actually invoked as this other user henry. Butler um okay well now let's kind of explore some other logs that are provided here to see did they genuinely steal cookies to access the rmm capability yeah okay looks like Henry Butler uh was working with oh their super specialized highly Advanced malware bypasser did in fact try to access another object or another file that would steal cookies from Google Chrome this is Bad News Bears and it's coming from the audit log on this host right okay object ID number from the audit log 4663 looks like an object was being accessed and hey the malare reached out to try and steal cookies we see this in the log entry and they showcas it in the lab now you might be thinking so what the remote monitoring and management tool ninja rmm or whatever case might be that probably has multiactor authentication configured right that's enabled that's enforced so that way you wouldn't be able to get into the rmm without the end user really without the it administrator allowing and granting that to authenticate however our threat actor under the compromis Paul Bowman user was able to steal these credentials from Henry Butler and they weren't just credentials though they were cookies they were session cookies and browser details so that hey that user Henry sure maybe they would manage the rmm utility but those cookies allow for access without logging in cuz it's for the session they don't need a multiactor authentication because they've already gained everything that is session hijacking right they could just slap it into their own web browser use a cookie modifying extension or even just go in like the developer tools and then put in everything they have now here's the final question say our thread actor they've gained their initial access they've done their lateral movement they've dropped some hour for that cookie stealing and information grabbing but what will they do with the remote monitoring and management access with the capability of that tool what will they do well they didn't just compromise one machine but they have compromised the entire domain because rmm that utility can manage every single workstation and endpoint it can push down other software other patches other tooling and do whatever they want it's super user access right it's the queen bee or the Mother Ship we can take a look at the logs here and again this is the rmm this is the remote monitoring and management tool and we see a couple actions completed oh running some system applications oh our super specialized highly advaned malare is ran on all uh all of the endpoints all of the hosts all the workstations every server anything under management of that utility looks like they did blast their malware out that was probably used to steal more information from other users hey can we access everyone else in the organization all the employees of the business but you can see finally one of the other last lines of the rmm looks like it did also push out another binary custom 2.exe on all hosts and that could very well be the rat the remote aess Trojan or toolkit maybe some Cobalt strike maybe some command and control whatever the damage is already done we can read here that the attacker has pushed out malware on every single host including the domain controller it's safe to assume every single machine is now compromised and this turned into that doomsday nightmare scenario where the rmm has been breached and can now push out ransomware cryptocurrency miners I don't know defacing utilities whatever on all the hosts the small breach has become a nightmare disaster and we'll need that incident response with that we followed this intrusion from Cradle to great we saw the initial Act by the thread actor their lateral movement and their escalation and then what they might do from now on sounds like we need to get that Paul Bowman user some security awareness training and uh maybe a password manager in place make sure we can have some stronger credentials that don't lead to a full-blown compromise again this is all free education from John Strand and black hills information security and anti- siphon training that whole tribe of companies and their pay what you can exercises material courses labs and great curriculum please please please check them out Link in the video description seriously you can decide the price tag and it's extremely valuable for you your team your organization and the whole industry we're making the place better making the whole world better by all getting sharper on cyber security thank you so much for watching hope you enjoyed this video like comment subscribe I'll see you in the next one