Red Team | Weaponizing Windows Crash Dumps

Key Takeaways

Alrighty everyone. My name is Jason Mo. I'm a cyber security architect with Lockep Technology Group. Got a background in consulting managed service providers. Came up initially in the network administrative system administrative place, but found over the years I really had a interest in security and that's kind of where I've stuck. I tend to be a bit of a jack of all trades with cyber security over at lockep, but the two areas I find the most interest in are going to be offensive security, incident response, understanding attacker behaviors and being able to map that to handle an incident properly or perform penetration testing just better understand a how threat actors do what they do. I graduated from Louisiana State University and the SANS Technology Institute. And what we're going to talk about today is actually part of some research that I did with SANS as part of my M's program. And this really came about as something that I discovered during a pentest shortly after doing my forensics 508 course. um trying to find ways to elevate privileges in a network for a client and I came across a crash dump file and I learned from forensics 508 how to interact with these crash dumps and the thought was I know that there's a lot of helpful information in here from a forensics from an incident response perspective how can I use this as an offensive security practitioner how could a threat actor use this So, what we want to cover today, um, for those of you that aren't overall familiar with interacting with memory dumps, interacting with crash dumps, want to do a quick crash course just to give you all some background on how to work with these files. Uh, want to talk a little bit about the value here. Why would you want to consider using crash dumps in an offensive context as opposed to some of the methods that are already out there? We've got a couple examples that we want to talk through of how do we use this data that we've discovered in practice. And because this is the hack and defense summit, do also want to talk through some different detection remediation options you can run through in your organizations to be able to find evidence of these files and determine how do you want to handle those? What do you want to do with those? So, first off, crash dump crash course. What do you do with these files? How do you interact with them? There are going to be three main crash dump types that we're going to cover today. The first one's going to be the complete crash dump. This one's really simple. It's going to be the equivalent of what's stored in physical memory when your device crashes. If you have a device that has 8 gigs of RAM and that device crashes, you would expect to see an 8 gig crash dump that's generated. Anything that's in physical memory at that crash is going to be in that crash dump file. Active dumps. It's a little bit smarter. It's designed to capture the vast majority of physical memory, but it is a little bit more selective. It's going to exclude data that's not necessarily going to be relative to any kind of debugging process that's going on. Because of that, it's going to be a little bit smaller, easier to work with. In my testing and my research, it came out to about 2/ird the size of physical memory. And then finally, we've got automatic dumps. This is going to be in general the default crash dump type, and it tends to really only focus on kernel memory. And in my testing, this is going to be the smallest type at about the eighth of the size of memory. And at that point, it's going to be a little bit difficult to find a lot of good data in there. But if you're using something like strings and you know very specific about what you search for, we was able to find some helpful information from time to time. So you can make some assumptions about what kind of crash dump you have just based on looking at the size of it and comparing it with the memory on the device. But if you want to find for sure what kind of crash dump type you're dealing with, something like wind debug I found is the best way to do that. And if you look at this screenshot, you're going to want to look at a specific string of text that's highlighted in red here. And that's going to tell you what specific kind of crash dump type you're dealing with. And then on this side here, the descriptions are going to tell you what you've got. So if it's a complete crash dump, you would expect to see full address space is available. For active memory, you would expect to see something to the effect of active memory is available. And then for automatic, you'd expect to see kernel address space is available. User address space may not be available. So that'll give you a good idea from the start. When you find one of these crash dumps, what kind of crash dump are you dealing with? There's a number of tools out there that you can use to interact with these crash dumps and explore what's in there. Valatility framework, I think, is probably one of the the de facto standards for memory dump crash dump analysis. One of the tools I came across in forensics 508 that I really liked was something called meroc fs. The reason I like me fs and the reason that I think it's very useful for people especially those coming into memory analysis for the first time is it presents everything in a file structure that you're probably going to be very familiar with and it allows you to really just kind of kind of poke around click in the folders and explore a lot easier. And for those of you that haven't used that, we've got a little bit of a demo here just to talk through some of the main points of it. So we go ahead and run me fs. In this case, we're referencing the forensics parameter. We're going to want to talk about that a little bit more later, but it's going to ahead and mount the memory to the M drive. And then we can go ahead browse to the M drive. And now we've got access to the crash dump. Processes are going to be split either by name or by pit. You can explore either way. In this case, we'll look at name. And then we'll go down to a Firefox process and explore what's in there. And if we go to files, we're able to view files that are in memory that are part of and associated with that process. So in this case, we can access the cookies.sqlite file, which we'll explore a little bit more later. And we've also got the ability if we want to to extract process memory for a specific process if we prefer to interact directly with that process memory as opposed to crashed dump memory in general. We've also got the ability to browse registry hive. So if we go to the registry folder, we can interact specifically with the different hive files such as the system hive software. Those are all going to be available to the extent that that data is in memory. We talked about the forensic parameter earlier. The forensic parameter is going to give us access to browse files in memory within the file format that they were on the system generating the crash dump. So we're looking here and we've got the example of the C drive going into users and being able to enumerate a user's profile and look at folders as they are in memory. So where's the value here? Why would you want to look at crash dumps as a way to get sensitive information as opposed to some of the more traditional methods out there? To me, a lot of it comes to lack of visibility and lack of detections for interaction with crash dump files. When I searched the uh MITER attack framework for memory.dump just to see what kind of detections were out there, nothing came up. I searched for crash dump. A few more things came up, but they were mostly related to other exploits and other techniques a lot around the Cisco arcane door exploit. So what this told me was there was really appears to be a detection hole in terms of threat actors, offensive operators that are interacting with crash dumps. That seems like a, you know, a hole and an opportunity to me. So, how does this scenario play out in practice? A thread actor would gain some level of initial access to an endpoint and get local admin on that endpoint. And from there, they're able to obtain the crash dump that's on a system. They could even use something like not my fault through CIS internals or some kind of custom PowerShell code to actually induce a crash to occur and create their own crash dump if it wasn't already there. From there, they excfiltrate that crash dump. If there's nothing looking at interaction with that crash dump file, with that memory.dump dump file, then there's probably not going to be a lot of detections that are going to be triggered outside of maybe some kind of detection that's going to be looking for uh an increase in network traffic from a specific endpoint. From there, and where I think this differs from utilizing live on system attacks, we're going to do all of our analysis offline. We're going to do process extra extraction, maybe interact with the LSAs process, interact with the registry, interact with files where we we would typically do that on a normal system, on a live system, we're going to do all of that offline without any kind of detections that we have to worry about. From there, we find some good information. Maybe we find some domain admin credentials that are cached in the registry or cached in Elsass. We find some credentials that are in a browser. Now we can go back to our live system and we can use that to elevate our privileges further throughout the network. So let's go through some examples here of how we can actually weaponize this. How can we take this from a theory and actually apply it? System credentials typically are going to want to be one of the more frequent things that are targeted by threat actors. We can obtain active directory and local credentials through the registry hives that are found in the crash dump. As we explor explored before pi cats is a great tool for this. It's a Python implementation of mimikats and using that we're able to feed it the system hive the software hive and anything else that's required with that to go ahead and get information extracted out like we can see in the screenshot here. Now the one caveat here is when I started going to smaller crash dump types like an active or an automatic I was running into situations where the registry high files I was able to extract from the crash dump were corrupted. So this is going to be an area where it's going to be most beneficial using this with a complete memory dump. And here we've got a more specific zoomed in example of how that looks. So, we run Pi Cat's registry. We feed it the Security Hive system software and then we're all set. In the example here, we've got access to cache domain credentials for what looks like an administrator and a user. So, now we can take those offline and attempt to crack those and get the actual password. But maybe you want to interact with the system a little bit differently. Maybe you don't want to use the registry on a live system. You'd expect to dump Elsass, but I think we all know that's going to be something that's going to be pretty noisy. But we already discovered how we can interact with process memory from a crash dump. So we can just take the LSAs process from the crash dump, feed that in the pi pie cats or mimic cats and get your data that way. The nice thing with this is I was discovered that getting access to these memory dumps for Elsas were available in active memory dumps. So I'm not in this case tied to using a larger memory dump format. By default, meroc FS does not do this anymore, but there are alternative builds on GitHub that do enable this functionality to interact directly with the LSAs process from the crash dump. So Melvin discussed this a little bit earlier, but web browsers are a great target for thread actors. There's a lot of good user-based information contained in there that's very helpful to a thread actor. And these tactics are recognized by attack as well. There's a number of techniques that look for interaction with these files. The screenshot I've got here is looking for SQLite interaction with the cookies.sqlqlite file for Firefox. So live interaction with these files that contain these browser secrets is something that a lot of detection frameworks are already looking for. But what if we got it from a crash dump? Firefox is going to store things like cookies and other sensitive information in unencrypted SQLite databases. But crashed up analysis is going to give us a stealthier method to interact with these files rather than interacting with them on the disk itself. So in this example here, we're browsing the file system and we're interacting with the cookies.sqlite file. And then when I open it in SQLite database browser, I can view the cookies. And here we've got an example of the ESTS off persistent cookie. So the ESTS off persistent cookie is going to contain a session cookie that allows access to a user's Office 365 session. This is going to be something that's frequently targeted in fishing campaigns to gain access to users Office 365 mailbox. This is just another method now to obtain that same information. Moving on, we'll look at Dappy for a bit. As Melvin discussed earlier, DEPAP is going to be a native encryption API that's built into Windows. Master keys are going to be stored per user per computer. And it protects a number of different things. A lot of thirdparty applications are going to use DPAPY to encrypt the secrets contained in those applications. So if we're able to get into decapy, decrypt it, we're able to then use that to get into other application secrets. I've got up here the master key location for users. And the general workflow is going to go like this. We obtain a copy of the crash dump. We obtain the user's password. We already discussed some ways to get access to user's password hashes. So, let's say we go ahead and we crack that password hash. Now, we've got a copy of the user's password. We're going to want to generate what's called a dappy prekey where we combine the user SID file and their password to get that prekey. We're going to want to determine the current master key file. So, Windows is going to rotate that master key file at regular intervals, and it'll keep older copies of that master key. So, we're going to want to determine which copy of the master key on disk is that current copy. And then we're going to want to take that prekey and decrypt the master key. Once we get that decrypted master key, we then have access to be able to decrypt application data. So where we talked about a little while ago a number of different ways to get access to Chrome data more on a live system. This is taking that same approach but now we're getting this information from a crash jump. So the login data local state data in Chrome we're going to want to extract those. We're going to use pi py cats to create the prekey file combining the data that we've already got access to. Then we use dpappy preferred key with pi py cats to find the current master key. We're going to use dpappy master key along with the current master key and the prekey to decrypt it. And then finally, we're going to use the pi pycats dpappy chrome command to combine the login data file and the master key file and the local state file. And in this case in the example, we now have access to c credentials stored within chrome. This isn't a new feature. This isn't a new procedure or a new way of doing things. What's new and what's different is the way we're actually obtaining this information off disk rather than targeting all of these specific files on the live machine itself. And as we talked about, there's a number of different tools and a number of different applications that use deepappy. So if we understand this process and we understand how to decrypt user and systembased master keys, that process can apply to a number of different applications that are out there. And then the last area I want to talk through is password managers, specifically password manager extensions in browsers. Password managers to me are a pretty interesting attack vector, but the timing kind of has to be quite right. Vault passwords are going to be unencrypted in memory when the vault is unlocked because at some point the data needs to be encrypted for the user to be able to interact with it. But what about once that vault is locked? What about the master password? How can I interact with all of that? So, one of the things I really wanted to research was what could I get from a password manager extension like BitWord. So, I worked with that a little bit and I was able to discover that even when a vault was locked, I was still able to find the master password in memory. I had talked with Bit Warden about this and they had pointed to some memory leak issues within Chrome. I talked to Chrome about it and they had claimed that this wasn't in their uh scope of investigation because they viewed it as more of a local issue that they're not really interested in looking after. So looking through this specific example here, we'll go ahead log in the bit warden and then I'll go ahead and I'll expose the master key just so we're all on the same page on what that is. And then we'll go ahead, we'll unlock the vault and we'll interact with some vault entries like a normal user would do. And now we're going to go in and we're going to look at process memory to see what we can find. So if I search for a vault entry, I can see some JSON formatted text that looks like how Bit Warden is typically going to store a specific vault entry. And then if I go back again and I search for the master vault password, I see a number of different areas of memory where that master vault password's stored. So now we'll go back and we'll lock the vault and we'll wait a couple of minutes. That was something that Bit Warden advised me was waiting a couple minutes to make sure everything was cleared out from memory. So, we go back and we repeat the same process over again. We'll search for the vault entry first. There's nothing there. So, we'll search for the master password again. But the password is still contained in some areas of memory. So the takeaway here is if we're able to obtain a crash dump from a point when a user was in their browser, they were working with the password vault, even if the vault was locked, there still is some opportunity to find and capture that master password in memory. And this was something I was able to duplicate a a number of other password managers as well. So we've talked about how to weaponize this. How do I detect it? What can I do to find evidence of this in my organization event logs can be a pretty good source. System event ID 101 is going to record crash dump creations if that setting is enabled. And I've also created a sigma rule that's been published that's going to search specifically for I that ID. So you can use that at scale if you're doing some retroactive searching of your SIS logs with something like chainsaw or you can build that kind of sigma rule into your SIM if you're ingesting your event logs and you want to get alerts like that when they fire off based on your s your event log ingestion. If you're tracking registry value creations in your EDR, you can also query specific registry keys that I've got up on the screen. And anytime those registry keys have created, I found those are indicative of a crash dump being created. So depending on how you want to handle crash dumps in your organization, you can treat this a couple of ways. If you've decided in your organization you don't want crash dumps at all, you don't see the value of automatically creating them and you create policies to turn them off and you create an alert around this. Anytime a crash dumps created, that's going to trigger that alert and that could be an interesting thing to investigate when it's created because you've decided as an organization you don't want crash dumps to be created. So that's an outlier. If you've decided that you do want to create crash dumps in your organization on demand by default, then maybe you configure this to run as a report and you can just get some general visibility at specific intervals about when crash jumps are created being created on what machines, how frequently. So you could use it as an indicator to find machines that are potentially unstable, crashing frequently. that could be indicative of system performance issues or some kind of malware or other threat on the machine. You've also got the ability to use PowerShell as a way to search the file systems to detect when a crash dump has been created. So, we don't necessarily want to search for C Windows memory.dump. It could be named something different. You can have processes where crash dumps get moved somewhere else for storage and future review, but you can search using PowerShell for file headers rather than specific name. So, we've got an example the bottom of this screen of specific file header of a crash dump. So, you can search for that header instead of just searching for a name, searching for a specific path. And then the output of that script is going to give you a clean list of a crash dump path size and a type. I've got a example on GitHub that I'll I'll post in a little bit. It's really currently configured to run per endpoint, but it's something that could be modified to connect to remote endpoints or be deployed through other remote management solutions. So ultimately, what do I want you to learn from this? What are some of the takeaways I'd like to see you bring back to your organizations, bring back to your companies? I think the overall thing I'd really like to think through is having you all think through crash dumps and making that determination in your organization of do we really need these crash dumps to be automatically created? Do we as a organization have the IT resources, the IT knowledge to analyze these crash dumps on demand proactively as they occur? Or do we really only interact with these crash dumps in scenarios where a user's reported an issue or we're working with a vendor on demand to troubleshoot an issue. If you're not proactively interacting with these crash dumps as they occur, I would challenge you to delete to disable these crash dumps and then if you have a need to work with a vendor to troubleshoot an issue, troubleshoot some kind of system instability, enable those crash dumps on that machine for the time period that you're analyzing the issue and then disabling them. I think there's a lot of potential for valuable information with this within these crash dumps, but you do need timing and a little bit of luck to I think always make good use of this. Again, automatic crash dumps, which are going to be the smallest, are the defaults. So, while we talked about a lot about what you can see in larger crash dump types, realistically, you're not going to be running into those all the time unless you're in a a situation where you've got the ability and rules of engagement say you can trigger a crash dump. And we only covered, I think, a couple of examples of the different elements of data you can find from a crash dump. I think there's more potential. There's more pieces of information, more sensitive information and different ways you can use these crash dumps for escalation. I really think they just require a lot of curiosity and some time and research uh to dig through those crash dumps to find useful information. So, I've got my contact information here. The link at the bottom is going to point to a GitHub repository where I'm going to have a copy of the presentation as well as as a couple of PowerShell scripts that I use to generate data and search for some of these crash dumps throughout my test environments. But with that said, any questions?