Detecting Ransomware

Key Takeaways

[Music] this is data skeptic time series the podcast about how to predict the future based on historical sequential data episode number the first known ransomware attack took place in 1989. yes 32 years ago a pc virus would hide and encrypt the file name though not its contents of all the files on an infected machine after this process was complete the virus would demand they pay a hundred and eighty nine dollars u.s to obtain a repair tool that's a small price to pay compared to the sums asked for modern malware attacks which encrypt the entire contents of your files effectively locking you out from them any serious it organization needs to be panicked about this attack vector and looking for ways to detect and thwart it how can signals at the hardware level give an indication of when your system has transitioned from its stationary usage into a sequence of operations suggestive that an attack could be underway today on the show i speak with nitin pundir about techniques for detecting crypto ransomware [Music] hi kyle thanks for having me my name is nathan punder i'm a phd candidate at university of florida i work in the florida institute for cyber security research under professor mark tehranipur and my specialization is in hardware security and we look into security solutions which can utilize machine learning so basically we try to develop machine learning based hardware security solutions well you said you're a phd student have you picked your thesis topic and title yet so uh basically i know what i'm supposed to do the topic is not finalized yet so basically i'm working on developing machine learning based secure physical design solutions for side channel attacks but that is not this paper is about this is paper is about something else but yeah my dissertation will be on using machine learning to enable the secure physical design that can allow pre-silicon based side channel and verification for the cad companies like intel synopsis and for the government chips and so on oh that's very interesting yeah maybe we could take a moment there before we get into the paper that's not something i'm familiar with is this new cutting edge stuff or are you just pushing it forward so it's a new cutting-edge stuff so you must be aware of flight channel vulnerabilities like a power side channel there was a spectre and meltdown right which create like huge buzz because all the intel architectures were like vulnerable to it so basically what happens is like all the chips which are manufactured there is like a lot of engineering effort before it gets fabricated right because once you get a chip it's like finalized you cannot make any modifications to the hardware so there's like a huge deal of verification going on in the back end like even before the chip gets fabricated by tsmc or other global foundries right so during that verification efforts they try to make sure that there are no hardware vulnerabilities in the design right such that like the design is not vulnerable to like powered side channel laser fault injection attacks and so on but what happens is current verification efforts are like limited right because you need to have the physical characteristics to make sure your design is not vulnerable because what you're trying to say is after fabrication your chip won't be leaking any of these side chain vulnerabilities but your chip is not fabricated yet right it's just in the pre-silicon area so my effort comes into that so basically we are providing all these cad industries with these verification frameworks where you can kind of emulate the post silicon environment in your pre-silicon test framework to make sure that your design or your chip layout is not vulnerable to these like side channel vulnerabilities and that's where my dissertation is more focused towards well exciting and important stuff for sure well the main paper i want to invite you on to talk about is titled rand stop a hardware assisted runtime crypto ransomware detection technique for anyone who's not familiar i think most listeners will at least be aware of crypto ransomware attacks how could you not but could you give a high level summary for what's happening there what are the security holes that allow such a massive takeover to happen in an organization basically what happens is these kind of attacks are like very prominent right everyone has heard of it and recently in florida they attacked the pipeline right and the entire like database was sabotage and they were asking money like a ransom in like bitcoin and everything right so what happens is like the attackers kind of exploit someone like loopholes in your system and they get access to your system right once they get access to your system they start encrypting your files and database and everything and in turn like in return to give back access to those files they ask for some money in terms of all these cryptocurrencies which are like untraceable and everything so since these attacks are kind of which sabotage or like make hostage like all your data files like critical files especially in like government agencies like the pipeline and even in hospitals because these areas are where which are like most vulnerable because these people don't come from like the cyber security background and so on so their systems are kind of more exploitable and they have like more critical data to save right so that's where the ransomware and everything is like very critical when it comes to all these institutions which are handling all the critical user data right absolutely a lot of organizations have turned to software solutions for this and it kind of fits with the way people are thinking right we've had virus checkers for a long time that are all software based why is a hardware assisted runtime a good solution for this problem if you talk about the solutions right like all the anti-viruses and everything they are software-based solutions right so basically how they work is they already know the malware or the ransomware which is already there they try to extract the software-based signatures out of this right and based on that they kind of perform the signature verification when you try to run something on your system and say hey this is kind of template matching and then they say hey this signature belongs to a virus so it should not be run but what happens is all these attackers they're also very smart right they apply changes to their malware in such a way that the signature changes and all of these anti-viruses and everything they are not able to detect those and so basically if you see in overall all these software-based verification methods they are not applicable to zero-day attacks right like new variants of i'll say crypto ransomwares if i attack or change those and just spread those in the network maybe the software based applications won't be able to detect them on time so basically what we plan to do like we aim to do was extract the hardware features basically like a signature so it will be like analyzing someone's handwriting and then analyzing different ways he writes the email and so but the key characteristics of that person like the handwriting remains the same so basically that way by analyzing the hardware-based microarchitectural events of these ransomwares we can protect against the zero-day attacks which will be like really helpful when i think of a hardware approach in my mind i'm thinking of something very low level you know the code i wrote maybe it's in python perhaps somehow got converted down to c and then to assembly language or there's operating system libraries involved and then on the hardware it's just bits going through integrated circuits it seems very low level and almost like it would be difficult to recognize malware from ransomware you were talking about the handwriting what does handwriting look like at the binary level when i say the hardware event so it's like as you said we are literally going to the lowest level possible right so what we exploited in this case are like took advantage of is all these hardware performance registers or counters provided by the architecture itself right so these performance counters are actually used by the developers to see the efficiency of their code right like this particular section has a bottleneck or something right in terms of uh cash message misses other branch executions and so on right so we took advantage of those hardware performance counters and extracted these metrics so some of these hardware metrics include like branch rate branch misprediction rate cash misses l1 miss rate and so on so there are like bunch of these uh micro architectural events which are like out there and could be utilized very interesting the branch rate if i understand that correctly at the assembly level branching is really like an if statement i guess at a higher level it's you know if this is true do this otherwise branch to some other location and it seems like code should be free to branch or not branch whenever it wants going into it what led you to believe that could be an insightful metric that would help tell the difference between some of these different types of software basically when we were planning on developing some of these features right i don't say features but a technique which could exploit the hardware micro architectural events so what we had in our mind was that a program does not change its semantics over a variant so basically if you see a crypto ransomware right what's its basic characteristics right so it's going to search for all the data files in uh like in the user directory or in the system and start encrypting it right so that is like a very basic characteristic and if you try to extract the microarchitectural events of this behavior or a crypto ransomware you will find some similarities making sure of these similarities by naked eyes will be like very difficult right so that's why we try to exploit the machine learning in this case so where we have bunch of these micro architectural events captured from different ransomwares and the benign programs and then we say hey this is my machine learning model can you identify some similarities between the ransomware variants and different ransomware programs and can you forecast it or classify it so that you know my model can distinguish between the ransomware and a benign program and then does that end up being a supervised learning technique so at this point this was a supervised learning so we already have a bunch of ransomwares from a database from virus share we have a bunch of benign programs and we had to be especially careful because i was running this ransomwares in my university so i had to tell my it administrator hey keep monitoring my port if something suspicious happens like make sure it does not you know get away into the entire university network oh that's an interesting challenge yeah and actually it did so he said someone was scanning ports from my network and i said i i was not doing that probably something escaped from virtual machines and was doing it oh wow well yeah then so to do that you need some labels you'd mention that there are you know some famous ransomwares out there what do you start as for your control group there are some benign software applications uh what's the sample there for comparison basically when we were choosing the benign softwares right so we wanted to a mix of benign software so like normal applications like opening a text editor writing like copying moving and all those but also we wanted some benign programs which have similar characteristics to a crypto ransomware right where you are encrypting a very large file or where you're decrypting a very large file where you're deleting a very large file because all of these characteristics are kind of similar to ransomware crypto ransomware and we wanted to have a mix of that database we wanted to check whether the machine learning would be able to identify like micro architectural events from these two databases like which are like very similar if you see from a human perspective [Music] thanks to this week's sponsor oval edge ovaledge is a cost-effective data catalog designed for end-to-end data governance privacy compliance and fast trustworthy analytics every business has personally identifiable information or pii not only do you owe it to your users to be good stewards of their data it's going to be required of you as stricter and stricter privacy laws come into play the good news is you don't have to take care of that solely on your own you can call the experts at oval edge their end-to-end data governance suite has flexible solutions at a fraction of what other providers charge starting at a hundred dollars per user per month their data cataloging solution uses artificial intelligence to crawl through all of your data sources and identify the parts that have pii you know this isn't a simple problem it's not just a matter of a couple of fields in a user table pii is scattered throughout notes fields and other subsidiary tables and places it's copied it's migrated if you really want to protect your customers you need a powerful tool like oval edge cdo's cpos cios and any leadership and data i want all of you to head over to ovaledge.com data skeptic there you can download and implement their data governance guides or even schedule a demo if you want to learn more that's ovaledge.com datasceptic [Music] are there any concerns you had about false positives i could see where like uh the dropbox application could be flagged because it's deleting or moving a lot of files around by design or maybe if i'm going to encrypt some documents the process of encryption might look like what the ransomware is doing could normal operating systems be flagged as ransomware when they're not actually yes so when we were developing this method right so that was the biggest challenge your false positives and false negatives are like two metrics right so basically when we are training the model generally everyone sees as the accuracy of the model right but in this case more than accuracy it's important that we divide that accuracy into false positives and false negative but if you see from like a user's perspective right if there is very high false positive it means that a benign program is characterized as a ransomware and then he will just have to go and say hey this is not transfer allow it to execute and then there is false negative where ransomware is characterized as a benign program in our opinion the false negative is most dangerous because you don't want even a single ransomware to slip by whereas the high false positive can be okay it will just be bothersome to a user right he'll just have to go and say allow it but at least his files will be safe right so that's why in our method we tried to reduce the false negative but we were fine with a false positive so if you will see like in the paper the false positive is like kind of in the range of 15 to 20 percent right well you've talked a little bit about some of the features that the machine learning algorithm will train on but they're kind of interesting features like deleting a large file i suspect a lot of the listening audience would not know how to go implement that themselves could you talk a little bit about the tools you use to get access to these seemingly low level features that you're going to train on how we did was so each core has some a hardware performance counter and all of these are capable of collecting the micro architectural events but because of the limitations only four or less than four micro architectural events can be captured at a time right so basically what we did was we used the software such as liquid which can go and integrate into the performance monitoring unit in the system and it's provided by intel and then basically it allows you to capture the micro architectural events based on different groups so these groups could be branches cycle activity data flops dashes l2 data and so on so if you go and see in our paper there are like 16 different performance groups which in turn has different metrics for example branch has four different metrics branch rate misprediction rate miss prediction ratio instructions per branch so that is about how we collected the micro architectural events the another thing was these are ransomwares right so we had to be like very very careful in where we are running this so what we did was if we spin like a virtual machine so all of these ransomwares were like for windows so we spin a virtual machine run the ransomware collect the micro architectural events delete the entire machine and bring it back to the previous state where it was before running the ransomware so we had to do it like continuously for all different executions and the problem i mean it's good it's kind of replicating how a normal user will work right because there are multiple things happening like different processes and everything is like running in the system and at the same time ransomware is running and in this case virtual machine is running different processes inside that virtual machine are running in their normal execution and on top of that ransomware is running or a benign program is running and we are like hooking up the performance monitoring unit from outside the virtual machine and collecting the micro architectural event so interesting thing here is like so what we are collecting are coming from cpu directly like from the core there is like lots of noise so there is a host machine which is linux and then there's virtual machine which is windows which is running the ransomwares and all these micro architectural events are collected so basically we had to do it so that you know if the ransomware even escapes the virtual machine it will have a unfavorable environment in the linux so that it cannot go into the network is there any overhead to be paid for the collection of all these oh yes there was like a lot of overhead in terms of collection since we had to delete the entire virtual machine and like copy the new virtual machine back to its previous date that was time consuming like so much time consuming then there was this you can only record like four micro architectural events at a time so to collect more micro architectural events the same ransomware had to be run multiple times with different performance group attached to the pmu so it was like a repetitive process and it took us like a long time to collect the data set and even if you see from the machine learning perspective this is not a huge data set but from the hardware perspective it took so much time to collect that even this small data set yeah so once you've collected it in my mind you can represent that as a time series it's these the history of the events what the event was and when it happened which is great because it's sort of all the raw data it tells the full story and you can replay the history if you want but there's kind of a logical leap to go from that raw data to something a machine learning algorithm or at least maybe today's machine learning algorithms know how to process generally have to do some aggregation or feature engineering or something along those lines are there any steps you had to do before the data became useful for ml what we had in mind was if you see we don't know what these different ransomwares are doing in the back end right because we just had this raw ransomwares from the database and we were running and our objective was in a limited time frame which in our case was 20 millisecond we should be able to detect whether this is a benign or a ransomway the thing is the ransomware may or may not be you know encrypting or sniffing around the network or searching for the files at those particular time instances could be just lying there dormant for a while and then start searching there is like the context switching where basically that process is not scheduled by the cpu so we don't know at what time instance that process was running but we know within this time period there are certain time instances where the data is important so that's why we wanted like a the model which could have the past memory and have the right memory but which is also insensitive to the time intervals so that's why we chose lstm and basically to process the data we just normalize the data because basically it's like we knew the data is there i mean the key features are there but it's like if we try to manipulate the data we may lost some of the features so that's why we we did like the least minimum pre-processing as possible and let the lstm layer try to extract the features and then we perform the global average pooling to reduce the dimensions so that our next fully layer can actually extract the important data from all this like multi-dimension time series we have [Music] today many small business owners are busier than ever time spent searching for and interviewing the wrong candidates for a job opening could be time better spent growing your business that's why linkedin jobs has made it easier to get to the candidates worth interviewing faster and it's free create a free job post in minutes on linkedin jobs to reach your network and beyond to the world's largest professional network of over 750 million people focus on candidates with the skills and experience you need use screening questions to get your role in front of only the most qualified people then use the simple tools on linkedin jobs to quickly filter and prioritize who you'd like to interview and hire i have personally followed that simple advice and been very satisfied with the experience whenever i'm hiring especially for an important role linkedin jobs is always one of the places i think to go to top of list everyone's on linkedin and i've always been satisfied with the quality and volume of candidates i get linkedin jobs helps you find the candidates worth interviewing faster did you know every week nearly 40 million job seekers visit linkedin post your job for free today at linkedin.com slash data skeptic that's linkedin.com slash data skeptic all caps to post your job for free terms and conditions apply [Music] ah very interesting yeah and makes sense why you would choose those architectures were those broadly speaking inspired by other techniques or did you really have to come up with something novel for this use case we knew like in the past people have used so basically we knew people are using hardware performance counters for the malware detection and so on there have been some techniques in the past but when we were doing this so we came up with the lstm like we i mean we knew the hardware performance counters are important we came up with the lstm based model but at the same time there was another paper published who also used the lstm based auto encoders for malware detection we and that paper kind of published at the same time so that was interesting yeah very interesting great minds think alike i guess sometimes well let's talk about the results we mentioned already that you were framing it in the context of false positives and fallen negatives could you share some numbers with us what sort of statistical achievements did the model turn out getting this was our preliminary results right so because the data set is small right and if it has to be you know done on a large scale like to be adopted by industries right like anti-virus companies they need to be very large data set and the models need to be like retrained with new variants and so on each day right because they need to learn so with us even with the small data set we were able to like achieve around 80 to 90 percent accuracy in terms of classification so at this point the model is purely supervised classification but we also envision that this could actually be upgraded to forecasting as well for example like by understanding the micro architectural events of program for a certain amount of time you can forecast like how micro architectural events will look like in the future and based on that the model will be able to say that whether it is a benign or a ransomware so that is like a next part of the research but again even with the limited amount of data set we were able to generate and everything so on an average we were able to achieve 80 to 90 percent accuracy the false positives with 70 training data were around 20 to 15 percent for most of the branches like for most of the performance groups and false negative were around one to two percent so basically it allowed like very few ransomwares to be classified as benign but it was able to detect most of them so we've been talking about these lower level features things like essentially measuring what the cpu is doing even though i understand intuitively like okay ransomware is going to maybe be encrypting more than other programs it doesn't obviously follow for me how a model would work what it would look at in those features and what it would consider in making its decision i guess you're closer to the data though do you find the resulting model interpretable can you understand the mechanism of it or is it just kind of a black box so machine learning is usually kind of a black box right so we tried to see the data set right i mean to me like to the naked eyes the data set kind of looked similar for the benign and ransomwares but obviously if you try to separate machine learning is obviously seeing something else which humans obviously cannot see and they were able to identify something because you see like the lstm what lstm is doing in this case is keeping track of the past features as well right so basically the processor is scheduling different processes it's not like the ransomware is always executing on the cpu core right other processes can also come in so basically the events which you are collecting is noisy right but lst model is having a past memory of some events and combining all of these it's creating some like new features which obviously the human is not able to perceive and that's how it's able to classify between the benign and the ransomwares if i were an i.t administrator or maybe i oversaw a data center something like that i would be chomping at the bit to get your software into my industrial application because the features you're looking at they don't invade privacy they're fairly you know agnostic to what the program is and that sort of thing i would love to get this on all those machines and then have some indicators about whether or not you know there seems to be a ransomware infecting my system how far away do you think we are from this being an industrial application in that manner so there was a conference in our university and intel actually presented the paper and their work on exactly similar things so they are looking into using these micro architectural events which they have already embedded different sensors into the cpu and collecting data from that to provide the security so industry is already doing research into this field and soon probably will have some you know real life application coming out and when that comes of course there'll be a cat and mouse game and maybe hesitate to ask this because i don't want you to give them too good of advice but once the malware and ransomware creators learn about these techniques do you think there are avenues by which they can avoid them so obviously hackers are smart right so they're like always one or two steps forward than the verification of the security engineers because what they need to find is one loophole and as a security verification engineer we have to like close all the doors right for them to come into i mean i'll say it will always be a race you know where we are trying to protect and all they need to find is one loophole and systems are very complex and they're getting complex and complex so let's have our fingers crossed for sure you'd mentioned a couple constraints on this like having the setup with the vms and doing this properly in a secured way has you know an overhead resulted in a small but still promising data set so there's lots of ways this could be scaled up if the right investment was put into it what do you think are the most fruitful directions that should move in the future that's right actually so because this data collection and everything if you see when it comes to providing machine learning based solutions the major constraint is the data set right so you need to have a really good data set to make sense out of it and the industry is like you know the current anti-virus industries the companies i think they are already investing into this direction providing machine learning based solutions but yes if they invest more and collaborate with the universities and so i mean we could have a really good data set and we could like rapidly increase like ramp up the development of such mechanisms of the frameworks where we can protect against the zero-day attacks because i think machine learning is the only thing which can kind of forecast or classify for something which is not already present and provide protection against this zero day attacks absolutely yeah this is a very modern and real threat to the world that we need solutions for well newton where can people find you online if you go to the university of florida and search for fix research which is short for florida institute for cyber security research you will see me as one of their students and also if you search me on linkedin you can find me and this paper is already on archive and my email id is available there if you like literally search nitin punder uf i'm pretty sure you can find me awesome we'll have some good links in the show notes as well thank you so much for taking the time to come on and share your work this is extremely interesting stuff thanks kyle that concludes another installment of data skeptic time series our guest today was nitin pundir our sponsors oval edge and linkedin myself claudia armbruster as associate producer vanessa blige is guest coordination and our host kyle pulich [Music] you