Creativity and Choices: Talking About Thinking

Key Takeaways

support for the blueprint podcast comes from the Sans Institute ever since the debut of my blue team fundamentals course in 2019 I've had students ask if there's a management specific course that pairs with it well the wait is finally over if you like the topics covered in this podcast and would like to learn more about blue team leadership and management check out my brand new 2-day course management 551 building and leading security operations centers this new management track course is designed for sock leaders looking to build grow or improve their security operations capab abilities through improved analysis technique false positive reduction better metrics and continuous stock assessment don't think that just because this is a management course it won't be Hands-On throughout the two days there are six Hands-On Labs that show you step by step how to use tools for organization and implementation of analyst playbooks sock use cases threat intelligence purple team assessment planning and much much more check out this new offering at Sans url.com 551 hope to see you in class this is the blueprint podcast bringing you the latest in cyber defense and security operations from top blue team leaders blueprint is brought to you by the Sans Institute and is hosted by Sans certified instructor John Hubbert and now here's your host John [Music] Hubbert on today's episode we talk investigation and Analysis with experts Steph Rand and and Chris Sanders if somehow you haven't run across these folks before stay tuned for a very interesting and unique conversation about how to understand and improve your own alert analysis technique on this episode Chris and stuff explained their recent research on understanding what happens in an analyst's mind as they approach triage analysis we cover topics like convergent and Divergent thought and how and when to use each their newly developed ADC model and how understanding your own thinking process can make you a better analyst all coming up today on the blueprint podcast [Music] all right thank you for joining us Chris Sanders and Steph ran today on the blueprint podcast so I'm really really excited about this interview thank you so much for coming on especially during this crazy crazy time I know you probably have a very busy schedule going on right now with everyone working from home and all the wild stuff going on in the world so really appreciate you coming on and explaining some of this awesome research you guys have done thanks for having us yeah absolutely so to get started I want to have you guys do a quick intro I know you both have a very interesting background as does everyone in information security all of us seem to have come from some kind of circuitous route into this field and I know both you have some interesting stories as well so Steph if you could go first and let us know your story and then what brought you to infosec and what brought you into this kind of research sure thing my name is Steph Rand and I'm a recent graduate from Augusta University I got my degree in it with a concentration in cyber security I did not plan on going into cyber security I thought General it was going to be sort of my scene but we had so many amazing professors and amazing classes at my University that infosec just pulled me in I think it was my first in to networking class we fired up wire shark and I got to see packets and I was like that's the internet this is the coolest thing I've ever seen and I think that's what did it for me my prior background this is my second career I previously did Clinical Psychology and got a masters in that in 2012 and that's how I ended up doing the research I went to one of my first information security conferences which was security onion and when I was talking to folks and they said oh you have a background in Psychology have you talked to Chris Sanders have you met Chris Sanders and so we connected and started talking about different research projects and it just sort of took off from there very very cool yeah psychology I know is an interesting field that has some really cool kind of intersection with what all this research is and I'm really excited to get into that before we get into that sort of thing Chris Your Story please so this is also my second career my first career was when I was 13 as a roadside fruit salesman I figured out pretty quickly that that was very seasonal and wasn't going to be something I could build a long-term career out of so I got into computers and seemed to have stuck with it and here I am for some reason but now I was in a computer from an early age I was hired by the school district I gradu uated from as their first network administrator I made what was maybe a mistake at the time of installing a very early beta version of snort on network that had never seen inent detection before as most did not at that time I saw a lot of things that I could not unsee but I kind of got interested in investigating things from there because I would run them down I would find evil and it was fun to find evil and kind of built a career on that from there so combination of that and eventually moving into some of the psychology focused facets of research into what we do which I'm sure we'll talk about here in a little bit but mostly focused on those things and and spend most of my time researching investigating teaching all of those things surrounding investigations yeah awesome so yeah with the Clinical Psychology thing my wife is actually in psychology as well which obviously I know nothing about myself but I try to soak in through being near her Brilliance and so this is going to be a really interesting conversation I hope when it comes to the research paper you guys did you're looking at the way the analysts were approaching analysis what they were thinking about how they were approaching the problem what was the thing that brought you to kind of bring up this question as something to research and why was it something you were interested in pursuing you know one of the interesting things that I found and I'm glad to admit to everyone listening I'm not a very smart person and I was really not a very good analyst for a long time and I really struggled with it one of the things I did a lot of was asking other analysts who were good at it I said why are you good at this how are you good at it and most of them couldn't give me an answer which I found very frustrating at the time and now that I think I'm at least marginally better of an analyst now it's still frustrating to me because we need to be better than that we need to do better than that so when you start examining our field on that level looking at all of these different facets about how we think about the things we're doing there's a lot of things that need to be researched and what we chose here with the use of Divergent and convergent thought was really just one facet of that because John as you know step certainly when we want to examine something kind of in a scientific manner you have to slice with a very fine knife you have to be very specific and that's what we did here with Divergent and convergent thinking just as one facet of how analysts think and how they use that in day-to-day investigations very cool so Steph having a background in Clinical Psychology and all that did that feed into how you designed this experiment and Chris as well how did you come up with exactly the experiment you wanted to do and formulate the tests that you were going to run and find the people that you were going to use with this experiment we didn't really do an experimental study we did a lot of interviews we did some qualitative research so since this is so new there's not a lot of research looking at how security analysts are approaching their investigations we needed to lay some groundwork we needed to have some like Baseline Theory and so we decided that the best way to do that was to interview a number of analysts and see if we could find some common threads that we could start to pull together and from that we could start to build a model build this picture of what analysts were already doing in their investigations and put some terms to it again give us that base to start more research from with a common Foundation a Common Language you know some definitions we could all agree on yeah and so speaking of definitions some of the things in the paper metacognition convergent Divergent thinking I think a good place to start for this discussion here would be kind of explaining some of those what they are and how they relate to the process of analysis let's start with metac cognition because that's I think everything kind of goes back to that and it's there are a lot of fancy definitions for it but I think the simplest for our purpose is thinking about thinking it's being aware of your own thought processes because if you're aware of those thoughts then you can kind of Leverage those to your advantage you can think different ways in different situations you can apply those and you can also teach those which is really helpful when you're teaching new analysts as well so that's metacognition and then Divergent convergent thought our own awareness and talking about those shows some level of metacognitive awareness which is pretty cool and those are really two sides of the same coin so I'll talk about Divergent I'll let Steph talk about convergent Divergent thinking is when you really have some type of prompt whether it's a question or a problem something like that and you just generate possible answers you're almost creating a list so for our purposes in the sock it's often an alert and we have an alert and there's a lot of things you can investigate or a lot of things you can pursue to figure out what's going on so with Divergent thought you take that initial alert and you just make a list of all the things you could do and that is diverging out into all these different areas of potential investigation and then the other side of that coin is convergent thinking which is kind of what it sounds like you have all of these different options and you Converge on the best one you take a look at the different possibilities and you sort of narrow down to what you think will be the one best choice or the one best solution for whatever problem or situation you are faced with at the time awesome so an example of that might be like someone gets a snort alert or something that says user X went to this website that is a known bad website and your convergent thinking might lead you to think oh maybe it's truly a bad website maybe it was a bad website but now what's you know an expired domain maybe it was a false positive that someone put on a thread Intel list something like that like the different options of what might be truly happening is that correct diver sorry diverent yes that's what I meant yes and then the convergent would be picking the best one of those options which one you think is the most likely gotcha and so in the paper you discussed how these kind of relate to creativity and how the creative process is kind of tied into analysis and conversion and diversion thinking Steph could you explain a little bit about how that works and why that creative piece is of interest when it comes to these two terms the creative piece is interesting because different kinds of thinking have been associated with creativity in different ways so traditionally Divergent thinking was associated with creativity and generating lots of different ideas but to get a better more holistic picture of what the creative problem solving process is like you do need to pull in these other different pieces so creativity is not just generating new ideas and New Concepts it's also being able to use your thinking to again do that convergent piece to to narrow your options down and find the best solution yeah so it kind of takes both pieces to well someone with higher creativeness maybe able to do both of these things better as opposed to just the Divergent piece which maybe traditionally associated with creativity correct it's both coming up with the options as well as evaluating the options it's it kind of plays a part in both of those is that correct it is and important to note too I mean convergent thought doesn't really exist without Divergent thought you can't pick the best thing from a list if you don't have a list so these things kind of wrap together into this notion that exists in the broader psychology field that's called creative problem solving because to start with this Divergent exercise you have to or at least it is correlated with creativity and larger amounts of creativity so theoretically if you're going to pick the best thing from a list you want the list to include the best thing and that may be more likely to happen potentially if you're more creative that's little outside the scope of our study but there's some relationship there and they're all kind of work together sure so given these definitions now with the research paper that you have done what was the main hypothesis or what was the question you were trying to solve around these terms in the process of analysis through this paper the main thing we were looking for was essentially we knew just through observation that analysts use Divergent to convergent thought sometimes and we really wanted to know just more about that the facets or the degree to which they so is it used at a specific time in the investigation process is something something people generally start with we looked at some different facets of that in terms of expertise whether you know junior level versus senior level more experienced analysts used this in different ways we looked at it from a few different angles not all of them produced enough meaningful results that were able to write about those but some of them certainly did at least in terms of when people choose to use these things compared to when they do not and what was the process you used to interview the analyst like what was the group of people how did you find them like how did you analyze what they were doing we'll be back after a quick break if you're enjoying this episode then you're undoubtedly interested in building the strongest security operations team that you can for those who want to go even deeper did you know that Sans has not one but two courses that cover security operations centers as well for the leaders managers and directors out there my co-author Mark Orlando and I offer 551 building and leading security operations centers this course covers building your team your physical and virtual workspace getting the right data into your tools and then focusing on security priorities through everyday EX ution of important security tasks and building the best sock team possible for the technical practitioners out there my course SEC 450 blue team fundamentals security operations and Analysis is designed to cover everything you need to jump in being the best sock analyst that you can be we cover important data types sock tools security logs malware analysis technique Automation and much much more in addition if you want to prove you can deliver the best on any security team both courses have an accompanying certification available from gak that's the gson for 551 and the gck for 450 check out both courses and free demos available on the Sans website you can get registered today for an in-person course at one of our many events or go to on demand and take either class anywhere at your own pace thanks for listening so we basically put out a call for volunteers we kind of had some baseline criteria they were pretty simple in that you had to have some degree of investigative experience we kind of picked folks across a few different domains of inv people who would consider themselves more stock analysts versus digital forensic analysts versus maybe like detection Engineers things of that nature we also excluded folks who had taken a CL my investigation Theory class because we didn't want to have Poison the Well so to speak because I teach about those things and within those groups we we basically split it into a couple of things one of those the one I mostly LED was the qualitative side of that where we basically interviewed people recorded those interviews transcribed them and then went through this process of coding them which you basically take out things they say and assign those to codes and look for themes look for things like co-occurrence sequence of occurrence stuff like that and we try to tie those to Divergent and convergent action so we would ask them some scenario based questions like okay here's an alert I would describe the alert to them and I would say you know how would you go about investigating that and basically through what they said we were able to code things and map those to instances of convergent or Divergent thought there was also a quantitative side as well which is Step will talk about here a little bit that was more her area I also helped put together the code book that we used to code the responses and that was actually really fun I got to go into the psych literature and find different traits and behaviors that were traditionally associated with the different kinds of thinking so one of the pieces that's tricky is if somebody's not already very metacognitively aware they're not going to be able to say oh I used Divergent thought to create a list and convergent thought to pick the best option we needed to listen for cues in the responses to sort of let us know that they were using those kinds of thinking so for example one of the traits associated with Divergent thinking is being able to come up with lots of different ideas so whenever we saw somebody generating a lot of different ideas to a scenario we could say oh that is use of diversion thought so that was a really really fun piece of the research I really enjoyed that okay so basically we were doing these questions and you were saying here's a scenario like describe to me what's going on in your head and then you were converting what they were saying they were doing into what you found to be in the literature divergent or convergent traits and then looking for patterns throughout that to try to come up with conclusions on well come up with a model actually of when and how analysts are actually doing this so through this process you guys did have some success and came up with an interesting model could you describe what you found with what you call the ADC model sure so ADC stands for ambiguity driven convergence which is a really fancy name but it really boils down to a couple of things and one is we found that analysts don't generally start with this whole interplay between Divergent convergent thought most of the time most analysts that we talk to started by just using their intuition using their gut that is they would immediately basically use kind of that system one immediate type thinking to say if this then that that's what they would go with and we can talk a little bit more about that later that's not always the best way to go about it there's some value and deliberate thought so we found is a couple things were happening one is most of the animals we talk to possess a little bit of a lower tolerance for ambiguity it's one of the things that drives them when I say ambiguity I mean things being unclear so when you combine that with what we identified as high stakes situations you saw instances where Divergent and convergent thought occurred now H situation can a number of things the most common we saw were when an analyst had run out of leads they kind of at their end of the width and they don't know what to do next another was social situations so situations in which the analysts were working in a team or when they had to report these to a superior or something like that so there are a number of hake situations we didn't capsulate all of them but those were a few so the combination of this lower ambiguity tolerance plus these hake situations generally saw analysts revert to some form of Divergent and convergent thought they didn't of course know that that's what they were doing they might say oh I made a list of things often times they would physically separate themselves from the scenario they would say you know I got up and I took a walk and of course what they were doing they're taking a walk was thinking of all these different possibilities all these alternate timelines they would say you know they went to lunch they took off for the day they went to listen to some music any number of things there's often some physical separation there as well although not always but nonetheless what we have is this kind of transition from an initial bit of intuition into the use of Divergent and convergent thought which we thought was particularly interesting because High situation you use this line of thinking well that means it's probably valuable right so there's value to it and that would lead to some of our recommendations later on gotcha so yeah I mean this made total sense to me when I read it I know you know I spent years as an analyst myself and I can remember the thousands of times that I saw an alert and I was like oh I know what this is I've seen it a lot of times before it's easy to say it's you know you're playing the stakes right you're saying it's probably this thing because I've seen it this many times and every other time 90% it was that so I'm going to kind of Chase that thing first but then they hit these high Stak situations and that's where I thought it was really interesting that makes a lot of sense to me like you would get to that point where you're like oh well now it really really matters right we're at a fork on the road here I need to be very very careful about this do you think that's a rational approach to take I mean obviously it seems to make sense in the moment but you kind of plan the stakes in the beginning and then later on you're like oh now it really matters right you found that there was some issues with that I guess and so what was it that you found in terms of the correct process that people should be doing versus what's right and what's wrong about doing it that way I hate to use the word correct process because that's exactly what a better process yeah I use the word correct but I will say what we found there were some processes that were more fallacious than others and one of those was this initial jump to intuition and intuition is not a bad thing but it's not always a good thing either and particularly for newer inexperienced folks like we all have some sense of intuition that's going to develop as you get experienced someone with one year of very centrally focused experience is going to have a lot poorer intuition than someone with 20 years of really Broad and diverse experience right so we all tend to want rely on our intuition but there's tremendous value in this slow deliberate thought and one of the things we found was we went through these exercises with analysts where we had them generate lists of things about how they would investigate specific scenarios that we gave them and then we had them basically tell us what they would do and when without any type of intervention or anything like that most of the folks who said what they would do initially well it wasn't really the best thing and what we would do we would kind of go through and I would basically rate these things I would rate the quality of the analyst responses here and from their list the thing that they would pick the first thing was gener not the best thing the best thing or some good things were certainly on the list they were there but the first responses or at least the first things on their list were not always the best so for instance in your scenario say you have a snort alert for users downloading something bad you may make a list of 20 different things you could pursue in that chances are the first one two three four five things are not going to be the thing that you're actually going to choose to investigate your intuition is not always going to be the thing that drives you in the right way so again kind of to summarize what we found was when people created a list there were good answers on their list but they were rarely the first second they were rarely the first few answers they were kind of things on down the list when these analysts were forced to be more deliberately thoughtful about what they were pursuing and it wasn't just by our estimation as well when we asked the analysts to pick what they thought was the best place to start in their investigation they weren't picking the first thing on their list most of the time they themselves would pick something usually from the second half of their list so without really consciously knowing it they were more once they had to think about it they thought the answers in the second half of their list were the ones they would go with they picked them as better yes that was one of the things that was really interesting to me because you know when you kind of play this scenario out in your head you kind of think well I see the alert and I've maybe seen this alert thousands of times I've chased it down and I found out what it was and it was this thing most of the time what it seems like you're finding is that when people are forced to go beyond when it likely is they actually are more likely to find that if they had gone with that initial answer it was wrong right does that imply that maybe what they thought that alert has always been 90% of the time in the past was actually Incorrect and maybe they've been doing stuff wrong or what builds that intuition if people are drawn intuitively to a certain answer and then you found that it didn't turn out to be where they chased when given a whole bunch of answers I guess how do those two things which seem to disagree with each other you know where does that come from so I think in terms of what builds intuition I mean intuition is to some degree directly drawn from experience and if we get the same alert a hundred times and we investigate it the same way every time we're building kind of this mental muscle memory into our intuition that says this is the way I'm always going to investigate it and if the result is always the same we're always going to jump to that conclusion right it's interesting when you see people break that intuitional mod that they've had created when they maybe move jobs because what happens when you move jobs is we have very different data sets and IND sock that you're going to be in different tools so I may have gotten this Alert in job a and I'm always going to use pcap to investigate it and I always get to the same conclusion well then I go to another job I maybe I don't have peap maybe that's just not something they have and I have to look at the host logs and then all of a sudden I run across or I miss something where there was this other facet of it that's occurring on the host that I missed because I was only looking at Network logs so diversity of experience is really important here and that we're not just looking at things the same way every time because intuition is good but intuition has blindness and that blindness is often expressed as bias towards one way of looking at the data one way of arriving at a conclusions so the more diversity of experience you gain the better equipped you are to combat that in our field tactically speaking that's generally going to be through different data sources different ways of looking at the data different ways of transforming it things like aggregation statistical frequency analysis things like that that diversity is really critical so I think I see where I might have been confused here what I was I guess reading and probably misunderstood was not not that it's the conclusion that they were coming up with as further down the list but it's the way that they were investigating it was further down the list and they found a better way to investigate not a different answer and so uh if you investigate the same way over and over you might continuously come to the wrong answer and that's not really the question you're chasing it's more of which way should you investigate to optimally lead you to whatever that answer happens to be all of our research is this is counterintuitive because we really do care about the conclusions of our investigations but really most of our research is is really about the path you take to get to them sure because that matters and I think there are a lot of people who are maybe not as metacognitively aware who would say well what does it matter if I get to the right place and I'm like well it matters for all the reasons we just talked about this diversity of experience the potential for bias to come in always looking at the data the same way versus looking at it in different perspectives the path certainly matters and I would say it matters the most awesome so we found that intuition is not the right necessary First Step some other things that you found the lack of metacognitive awareness that was another really interesting section of the paper could you explain what you found kind of under that umbrella and how that ties into you know the convergent and Divergent thinking and metacognitive awareness of newer people in the industry versus maybe those who are more experienced because there were some really interesting things in there as well around stunning Krueger effect and all that one of the things that we did was we did do some quantitative analysis so we gave our participants some tests to take and one of the tests we gave them was looking at metacognitive awareness and we also had our participants rate their own level of experience and we rated their levels of experience and we kind of created these different experience groups based on their self-report and also our perception of their experience level and those in the beginner group believed themselves to be more metacognitively aware than those in the expert group so the beginners had higher scores on the metacognitive awareness inventory than the experts did and they were much higher than the intermediate level skill Group which was the lowest of the three as far as their own perceived metacognitive awareness they were saying they thought they were more aware of what they were doing than those who were experienced thought they were aware of what they were doing yes yes which is a good example of the den and Krueger effect which is again not just something that we see in cyber security that's something that happens in a lot of different disciplines where beginners I think it's partially because they don't know what they don't know but beginners do tend to overestimate their level of expertise and their level of knowledge yeah that was a really interesting result when I read that I was thinking wow like scientific proof of the D in Krueger effect right we always joke about it in infosec presentations and we see the dip and then you know the oh crap I didn't realize how much I don't know you actually found that and put it into quantitative terms that that does actually occur in the real world as a real thing so I thought that was a really kind of fascinating thing that that came up with when it comes to combating these problems the intuition piece that leads into these investigations seems to be the thing we don't want to do like that's the first step that where things maybe go a little bit off the rails how can we begin to correct that kind of the issues that that can bring up what would you suggest as a way to start putting things on the right path right from the start so there are really two things here I'll let Steph talk about playbooks here in a minute I think the thing I want to talk about is this notion of deliberate practice and that's getting used to using Divergent and convergent thought as part of your process so not just relying on it when things get tough you're in this high stake situation clearly we know it's important because that's when we're using it anyway so let's put it right at the Forefront and let's use it as part of the process and also as a training exercise so there's a couple ways to go about this one of the things I like to do as an exercise and socks is do what we did kind of in the interviews with our research study is give people some type of input often some type of alert and it could be a snort alert circot alert it could be a sigma rule something like that and just take that maybe break into groups or individuals either one and come up with a list how would you investigate this what data sources would you look at the way I like to do it because I like doing things question based is say what questions would you ask to investigate this alert and you know you spend five or 10 minutes on that you list the questions rapidly you don't really stop to think about them or discuss them so much just as they pop into your mind you list them down and you come up with well maybe a pretty big list 20 30 40 different questions you're going to pursue and then from there you switch kind of into this convergent mode you say of all these questions I created let's talk through this let's be deliberate what are the maybe top three that I would pursue if I were actually in this investigation and why and from that point that becomes a pretty useful exercise because now you're deliberately practicing this thing particularly as a group because then you can talk about it and learn about how other people are coming at this problem which is going to be a little bit different than you are you're getting some of that diversity of thought which is absolutely critical and repeat this certainly do it maybe throughout your day but at a minimum if you're in a group in a sock environment do it once a week with everyone it's a really quick thing you can do it in maybe 20 minutes and by doing that deliberate practice it gets you conscious of how you think it gets you able to switch into Divergent and then convergent mode and while you're doing this you're also building kind of this library of humanistics which is kind of useful for Playbook development and I'd also like to throw out two thumbs up for this group exercise for giving your less experienced analysts a chance to hear how the more experienced analysts are thinking about problems and approaching them you know that's one of the ways that the metacognitive awareness piece really can be leveraged to make our field a lot better is being able to train new analysts and get them up to speed more quickly and by doing these exercises as a group and getting the less experienced analysts some more insight into the process they will get better at their jobs quicker which is great and that's another thing that the playbooks do is as you sort of develop these good investigative choices for these different scenarios you can start to again have these fistic create these lists have these wonderful choices that have been crowdsourced by your team as places to start when you have different kinds of alerts and different investigative inputs and we don't want to say that if it's too prescriptive if it's too strict then you don't give your particular your less experienced analysts the chance to explore and gain some more experience and really kind of explore the space but it is helpful it gives them structure it gives them a place to start and sort of boosts the quality of their investigative responses up a little higher a little faster which is wonderful yeah so I'm really excited you guys took this into the realm of playbooks because this is one of the things and I'm glad that's the conclusion you guys found as well because that's one of the things I'm constantly saying in classes is playbooks are kind of a double-edged sword they're great because they lead newer people down the generally right path right but if you take them too far and you overfender you can't manage that kind of a Thing If you say these are the steps Thou shalt follow these steps then people can't be creative it doesn't allow for the FL an aren't robots right yeah exactly and nor do they like feeling like them right exactly and that's one of the reasons in some places the job of a sock analyst isn't so fun because they're expected to be robots to just go through this prescriptive set of steps and that's not what so analysis should should be it's not what it has to be and I feel so bad for the folks who are in those jobs and they hate it and they're like how could I ever you know they look at a guy like me who spent my career as in and around sock and like how could you do that and I'm well let me show you the lot brothers and sisters this is how it can be done we're not robots it's humancentric let's take advantage of the fact that we're humans with these really crazy powerful thought processing things in our head capable of Divergent and convergent thought let's use those things let's do the thing I'm talking about where you're deliberately practicing this stuff which de facto helps you do the thing that Steph is talking about where you create these play books that are just enough to keep you going just enough to keep you thinking cognitively diverse but still letting you do the fun part of this which is running down evil catching bad guys and having fun yeah that's one of the things I'm consistently saying in nearly every talk in class I give is people hate feeling like robots if you want to grind your sock down and have a retention problem you know that's the fastest way to do it right and so playbooks are one of those things it's like everyone wants them because we want to say we have repeatable process right and we do want semi- repeatable process but we have to have that flexibility built in there with what you found here what level of specificity and like what would be an example of a Playbook that you think like what would the steps in kind of General flow of like how specific would you get with a Playbook could you give an example of what one of those things might be or this level of specificity you would have in one sure so I think when I recommend folks to kind of get started with playbooks we're not doing this down to an individual rule or alert level we're generally doing it in terms of categories so you may have a Playbook that is for all HTTP based malware or for all incoming spam that somebody clicked on the file or is for you know a back door on a system or weird beaconing break it down into categories which most of your rule sets are already in these categories anyway so you can kind of map your detection to this sort of thing which is kind of slick and then from there again I do think those things question based I believe that a question well stated is a problem half solved so if you just list out the questions you would want to answer and you can probably tie those to data sources if you'd like but really just list out the questions so for instance if I'm dealing with HTTP based malware well some things I'm probably concerned about was what was the user browsing when they got there what type of file was downloaded you know is there some type of exploitation in that file what happened immediately after this right we're concerned about the timeline coming into this the timeline going out of it what is the properties of that host what is the nature of the user that uses that system are they you know in finance are they in it is this something we expect this user to download a Powershell script right is that something within the scope of what they would do so really just these kind of high level questions the idea is you know we could talk for hours on what makes a good investigative question but really it's all about you want to ask a question that is specific enough to be answerable with the data sources you have available is the general idea so keep it to simple questions you can list out the data sources that will answer it really no reason not to if you're in an individual sock that may be harder at the mssp level but as long as you keep it to those levels of questions and you're flexible on those questions and you give analyst the ability to add to and modify and change those questions and freestyle a little bit from them then I think you're headed down the right path awesome yeah I love that advice you know specific enough to be answerable in kind of a meaningful way one of the other things I usually do when it comes to Playbook creation is I have all those investigative questions and I tend to divide things into and maybe this is a good idea maybe this isn't but the questions for investigation to figure out what happened as well as the like response actions you need to take in order to get in the way and disrupt that attack so you know taking emails out of inboxes if it's a fishing wave and checking if people went there blocking it with the proxy and all that kind of stuff that's usually how I approach Playbook creation I think that probably meshes fairly well with that sort of thing it's kind of like the questions need to answer the stuff you need to do and together hopefully it creates one kind of comprehensive response is there any other like followon questions that this research has brought up in your mind oh my goodness so many I mean there's directions that we could go in from here how do you measure analyst Effectiveness like how do you measure when they have this metacognitive awareness how can you tell whether their investigations have improved or not that's a question that we need to answer how can we get better at training you know what kinds of training is effective and what do we need to start to do to really develop this kind of medic of awareness in the discipline how do we make this bigger how do we get it out there so that's definitely some of the questions that I have yeah as Steph mentioned earlier our research here was non-experimental we didn't introduce out of control group and an experimental group there's no intervention that occurred but I think that's probably a next logical step is you know we've talked about some things that we've seen work and I've observe them work in the socks that I've looked at but it'd be cool to measure that because I want to see not just who does it work better with versus who does it not work better with to what degree does it work how do we make people better analysts and just answering that question involves a lot of things because if I want to make someone a better analyst I have to be able to measure how good of an analyst they are in the first place anyway and you know there's some very broad attempts at doing that most have not been super effective we kind of explored some rudimentary ways to do that as part of this paper which worked for our basic purposes but really that's a study in itself is how do you quantify the expertise of an analyst and to answer that question you have to ask more questions such as what does an analyst do and you would be surprised at how many people can't fully articulate what an analyst does part of my I'm in the middle of a doctorate right now and my dissertation is basically a cognitive skills assessment of the digital investigative role and that's spanning all investigative roles so analyst malware reverse engineer incident responder all those things because we like to think they're a lot different but they're really pretty core similar just with some tool differences so mapping out the task that we do the thought process that go on that's going to be a start point a kicking off point to a lot of other research and it's going to help us answer all these questions that's kind of the Beauty with this kind of research is it makes you ask cool questions but then answer them you have to ask a lot of other questions and answer those you have to ask a lot of other questions and before you know it your entire career is mapped out and there's too much work to do good problem there to have right more work than we know what to do with so very very exciting stuff glad you guys were able to come on the podcast and explain all of this this is some really cool research and I'm really thank you for coming out and breaking this down for us I think this is one of the things that the industry really needs and like I had kind of said at the beginning of this I believe you two are some of the only folks investigating this so I hope to kind of spread the word about this kind of research and maybe have people help you next time around for follow on Research or Inspire others to do the same because like we've all kind of said here you know analysis can be really really fun when done right and we can make it better in a more kind of scientific process once we all kind of understand the factors that go into that and where it can go right and where it can go wrong in terms of other things you guys are up to Chris what kind of stuff you got going on besides this any other research or other I know you got your doctorate so you're probably super busy but anything else going on right now yeah the doctorate is the big thing in the dissertation that's probably still a couple years out which is you know that's closer than it was last year so we're getting there we're making progress I'm hoping to release new paper later this year that's going to be built in part on some of the research Steph and I did along with a little bit of new research where I'm talking about something called diagnostic inquiry which is the investigation process as I teach it and as most folks are conducting it so I've written about that before on my blog at Chris sanders. org but I'm formalizing that up as part of a paper and hope to have that out this year so that's really the thread I'm currently pulling right now along with the dissertation thread and again my hope is once the dissertation completed that opens up the gateway to a lot of other research some mindes some steps some of us together some by a lot of people we've never met or never heard of the idea is you know infosec and Academia have not really played well together for most of the existence of infosec a lot of good reasons for that a lot of crappy reasons for that so we're trying to bridge the gap in that a little bit because we need each other we need academic methods to get better at thinking about thinking and make our field more solidified and make it better for the next group of folks because we all know it's important it's only going to be more important so that's the stuff we're trying to do fantastic Steph what's next for you well I'm still working on kind of getting my own investigative process honed because I just started working at firey mandiant in inite response and so I'm trying to take all of these lessons that we have learned from other analysts and apply them to my own investigations which sometimes is you know much easier said than done because that intuition super powerful it looks shiny you want to follow it and so I'm getting some good opportunities to practice what I preach and get some more experience in the field in general yeah I'm sure you will see some very very interesting scenarios and kind of be on the rocket ship of interesting learning and situations you may find yourself in in that kind of role oh my goodness it's been a blast it's been fantastic yeah I can only imagine the kind of cool data and stuff you get to see over there so that's fantastic Chris I know you involved in the RTF created the RTF R technology fund can we get a shout out for that that's a outstanding kind of resource for a lot of folks I want to make sure we get that squeezed in here so everyone knows about that yeah absolutely so I've been fortunate to be able to travel all over the country I grew up in a very rural area and one of the things I've learned is that Talent is distributed pretty equally but opportunity is not and particularly in rural areas as it pertains to not just cyber security but really any type of technical computer oriented education there's just not nearly enough opportunity for kids coming out of those areas I was one of them and fortunately that's still it was an issue for me it's an issue still for other people so the RTF is an organization I founded a little over 10 years ago we go into rural areas work with teachers and give them the things they need to get kids interested excited about technical careers and if they're already excited we give them the things they need to pursue that and have a leg up so that they will you know pursue a career in that ideally pursue some additional education in that usually at the college level and so that they'll be able to compete once they get there so they don't go in start out find out their outmatch by their suburban and urban peers and drop out because that's a big problem we deal with as well so we work with them we donate things like 3D printers raspberry pies arduinos robotics kits all over the country we're in all 50 states to this point we put technology into the hands of over 100,000 kids so we're pretty proud of the work and if you're interested in learning more about that you can check us out on social media at R Tech fund or rtech fund.org outstanding Steph any organizations you are a part of or any other causes you want to shout out I've gotten to help out for Rural technology fund and do some fundraising and I just think it's a wonderful project that would be I'm GNA use my shout out for RTF as well it's fantastic I promise I didn't pay her to say that I'll also shout out a couple other things I want to make sure people read the the paper because if you're interested in the stuff we're talking about here well we wrote a whole paper about it which is really great I'm biased but it's really good it's