Advancing AI in Cybersecurity with Databricks & Deloitte: Data Management & Analytics

Databricks · Intermediate ·🔐 Cybersecurity ·1y ago

Key Takeaways

Advancing AI in Cybersecurity with Databricks and Deloitte, leveraging Databricks to extend SIEM capabilities and integrate advanced AI and machine learning solutions for scalable, cloud-native architectures.

Full Transcript

All right, welcome to the breakout room session and I'm super super excited to hear from Kieran and Chris on advancing AI and cyber security with data bricks and deote and I'll hand it off to them. Great. Thanks. You got to you got to stay on your side. I'm going to take this one. Um, thanks everybody for joining us today. I appreciate you spending the time. Um, so we're going to talk a little bit about uh a couple of trends we've seen in the industry over the last few years. Starting with the fact that SIMs have in many cases, certainly traditional SIMs have somewhat hit kind of the the strain of their life cycle. Meaning people started dumping more and more data into their SIM over time. And eventually that's led to excessive costs as well as some performance issues and other challenges and getting SIMs to perform all of the analytics that we want them to do. Um, you add to that the the world of AI and a lot of the desires to use AI to do more advanced use cases in around and on top of SIM and the you know the volumes of data are simply increasing right the processing going along with it is also increasing as you consider you know trying to use more telemetry in your environment to get at some of these advanced use cases. So, so along those lines, we've been thinking about, well, you know, if the the data is going to continue to expand and SIM is still playing a particular role, how do we how do we make for a more flexible uh and agile architecture? And that's kind of what we're going to talk about today, right? So, um just a quick intro before I get into the slide. So, Kieran obviously uh been with a Deote for quite some time. I am the cyber AI leader for Deote in the US. Um and so in that role I spend 50% of my time working on solutions that we're taking to market and the other 50% of my time I spend on building solutions internally for Deote to basically transform the way we deliver services to clients and so I've got kind of the builder and the buyer hat uh which is helpful. Um I've uh been in the IT space for about 30 years to cyber for more than 20 and so I've certainly seen a little water under the bridge and somewhat have the gray hair to prove it. Uh Chris, do you want to introduce yourself real quick? Yeah, thanks Karen. Um, I'm Chris Knackstead. I'm a managing director with Deote. Um, I'm I work within Kieran's team to build and implement big data and analytics solutions for our clients to include a lot of solutions that are now being built upon some of the new trends and and um applications and artificial intelligence. Um, by way of my background, you know, very similar to Kieran, 20 years in um technology, uh, 10 in cyber security. Um through my work with um with Deote, I've implemented most of the SIM systems that have been on the market for the last decade um and continue to work with a lot of these new products that are now hitting the market. I'm sure many of the folks that are here um if you do work in the sim market, you've seen just this explosion in the technology space around here and a lot of the new technologies that are hitting the market and a lot of the new approaches that are now coming to market with regards to building um additional capabilities on the back of SIM, whether it be within the new capabilities that these SIM systems are offering on board as well as augmenting them with other, you know, general purpose data and analytics platforms. And that's really what we want to be able to present to you today is is how we're seeing the market adopting these general purpose data and analytics platforms. Data bricks being one of the ones that we're seeing in the market um a lot and how they're using that to augment the capabilities of of the traditional SIM system. Yeah. So um and we'll be kind of going back and forth during the course of the session. Uh we'll also leave time at the end for questions. A quick show of hands. How many of the folks in the room are actually in cyber in one form or another? Okay. Uh what about data science? Okay. About a 5050 split there. Developers, infrastructure, other Okay, there we go. All right. Got a few there, too. All right. So, we'll try and tailor it to the to the crowd. Um so, go ahead and flip a slide, Chris. So, I I won't talk through these because I know you guys are probably, you know, familiar with a lot of this, right? And there's a thousand things that have gone on that make cyber harder. Um, you know, the way I summarize it is I've been in the, as I say, been in the business for more than 20 years. I'm not sure as an industry we're better off today than we were 20 years ago because we're still doing the same stuff. We're still running around and patching systems. We're still looking at vulnerabilities trying to know where our assets are, etc. Um, so a lot of people look at AI and say and the data requirements for AI and they think about well you know how are we going to manage to to use that? How are we not going to create more vulnerabilities in the system etc. And the reality is the only way we're going to get do better is to fight fire with fire. So you're going to have to use AI in the cyber function in order to be competitive with the advancing threats that's coming at the enterprise. Right? Part of that message is also you can secure those systems obviously leveraging um cyber is the business aer or business enabler of adoption of AI um but that's a different talk and if you have interest in that space happy to chat afterwards um so as we think about kind of where we're trying to go we're anticipating that we have both sort of advancing threats as well as advancing needs right and the advancing needs is what sort of run outrun the architecture that we've traditionally used okay So you want to hit the next slide? So there are you know probably a few use cases we see clients most interested as it comes to leveraging AI in the business of cyber. The first one is threat detection and response right and that's what we're going to talk about today. But just to to kind of give you a perspective on where the industry is headed I'd say the second probably top priority is security compliance optimization. A lot of people focused on that. Third area would be attack surface management, right? The whole config, patch, asset, end toend process and then secure software development, right? And incorporating this into the secure off software development life cycle. Um, all that's going to require a large amount of data and beyond the telemetry we've historically used, right? Um, so let's talk specifically about AI and security operations. So you can see these are the the the statistics from a recent study where they asked a number of professionals as to where they think they can leverage AI to make advancements from a threat detection and response perspective. Not surprisingly improving detection of new or unknown threats came out number one. Um obviously there's you know using you can use a lot more telemetry and data. You can take a lot more approaches the IML uh than historically maybe have been able to. Um that goes to improving threat detection in general. You know, we start to get into autonomously responding to threats. Clearly, that's also where you start to overlap with what we're seeing from an agent perspective. Um gets into accelerating threat investigation, you know, automated remediation and recovery, reducing alert fatigue, and then, you know, fishing attack simulation being one of the questions they asked, but probably a smaller piece of the pie. You know it surprised us a little bit here is this reducing lurk volume fatigue. We see that as a much larger problem at our clients today and so surprised that that wasn't right up in the t in the front because that's the pain a lot of our clients have. So they're they're running their sock. They're not having any issues generating incidents or alerts. They're having issues triaging and responding to those alerts and tackling them within SLAs's because the volume is simply too high. number of false positives are too high etc. And again that's where data is going to be the only thing to allow us to sort of change that game along with AI. So so we think about what we're going to try and achieve there is sort of a you know kind of a hierarchy of needs right so to do AI in an advanced fashion you have to have the data to do it your AI is only as good as your data like I can't sort of express that point enough. A lot of our clients that we work with will go in and say they have, you know, use cases they want to go after, etc. And then you you ask the question, well, where is your data to allow that use case to be addressed, they don't know or it's not available or it's in three different systems or five different systems or 10 different systems and they don't have a way to pull it all together, make it accessible and make it available in a format that you can repeatedly build off of it. Right. So that's why this becomes so important and I'll turn it over to Chris to talk a little bit about the slide. Yeah. No, thanks Kieran. Yeah, to echo a couple of things that Kieran just mentioned, right? Like, and I'm sure that that many of you can um attest to this that a lot of our clients and a lot of folks within the industry are very much focused on the tip of the pyramid, right? They want to be able to build AI, implement AI. They think that AI is going to solve all their problems. And, you know, based on the prior slide that Kieran just went through, I think that there's a lot of um support for that comment, right? A lot of people are very bullish on the fact that AI are really going to help us solve a lot of the problems that we're having now by way of its power to um assess its power to extract sentiment from from various vast amounts of data. Um it's a its ability to help with solving a lot of these problems um that have been so foundational to cyber security. I think a lot of people at least a lot of people outside of cyber security perhaps don't realize the amount of effort and the amount of coordination that takes in order to get to a point where you actually are implementing AI and that includes you know some of these layers down here in the bottom of this pyramid which is really circumvents around or circulates around the ability to put together a data system that will allow you to collect data on an ongoing basis to do that in a a fashion so that you can handle the vast amounts and volumes of data that cyber security solutions and and you know telemetry producing solutions are are throwing on a daily basis and this is in the hundreds to multi hundreds of terabytes a day for just you know your average size corporation. Some larger corporations you can start going into pediat byte scale with regards to the amount of data that's being collected on a daily basis um in order to serve all the mission that a lot of these cyber organizations are now dep demanding of the data within their their agencies and organizations. And I I say that people outside of cyber security may may not have an appreciation for this because I know that within cyber security at least anybody who's worked in a sock can probably um attest to the fact that we have been laser focused on trying to solve the big data problem in cyber security for a long long time. And that starts off with your log management systems of a couple decades ago to your SIM systems. And now, you know, new technologies and new approaches are now coming in vogue that take a lot of the sentiment of what we what you show here on this diagram and it takes it to another level. And you're talking about some of these new age SIMs that have data lakes built into them as well as like these standalone data and analytics platforms that we're going to talk about. And so a lot of the things that we see in the market are, you know, clients really interested in being able to get the power of AI but not really understanding the uh the work uh that goes into building the foundational elements below that. We used this diagram here because it's a pretty standard diagram. Anybody who's a data scientist has probably seen this on the open internet, the AI hierarchy of needs. I I really like this diagram because it can really apply to a number of different domains and spaces and cyber security is surely one of them. Again, as I mentioned before, like some some folks in our organizations can make an entire career out of building these uh data transit systems, these brokering capabilities, and ultimately being able to get to a point where you can store and manage all the data that goes into the various use cases that organizations are now demanding of these data. And so we're going to go into a little bit um on both uh both ends of the pyramid. So we're going to talk a little bit about um what we're seeing in the industry and how organizations are kind of starting to fight this big data problem with the use of modern technologies such as data bricks as well as building in some of the capabilities that we're seeing on top of data bricks in order to start serving more more complicated AI problem. So we just wanted to to start off with just a basic diagram of what you know your traditional SIM system is. And for those of you who are not in cyber security, this just shows, you know, what we call the security event to insight pipeline. Meaning at one end of the uh the pipeline, you see all of these different uh data producing agents, endpoints, uh edge solutions. These are all the tools that are either producing an event that some sort of digital activity is happening or it is producing some sort of an alert to give somebody a trigger to say look there might be something here that you'd want to take a look at. Traditional SIM systems have always brought that data through a standardized pipeline which you can see here represented here by all these multicolored arrows and it brings into a data repository. Now, the reason why we represented this as a as a closed pipe as opposed to something more open is that a lot of traditional SIM systems operate just as that. There are these closed systems, you bring the data in, it's really hard to get the data back out and if you do want to bring the data back out, you have to kind of reformat it so they you can use it for a different purpose. And so a lot of the traditional analytics that are being built within SIM systems and a lot of SIM systems have spent decades putting together really curated um cyber security uh workflows for detection and response and remediation. Um most of that content stays within the SIM system and it's really difficult to bring in outside content um into traditional SIM systems and it's really difficult to be able to democratize that data outside of people who are just working in security operations. And so we see that as a a big limitation um to SIM systems, especially as cyber security organizations and other um adjacent risk uh capabilities and risk uh organizations within your enterprise really see the value of this rich telemetry and really trying to use this in order to build uh additional analytics use cases to include artificial intelligence use cases. And so Chris, just a a couple of points to add there. So certainly cost storage cost within the sim has been a challenge. I don't know how many folks have been, you know, challenged by the CFO or someone else as to why they're paying so much for storage. That's that's certainly been a topic in the last few years. And then the other issue is performance, right? And you even if you have the the budget, if you start shoving all the data in there, it's just not built to be that kind of platform, right? It is is intent to serve a purpose. It's not meant to be a general data platform in most cases. No, it's a great point, Karen, and that's actually a good segue to the next slide where we introduce the the concept of a a data lakeink. And a data lake is not a new concept. It's it's a term that's been around for quite some time. Um, cyber security is starting to adopt the the use of data lakes more and more um in order to help service some of these uh these you know data management issues and data analytics issues um that Karen had just mentioned. So one lowcost storage, right? SIM systems um more and more as they continue to build in a lot of these really intricate workflows and a lot of this content which is is really good content across these platforms that really hone in on key cyber security problems and um issues that organizations need to continue to monitor on an ongoing basis. the cost to store data in these SIM systems is becoming um you know really cost prohibitive and it's getting to the point where there's so much data and organizations are using SIM systems in order to store this data that it's not being cost competitive at all and it's really causing an issue um because a lot of these SIM systems just can't handle the capacity of data that's required. Um Chris just to jump in real quick. So, so number one, we get called like basically for, you know, family guidance services, right? So, someone wants to divorce from their SIM and they're like, "Hey, this isn't working and blah blah blah." But you find out a lot of the issues are not necessarily the SIM. It's the underlying process and approach that they're built on that is now no longer serving the organization very well. I'm just curious out of the crowd for those who know or have a perspective, how many of you feel like the SIM in your environment is being optimal way? Okay, just for the record, no one raised their hand. So, either nobody's involved or this is probably a fairly common experience, right? The other thing I'll mention is that um as you get kind of more advanced in understanding the threats targeting your organization, every business is different. So, a single product is not going to meet the needs of a very diverse set of businesses in this room, different geographies, different, you know, delivery channels. I mean, it's it's gets very nuanced. And so if you want to start looking at identity data, comparing that to what's happening over your IoT network and comparing that to transactions that are going through, you're going to have to do that on your own because no one is building that use case for you, right? So an important point. Yeah. No, no, absolutely. And you know, a lot of SIM systems are continuing to expand to include other risk domains just outside of the security operations system, which is where many SIMs have been targeted, a lot of their capabilities, but they're not really keeping pace with again the way that data is is um increasing within the organization and a lot of the demands that are now being put on those data. Um, we'll get to there in a second, but there's a number of different, you know, implementation patterns that we're seeing in the market. Some of which are very much directly tied to a um, a phenomenon that we're seeing in the market where the SIM systems are now building data links directly into their SIM platforms, which I think is a positive thing. Um but by and large the traditional SIM system which is more or less in the architecture that we just showed a moment ago have some limitations with regards to the capabilities by way um that it has to store the data to present the data to other users outside of um the SIM you know stakeholders themselves as well as being able to um adopt some of the more leading approaches you know particularly as it relates to machine learning and artificial intelligence to be able to build those in and apply those seamlessly on the data in the sim system. Again, Chris, sorry to keep interrupting, but you're facing the other way, so I can't I'm um so uh again a couple of points to add. So curious out of the folks in the room who raised their hand for data science, how many of you work on cyber specifically? Are do you specifically work on cyber use cases or are you doing broader? Right. So I think two people raised their hand out of that group. That drives the point home that the data scientists in your organization aren't on loan to the cyber team. Right? So, if you're picking your own path and you're trying to build something on your own, that's great, but you're going to be limited to resources who know your SIM platform. Right? If you tag on with something that's bigger, probably footprint in the organization that's generally being used for data science, you likely have greater pull through as a cyber organization to get help from others as well as integrate to to what they're doing across it in general. And so again, that's I think that's important because historically cyber sort of and you know, I've been in cyber for a long time, so I point finger myself. We've run off and we had our own toys, right? Nobody gets to touch our toys. There are toys and it's too secret and you can't touch it. But that doesn't actually work well in the data science world. No, that's a great point and and I think that this represents a convergence that we're seeing more widely in the market where a lot of the data science tools and capabilities are now converging with cyber security tools and capabilities. Um, as Kieran mentioned, you know, as SIM systems have grown and evolved, they've kind of done so in the image of the people using them, right? So, these are the security practitioners, the network engineers, those that ultimately worked in the security operations center. And with that, they spawned their own languages, their own syntax, you know, their own data models and schemas, right? Very unique to the to the trade craft and very unique to the domain. Um, as Kieran mentioned just a moment ago, a lot of what we're seeing now is a lot of these general data science, data management practices and principles, things that kids are learning in college now are now being applied to this domain. And so you're seeing now these really performant data systems like data bricks that will allow somebody to come in with just general knowledge of um standard SQL and Python and can really be really effective and impactful building out security use cases building out code that would serve security um problems. And with that, as we continue to progress in our conversation, you know, some of the things that we want to talk about specifically is how data bricks is now covering down on a lot of the capabilities that we're seeing lacking in the in a lot of the traditional SIMs. So, the ability to bring in a, like I said, a general purpose data system that provides the tools and the capabilities to build out more advanced use cases that gives you the ability to store volumes of data at a far lower cost than you would in a traditional SIM system and have more ready access to those data um with more traditional data science tools that you're seeing more openly adopted in the market. Also seeing the ability to basically proliferate the data um across the organization. So, not only can you use tools that are directly um connected to data bricks and those that are in the data bricks ecosystem, but you also now have the ability to connect the data to other tools, many of them being like your traditional cyber security tools like a SIM. Um, so we're seeing a whole lot of flexibility now being opened up with the advent of the the data lake and some of the capabilities that are now being offered on these platforms. And so with that, we'll take you to the next slide. Uh I mentioned this a moment ago but trends that we're seeing in the market. Um so as we think of data bricks as your cyber security data lakeink capability we're seeing our clients adopt them in a couple of different patterns. The first one that I'll speak about is on the very far left which we call the bolt-on model. And so this is um really as it as it shows in the picture. an organization has a SIM, they are looking for a way to relieve cost pressures and a way to um store the vast amounts of data in a different, you know, lowcost, high volume array. And so they're adopting data lakes like data bricks and they're basically just dropping the data out of the back of their SIM and they're putting it in a data lake. Now, there's both pros and cons to this uh solution. One of which the pro is is that it doesn't really disrupt the data pipeline that you have in place and all the other um you know technical debt that you've built up around the sim around the data pipelines and the data acquisition um scripting and things like that. And really what it does is it just bolts on a a uh a storage array to the bottom of the sim. What you lose in this particular case sometimes is that you're not able to maintain par between the SIM and the data lake and you're not really able to use a lot of the data that's going into the data lake because oftentimes that data is getting stored into the data lake in some sort of proprietary file format that doesn't allow like a lot of your general purpose tools to access and analyze the data. With that we we are seeing now the second model come about which is what we're calling the parallel pipeline. So in this model you see here that data is being pushed directly into your SIM to run your cyber security workloads as you normally would to use all the different logic and all the different workflows and things like that. And then you have data bricks or a data lake um ingesting uh either a parallel data set or some um derivative data set so that you can bring that down in there. And oftentimes we're seeing that the data that's going directly in the SIM gets very much paired down to only those data that are required to execute the alerts to run some of the workflows and to do some of the analysis that security operations folks generally need which generally consists of having data somewhere in the neighborhood of like 30 to 90 days old uh very specific to certain um you know trends uh alerts and um notables. The rest of the data goes into a data lake or into data bricks where your security data science team some of your other researchers can then use that data in order to explore the data further to build out other use cases or in the case of incident response you can have an incident response team you know have access to all the data to include all the raw logs so that they can parse through that data. That's the second model. Again the pros are is you're able to scale back the data that's going into the sim. So you don't have to bring all of your data into uh your SIM. And with a lot of the pricing models that are out there with SIMs, you get charged for the data that actually comes into the SIM. Whether it be, you know, the actual data that gets pushed into the SIM or how you process that data to normalize it against some sort of schema once it gets in there. Instead, you can only bring the data in there that you really need. You can bring the rest of that data into a solution like data bricks and you can manage it there. The cons are is that it includes some sort of duplication of data, right? So, as the the the picture represents, you're bringing at least some, you know, version of of both uh data streams into both the SIM and the in the data lake so that you can use them in parallel. So, with this phenomena, we're seeing the newest pattern emerge, right? And this is what we're calling the subscriber model. And I think that this is more at least at present this is the modern architecture that we're seeing that can really help with expanding the use of the data not only for the SIM system but also the ability to continue to use that data um for other capabilities outside of the SIM. And so what it does is it really effectively separates the data management capacity of a SIM system to the actual data and analytics workflows. And um it keeps them separate so that you manage all the data in a central data lake capacity and then you can use the SIM in order to run the analytics workflows um you know that uh that SIMs come with either out of box or that you can program into them. So Chris I would just jump and make a couple of points. So in the last scenario, um, chances are if you do start to get into more advanced use cases, leveraging I and start building agents and so forth, they're going to have to consume from a data lakeink as opposed to from a SIM, right? Or similar technology in that case. Um, so in our case, we run uh multiple flavors of security operations centers for our clients. And so because we're running, you know, across multiple clients, uh, we don't have a sing and they're not all using the same tech. So we don't have a single system that they're all using and touching. So we have to build and operate a level above that right. So we have to correct you know collect telemetry from all the systems deployed within client environments bring them in and uplevel it to you know a standardized approach across all of them so that we can obviously execute in a standardized way and do so efficiently. So if you're in a large organization if you have multiple divisions and you've got different tech teams out there etc. you might be in a similar circumstance. So again, the data lakeink model might be, you know, the smarter way to approach it. Yeah. No, no, agree. And, you know, to to Kieran's point, I think that the subscriber model that you're seeing right here is definitely aligned with where organizations are starting to build out their um agentic AI um ecosystems, right? in really being able to use a data lake uh to manage the data, manage the access to the data, and then have a lot of these agent-based systems and ecosystems just pulling the data directly from everyone's got to do a shot every time they hear Gent. That's right. Yeah, we wouldn't be here, right? We wouldn't be here if it wasn't for that. Um and so this diagram right here kind of represents that concept of the subscriber model. And so again, you know, using the security event to insight pipeline like we used to illustrate the traditional SIM model. We're showing here now the idea of this um separation between the data system, the underlying data system, which you can see there in the middle built from data bricks from all of the different consumers of those data, which would include up here in the top a SIM system could also include a soore uh a case management system. We have down here this SPCL are specialized tools. So say if you have a fusion center and you have folks that are doing fraud management, if you have other folks doing compliance, if there's other things that are being done around supply chain management, um these are all tools that can consume data directly from the data lake. And then also to Kieran's point just a moment ago, uh we have this represented in a different color because we we're trying to represent some of the capabilities that are available on the datab bricks platform. But this also includes the ability to build and develop and serve AI models using the data lake as the storage repository to pull those data from. In this particular representation, as you can see here on data bricks, we're using the Unity catalog in order to manage the schema in order to manage the the data. Uh Delta Lake is the storage array. Um again uh based on your hyperscaler of choice and then you know for those of you that are familiar with data bricks and I assume everybody is here we're at the conference um leveraging the various um storage levels in order to build out your raw log uh storage capacity some of your parsed and normalized as you start applying data models on top of it and it can be multiple data models. Everybody understands that um a lot of these SIM systems have their own proprietary data models and if you want to use more than one SIM system or if you want to use more than one system. The data lakeink allows you to develop multiple different data models and integrate them. And then the curated and structured data sets are really those um data sets like a feature vectors that are used for building out and running machine learning models and things like that. So being able to have those um built in there as well all in the same uh capacity and then use various methods in order to um you know establish what we call like a data and AI fabric to ensure that those data can be shared back and forth between the solution systems um uh you know back and forth again. Yeah. Can you really get the question like who keeps raw logs anymore? Um, yeah, the answer is like if you're not, you should be because there's a lot of there's basically a lot of context that's lost as you go up that chain. And in many cases, you're going to find that the what's lost is important for an AI use case. And so, you know, storing the the data in multiple ways is also a key strategy because if you summarize, you lose fidelity, right, in many cases. And so that's going to be, you know, probably an ongoing need. Yeah. I mean for for cyber security the mantra you know way back in the day is we want to keep everything and then we started running into storage capacity issues and they were like well maybe we can't keep everything and now that a lot of these capabilities and the storage is becoming cheaper um we're now getting back to the to the state where well maybe we can keep everything just in case we need it. And again I don't know about you Karen but as we continue to push forward in AI we continue to uncover new you know ideas for use cases to use data that we've you know hadn't used previously. So being able to maintain those data and keep those data not just for compliance reasons but also for you know research and development reasons has become very very impactful. One thing we did want to double click on just a little bit and again this is one of the things that you know you needs to be um explained a little bit as you start building out that subscriber model with a SIM system, right? And I think you know many people within this room kind of understand some of the data sharing capabilities within the data bricks platform, right? If you wanted to be able to run you know organization that has multiple different data bricks installations across multiple clouds and using a lot of the integrated features right there. Um this um diagram right here shows an integrated data mesh between a datalix platform and a commercial SIM system right which all the time they don't necessarily sit in the same cloud one might be on prem uh your data bricks um installation will be able to communicate with that via this what we're calling the data and AI mesh and again you can do so in a couple of different ways dependent on the sim system right you can connect it using um APIs other various connectors Um I think folks are familiar with um uh lakehouse federation as a sharing service and um Delta share as another sharing service between data lakeink and some of the other solutions. So data bicks is making it very easy to democratize data to other platforms that are not natively sitting on data bricks and um within a traditional SIM system really a lot of the capabilities that are required is is can you search the data from your SIM system to somewhere else and so that brings about this uh capability of federated search there in the middle below that you know there is you know principles of being able to share the data back and forth between the two platforms right because remember SIM systems also have their own onboard board storage facility that it uses. And some of those use cases that you would want to execute or some of the research that you might want to do, you might want to do it directly in your SIM system. So, but having a a data sharing and ETL capability from between data bricks and your SIM systems gives you the ability to birectionally share the data back and forth between the onboard data repository within a SIM system and then data bricks. And then lastly, some of the emerging capabilities that are coming that draw more parody between the SIM system and and the data lake and in this case data bricks is being able to do federated an analytics. So do I have a model? Do I have some code that I want to initiate in my sim system and execute in in data bricks? And so those are three different scenarios for which you can really draw a really tight connection and build what this AI and data mesh between your traditional SIM system and data bricks. Yeah. just and I know we're talking about SIM because that's kind of the center of the sock and a lot of focus from a cyber and an analytics perspective. I do want to point out though it's not just about SIM. There's a ton of data for example in your identity management systems, right? A tremendous amount of data, a lot of value. Historically, we've not seen clients integrate that into the SIM because of some of the challenges we've talked about already. And so really this is setting you up to start expanding into other data domains that can make you more effective and allow you get to some of the things we talked about in the beginning in terms of better threat identification and better response time and so forth. Thanks Karen. Yeah, before we open it up to questions, we're going to share one more slide with you guys and you know kind of continuing to persist this idea of drawing the integration between your SIM system and data bricks. Uh this next diagram illustrates how the data bricks data management system and again we've summarized the security event pipeline here to just show data management serving and insight consumption. But you can see here that data bricks can be used to manage the data help with serving the various data needs in order to run a lot of the onboard analytics that you traditionally find within a SIM system but also gives you the ability to build your custom AI models. And in this particular picture, it's it's the model itself is scoring data in a production environment and sending those insights by way of a score or uh a range or a rank and it sends it back into the SIM system um or sorry sends it back into data bricks so that it can ultimately be consumed by the SIM system and integrated with the various workflows that would require it. And you can see here that uh definitely continues to serve the security operations center. Um the models themselves can serve you know security data science teams research but then again also um and again through some of the work that we do we've built out various models that would um you know uh execute use cases that we just couldn't find a way to do so in the sim you know because the sim was either not able to process the amount of data that we needed or it didn't have a capability such as like access to a deep learning algorithm or something like that in in the um sim itself. So we would do that work in the data lake, send the insights back um into the storage array and then ultimately reconsume that into the index.

Original Description

Deloitte is observing a growing trend among cybersecurity organizations to develop big data management and analytics solutions beyond traditional Security Information and Event Management (SIEM) systems. Leveraging Databricks to extend these SIEM capabilities, Deloitte can help clients lower the cost of cyber data management while enabling scalable, cloud-native architectures. Deloitte helps clients design and implement cybersecurity data meshes, using Databricks as a foundational data lake platform to unify and govern security data at scale. Additionally, Deloitte extends clients’ cybersecurity capabilities by integrating advanced AI and machine learning solutions on Databricks, driving more proactive and automated cybersecurity solutions. Attendees will gain insight into how Deloitte is utilizing Databricks to manage enterprise cyber risks and deliver performant and innovative analytics and AI insights that traditional security tools and data platforms aren’t able to deliver. Talk By: Chris Knackstedt, Managing Director, Deloitte & Touche LLP (Sponsor Speaker); Kieran Norton, Partner, US Cyber AI Leader, Deloitte Consulting (HQ) (Sponsor Speaker) Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements Connect with us: Website: https://databricks.com Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/databricks Instagram: https://www.instagram.com/databricksinc Facebook: https://www.facebook.com/databricksinc
Watch on YouTube ↗ (saves to browser)
Sign in to unlock AI tutor explanation · ⚡30

Playlist

Uploads from Databricks · Databricks · 0 of 60

← Previous Next →
1 Building AI Agent Systems with Databricks
Building AI Agent Systems with Databricks
Databricks
2 Databricks Workflows
Databricks Workflows
Databricks
3 Automate Unity Catalog Upgrade with UCX Part 1: Overview
Automate Unity Catalog Upgrade with UCX Part 1: Overview
Databricks
4 Automate Unity Catalog Upgrade with UCX Part 2: Installation
Automate Unity Catalog Upgrade with UCX Part 2: Installation
Databricks
5 Automate Unity Catalog Upgrade with UCX Part 3 - Assessment
Automate Unity Catalog Upgrade with UCX Part 3 - Assessment
Databricks
6 Automate Unity Catalog Upgrade with UCX  Part 4 - Group Migration
Automate Unity Catalog Upgrade with UCX Part 4 - Group Migration
Databricks
7 Table Migration and Catalog Design with UCX | Part 5
Table Migration and Catalog Design with UCX | Part 5
Databricks
8 Setting Up Azure Access for UCX Table Migration | Part 6
Setting Up Azure Access for UCX Table Migration | Part 6
Databricks
9 UCX Table Migration: Creating Catalogs and Schemas | Part 7
UCX Table Migration: Creating Catalogs and Schemas | Part 7
Databricks
10 Automate Unity Catalog Upgrade with UCX  Part 8: Code Migration
Automate Unity Catalog Upgrade with UCX Part 8: Code Migration
Databricks
11 Streaming to Kafka Just Got Easier with DLT Pipelines
Streaming to Kafka Just Got Easier with DLT Pipelines
Databricks
12 Data Engineering From Data to Dashboards with DABs: Crunching the Cookies Dataset
Data Engineering From Data to Dashboards with DABs: Crunching the Cookies Dataset
Databricks
13 Epsilon helps businesses connect with their consumers using Databricks Data Intelligence Platform
Epsilon helps businesses connect with their consumers using Databricks Data Intelligence Platform
Databricks
14 Unilever transforms operations with GenAI using the Databricks Data Intelligence Platform
Unilever transforms operations with GenAI using the Databricks Data Intelligence Platform
Databricks
15 ActionIQ enables businesses to unlock customer data with the Databricks Data Intelligence Platform
ActionIQ enables businesses to unlock customer data with the Databricks Data Intelligence Platform
Databricks
16 Mixed Attention & LLM Context | Data Brew | Episode 35
Mixed Attention & LLM Context | Data Brew | Episode 35
Databricks
17 Inside Databricks SQL: Engineering innovation with Hans
Inside Databricks SQL: Engineering innovation with Hans
Databricks
18 Inside Databricks: Engineering innovation with Michael Armbrust
Inside Databricks: Engineering innovation with Michael Armbrust
Databricks
19 The Money Team at Databricks: driving revenue and customer growth
The Money Team at Databricks: driving revenue and customer growth
Databricks
20 Unity Catalog unveiled: engineering data governance at scale
Unity Catalog unveiled: engineering data governance at scale
Databricks
21 Create a view in Databricks and share it with Power BI using Delta Sharing
Create a view in Databricks and share it with Power BI using Delta Sharing
Databricks
22 NDUS leverages Databricks Data Intelligence Platform to revolutionize higher education management
NDUS leverages Databricks Data Intelligence Platform to revolutionize higher education management
Databricks
23 Démo Databricks de AI/BI
Démo Databricks de AI/BI
Databricks
24 EMEA Data + AI World Tour 2024
EMEA Data + AI World Tour 2024
Databricks
25 GenAI: The Shift to Data Intelligence - Customer Panel on Industry Use Cases
GenAI: The Shift to Data Intelligence - Customer Panel on Industry Use Cases
Databricks
26 GenAI: The Shift to Data Intelligence - Ft. Ash Jhaveri, VP of Reality Labs Partnerships at Meta
GenAI: The Shift to Data Intelligence - Ft. Ash Jhaveri, VP of Reality Labs Partnerships at Meta
Databricks
27 Virtue Foundation leverages the Databricks Data Intelligence Platform to advance global health
Virtue Foundation leverages the Databricks Data Intelligence Platform to advance global health
Databricks
28 Announcing Synthetic Data Generation in Mosaic AI Agent Evaluation
Announcing Synthetic Data Generation in Mosaic AI Agent Evaluation
Databricks
29 AI/BI Dashboards Embedding - A tutorial
AI/BI Dashboards Embedding - A tutorial
Databricks
30 Bayer transforms global data management with the Databricks Data Intelligence Platform
Bayer transforms global data management with the Databricks Data Intelligence Platform
Databricks
31 Databricks at AWS re:Invent 2024
Databricks at AWS re:Invent 2024
Databricks
32 Hive Metastore and AWS Glue Federation in Unity Catalog
Hive Metastore and AWS Glue Federation in Unity Catalog
Databricks
33 Data + AI World Tour Paris 2024
Data + AI World Tour Paris 2024
Databricks
34 Retail reimagined: Currys data-first strategy to driving growth and improving operations
Retail reimagined: Currys data-first strategy to driving growth and improving operations
Databricks
35 Mixture of Memory Experts (MoME) | Data Brew | Episode 36
Mixture of Memory Experts (MoME) | Data Brew | Episode 36
Databricks
36 Verana Health Data Curation and Innovation with Databricks and AWS
Verana Health Data Curation and Innovation with Databricks and AWS
Databricks
37 Securing SaaS Applications: Obsidian Security on Their Journey with Databricks and AWS
Securing SaaS Applications: Obsidian Security on Their Journey with Databricks and AWS
Databricks
38 Twilio Eng VP on Data Intelligence & AI at AWS re:Invent 2024
Twilio Eng VP on Data Intelligence & AI at AWS re:Invent 2024
Databricks
39 Chegg Eng SVP on Data-Driven Approach to Student Success with Databricks and AWS
Chegg Eng SVP on Data-Driven Approach to Student Success with Databricks and AWS
Databricks
40 Ibotta Personalized Rewards Innovation with Databricks and AWS
Ibotta Personalized Rewards Innovation with Databricks and AWS
Databricks
41 Simplify AI governance with #databricks AI Gateway
Simplify AI governance with #databricks AI Gateway
Databricks
42 Databricks SQL and Power BI Integration
Databricks SQL and Power BI Integration
Databricks
43 Databricks Serverless SQL Warehouses
Databricks Serverless SQL Warehouses
Databricks
44 7 West powers audience growth with the Databricks Data Intelligence Platform
7 West powers audience growth with the Databricks Data Intelligence Platform
Databricks
45 Secret to Production AI: Tools & Infrastructure | Data Brew | Episode 37
Secret to Production AI: Tools & Infrastructure | Data Brew | Episode 37
Databricks
46 Skyflow CEO on Data Privacy with Databricks at AWS re:Invent
Skyflow CEO on Data Privacy with Databricks at AWS re:Invent
Databricks
47 Databricks Clean Rooms Product Demo
Databricks Clean Rooms Product Demo
Databricks
48 Dun & Bradstreet Enrichment & Monitoring, powered by Delta Sharing & Databricks Marketplace
Dun & Bradstreet Enrichment & Monitoring, powered by Delta Sharing & Databricks Marketplace
Databricks
49 Unpacking Libraries in Databricks
Unpacking Libraries in Databricks
Databricks
50 Providence uses an AI agent system from Databricks to help doctors improve their communication
Providence uses an AI agent system from Databricks to help doctors improve their communication
Databricks
51 How State Street Uses AI to Transform Millions of Trades Daily
How State Street Uses AI to Transform Millions of Trades Daily
Databricks
52 Vevo Therapeutics CEO on Curing Disease with Data at AWS re:Invent
Vevo Therapeutics CEO on Curing Disease with Data at AWS re:Invent
Databricks
53 Over Architected with Nick & Holly: Databricks updates for Feb 2025
Over Architected with Nick & Holly: Databricks updates for Feb 2025
Databricks
54 The Power of Synthetic Data | Data Brew | Episode 38
The Power of Synthetic Data | Data Brew | Episode 38
Databricks
55 Use Databricks Lakehouse Federation to break down data silos
Use Databricks Lakehouse Federation to break down data silos
Databricks
56 AI's rugby score: National Rugby League rallies fans with analytics and unified data
AI's rugby score: National Rugby League rallies fans with analytics and unified data
Databricks
57 Open Variant Data Type in Delta Lake and Apache Spark
Open Variant Data Type in Delta Lake and Apache Spark
Databricks
58 How would you sort Ætheldred in the alphabet using Databricks?
How would you sort Ætheldred in the alphabet using Databricks?
Databricks
59 A guide on how to operationalize the Databricks AI Security Framework (DASF)
A guide on how to operationalize the Databricks AI Security Framework (DASF)
Databricks
60 Future-Proof Your Asset Performance Management with Generative AI - Field Assistant Live Demo
Future-Proof Your Asset Performance Management with Generative AI - Field Assistant Live Demo
Databricks

Deloitte and Databricks are advancing AI in cybersecurity by extending SIEM capabilities and integrating advanced AI and machine learning solutions for scalable, cloud-native architectures. This approach enables more proactive and automated cybersecurity solutions. By leveraging Databricks as a foundational data lake platform, Deloitte helps clients design and implement cybersecurity data meshes and deliver performant and innovative analytics and AI insights.

Key Takeaways
  1. Design and implement cybersecurity data meshes using Databricks
  2. Integrate advanced AI and machine learning solutions on Databricks
  3. Extend SIEM capabilities with Databricks
  4. Deploy scalable, cloud-native architectures for cybersecurity
  5. Analyze cybersecurity data with AI and machine learning
💡 Leveraging Databricks as a foundational data lake platform can help unify and govern security data at scale, enabling more proactive and automated cybersecurity solutions.

Related Reads

📰
Top 20 Cyber Security Interview Questions for Freshers
Learn the top 20 cyber security interview questions for freshers to boost your career in cyber security
Dev.to AI
📰
Managing a Million Security Profiles at the Edge
Learn how to manage a large number of security profiles at the edge of a global network by predicting site defenses and traffic patterns
Medium · AI
📰
The First Five Minutes After an Employee Clicks a Phishing Link
Learn what happens in the first five minutes after an employee clicks a phishing link and why it matters for cybersecurity
Medium · Cybersecurity
📰
IDOR simple way with no burpsuite banged P3
Learn a simple way to identify IDOR vulnerabilities without using Burpsuite, and understand the impact and reporting of such vulnerabilities
Medium · Cybersecurity
Up next
Best VPN For China 2026 — Which One Actually Works?
Tutorial Stack
Watch →