Rethinking Web Application Firewalls

Key Takeaways

[Music] you're watching the new stack makers a podcast for people who develop deploy and manage at scale software for more conversations and articles please visit the new stack dot io now on with the show hi gara the inventor and maintainer of open source calico delivers calico cloud the next generation cloud service for kubernetes security and observability tigera and the new stack are under common control hey it's another episode of the new stack makers and today i am joined once again by rattan tipanini who is the president and ceo at tigera rattan thank you so much for joining today thank you alex thanks for having me here good to see you again definitely great to see you and this is the second part of a discussion that we had and some of the topics that we discussed in our first discussion we talked about how the workplace is changing quite a bit how that's really forcing a whole new move to the to cloud services we're seeing the growth of cloud native applications in the process but the question still remains about the approach to securing those cloud native applications and how is that actually done so how do you identify how do you assess how do you prioritize how do you adapt to risks across the application how are organizations really thinking about this from a technical perspective and so retin i wanted to follow up and maybe the first thing we can do is talk about one theme that we really touched on and that's related to web application firewalls now we talked in february and in february we have since february we've had a number of vulnerabilities uh discovered uh what has been your thinking about the evolution of of waf when you think about what we're seeing in the attack landscape and the preventative measures that companies are taking or not taking at all yeah so alex i think vaf has existed for a long time for several decades that technology has been very impactful in thwarting attacks at the application level however in the context of cloud native applications there is a fundamental assumption of traditional baf architecture that breaks down and so there's a need to rethink how that layer of security should be applied so more specifically waffles always applied the security control was applied at the edge either in the cloud or through like cdn players and that worked really well when you had a set of static applications and there was a concept of a pentameter however with cloud native applications with the microservices distributed architecture you have to assume that something inside your cluster has been compromised so just sitting behind a vat that's sitting in the edge doesn't give you adequate protection you have to assume that every single micro service container is almost open to the internet metaphorically speaking so then the question really is how do you apply graph controls and just to complicate things these these down these workloads happen to be dynamic so you can't really statically program web controls around any specific workload so that's where you know we feel architecturally that the vav controls have to be workload centric where every individual workload has its own valve so to speak and when an application or container gets when a microservicer container gets spun up the valve controls automatically get spun up around that so that way even if something inside your cluster is compromised or if you're exposing some of the services to the internet it really doesn't matter because you're not protected by the vaf and so that's a fundamental architectural change that is required now the additional benefits are with this type of an architecture uh the vaf now has complete context around the application uh which is not feasible when the graph is sitting at the edge or on the cdn so so that really is one of the fundamental changes that has to be driven when deploying security for cloud native applications so you have to rethink your architecture for the waff and start to apply them those security controls at the workload and you need a workload-centric graph you need a workload-centric graph and there's lots of different types of workloads so is that part of the re-architecture to consider those different types of workloads yeah exactly you know and and again you know when you have a powerful orchestrator like kubernetes you know spinning up these containers with these workloads uh you have to be able to think about graph controls in a similar way it's programmatic those get spun up dynamically uh whenever a workload gets spun so what are some of the fundamental differences that we see in waff as it evolves yeah i'd say you know let me start with what remains the same right so if in terms of the basic the security rules like the owasp top 10 and a bunch of intelligence that's been built to catch some of the traditional application centric attacks the good news is i think a lot of those a lot of that software those libraries those rule sets are still applicable right so that's the good news right you get to leverage all that stuff but the difference is you have a non-trivial challenge now of how to actually operationalize deploying a workload-centric wave and that's that's what has changed significantly and it's very different and we're hearing from customers that the vaf solutions they've been relying on for the last decade or two are really ineffective uh inside kubernetes and microservices-based cloud-native applications and they're looking for something different because some of the verbs sitting on the edge first they miss the context of the application they're getting a lot of false positives and so so that's not very helpful and worse they don't have complete visibility to all the traffic that's coming into these cloud native workloads so those are the three big challenges we're hearing from customers and architecturally you just need to be able to think of a different way of deploying workload center graphs and uh you know that that's what we're doing at target so i know that uh you know the firewalls have changed considerably uh that's clear um it's not perimeter based anymore uh but what about uh what about detecting uh anomalies and and issues uh what uh what attention do you have to put on the behavior um you know that you see the behavior patterns that you see in the network how do you get beyond uh you know practices that once were effective such as uh signatures right and you know there's other issues too you know such as you know it once you it seemed like you could trust any api but you know now it's increasingly being uh a subject of attack uh those apis themselves yeah so let me answer it in two parts so the first part is i mean you're absolutely right the game has changed and let me start with a very simple example if you think about vulnerability scanning to detect known vulnerabilities uh that was a pretty well understood problem in a pretty straightforward solution for traditional application architectures however with cloud-native applications and also with the explosion of vulnerabilities it's no longer sufficient to throw out a report that tells you about all the vulnerabilities in your system because that report is not actionable because what people who are operating these services are discovering is that the amount of time and effort it takes to remediate all these vulnerabilities is incredible right so what they're looking for is they're looking for some level of prioritization in terms of way to start which are the vulnerabilities that i need to first attack which are likely to cause the most amount of damage in your system and so this is this is a fundamental change so the rules of the game have changed so the the important challenge right now that customers are struggling with is can you figure out the blast radius of a known vulnerability in the context of your system it has to be contextual in your system and you can only do that at runtime and then once you understand that once you understand the blast radius and the criticality of some of the workloads that maybe a potential vulnerability is actually impacting it automatically gives you a sort of list of vulnerabilities that you have to manage so that's the first part the second part really is related to that understanding you know the the the surface area and how you minimize the surface area of attacks is super critical because again in this world of cloud native applications customers are discovering very quickly that trying to protect every single thing when everything has access to everything else is an almost impossible task so the higher order bit for them is to start with understanding how they reduce the attack surface with a denial model and only allowing microservices which need to talk to other microservices to be able to talk to each other so that is a very highly leveraged activity and a security control that can actually stop a lot of these attacks now the third part which really coming to more specifically to your question is after having all done done all that stuff you're still you have to assume that you're still going to get attacked mostly because you know there's always the threat of an insider attack and in that situation you're looking for patterns of anomalous behavior where there's unusual activity maybe at the process level at the file system level uh or maybe the system call level and you're looking to baseline what the standard behavior is and you're looking for deviations from the behavior and then trying to tease out some signals which are indicators of either attack or indicators of compromise and and once again you know you do have to lean on some level of machine learning to be able to do that and and just to add to that maybe a simpler use case of that is to constantly be able to monitor at runtime for uh known bad uh hashes or files or binaries that are known to be bad right so so the challenge is is the the good news is that you know in the security community there's there's a finite list that you know the community contributes to we we all know what that list of hashes are of bad files are the challenge is how do you constantly monitor and recognize those hashes inside your system and you have to be able to implement that security control which is another layer of defense so so just to summarize you know this is this is what i've just articulated is defensive depth there's no single silver bullet you have to be able to do multiple things to keep your applications safe inside modern cloud native architectures interesting uh you know perspective there uh what is that forcing you know network providers to do and you know uh because those those behavioral differences can be quite granular uh you know and it to me it speaks to uh the evolution of observability no you're right you're absolutely right now in terms of the network providers or cloud providers uh just given what you said you captured in your question they're limited in what what they can do because a lot of this is contextual and it depends on your applications your services so some of these security solutions are to be specific to the workloads you're deploying so there's only so much i think that cloud players all the network providers can do the onus is really shifting to the companies and teams deploying these applications and really they're the ones who have to go deploy these solutions okay uh and so how do you see observability evolving oh yeah as a practice sure yeah so sorry i missed the second part of the question you asked earlier observability is very interesting right i think with we tend to think about it is first of all if you look at the last 10 years observability security have been two silos and i think last time we spoke i talked about you know those two silos converging and specifically it's no longer sufficient to report on security incidents in isolation where you spit out a report of all the vulnerabilities or all the issues inside your system you have to contextualize it and so the contextualization like as an example in our case uh we have a dynamic service and third graph which shows visually which services are talking to each other services it shows an aggregation of name spaces it shows which services are actually talking out to services outside the cluster sitting on the internet or which services are accessing ips and clusters of ips so there's a nice visual representation of what is happening inside the cluster and it's like magic because you know as soon as it pops up you know people uh actually get the first time understand what's actually happening inside the cluster now you start to overlay security information on top of that it starts to become really meaningful for them because you can talk about a service a shopping cart service that probably has a vulnerability but then you can actually visually start to see which other micro services that shopping cart is actually talking to in the context of the dynamic service and thread craft and when you do that you automatically have a perspective on what the blast radius is for the vulnerability and based on which other microservices it's talking to you can then decide if the other workloads are mission critical or not and if so what mitigating action you take maybe you decide that the blast radius is pretty huge in terms of the impact to mission critical workloads and you need to actually put some remediation or mitigation in place as a short-term measure before you get some remediation and you actually stop you you roll out some security controls to stop the flow of traffic to these other microservices from your shopping cart service until you have a remediation right so that's a great example of how observability and security kind of work with each other and once you roll out the security control you can actually then visually see through your observability feature set whether the traffic indeed has start going to these adjacent micro services so that's a very simple example of how observability security are starting to converge and i think from a trajectory of where we feel the industry is going all conversations about security will happen in the context of observability or something like a service graph because without the context just talking about absolute data about security is not as meaningful or as valuable to security analysts or to the people operating this infrastructure and applications [Music] and so this is these are early times what are some of the tools tool development that you're seeing there's a need for or what and what are people relying on in the meantime yeah actually you know there's there's quite a bit of complexity in what i've just described i've articulated a pretty simple case yeah but uh in reality when you have maybe hundreds of microservices running on thousands of containers across multiple clusters uh the amount of you know the data in what a human has to process is pretty staggering and even if you have the best visibility tools it can still cause and cause an information overload so so a couple of things you know one is we refer to it as the leanback experience where we try and take the burden away from the user and try to do a lot of the inference ourselves and present conclusions to users of what they can what they should be doing so so that's a huge opportunity for innovation over the next decade uh what we represent in the and the that is in terms of the inference correct exactly is that inference um i've i've heard a lot of discussions about inference at the edge for instance um but we're talking about inference on just the overall network correct uh the yeah and and the micro services running inside the network and how they're talking to each other the potential impact than indicators of compromise or indicators of attack so inferences on any of these dimensions are very powerful and the second related concept to that is so you discover something and but it could take a significant amount of time to remediate it because remember that the the practical aspects of remediation are you've got to go back to the source of the soffit the developer get her to fix it and come back and test the fix and roll it out that could take days weeks and months right who knows but in the meantime you're faced with a dilemma like do you shut down those services or do you take the risk and keep them open knowing that you've got a pretty big security hole inside your cluster so that's where the concept of mitigation comes in so the meeting the concept of medication is that once you've identified a security hole can you put in place controls that temporarily maybe block or quarantine a specific service or container or microservice that's been identified as questionable until you have an opportunity to get a permanent fix that's mitigation and second can you automate the mitigation can the system automatically uh propose uh controls for you to quarantine that specific container in question without you having to manually go configure and then can you test it by staging it right can you stage it and just watch the traffic flow and say has this thing really been quarantined or not and then promoted to production right so applying a lot of the software engineering principles to security where think about it like a ci cd system but you're doing this during operations and you're automatically not only detecting these you're trying to figure out the blast radius then you're trying to figure out you know how do you apply mitigating controls you have the system proposed what the mitigating controls could be you test those mitigating controls by putting it in staging if you're satisfied dive with that by visual inspection through observability you promote that to production uh a lot of this is is is right for innovation over the next decade so software engineering principles apply here uh to the management of for instance inference models uh the mitigation controls that you're putting in place and really the decision making that you have to do who are these teams that you're seeing do this work yeah so the teams really are you know anyone managing security operations are really the teams doing all this heavy work and just going to maybe build on the first part of your question there for a second alex uh you know in terms of applying software engineering principles to security in fact i'd go so far to the extent to say that honestly that is the only way we have any hope of being able to secure these cloud data applications because when you think about why the traditional security models like firewalls are failing they were all hard-coded right they were saying hey this specific ip address you let traffic in the specific ip address you don't let traffic in so those are examples of hard-coded rules and there are customers with i'm not exaggerating tens of thousands of these rules and they're afraid to deprecate any of those rules because they don't know what's going to break because some of those rules were configured 10 years ago right and so the modern way of handling security is to treat it like software and you assign labels to workloads and you say these workloads are red workflows these clouds are blue these are pink reds can talk to blues and blues cannot talk to pings and then if you need to change some of the rules you're just changing the software or the rules around that and everything gets reconfigured or if you introduce a new workload you say this workload you tag it as blues or reds or pins or a combination of that and they inherit all the rules so i mean thematically you can see how easy some of this is once you start to apply the principles of software engineering to security [Music] so what is the level of seniority on these teams that you're seeing really adopting these technologies and like how do you see those teams um being composed who are the people on these teams but you talked about security op security operations people is is that the whole team or is it is it something different or or what so it's a little bit of a hybrid but the first part of your question in terms of seniority so so one thing that's interesting is that a lot of this adoption and a lot of the innovation and the thought leadership is happening uh bottom-up right it's not being driven top-down so you may have someone without any fancy title but they have the vision and they know how to do this and they're really driving it they're getting the rest of the organization to adopt it so this has very little correlation to someone having fairly senior titles the second part of the question is it is very cross-functional in nature and that mirrors how the software for microservices and cloud native is getting deployed it's a combination of developers uh it's a combination of some someone who has responsibility for handling the platform they may have different titles and it depends on the size of the organization definitely someone in the security organization and someone inside the devops team now i've listed four roles but it's not unusual in smaller companies maybe all these four roles are rolled into a single person and she's handling all these roles right so that's very possible but as you as you get into larger organizations each of the roles i talked about maybe there's a whole team behind that but they have to work together to pull this off they can't work in silos what's the intersection of what we're talking about with these software engineering principles and how they apply to for instance the management of inference models and other these topics too what we are seeing in the evolution of of laughs and protecting the the network over the not just protecting the overall network but the applications and the services themselves yeah so i i think if you start to apply i mean think about software engineering most of software engineering programming languages are built on a concept of abstraction i think you know if you look at some of the advances we made in software i would argue one of the core reasons is better tools and better programming languages and what does that mean better abstraction because just cognitively a programmer can do more things because a lot of the complexities hidden from the programmer you know you don't have to worry about the registers and the bits and the bytes you're operating at a different level of abstraction right so that's how in the last two to three decades you know our other the software engineers today are so much more productive compared to like a decade or two ago because of that level of abstraction so the concept is very similar i think in cloud native security that you bring in that level of abstraction and you start to as a simple example you start to assign labels in identity to workloads and then you start to configure rules that that allow you to dictate the behavior of these workloads what they can and cannot do so what that takes away is it takes away the physical dependency on where does that workload live right it's no longer attached to an ip address a particular system or particular infrastructure it really doesn't matter right and when you need to change the rules you probably need to just go change one rule instead of going in changing 53 different hard-coded you know points of reference to a specific ip address right so that's a very simple example of of of what i'm talking about and some of the inference we talked about is really built on top of or is layered on top of that once you start to baseline the data of how workloads blue workloads have been talking to red workloads for the last 23 days and suddenly you see a difference in how those two types of workloads are communicating with each other or some of those workloads start to make some calls out to the internet that you've not seen in the past 23 days it starts to bring up a question of why why are the blue workloads behaving differently right because you now have data from 23 days that tells you this is supposed to be even a different way and then it gives you the ability to probe deeper through signals from process behavior file system behavior system called behavior and then start to narrow down and get to what could be different and what could be the source of the problem and then the rest of what i talked about so if you identify the problem then what do you do about it from a mitigation perspective before you wait for a final remediation that could take weeks or months can the system tell you automatically what rules and what policies security policies to configure and allow you to test them out through staging and then you promote them into production that entire workflow uh so all those i'd say i'd say our our principles in some form or shape we're borrowed from software engineering so that's the beauty of it right we don't need to reinvent a lot of these things we can actually dip into the rich history in the repository of software genetic practices that's a really fascinating topic and i could talk about it for a while you know i'm particularly interested in the ramifications for how this may positively affect an organization so if you can start seeing those calls out to the internet you know that you can start tracing those calls to see need to learn more really and that goes with them way beyond your network itself but also into the extended network and the extended supply chain really and uh that seems to be a a kick off for another conversation another time but i want to thank you very much john it's been a it's been a real interesting discussion i always enjoy getting into the teams and how they're working because i think you learn a lot through the work that they're doing because they're the ones who are really working on the on the projects so thank you for that perspective absolutely no thank you and we learned a lot from our customers and working with them so i completely agree with you alex has been fun thank you so much for the time if you like this video please give us a thumbs up and if you'd like to see more videos like this you can always subscribe to our youtube channel we're on all the major social media platforms you can always find us at the new stack dot io we hope to see you soon [Music]