Understanding the Application Pattern for Effective Monitoring

Key Takeaways

I [Music] cisco is a sponsor of our ebook series on the docker and container ecosystem learn more about Cisco's perspectives in our latest ebook about monitoring and management it's available for download now at the new stack I oh hi this is benjamin ball with a new stack i'm here today with ken owens CTO at cisco we're talking about container modern for our upcoming ebook about container monitoring and management in the doctor and container ecosystem ken how's it going today I'm doing great Benjamin thank you um it always good to be with the new stack I enjoy spending time with you guys thanks for having me back this is I think now the fifth of ebook podcasts we've done so we're starting to rack them up so we're talking a lot about modern today that's really the primary focus of the book as the fifth book in the doctor in container ecosystem series and sort of a fifth topic for a reason here it's sort of the late stage I think of adopting containers it's something I've I think learned more and more about that it's maybe even one of the hardest parts anyway because it is sort of the last thing that you're doing when you're adopting containers so really kind of the first question I have it's really about what's different when you introduce containers into this monitoring landscape we talk a lot about monitoring in general I think and it's sort of a well a lot of people would say well understood some of the difficulties and monitoring applications systems and safety aims as well but I really want to know what you think changes when you introduce containers and I thought about this sending out this question Ben I can I think of you know application monitoring is very well known and understood and the idea of application monitoring is sort of looking at the processes and the performance of those processes to kind of detect and isolate any sort of abnormalities or shortcomings of the performance of the end user experience right network monitoring is also something that's very well known it's more of a science mouth and its application mountings more of a kind of an odd whereas network monitor is more of a science containers is sort of interesting to me and what's different here in general if I get too specific differences is that of civic changes is it contains are a small unit of capacity it helps have capability that results when you decompose an application into a set of services and in those resulting services you kind of create a set of micro services which you instantiate as containers and so connecting these together managing these and understanding the availability and reliability and the translation to these different small micro services which make up an application is much more complicated and much more difficult to do than what we've had to be able to do in the past and I can't look at that and like I said before in these three areas one is just the sheer amount of data when you get metrics for like 20 VMs we can you translate that to you know containers you might have a thousand containers right and so what used to be just monitoring 20 different instances now become thousands of instances to monitor the other pieces that containers get recycled right name they move around quickly or they're just destroyed and these operations are not very short lived in general and so how do you sort of manage these conditions that are more like race like when you have a very limited period of time to actually detect a problem and then the last one is that the correlation of the of the services and the complexity of the collation of those services and I think that that's especially true when you think about how quickly the number of services can grow into the thousands of services trying to correlate an issue between any given set of services becomes an in squared problem very quickly these kind of seem to be a lot of the issues I think reflect containers a lot in general it's basically seems like all challenges stand from a lot of these points the food scale that painters are relatively short-lived and that there's a lot of complexity and having numerous services is this something that when we talk about a lot of these same issues with micro services would you say here that when people talk about monitoring microservices that same kind of logic applies to moderate containers I think you can't have to look at the sort of a services architecture view of micro services because to me like containers are more about a specific given instance of a of a micro service or part of a micro service right was the micro services will leave the combination of all of these different containers working together and being orchestrated together to deliver an application and so to me that the level of complexity goes up when you talk about Michael services versus individual containers because whether or not a container is up is easy to determine what's the impact of that container not being up is the Micro services problem that's very difficult to determine the data that you're looking to collect from the application is that different significantly or is it the case where a lot of the of what we're looking for is the same we just collect it in a different way yeah I think it's a boy I look at more like a superset so some of the data that you collect from from a container is very similar to what you would collect in a vm I think worthy again the nets or stuff like memory and CPU usage and up you know kind of the status of the of the services running or the processor is running but with a row I think it is a little bit different is that the micro services or that oh this container model sort of wants you to start looking at different aspects that are more application specific around you know the network traffic the type of response times you're getting for the service on the interoperability for the for the data and that where that data how you get access to that data these sorts of more complex distributed systems problems come into play and I think that's where you get to that complexity application monitoring but you're doing it at a finite set of individual services that you have to now correlate so that's where you sort of get to that complexity is related like we mentioned and like you mentioned like we mentioned in the Manning right that correlation of this data and these this amount of information you're collecting for the monitoring it's being done across a set of different endpoints and systems and services that are in themselves to live and difficult to understand what the dependency is between them so this is something I've kind of been asking a lot about and one of the points that you mentioned earlier because of containers being short lived what really is the approach to monitoring them if you're not really sure that container is a actually exist at that point or not is it is it more about monitoring the relationships between certain sets of containers or I realize there's a difference between monitoring a single container and monitoring a bunch of different containers that are doing related things a couple of different approaches for kind of monitoring containers I think you know the first thing that you want to sort of understand is the weather like to cost sort of the control plane or the ability of the services to provide you with that data and so being to understand the health and the connectivity model between the services and think about sort of like a control plane problem like how do I understand what these components are and and this is very difficult because it it takes a lot of in my opinion it takes a lot of training if you will so because of the lack of visibility to just built into these systems because you're using a bunch of despair components to kind of build these services just understanding these basic service assurance aspects and the health and issue and problem management type of space right it it's not easy for like to get to a really different understanding of just what are the basic monitoring points of interest in this environment right and so I think the learning curve is pretty high in order to sort of understand you know when you request this service to be deployed there's a lot of individual systems they get hit and there's not really a lot of us there's not much of a feedback loop today to tell you when this this container is is spinning up incorrectly as spun up and you didn't expect it to spin up this way if when it spins up if the services are all up and available and when responding the way they should respond right it's sort of a fire-and-forget sort of model today and so while on one hand it it's great that you can sort of automate and orchestrate these set of services you know and very and very unique and interesting and high-performing ways it sort of becomes a very complex problem to solve when something breaks because it's very difficult to identify and isolate those problems so that points out one of the challenges that guess then it which is not only making sure you're collecting that data and actually of understanding it but but knowing how to go back and fix something when I think it's wrong essentially right that's I think the next step with that bin is like how do you even to me it's a lot of what I've been trying to do is like identify these patterns because I really if you think about you know system theory you go back to my system theory comment right every application has a known pattern and so therefore a micro service which is a set of services make up an application would have a set of similar patterns right the application pattern better equal the patterns on the individual services that are giving you that application right and so it like a statement but I think it's a really important statement because if you understand the application pattern you're trying to achieve you know how to monitor that pattern now you just have to understand how to apply the monitoring point that you know you need how do you get that into that micro services infrastructure component tree so you can actually get the time entry you're looking for it to deliver that end result that you need for the business the application pattern way of thinking about that you mentioned is actually reminds me a lot of some of the topics we discussed in the last ebook around container security and that kind of one of the propositions was that if you know what a container is supposed to be doing and you you look for it to be continually performing that set of tasks or that behavior you're really just looking for spotting the difference essentially is it's sort of like that where you're essentially looking for a predictive behavior that's maybe a very valid thing like there's a lot of predictive behavior in here that you're looking for i think the you know they like the common image you know I some people call this a corner case but in my experiences as a common use case it's not a corner page right is the slow response time for you know my my application isn't responding or doing what I expect it to do right the those sort of like issues and problems happen often right and so do too I I kind of understand why the performance isn't the way it should be is something that I think you know a lot of companies a lot of big companies have spent a lot of money with application performance monitoring you know a PM says we should try to help solve that problem but that doesn't really apply in my opinion to containers because containers are still sort of you know to too many levels of complexity below what the APM tool is looking at and looking for right and so I mean I think it is sort of a area that is common is in this pattern detection with known patterns in the area that unique it's this how do we how do we evolve monitoring for containers to address more this microservices application awareness and performance model kinda like an APM light sort of model that helps you type the application services back down to some of the underlying componentry that you already have in place that is based on known patterns so that what you're only focusing on is the anomalies that you can detect pretty quickly with with the analytics capable a that we have today but not really focusing too much other than where to place these solutions in terms of the pattern recognition right because most of the patterns that we talked about a common so you can sort of identify the locations to place the solutions that you want to work on or work with for your monitoring allow that to detect your your you know your set a well-known patterns and then focus your efforts on more of that next level of how do i do what type of analytics do i need to generate and analyze in order to get to this behavior analysis piece I wouldn't get too yeah that sounds very familiar essentially and you mentioned a p.m. and I'm kind of glad you did application performance monitoring is sort of a lot of ways what I've thought about as traditional monitoring it's the monitoring subject that I'm used to hearing about and so when we segue into container monitoring I was strange to get out of that mindset of thinking about application performance and thinking about there's different things to target here and there's really actually a different level of monitoring going on entirely you mentioned it being kind of levels of abstraction away from from from application monitoring so that's interesting for me to hear it seems to validate a lot of what other people are saying is this something where you think it's a fundamental difference from how may be a traditional monitoring of vendors have approached the subject yeah because you know there's definitely a school of thought out there were two additional monitoring vendors that they can just kind of you can leverage their existing solutions and get good enough monitoring data to manage consent container or microservices approach and in my experience I haven't had really a lot of luck with with leveraging existing monitoring solutions in this space at the same time though I don't think that this industry has the luxury to sort of go off and you know gold and consume new monitoring so this is although there are some good open source projects you know to take a look at like Prometheus but you know I am uitat cisco we you see advisor for a lot of our container work that we've been doing so there's there's definitely projects and things out there to look at but you know me the key with with that whether it's L your existing infrastructure and monitoring solution or your application monitoring solution or it's you know this news a new open source tooling or a combination of the two I think the most important piece of this is the integration and tie in these things back into the systems at your ops team to understand and if you can't tie it back and maybe providing some kind of a single pane of glass or some type of a solution that lets you you know bridge the gap between your existing tooling and the new container type of toiling that you may need in my opinion is sort of an important goal for an enterprise to have how do you sort of provide both views because you need both uses it's not a single view of of this problems going to solve it right you're going to need to sort of correlate and bring together multiple different solutions that you have in place to kind of help identify the right you know the right issue that you're trying to address yeah and you're getting into a topic where there's I think a lot of complexity to it's the you mentioned both see advisor and / me is both open source solutions we hear a lot about we've talked a lot about prometheus recently and probably will talk more Kubek on it's something where there's a seems like there's a lot of solutions and that maybe there's some difficulty in knowing what solutions are going to work best for you or work best for your stack or whatever actually you need to target you mentioned C advisor I wonder if you have any advice for people who are who are thinking about what tools are going to actually benefit them you mentioned integration and it seems like that's kind of the number one point is you want whatever solution you pick to actually fit in with what you're using and what you seem to think is important right yeah and then definitely you know include you know include your ops team and in the discussion because they you know they're going to have a very specific you know point of view one on what they need to be able to see and how it needs to fit into the existing dashboard right because I the last thing you want to want to do is you know take some nice job is already a complex typical job and make it even more complex and difficult right so you know be I think be conscientious of your of your Ops teams you know whether it's dev ops ops right those are doing the same the same job I think need to be able to kind of understand the information is coming in to make enough sense of it to make a decision when they meet two and so I'd say you know that's the second advice I would give I think the third piece of advice is don't try to overthink it too much right i think the i always I always kind of joke back in my previous days in working with the service in the service of IT industry we were trying to create a cloud service offering and so the way we structured our service we could put at the time this is about seven years ago now 70 years ago now we could put about 25 virtual machines on a physical host and and that was without oversubscription fact we will with and when we put all of our monitoring and all event statistic election tools and all that the security agents and all the things we needed to really hard in and make sure the environment is the way we wanted it we were down to about 10 B and we could put on a physical hope so 15 were for management and monitoring inc and security and 5 10 of them were for customers and so it was like a joke that I started kind of telling people that if you overkill this solution you basically you know hide your performance you hurt your availability and you had to your ability to make money because you you're spending all of you your CPU and memory cycles supporting the environment thanks for trying to to make high performing in the first place right and so don't overthink it too much where you have to have the exact same heavy solution in every place try to architect in a way that you can you know leverage commonality to leverage the network presidents most of if you create the right sort of network environment you could probably put a monitor in your network that would give you eighty to ninety percent of what you need for that environment you don't have to touch you your servers or touch your services at all right and so I think there are basic lessons that we've run in the past that we should apply to containers because it's the same sort of environment right it's all very much a connected and and heavy Lee you know utilize compute environments with big networking pipes in them so take advantage of what's there and try not to overthink it in the process of not overthinking it it is part of that actually choosing different solutions or choosing different tools when you need essentially a different result this is something we've talked about where people are actually using a couple different monitoring tools essentially different things is that related at all to what you're talking about it is yeah it's definitely related to what I'm talking about in that you know if you have if you have a good monitoring solution in place today there's no reason to rip it out and try to replace it with five or six different tools right if it gives you what if it does a good job today leverage it and just augment it where you need to and enter your point right there's sort of two scores of thought and I'm I'm probably more than the school of thought you define which is you know find the right tool for the right job get get you know I'm very much one of these guys that says if you if you need to monitor x and this solution will give you the best solution for monitoring X and and you really want that monitor X and get that tool right the other school of thought is you know good enough right there good enough school of thought which is you can just get one solution and you'll get you know they say eighty percent I think it's more like sixty percent of what you need but you'll get you know and for what you want to look for and the rest of it you just have to have your release my guys dig in to the problem and spending days to try to figure out what the problem wasn't fixed it right and so you know I think to me that's sort of how I classify those two schools of thought the one that's the right tool for the job is going to save you time and and energy and effort to solve problems the the one that's a good enough schools of thought is going to require a lot more intervention by by humans to try to figure out complex problems and that does seem to be some of the factors that go into the discussions like build versus by 44 different monitoring solutions and that's something that we write I think hear people talk about a lot which is essentially what a what a package solution be better for me but lately we've I've been hearing this question more as what should I build and what should i buy and rather in rather that it's not I'm either going to build everything open source or buy one big package I might use a combination of things right white every time I have this conversation I like to kind of it always reminds me of the conversations I have with my kids right and the conversation no matter is sort of what the topic is it always when it comes back to just because you can do something doesn't necessarily mean that you should do something right and i think i think that's advice that that we should all consider very carefully when we make these choices because it's these are not these are not single decision metrics right when you decide to put a solution into your environment that's going to do a certain function that function is going to be there for a period of time through lots of changes through lots of environmental impacts and in fact right and so I think when you make these decisions go into them knowing that you're going to have to if you're going to do this yourself you're going to have to update you know update this on a pretty regular basis you're going to keep track of security vulnerabilities and issues to come up in the community you're gonna have to take on some of the hardening that you know open source projects don't take on for you if you buy a solution from a vendor you know you have to wait for them to patch things or they presumably with the new release and sometimes then new releases are not you know compatible with the old releases right and so there's never a really easy you know 11 answer for all but I think thinking through the problem and then knowing like which side of the fence do you fall on and you're going to do something that is good enough you're going to try to do something that's the right solution for the right problem not definitely that that build buses by I think definitely go to feed into that same thought process and should be a big part of the consideration because it it could definitely change your view on that very quickly so you mentioned Cisco using a sea advisor earlier I was going to ask if there is any cisco specific perspective on how to go about modern containers or if there's anything that you're actually working on their relates to monitoring I think there's a couple of of perspectives you know from from like panda Cisco Ward V of you all I think that probably the first one is when you are you deploy we did this open source project mantle and when we when you go into POI and stop running a micro services environment these are developing applications and deploying those applications into that environment you know it is probably nothing better in terms of like finding the problems and the issues and then the lack of visibility that you have in these environments then we start using it right and so we have kind of what i like to call on the job experience you know here are the problems you go run into when you try to deploy no microservices into any environment right doesn't matter if it's a public cloud a private cloud or do it yourself you know you know model you're going to run into these same issues and so there are no specific things that we try to monitor within the microservices container control plane itself and we've isolated the control plane from the data plane so in general when you think about micro services and containers you think about flat networks or overlay network so you sort of lump the control and the data into the same flat space you have this big IP address you sort of push out to anything external from that server environment and from a mind fusion if all of your micro services are sitting behind an IP range that I don't get access to it's hard to determine what services are up what services are down what problems you're having inside that that black box to a monitoring solution and so I would say the the more you can isolate the more you can provide role a real IP address to the key services and the key control plane components of your of your pods that need specific monitoring of actual services you depend on for delivering orchestrating within that environment that's that to me is probably number one key lesson two we've learned that we need to do you need to make sure you have really good monitoring on the control plane and the components that make up a micro services deployment architecture the second thing min is I think the other piece is not as well understood with containers has to do with the the networking and how these services are discovered and how you consume and kind of compose different services together in a set of joint you know loosely coupled but still are sort of dependent on each other way to deliver an application and the networking piece is something that we've spent a lot of time obviously a cisco looking at you know things like project conceived and FDI oh uh virtual packet processing open source project just sort of look at how do we bake n just very you know I put a network administrator for a network team they're going to see the same types of metrics in the same types of data available to them and a containerized model as they would in a physical model because you know to me and I've mentioned this in previous podcast right there is no container networking it's just networking and so now what he needs to look like networking because if you have to do special networking then you're in trouble because anytime you do something special we all know that that's going to be a problem so so we are doing a lot in open source to try to ensure that networking acts and looks and smells like networking whether you're running in containers and virtualization or in a physical node and that makes sense and that that does as you say kind of hearken back to some of our previous discussions is there any uh basically anything you would point out about Cisco's customers may be any of the challenges they've been having as they've gone to monitor their in container environments yeah definitely we've had on we've had quite a few customers sort of working looking not not just with some of the cisco open source but just in general with a you know with myth mesosphere with docker with you know coil as with ranchers we've done a lot of work over the last year year and a half with the civil of our customers that are going down this journey towards 12 native and the charges that seemed to kind of blow up to the top is theirs I'm for them it's a lack of visibility of what's happening under the covers they say that there's a lot of basic service assurance aspects that are missing and so like the the health of an application is unknown they can get out of an individual service but they can only get health of an application and in these environments these I've issue in problem management become very complex for them they don't really have a good way in a containerized ward of applying some of the basic best practices are on operating and environment so that that's been trying to top complaint I would say and then any because I kind of mentioned earlier to the the biggest issue I hear from our customers and it's pretty much common across all of them is that it's very difficult to to sort of you know get to the base level of running an application or set of applications in this sort of a model to very steep learning curve just to get to that basic level of understanding the flows and then once they get to that basic level what I'm saying the flows it's not obvious to them where they should be monitoring different interests right some of the interest points if they want to look at it where those interests point should lie but then that architecture is not obviously apparent to them always where they should put those monitoring components and then it's sort of a you know we'll try it here see if it works too at the output is so if we can make value nor that solves our problems oh that didn't solve a discredit here so those are sort of like the main like challenges they faced just to kind of like as an aside I think it's important though one of the things that that we we started doing in our open source project was we started doing synthetic monitoring and and the point of that was if if you think about these different systems are interacting to in order to you know to put a moment you to play the set of containers there's a number of different services that are in the control plane that is set up to go and actually execute those commands and so what we did was sort of started doing synthetic monitoring and every new every single you know action that could cause a change in that control plane we would exercise those a synthetic monitoring event very rarely so that we would know if something started to degrade or the service started to have no memory least it was causing it to sort of have a slower along a response time and it it only improved our liability very quickly because we could identify issues that were about to happen and we could basically shut down those services and spend up new services so that we'd never ran into those issues that we could automate that whole process that makes sense and basically I think one of the other things I want to ask about is is there anything sort of on the horizon with Cisco is there anything you're working on thats related container monitoring we've talked about in the past of the acquisition of container X is that something that might be part of the future pipeline yeah absolutely so like you know that they came to contain notes acquisition that we just completed was very much around looking at at the space of of the concerns that our customers are talking about with containers and just so you know there was like a study that did read had commissioned not too long ago from with Forrester but can't talk about the top concerns and management was you know 61 percent of responses were like what the management of the space is difficult and then I kind of love monitoring to some degree into that does though other things like you know performance and integration that were just as high that you could say were you know I to the monitoring solution too but you know the whole point of what container X was doing is they kind of were looking at at this space from a i would say sort of a an enterprise window Enterprise view looking at kind of you know the way you know the monitoring and the operations teams need a single view or single pane of glass and tons of what's happening within this container space and so one of the reasons why we acquired them had to do with not so much that the product that they had but the way they thought about ensuring that monitoring and the management integration in between what new with containers sits well with what's existing in the enterprise today and so I think you see in you know in the not-too-distant future you see some plans we leave some syscon what we plan on doing in this space with you know and the bringing together of container X with the work that my team was doing on mantle with the work that the networking team was doing with konchi where you can see sort of this all come together into a coherent set of open source solutions or projects if you will plus some products that will be able to sort of meet the needs of our enterprise customers definitely really excited to hear about any news that relates to that I think that's going to be really interesting and I'm sure we'll talk quite a bit about that when that happens and and if that's something that develops in the future definitely really excited to see how that all integrates together and offers a full package so it can be a commitment yeah great well thanks so much for talking to us today can we've we've done this quite a few times now and it's starting to feel pretty familiar great thanks for your time again Benjamin tonight I might happy to spend time with you guys keep up the great work all right and thanks for listening a new stack we'll talk again soon cisco is a sponsor of our ebook series on the docker and container ecosystem learn more about Cisco's perspectives and our latest ebook about monitoring and management it's available for download now at the new stack io [Music]