Talk: Multidimensional analysis of cloud-native software based on large-scale operation data

Key Takeaways

hello everyone i'm xingpong from foden university china welcome to my talk multi-dimensional analysis of cloud native software based on large-scale operation data in this talk i will first introduce the concepts of cloud native software and the microservice architecture then i will explain why cloud native software evolves to complex ecosystem after that i will analyze the challenges and opportunities of cloud native software analysis and finally i will share some fast and ongoing work for multi-dimensional analysis of cloud native software cloud native software has been a very hot topic in both industry and academic cloud native means that the design and the implementation and the deployment of an application fully exhibit the characteristics of cloud computing for example distributed and elastic cloud native means a lot of things among them i think the most features include microsoft's architecture container-based deployment and operation and develops and continuous delivery according to idc by 2022 90 of all new applications will feature microservice architectures and 35 of our production application will be cloud native uh microsoft architecture is very important for the cloud native software microsoft agitator means to implement an application as a set of small grain services each service can be developed by a small autonomous team and can be independently released deployed and scaled a collaborative software can be very large we can see in some very large companies their cognitive software can include tens of thousands of services this kind of cloud native software includes many applications these applications share common service infrastructures and the boundary of the applications phase and in this kind of software system new features and the business continually emerged every day under the services of the system continues delivered and added and updated so we think this kind of complex system can be regarded as a new kind of software ecosystem and this kind of software ecosystem i think is quite different from traditional software ecosystem like android and ipo ecosystem because for this kind of cloud native software ecosystem the whole applications and the services are continuous in operation and their evolution are mostly online we all know that software analysis plays an important role in software engineering they can be used for different purposes for example reverse engineering fault detection and the testing design issue detection and the debugging and traditionally the techniques used for software analysis include static analysis dynamic analysis and history analysis among then static analysis which means to use some kind of static analysis tool to analyze the source code and history analysis which means to analyze the evolution history of a software usually by analyzing the community history of a repository these two kinds of techniques are mostly used dynamic analysis is less frequently used in traditional soft analysis because dynamic analysis is usually very expensive but for cloud native software we will be faced some new challenges because the services in cloud native software are developed independently by different teams and in different repositories and running in a highly dynamic and uncertain environment and for this kind of cloud native software the complexity of the system has been shifted from individual services to architecture level because we know that we all we know that the services of the system can be very small so the development of the individual services is not so difficult but the complexity is quite large in at the architectural level because as i mentioned just now for a large software cloud native software it can include tens of thousands of services and this kind of architecture cannot be managed in a top-down way because the system is too large and the different parts of the system are developed by different teams so nobody can have a very complete picture big picture of the whole architecture and for this kind of software static analysis or history analysis cannot help because we know that each service in cloud native software can has its own repository so it means that if we analyze the source code or the evolution history of a repository we can only understand the individual services we cannot understand the interaction between different services but at the same time the analysis of clone's cloud native software has some opportunities first cloud native software follows microservices architecture so it means that this kind of software has a explicit software architecture because the architecture of cloud native software a knight is naturally defined by the services and the interactions so we call we say that uh this kind of cloud native software has an exclusive software architecture at the same time cloud native software usually has very good infrastructure including the so-called observability infrastructure obviously for cloud native software and the microsoft architecture includes three peanuts they are tracing logging and the matrix so it means that based on this kind of infrastructure we can obtain very rich and large amount of data from their their long-term long-term infrastructure including locks traces and the metrics and this kind of runtime data that is so called the operation data in this talk can can helps us to do different kinds of analysis so based on this kind of operation data we can do a lot of analysis so here i will talk about uh multi-dimensional analysis from four different aspects the first is anomaly detection the second is root cause analysis third is architectural evaluation and the force is business understanding normally detection and the root cause analysis both are concerned with short-term problem handling for example detect the anomalies and identify the root cause in a very short time to help the developer to uh resolve the the problem or incidence in a very short time activity evaluation the business understanding are more about some median or long-term goals for example architecture evaluation will be concerned about the long time the long-term evolution of the software architecture anomaly detection means to detect possible anomalies or incidents in the running system in traditional distributed system log analysis has been widely used for anomaly detection usually they will extract long events from raw log messages and the twitter log as a sequence of log events then they can learn log patterns from normal execution and detect anomalies when long patterns deviate from the trained model in recent years there are some researches focused on transfer anomaly detection they treat our tracks as a sequence of service invocations and learn similar to log anomaly detection they will learn trace pythons from normal infusion and detect anomalies when trace patterns deviate from the trend model at the same time we think that the current log anomaly detector and the trace alumni detection approach have some limitations because we know that a micro service system can include many different uh many macros many services running in different nodes so to understand the whole uh behavior of the whole system and to detect the anomalies and the uh we need to combine the behaviors of all the services and also their interaction so we think that we need to combine logs and the traces together because the logs reflect the intra service behavior and the traces reflect the inter-service interactions we need to combine them together and also we need to consider the complex structures of traces because we know that a trace indicates a invocation chain for request in the trace there are there there may be many service invocations in the higher in a hierarchical structure and at the same time a trace may include a parallel or a synchronized invocation so we need need to consider the complex structures root cause analysis means to localize the root cause of a given incident or anomaly usually we can combine different kind of operation data including logs traces and the kpis to for for this for this purpose in the recent work we use service dependence graph to do root cause analysis for specific types of anomalies including performance reliability and traffic in this work we construct a service dependence graph in this graph we have service dependencies and the related kpis which are captured from recent service invocations the basic idea of this work is to traverse the service dependence graph from the entry node for root cause localization the np node means that where the anomaly or instance is reported the key techniques in this work include single service anomaly detection which means to to determine whether a dvd individual services is anomalous based on the kps and traversing strategy means how to traverse the uh surface dependent graph in the efficient way a promise rate which means to uh prong the the uh prong during doing some kind of plumbing during the traversing because we know that the graph can be very large so if we have no planning strategy the whole process will be very inexpensive candidate root cause ranking which means that to rank all the candidates root crosses uh in the uh in the last step so the challenge in this work is how to balance between the accuracy and the time of root cause analysis because we know that if we want to obtain a very precise result of rule called analysis we can use more time for example to traverse the the service dependent graph in the in a better way but at the same time we need to control to limit the time of the nicest because we know that we are handling long-term alumni and incidents so uh the developers need to resolve the problem in a very short time microservice architecture is likely to decay and i think the courses for this problem maybe come from two aspects uh first microsoft architecture usually is ongoing a very long evolution and during the the process as the different parts of the architecture are maintained and developed by different teams different kinds of issues will be introduced to the architecture at the same time as i mentioned just now the microsoft microsoft architecture of cloud native software can be very large so it means that usually there are no global view about the whole architecture so different kinds of architectural issues may emerge for example we can see in some very large industrial micro service systems there exists a very long evocation chain for example there uh i i have seen some evocation chain include about 500 hopes in a single trace and also there are some psychic evolutions in some traces and also as you can see in this slide there are also other kinds of architecture issues uh for example hub-like dependence and also no cohesion and the high country of services and also there are some architectural issues related to performance availability and scalability architecture evaluation for cloud-native software of based on microservices architecture highly depends on operation data node source code and evolution history we know that articular evolution evaluation you usually conducted for traditional software based on source code analysis and evolution analysis analysis for example we can analyze the evolution company relationships between different kind of four files based on the repository but as i mentioned just now because in for microservices architecture each service account has uh have its own repository so means that we cannot uh analyze the relationship between different services from the repository because each repository only records the history of a single uh service uh also at the same time source code analysis cannot help a lot uh for similar reasons so uh i think for microservices architecture uh the evaluation i think highly depends on operation data that means that we should consider to use traces logs and also kpis to identify the potential potential uh architectural issues uh for soft uh native software uh basic understanding in a similar i think uh for similar reasons usually the developers cannot have a have a complete picture or view about the whole microservice system so uh similarly i think we can use the operation data to understand the business uh in a recent work we have uh have present approach that uh conduct a multi-layer aggregation of traces uh to help the understanding of business uh in this work we conduct uh multi-layer aggregation that means that uh in one layer we aggregate aggregated the traces by the topology that means that all the traces having the same tree structure will be aggregated aggregated together and in another layer we will further aggregate the traces based on business scenarios the big series here can be indicated by key services for example uh for given given address if the trace has passed some key services for example uh order submission and also payment then we can we can say ah this trade is about uh online shopping yeah so based on this kind of multi-layer allegation we can uh we can have a more efficient view about the trades because we know that the raw traces can can be very large uh for some big microservice systems they can produce uh billions of traces per day so we need to use this kind of aggregation approaches to to provide a more efficient view for the understanding of the whole system so in this work we combine graph databases and analysis analytics databases for efficient data analysis and uh also we provide some uh visualization uh for the developers to understand the business flows and they're changing so we can see uh based on this kind of analysis we can provide a visualization of the current business flow and also we can uh we can highlight the changes for example uh maybe this morning the the the developed development team have do some uh they deliver some new features and then the business flow have some changes so based on this kind of visualization we can understand what changes have happened uh for the system so conclusion uh multi-dimensional analysis of cloud native software is very important so but we need to know that large-scale connective software has evolved to complex microservices ecosystem and for this kind of complex ecosystem multi-dimensional analysis is required for various purposes such as anomaly detection root cause analysis architecture evaluation and the business understanding but at the same time different from traditional software analysis based on source code and the evolution analysis cognitive software analysis highly depends on operation data including trace log and the matrix our vision uh for multi-dimensional analysis of cloudative software including that uh operation data analysis can be used a means for observing and governing the cloud native software ecosystem and at the same time the so-called ai ops may need to extend its focus to include such as architecture the decades and the business review recovery and we need to consider to introduce more and the richer operation data by new sensors to improve the observability and also we need to consider to use more intelligent operation data analysis techniques by combining trace log and matrix thanks